The Problem With Notifications
Any experienced QA team will tell you: the hardest thing to test is normally notifications, for a seemingly inexhaustible list of reasons. The primary one being that even though the notifications go out through email, you’re often stopped from sending said email.
That’s because you can’t actually send a notification, as you don’t want users to receive something that’s not actually worth sending yet or which hasn’t been debugged.
There’s also the issue that the form notifications are delivered in are very client-specific, meaning that you actually need to see them in an email browser to know if they’re formatted correctly.
Now, let’s take a minute to talk about how people normally solve this: lots and lots of fake email addresses. They send notifications and test emails to a long list of fake email addresses that they have.
Unfortunately, this can lead to a few issues. First, it means that they’re wasting time creating an untold number of email accounts that they’ll very quickly forget about (or forget the passwords to). Second, they’re probably going to end up creating all of these accounts on one email service, meaning a small sample test size for their test. Third and finally, they’re going to spend more time dealing with the email clients than with the actual testing and development of the notification email.
In a nutshell: testing notifications is hard, the way people do it now is broken, and there’s confusion on how to fix this problem. Let’s change that.
The Types of Notifications You’re Testing
To start, we need to understand what notifications are and what flavors they come in. While you could break it down into super specific categories, we like to think of notifications as one of three types, with an optional fourth one depending on what you consider a notification.
1. User-Specific Notifications
These are notifications that are specific to a user, meaning that they are heavily personalized and not being sent to a large audience. An example of this would be the notifications you get from social networking sites like Twitter, Facebook, etc.
For example, let’s say you get a notification or roll-up email showing the latest tweets they’ve missed or status updates they haven’t seen yet. We all receive these, they get people to engage, and they’re sent fairly regularly. As it relates to testing, you’d want to see these notifications in the same way a user does, not just a demo account’s representation of what a user could theoretically see. You want real results.
2. Event-Specific Notifications
Think of these as responses to specific events, like clicking on a button, submitting a form, abandoning your shopping cart on an ecommerce site, or falling off an onboarding flow. For each of these events, you’ll want to trigger an email to be sent to that user, prompting them to re-engage with the process.
Unfortunately, these can be notoriously difficult to test.
For example, let’s say that you want to send a notification to all users who click on a rain jacket product, add it to their shopping cart, and then abandon the shopping cart without finishing their purchase. To test this notification, you’d need to go through that test flow which is already time-consuming, and then reach the point where you test the notification itself.
This is just one example of event-specific notifications, of which there are hundreds of thousands depending on your type of business.
3. Campaign-Specific Notifications
Moving beyond what specific users do or what specific events are triggered, the third overall category is campaign-specific notifications. These are less about personalization and more about strategic goals that can be reached by using notifications.
A great example of these would be a promotion or time-limited offer.
Let’s say that you run a software company and are temporarily cutting the price of your software by 15% for everyone who signs up within the next three days. In order to notify people (since, clearly not everyone is going to be visiting your site in the next three days) you’ll want to send an email notification.
This notification needs to be personalized with the contact’s information (as Kissmetric helpfully remind us) in order to have a high conversion rate, which means you’re going to want to test it with a real account. That means sending an email to another test account.
4. Password Reset Notifications
The last type of notification email is less of a high-level category and more of a one-off notification that everyone needs to send at some point. These are password reset emails, a standard security practice for every company that hosts user accounts.
In order to test these, you don’t want to end up actually resetting a customer or user’s password, since that would be seen very negatively by them. Instead, you’ll want a dummy account and a test email address to test this, just like you would for testing notifications.
What You’re Looking For When Testing Notifications
When you’re testing your notifications, there are a few different things you’re going to be looking for. While you could technically test notifications by having hundreds of dummy email accounts on Gmail or Hotmail, you’d quickly run into logistical issues managing them (not to manage Terms of Service issues). Testing email notifications correctly means planning ahead of time, not creating dummy accounts in the moment. That means you’ll want an automated solution for testing notifications.
When you’re evaluating your options, there are four things you should be looking for. These aren’t the only features you’ll want, but they are characteristics that must be there in order for you to successfully test notifications.
Immediacy
First and foremost, you’re going to want an immediate result for your test notification. There is little that is more frustrating to a QA engineer or analyst than being in the middle of a long list of QA tests that need to be performed, hitting a groove where you’re methodically checking off each test, and then hitting a bottleneck as you repeatedly refresh a third-party service.
What you want to have is the ability to trigger a notification to be sent a pre-determined email, be able to go to that service and see the email sitting there, and then be able to either approve the results of the notification or to dig in and find out what issues exists
It’s not the sort of thing that most services advertise (how would you quantify speed in most cases, anyway?), but when you’re using a service’s free trial to test them out, make sure you make a note of any lag.
Accuracy
The results of a test notification need to be identical to those that users will receive in their consumer email clients. It doesn’t do anyone any good if your email client supports CSS libraries that no one else supports yet, since that will make your email look better than they will actually end up appearing.
Instead, what you want is an accurate representation of what your email will look like (and how it will work) in a user’s browser. A QA test is worthless if it isn’t accurate, which is something anyone testing notifications needs to keep in mind.
Reproducibility
If you send a test notification to an email address at one moment and then send it to another email address at the next moment, you’d expect the results to be identical. Of course, there are fluctuations between different email providers, but overall the results should be the same.
If that’s not the case with the methods you’re using for your email tests, then you need a different method of testing your emails. It’s just that simple.
Flexibility
When you’re testing notifications using a third-party service you’ll want to be able to use your methodology, workflows, and processes.
If the tools you’re using are great at testing email notifications but make you go far out of your way to use them, then you should keep looking for the right option. What you want is a tool that is strict enough that it’s easy-to-use, but not so strict that you need to extensively modify your workflows
A Few Final Thoughts
Of course, we would be remiss if we didn’t finish by saying that Mailosaur helps with this. While there are competitors, we are laser-focused on solving the specific problem of helping transform your email QA testing processes.
However, even if you don’t end up working with us — there is a free trial — these suggestions should be able to help you test your notification emails in a smart, efficient, and informed way.