How to Choose Email Testing Metrics

We all want to do more email testing, right? But what does that actually look like? And how do we effectively measure the success or failure of tests within email?

Let's start with the tactics we often like to use – and then identify metrics that can answer the questions.

Before you get started, regardless of whether or not your email tool offers A/B testing suggestions, math or chooses automated winners, I recommend that you know how to and feel comfortable doing your own math. More importantly, know how your email tool records and calculates its metrics. 

For example, here's information from MailChimp, Luminate Online, Marketo or Engaging Networks.

I want to run a Subject Line test.
 

What's a driving force behind this test? Could it be (1) answering *compelling* content questions, (2) getting better visibility in the inbox, or (3) you were just told to run tests?

The key metric for subject line testing is open rate, but before you begin, you'll need to make sure that your email deliverability and inbox placement are in good places with the audience to whom you're sending this test. Otherwise, you can incorrectly interpret the metrics that your email platform is showing you.

A key misstep in a subject line test is ignoring the preview text options. This section is critically important! Is should be adjusted to reflect the differences in the subject lines themselves. Litmus is a solid resource for getting your preview text spot on. Can you spot the typo in this screenshot?

Another important consideration with subject line tests is the audiences that respond or don't. Often prospects and active donors react differently so running a subject line test may provide more value with the larger audience segment.
 

I want to test the layout of my email messages.

What's the driving force behind this test? Have you been using the same layout for X years? Are there significant layout differences between your types of emails?

The content of these emails must be the same in order to be correctly measured. The key metrics for this type of test are click-thru rate and response rate, with a few caveats.

If the layouts are drastically different and you're adding more calls-to-action, you will need to also consider math! Consider a few things as you configure the HTML of the email like adding an additional tracking parameter that means something unique to each link in each variant, like &link=header-c vs &link=header-t, &link=image-c vs &link=hero-t. This will allow you to more accurately measure click through on specific design element differences.

If the test layout contains more calls-to-action, then response rate became more important than click-thru rate. Response rate can also help if you're looking at email messages over an email series within a campaign, because sending one stand-alone email with different layouts may not provide enough information for firm decision making. The performance difference could be the email content, the visual, the timing or other factors which are at play.


Are you smart, motivated, and eager to solve problems in a collaborative environment? 

If so, we want you! Join our team!

See Our Current Career Opportunities