Editor’s Note: In a recent interview with MarketingSherpa Editor Sean Donahue, Research Analyst Corey Trent outlined errors even experienced email marketers make when conducting tests. (My personal favorite – #5). We thought this information was valuable, and wanted to share it right here on the blog for those who do not have a MarketingSherpa membership. Special thanks to our sister company for allowing us to republish the below article…

SUMMARY: Before you conduct your next email test, make sure you’re not falling into a trap that can muddy your results or limit the gains you might otherwise achieve.

We spoke with an email testing expert from our sister company, MarketingExperiments, to uncover common mistakes marketers make when running email tests. Read why good analytics and segmentation are crucial forerunners to testing, and why a blockbuster discovery from one test actually can be a risky thing for a marketing team.

by Sean Donahue, Editor, MarketingSherpa

Testing is an essential component of a strong email marketing strategy. But only if the tests are conducted and analyzed properly to ensure you’re helping – not hurting – your email performance.

“There is a cost for bad testing,” says Corey Trent, Research Analyst, MarketingExperiments. “Bad assumptions based on bad tests can cost you a lot of money and cause you to lose out on a lot of business.”

Trent routinely conducts email tests as a member of the MarketingExperiments sales and marketing optimization research team. Through this work, he’s seen how mistakes, misconceptions and simple oversights can derail a well-meaning marketer’s testing strategy.

We asked him to share his advice for avoiding testing pitfalls, so you can achieve your goal of improving email performance. Here are seven common mistakes he’s observed:

Mistake #1. Not having the right analytics behind your tests

Sometimes marketers must recognize they are just not ready for email testing, says Trent.

In particular, he says, marketers often don’t have the processes or the analytics systems in place to capture a complete data picture — you must be able to tie your email metrics to business goals, such as revenue.

For example, if you’re only measuring email metrics, you might test a new subject line treatment that increases open rates by a few percentage points. Sounds exciting, but without tying subsequent clicks from those messages to website sales, you might not notice that you’re actually decreasing conversions and total revenues because the subject line is creating the wrong expectations about your message.

The good news is that it is now relatively easy to track email activity through to website metrics using tools such as Google Analytics. This program offers marketers ways to track email messages through specific goals on your site, such as product pages viewed or time spent on site (see Related Resources below).

“Email tracking all the way through to business goals is available for everyone, and doesn’t take a long time to set up or require having an IT team to do a lot of programming for you,” says Trent.

So make sure your analytics systems are robust enough to support your objectives before you begin email testing.

Mistake #2. Not segmenting email lists for tests

Many marketers have large email databases, but don’t know a lot about the records held within. In these cases, they may conduct an email test using their entire database – and create a muddy results picture.

“Without segmented lists you don’t get good test results,” says Trent. “You get all these people responding differently to your emails, which pulls your results in all different directions.”

Spend the time to segment your database and understand the different characteristics of the segments before you embark on email testing. The more you know about unique segments within your database prior to testing, the better chance you’ll have of finding the right messages to appeal to them.

Mistake #3. Stopping tests after one big win

“With email more than anything, we see people get a big win and stop testing,” says Trent.

As exciting as those big wins may be, they shouldn’t be the end of your testing process. The makeup of your email lists is constantly changing; external factors, such as the economy, also impact subscriber behavior; and your competitors’ campaigns and tactics are always changing as well.

This constant state of change means you must routinely work on the messaging, layout, calls-to-action and other elements of your email messages to ensure you’re getting the full benefit of a testing program.

Mistake #4. Testing too often

The flip side of stopping tests too quickly is falling into a pattern of repeated testing, to the point that you simply can’t keep up with all the data you’re generating.

When finding the right pace for testing new email treatments, think critically about the tests you want to perform and their potential impact. Focus on the ones that have the greatest potential impact on your specific business goals.

Trent suggests researching your current email strengths and weaknesses: Examine your own email and website metrics and study industry research to benchmark your own performance against your industry peers. This process can uncover where your email programs are delivering weak performance – and where you have a bigger chance of improving your company’s bottom line through testing.

“Those deep dives into the data you have available help you prioritize that list of potential tests,” says Trent.

Mistake #5. Overlooking email copy tests

Today’s email messages are so versatile that marketers have a wealth of options to test: Layout changes, images, fonts, colors, social media integration and so on.

But the ability to tweak these elements can make marketers forget about the old standby: copy tests.

“What often gets left out is how important the copy and communication pieces of an email are,” says Trent.

Copy changes can deliver surprising results, so keep them on your list of test options. You can try using a different voice for your message, long copy vs. short, different copy layouts, or even a personal letter from someone in your company.

Mistake #6. Always testing additions, rather than subtractions

The pressure to generate more engagement with subscribers can push marketers to continually test new additions to their emails: More images, more buttons, more links or more colorful text.

Too many additions can actually confuse recipients, making it hard for them to decide what to do with your message.

Instead, Trent has seen marketers achieve strong performance gains by eliminating elements from an email and reducing the number of decisions a recipient has to make.

By focusing a message on a single goal — such as inviting recipients to an event or generating sales within a specific product category — you can see whether there are extraneous elements that might be distracting recipients from the desired action. You can then test the impact of removing those distractions.

Mistake #7. Not asking other customer-facing teams for test ideas

Great test ideas don’t always come from the marketing team. It’s likely you have other sources within your company whose insights can shape a good email test.

Trent recommends talking to sales people or customer service representatives to learn what they’re hearing from customers. The feedback these teams receive from customers about your products and services — or why they chose to buy from you — can spark ideas for new subject lines, email copy, or specific products to feature in email promotions.

“Don’t be scared to go down and talk to those people to get some good ideas.”

Related Resources

The Magical Metrics Tour: Demystifying the secrets behind analytical “tricks” to help you drive ROI (describes how to track email clicks with Google Analytics)

What Else Can I Test….To Increase Email Click-through?

Discover the Best Time to Send Email: 4 Test Ideas (for MarketingSherpa members)

Improve Your Email Programs: 5 Test Ideas (for MarketingSherpa members)

Email testing produces some of the most interesting results I see here at MarketingExperiments. The cause for this is a combination of constantly changing variables.

For one, content within email tends to change more often than your typical landing page. This makes optimizing for content more challenging as different topics are likely to garner different levels of interest from the segments within your email list. So results will change each month based on the content alone – making A/B testing the only reliable method for measuring progress.

In addition, email lists themselves prove to be a challenge, as what works for one list may not work for another list. Even within lists, especially aggregated lists, you will see different results based on the value proposition, content, layout, and calls-to-action (CTAs) used in your email.

And to further complicate matters, you are still dealing with a funnel process in which your email must first reach a user (avoiding spam filters, personal filters, etc.), your subject line must interest the user enough to open the email, your email must display properly (with images on and off) and be compelling enough to achieve a click-through to your landing page where the battle for a conversion wages on.

In today’s world of overloaded email boxes, people declaring email bankruptcy, spam filters and everything else, this game is only getting more difficult – for marketers and users alike.

With that said, I’d like to offer up my own favorite email testing tricks and tips. It’s important to understand that what works for one segment, list, or industry will not necessarily work for another. In fact what works one month for a list may not work next month. It’s an ever-evolving process in which you must always challenge your own best practices to maximize your results.

1. Text-only email

This is my silver bullet of email optimization. In a world where everyone wants their logo, business cards, websites, and emails to be as shiny and pretty as possible, it’s easy to forget that email is a text-based medium.

Out of all the emails you actually read each day, how many are HTML vs. text-only? The important emails you receive each day – the ones from your family, friends, and co-workers – are all likely text-based. This means the HTML emails you receive are most likely not from one these aforementioned groups and thus likely less important to you on a personal level.

Obviously there are caveats to this – such as an email from your bank or a Facebook friend request – but the truth is, when you send an HTML email you are already fighting a certain level of banner blindness. If you currently only send out an HTML email, I’d challenge you to A/B split test against a text-only version of your email and measure the results.

2. Story format

Emails are a form of value exchange. In exchange for someone’s time and interest, you must first provide something of value.

One of the easiest, most interesting ways to provide value in an email is to tell a relevant story. Not only can telling a story create interest in your topic, but it also can provide you with a natural sounding CTA of “Continue Reading” or “Read More” that requires less commitment from the user before clicking through to your landing page.

The downside to the story format is it requires you to have an interesting story to tell as well as an audience willing to read through the text.

3. Big button

Sometimes if your offer is compelling enough, all you need to do is give people a place to click. It helps if your email list is already familiar with your brand, is interested and familiar with what you’re offering and needs little explanation before clicking through.

For example, our Marketing Director is drawn to J. Crew’s buttons like a moth to a lightbulb…

The focus should be on the value proposition and the CTA copy as this will determine the commitment level required to achieve a click-through. A “Learn More” CTA will bring more clicks, but less qualified traffic than a “Buy Now” CTA. Experiment with this CTA copy to see what works best for your particular audience and dial in the right amount of click-through vs. conversion.

4. Multiple CTAs

I’ve found that when dealing with large, aggregated email lists, including multiple, different CTAs can help increase click-through on the basis that different people will be interested in different aspects of a topic.

For instance, if you are offering a free PDF download on a topic that you hope will encourage people to provide their email address, and you also have a webinar that does the same thing, go ahead and include a link to “View the PDF” and another CTA to “Sign-up for the Webinar.”

What you’ll find is that offering multiple contrasting calls to action will result in a higher click-through rate as some people will be interested in downloading the PDF, while others will prefer the webinar – two different mediums that attract two different groups of people but achieve the same objective.

5. Digest

If you currently send out a longer email newsletter, I’d encourage you to try a digest format. In the digest format you offer several links at the top of the email that either direct users to a particular article on your website or anchor down to an excerpt within the email itself, followed by a CTA to “read more.”

I’ve found digest emails work well when you have a lot of content to offer and you are able to effectively arrange this content so as to attract a wide variety of clicks. It’s a combination of the story format and multiple CTA emails mentioned previously.

Related Resources

The Five Best Ways to Optimize Email Response: How to craft effective email messages that drive customers to action

Optimize your Email in Three Steps: How one marketer tripled revenue from their house list

Order your custom Email Response Optimization Package

Over the past couple of weeks, our content and production team has been mulling through research from multiple companies, trying to find the best experiments for today’s web clinic. It’s always exciting trying to find the right illustrative research that will serve our audience. In the midst of this, a very interesting case study came across my desk that I just had to share with our blog community.

The experiment came from the research library of ThinData 1:1 and was with a large airline that had a strong online presence. They had the goal of improving the level of user engagement (through higher click-through rates and lower unsubscribes) for their customer newsletters. They wanted to test a 1:1 marketing approach and see how much impact a personalized conversation with customers would have.

The Original Newsletter (click-to-enlarge)

First, it will be helpful to see what the original newsletter looked like. There are a few key things to note:

1) The goal of the email was to create enough interest on the part of the recipient to get them to click through to a specific flight offer.

2)  Every element on this email was sent generically to large subscriber segments and included offers without any reference to a recipient’s personal preferences.

3) The only (and it’s debatable) 1:1 marketing technique that the original email design included was the name of the recipient.

The 1:1 Marketing Strategy

One-to-one marketing conversation requires a couple of key ingredients, one of which is insight about your customers. So, to understand their customers better, this company created a “customer preference tool.” This tool took previous behavioral/purchase data and mapped out preferences in five key areas (destinations, airports, hotels, etc.). It also gave customers the ability to update their preferences for each of these areas.

So once they had some solid insight about the preferences of their newsletter recipients, they were ready to start a better conversation.

The New Design (click-to-enlarge)

A new email template was created to customize content according to the five key preference categories of the customers. In fact, the tool was inserted at the top of the email and functioned as a table of contents for the email. It also gave recipients a portal to their own preference center to make any changes.

The rest of the email was organized by an algorithm that gave higher placement to offers with higher preference status to the email recipients.

Instead of displaying all possible offers for a category, offers were filtered according to the preferences of the recipient.

Overall, the new email design created a 1:1 marketing conversation that matched the specific motivations of the recipients.

The Results

As mentioned in the title of the blog post, the results were impressive. For a large company that no doubt had already spent time improving these newsletters, they were able to increase the click-through rate generated from the emails by 21%. And this came by subtracting, not adding more links within the email. The new design also decreased unsubscribes by 60%, which the marketers interpreted as a big thumbs-up from their subscriber base.

Overall, the results of this test underscore the value and potential impact of striving to have a 1:1 conversation in our marketing campaigns.

What does this mean to you?

Ideally, you would have as much information as this company had about their customers, and if it were a perfect world, you would have unlimited technical development capabilities at your fingertips. Well, most of us don’t live in that perfect scenario and creating a true 1:1 marketing campaign like this company might seem a bit out of grasp.

Here’s the good news, 1:1 marketing has a path leading to it and everyone is somewhere on that path. And every marketer, regardless of company size, can improve the conversation they are having with their customers. Today on our web clinic, we will look at the strategic things that marketers of any level can do to improve their 1:1 marketing campaigns. So join us today as we talk about this case study and more during our free web clinic: 1-to-1 Marketing at Four Levels: Strategic ways every marketer can enter into an online conversation with customers

Note: To give credit where it is due, I’d like to point out again that the case study mentioned in this post came from the research library of of Thindata 1:1. Thindata 1:1 is a training partner of ours that provides one-to-one marketing automation solutions. They have done some interesting research in the area of one-to-one marketing and I would encourage you to check them out at www.thindata.com.

It’s a little over-simplified, but an email marketer’s job is to get the right message to the right person at the right time to achieve a specific goal. Doing that means understanding what motivates subscribers to open a message and engage with your offer – and that’s where the process gets tricky.

Like our colleagues at MarketingExperiments, we at MarketingSherpa believe that nothing provides the better insights into the “right” approach than a good test. A marketer’s personal bias, best guess, gut instinct or assumptions aren’t enough. In fact, they’re often wrong. You have to be willing to let your audience SHOW you what motivates them.

Today in Munich, MarketingSherpa is hosting its second Germany Email Marketing Summit, which features a Case Study that demonstrates the power of testing to determine customer motivation. VNR.de, a publisher of lifestyle and professional advice from experts in their fields, is sharing the results of a list-cleansing/subscriber reactivation campaign they recently conducted.

Winning back “inactive” subscribers

The campaign targeted “inactive” members of their list, which they defined as subscribers that had not opened or clicked an email in 120 days. They wanted to either reactivate those subscribers, or else determine that they were truly inactive and remove them from the list. So they set up a four-message reactivation campaign to encourage a response.

Each message took a different approach to the reactivation effort:

- The first was a survey about email preferences
- The second was a request for subscribers to update their personal information
- The third was a contest to win a book
- The fourth repeated the request to update personal information

What is more appealing than FREE?

Going into the campaign, the team believed the contest offer would have the best response. After all, people like getting free stuff, right?

Maybe not: The contest offer had the weakest open rate and clickthrough rates of the four messages. Its open rate was 60% lower than the best-performing email – the survey about email preferences. And the contest offer’s CTR was 82% lower than the best-performing email.

The good news is that the reactivation campaign was a success overall. They reactivated 9% of the inactive subscribers they targeted – and they won a MarketingSherpa Email Marketing Award for it.

They also learned important lessons about what motivates their subscribers. Their conclusion: “People seem to be most interested when we are interested in them.”

Final lesson: Assumptions are no match for results data. So get testing!

Sean Donahue is the editor of MarketingSherpa, a research firm publishing Case Studies, benchmark data, and how-to information read by hundreds of thousands of advertising, marketing, and PR professionals every week.

I’ll admit that I am a Twitter novice. Compared to social media gurus, some of whom have tremendous experience with the platform (up to two* years!), I am still very much in the learning-by-doing phase. Then again, aren’t we all?

As I try to be informative and give back to the Twittersphere, one of my email-related tweets was picked up by a Florida marketing agency that services several metros nationwide. With our Email Optimization clinic series underway, I was more than happy to provide an analysis of a broad-spectrum campaign that they had planned. Luann, their president, was as excited as I was about making a Twitter connection.

With Luann’s permission, I wanted to share my thoughts and recommendations with our readers. Here is an edited copy of the email response that I sent to her:

Hi Luann,

(click image to zoom)

Here are a few thoughts based on the email message creative I got from Noele, along with the requisite assumptions I’ve made. I hope they will be helpful.

There are two important caveats:

1. I don’t believe in best practices. Everything I recommend is normally tested until I find out what really works for the particular product and customer segment.
2. I want to be as helpful as possible, so I am not pulling any punches; the comments below are not a reflection on your company’s competence or reputation—just how they are communicated via this email message.

The fundamentals: Optimizing thought sequences

In optimization, our objective is not to create better design or copy. Our objective is to affect different thought sequences, and design and copy are our tools. A useful way to examine the thought sequences we need to address is through three simple questions that arise in the mind of the email recipient immediately, whether consciously or unconsciously:

1. Who is sending me this email?
2. What is it asking me to do?
3. Why should I do it?

Our job is to answer these questions as directly and quickly as possible using copy, graphical elements, and layout of the email.

Without specific information about your list, I am going to assume (based on email content) that it contains a large segment that has never done business with your company and perhaps has never heard of it.

Communicating Efficiently: Make it an easy read

The body of the email appears singularly focused on its graphic design and a clever visual way to represent what you do. I suspect that your target customers would prefer a plain-English explanation instead.

They would also likely appreciate it being summarized into a strong, benefits-focused headline, supported with several key reasons why they should use your company’s services, rather than your competitors’.

How it appeared in my Outlook preview pane. (click image to zoom)

I am making an assumption about your target customer segment(s), but from my experience—especially with B2B—black text on a white background works best most of the time. There’s rarely a better way to communicate with busy professionals.

Relying primarily on text, rather than images, will likely work better for you because in default Outlook setup with a preview pane, most people will see blank white boxes instead of your message—and promptly delete it. Alt text helps, but not as much as well-formatted HTML text. You need to make sure that your email degrades gracefully: it needs to read acceptably with images turned off and in plain text.

Communicating Value: Make it clear why you are the best choice

Again, there is no real headline here. The question “Is your business missing something?” is so generic that I can’t imagine it being compelling at all. You can have a successful question-format headline, but it needs to point to a specific problem that you know your customer has.

A great way further to support your value proposition is by telling the reader what your customers say about you. It’s more powerful than anything you say yourself.

There is another challenge with communicating value: you are offering a range of very different services. Sent to a large enough list, this will get you calls, but I would invest some time into 1) trying to segment your list and offer only the most relevant services to each segment, and 2) if you can’t segment or still end up with a large “general” segment, help your reader understand which service is right for them.

Communicating Action: Make it clear what to do next

You don’t want to leave this up to the recipients to figure out. That’s what we call “unsupervised thinking.” You need to do most of the work for them—or you won’t get the click.

There is no clear next step. Here’s what I can picture a recipient thinking: “It looks like you just want me to sign up for the newsletter. It’s the biggest CTA (call to action). But I don’t know who you are. I really don’t care about getting latest news postings on your website. If we already have a relationship, why am I getting this generic email?”

In the end, you are not giving the reader a specific reason to contact you. This goes back to building the problem, explaining why you are the best solution, and telling the reader what they’ll get by clicking where you want them to click.

If this is an email to an unsegmented list, I suggest two options to test:

1. Have only one CTA (you can repeat it at the top and at the bottom, but ultimately you should be asking them to do one thing). The job of this email will be to build enough confidence/interest in your company to get a click. Then you can provide options (if relevant) on the landing page.
2. Have several distinct offers, making very clear which one applies to which customer segment or specific problem it’s solving (even if you can’t segment the list, you should know what the key segments are). Then the job of this email is to help the reader quickly decide which offer is most relevant, and click on the corresponding CTA.

I hope these insights will be helpful, and I look forward to hearing about the results you were able to achieve with them.

Sincerely,

Boris Grinkot

To see more email optimization ideas, you can listen to the replay of our last live web clinic, where the MarketingExperiments team offered testing ideas for audience-submitted email marketing messages.

* I’m not counting 2007—come on!

Many marketers I talk to are quite interested in optimizing the content of their email messages. They test images, calls to action, subject lines, and the tone of the email. However, how many companies test the timing of email sends and how this affects readership?

Proper timing = greater relevance

To illustrate how timing might affect open and click-through rates, think about how you read email.  In the afternoon when the day is dragging on and you need a break, do you give each email message a little more time than when you first get into the office in the morning and are confronted with 20 hot items bursting from your inbox?

So would an email with a more complex conversion goal (such as signing up for a recurring subscription) do better with you in the afternoon while a simple conversion goal (like signing up for a free web clinic) might have a better chance in the morning when you’re plugging and chugging and not putting as much thought (and perhaps doubt) into your actions?

While you were sleeping

If you subscribe to our informative email, you know that we send it in the middle of the night. By testing, we learned that email messages sent before 9 a.m. EST dramatically lifted click-through rates for our list. Here are the key takeways from our testing:

• Late-nighters in the management level and ‘indiepreneur’ crowds on the West Coast are opening work email up until the midnight hour. East Coast execs are responsive in the ‘early bird’ hours.
• Subscribers based in Asia and Europe respond to email messages that don’t get buried in their inbox during non-work hours.
• Time zone segmentation is worth a test for any marketer with a substantial international list – especially B-to-Bers.

What works for your audience?

Keep in mind, that for every demographic and persona that is part of your readership, their habits and optimal send time might be different. Test sending out at different times to see what affect that has on not only readership, but conversion – because even in these “tight time zones,” people might just glance at the email, (giving you the open metrics) but save the action for later. However, we all know sometimes “later” never comes.

Speaking of testing, it is not just good enough to just try different send times for entire lists. Aggregate testing like this can get you subpar results and hide the real conversions nuggets. Narrowing the scope to particular segments in your list (which you should always be doing…) will help you see how certain segments respond to timing and allow you to make stronger conclusions.

Sometimes the conversion gems that are waiting to be discovered are not only in the message itself, but how and (in this case) when it is being delivered. It is like when you asked your Mom to borrow the car – you knew not to bother her when she was busy if you wanted a good response.

Good luck in testing.

For a deeper discussion about timing and relevance, you can join our Senior Manager of Research Partnerships, Andy Mott, as he explores Increasing Conversion with Right Time, Right Message Strategies on Thursday, March 11 at 2 p.m. This free BtoB Magazine webcast is sponsored by Eloqua.