MarketingExperiments Blog: Research-driven optimization, testing, and marketing ideas

Editor’s Note: In a recent interview with MarketingSherpa Editor Sean Donahue, Research Analyst Corey Trent outlined errors even experienced email marketers make when conducting tests. (My personal favorite – #5). We thought this information was valuable, and wanted to share it right here on the blog for those who do not have a MarketingSherpa membership. Special thanks to our sister company for allowing us to republish the below article…

SUMMARY: Before you conduct your next email test, make sure you’re not falling into a trap that can muddy your results or limit the gains you might otherwise achieve.

We spoke with an email testing expert from our sister company, MarketingExperiments, to uncover common mistakes marketers make when running email tests. Read why good analytics and segmentation are crucial forerunners to testing, and why a blockbuster discovery from one test actually can be a risky thing for a marketing team.

by Sean Donahue, Editor, MarketingSherpa

Testing is an essential component of a strong email marketing strategy. But only if the tests are conducted and analyzed properly to ensure you’re helping – not hurting – your email performance.

“There is a cost for bad testing,” says Corey Trent, Research Analyst, MarketingExperiments. “Bad assumptions based on bad tests can cost you a lot of money and cause you to lose out on a lot of business.”

Trent routinely conducts email tests as a member of the MarketingExperiments sales and marketing optimization research team. Through this work, he’s seen how mistakes, misconceptions and simple oversights can derail a well-meaning marketer’s testing strategy.

We asked him to share his advice for avoiding testing pitfalls, so you can achieve your goal of improving email performance. Here are seven common mistakes he’s observed:

Mistake #1. Not having the right analytics behind your tests

Sometimes marketers must recognize they are just not ready for email testing, says Trent.

In particular, he says, marketers often don’t have the processes or the analytics systems in place to capture a complete data picture — you must be able to tie your email metrics to business goals, such as revenue.

For example, if you’re only measuring email metrics, you might test a new subject line treatment that increases open rates by a few percentage points. Sounds exciting, but without tying subsequent clicks from those messages to website sales, you might not notice that you’re actually decreasing conversions and total revenues because the subject line is creating the wrong expectations about your message.

The good news is that it is now relatively easy to track email activity through to website metrics using tools such as Google Analytics. This program offers marketers ways to track email messages through specific goals on your site, such as product pages viewed or time spent on site (see Related Resources below).

“Email tracking all the way through to business goals is available for everyone, and doesn’t take a long time to set up or require having an IT team to do a lot of programming for you,” says Trent.

So make sure your analytics systems are robust enough to support your objectives before you begin email testing.

Mistake #2. Not segmenting email lists for tests

Many marketers have large email databases, but don’t know a lot about the records held within. In these cases, they may conduct an email test using their entire database – and create a muddy results picture.

“Without segmented lists you don’t get good test results,” says Trent. “You get all these people responding differently to your emails, which pulls your results in all different directions.”

Spend the time to segment your database and understand the different characteristics of the segments before you embark on email testing. The more you know about unique segments within your database prior to testing, the better chance you’ll have of finding the right messages to appeal to them.

Mistake #3. Stopping tests after one big win

“With email more than anything, we see people get a big win and stop testing,” says Trent.

As exciting as those big wins may be, they shouldn’t be the end of your testing process. The makeup of your email lists is constantly changing; external factors, such as the economy, also impact subscriber behavior; and your competitors’ campaigns and tactics are always changing as well.

This constant state of change means you must routinely work on the messaging, layout, calls-to-action and other elements of your email messages to ensure you’re getting the full benefit of a testing program.

Mistake #4. Testing too often

The flip side of stopping tests too quickly is falling into a pattern of repeated testing, to the point that you simply can’t keep up with all the data you’re generating.

When finding the right pace for testing new email treatments, think critically about the tests you want to perform and their potential impact. Focus on the ones that have the greatest potential impact on your specific business goals.

Trent suggests researching your current email strengths and weaknesses: Examine your own email and website metrics and study industry research to benchmark your own performance against your industry peers. This process can uncover where your email programs are delivering weak performance – and where you have a bigger chance of improving your company’s bottom line through testing.

“Those deep dives into the data you have available help you prioritize that list of potential tests,” says Trent.

Mistake #5. Overlooking email copy tests

Today’s email messages are so versatile that marketers have a wealth of options to test: Layout changes, images, fonts, colors, social media integration and so on.

But the ability to tweak these elements can make marketers forget about the old standby: copy tests.

“What often gets left out is how important the copy and communication pieces of an email are,” says Trent.

Copy changes can deliver surprising results, so keep them on your list of test options. You can try using a different voice for your message, long copy vs. short, different copy layouts, or even a personal letter from someone in your company.

Mistake #6. Always testing additions, rather than subtractions

The pressure to generate more engagement with subscribers can push marketers to continually test new additions to their emails: More images, more buttons, more links or more colorful text.

Too many additions can actually confuse recipients, making it hard for them to decide what to do with your message.

Instead, Trent has seen marketers achieve strong performance gains by eliminating elements from an email and reducing the number of decisions a recipient has to make.

By focusing a message on a single goal — such as inviting recipients to an event or generating sales within a specific product category — you can see whether there are extraneous elements that might be distracting recipients from the desired action. You can then test the impact of removing those distractions.

Mistake #7. Not asking other customer-facing teams for test ideas

Great test ideas don’t always come from the marketing team. It’s likely you have other sources within your company whose insights can shape a good email test.

Trent recommends talking to sales people or customer service representatives to learn what they’re hearing from customers. The feedback these teams receive from customers about your products and services — or why they chose to buy from you — can spark ideas for new subject lines, email copy, or specific products to feature in email promotions.

“Don’t be scared to go down and talk to those people to get some good ideas.”

Related Resources

The Magical Metrics Tour: Demystifying the secrets behind analytical “tricks” to help you drive ROI (describes how to track email clicks with Google Analytics)

What Else Can I Test….To Increase Email Click-through?

Discover the Best Time to Send Email: 4 Test Ideas (for MarketingSherpa members)

Improve Your Email Programs: 5 Test Ideas (for MarketingSherpa members)

Email Marketing, Research Topics

As a presenter on our recent technology-focused web clinic, I had the pleasure of learning about an experiment devised by my colleague, Jon Powell, that illustrates why we must never assume that we test in a vacuum devoid of any external factors that can skew data in our tests (and even looking at external factors that we can create ourselves).

If you’d like to learn most about this experiment in its entirety, you can hear it firsthand from Jon on the web clinic replay. SPOILER ALERT: If you choose to keep reading, be warned that I am now giving away the ending.

According to the testing platform Jon was using, the aggregate results came up inconclusive. None of the treatments outperformed the control with any significance difference.  However, what was interesting is the data indicated a pretty large difference in performance with a couple of the treatments.

So after reanalyzing the data and adjusting the test duration to exclude the results from when an unintended (by our researchers at least) promotional email had been sent out, Jon saw that each of the treatments significantly outperformed the control with conclusive validity.

In other words, if Jon had blindly trusted his testing tool, he would have missed a 31% gain. Even worse, this gain was at the beginning of a six-month-long testing-optimization cycle. If Jon had assumed he had learned something based on inaccurate data that he really hadn’t, this conclusion more than likely would have sent Jon down a path of optimizing under false findings and assumptions.

In other words, to create a simple pre-GPS era analogy, if you make a wrong turn at the beginning of a 600-mile road trip and keep heading in the wrong direction, you will be much farther off the mark than taking the wrong road when you’re just a mile away.  However, in our cases with many businesses, wrong turns and mis-directions can cost from thousands to millions of dollars in lost time and revenue.

Worst of all, this email came from the Research Partner itself. As we run into many times, they unwittingly sabotaged their own tests. With the Internet being a dynamic place, it is next to impossible to avoid every external validity threat to your test, but at the very least we need to make sure that we are not introducing threats with internal campaigns to the same audience.

This is not to say we stop those campaigns, but just be aware of the potential effects on testing. That awareness, at least until computers become sentient beings, requires human involvement. Of course, that’s just one area where a little human curiosity is essential…

Do not let testing tools overshadow the human element of creativity.

Sure, many tools are now evolving to the point they will create “treatments” for you based on combinations, uploading content etc. But what this can create is a “perfect” sub-standard general design. These tools are limited to the inputs we give them so the optimization that can occur is constrained, where a human could take findings and radically change an entire process.

Begin by taking a step back, putting yourself in your customers’ shoes, and taking a human look at the big picture. Ask, “Is this even the proper overall design?” rather than taking the easy shortcut of testing a randomly generated combination of calls to action or headlines.

Multivariate testing (MVT) has its place. In fact, here at MarketingExperiments we use it frequently. But as with any tool, the result is only as good as the craftsman. So, when using MVT, make sure you have not ignored the big picture of what your users want by using the same sub-standard message presentation you’re trying to optimize in the first place as the base that you build your tests off of.

So how do we trust our tests? Here are steps for better setup.

• Sound Test Design – The test you are performing must represent the same environment where you are going to potentially apply the results. Many times we find people stretching the finding to different audiences, and then wondering why the results do not translate. For example, are you taking the lessons learned in email testing and applying them to your PPC ads? Well, they could each have different audiences that react in different ways.

• Research Question – Have you set a clear and specific objective for testing? Without establishing a clear objective, it is possible to get lost looking at a vast array of data points and trying to correlate them all. The research question also provides guidance on what items should be included in a test and what should be reserved for later.
• Proper Execution – Are you selecting the right test audience? Based on this audience, will you be able to apply the results to other aspects of your web communications?  Beyond that, you must ensure you have enough of this audience to reach a statistically valid conclusion, i.e. really learn something not just think you learned something. To do that, you must be recording accurate measurements. Ensure you double check your metrics technology before launching a test (more on that in the checklist).

• Confidence – Establish a standard for your results to uphold. Simply, you are trying to arrive at a finding that you have seen replicated enough times that you can confidently say, “we have sufficient information to make a conclusion on the research question we sought to answer.” The amount of times you need to measure will be a decision based on the volatility of the experimental environment and other factors. At the end of the day, though, it will also boil down to a business decision to continue or move on. This is something that needs to be agreed to and developed in house. Just understand that while setting this mark low carries some risks, some processes with low traffic or time sensitivity necessitate that we move on with lower confidence levels at times.

At MarketingExperiments, we try to stick to a 95% statistical significance as much as possible.  However, there are times where we have to accept a lower mark.

But remember, statistical significance from a piece of software cannot alert you to data that is inherently wrong or warn you that something else has influenced (and perhaps invalidated) a test, it only tells you that the results were unlikely to happen by chance.  Omniture (interesting alert for segmented data) and Google Analytics (GA intelligence) have been dabbling in this area, but still require human interaction and do not cover all aspects.

So make sure that you perform your due diligence with tool setup, test design, and data analysis – because it is very easy to gain confidence in the wrong decision with bad data from a tool that says it is 95% confident. Again, it is so important for us to invest greatly in people along with tools. As Avinash Kaushik says, you should invest 10% in tools and 90% in people.

Technology options/features that can trip you up.

• Metrics calculation process – Know how conversions are calculated (for example, visits vs. absolute visitor vs. page views, etc.). Many tools allow you to change how metrics get calculated, so make sure you are looking or pulling data using the same measure or comparison items throughout the test. Also, realize that individual tools may calculate conversions slightly differently.

• Default validity confidence levels – Understand your testing tool’s default measure of confidence and make sure that it matches your own internal measures.

• Default summaries – One of the most dangerous items in testing is the summary or dashboard view. Most of the juicy test details are hidden so problems that might be occurring in the test are tough to spot. Jon’s experiment is a great example of this. Looking at more specific data (like day-to-day metrics) will give you a better health check of what is happening with the test.

• Uniform sample distribution assumption – Tools assume that the data we are going to receive within tests will be uniformly distributed. However, if you have run your own test you know that this is not always true. As mentioned earlier, testing software has started adding some intelligence tools to try to spot “interesting” data points, but in our experience not many people use these tools. Non-uniform distribution can drastically affect validity and needs to be monitored…which means you need to pay attention to data closely (not in aggregate).

For a five-point testing technology checklist, and to learn more about other technology blind posts and how to address them, view the replay of our latest webclinic.

Related Resources

Online Marketing Optimization Technology: We have ways of making technology talk, Mr. Bond

Optimizing Site Design: How to increase conversion by reducing the technology barrier

Essential Metrics for Online Marketers

Clinic Notes

Back in March, Google got more serious about protecting user data privacy (as it should be), and to that end has announced plans for a browser-based opt-out for Google Analytics.

In typical Internet fashion, the blogs and Twitter lit up with doom and gloom news that web tracking was dead…run for the hills web analysts. In fact, it was very reminiscent of the reaction people had when Germany announced they were going to investigate the legality of Google Analytics and collecting data on their citizens. But, as with past incidents, people calmed down and life went on.

Personally, I am all for Internet privacy. We as businesses and marketers need to respect users’ wishes if they decide they do not want to be tracked (even if retail counterparts do not honor this). If a significant amount of people are choosing to opt out, then we need to adapt and find other ways to determine what our users want and need. Heaven forbid we talk or engage them more personally (see blog post of different ways to do this online).

But in talking with people that have concerns about being tracked online (especially by Google Analytics), I typically find that they simply misunderstand what the tool does. Most people with concerns feel like it is a Big Brother tool that tells us exactly who they are, tracks them after they leave our site, and relays to us every website they visit.

They lighten up significantly when I tell them that the tool is really used to anonymously look at users of our website and help us understand how to make our process, products, or websites better.

In fact, we cannot even see (at a personally identifiable level) who these people are if we are following Google Analytics’ terms of service. Once they hear this, most skeptics see the value and how it can really make the Internet a better place without negatively impacting privacy.

What I hope Google does

When Google releases this opt-out feature, my desire is they also release an FAQ page or other educational outlet to debunk the privacy myths about Google Analytics. I truly believe it is urban legends, what people do not know (fear of the unknown), or other more invasive measurement tools that cause the privacy scares. Perhaps if people are properly educated they will be less reluctant to opt out.

Whatever it decides to do, Google should err on the side of too much information. When it comes to anxiety, perception is reality. And as with any anxiety, we must overcorrect to overcome it.

But in the end, it is ultimately up to the consumer. And as painful as this is to say as a web analyst, we need to respect our visitors and their desire for privacy, even if it is a poor or wrongly informed concern.

Related Resources

What the “opt-out” option means for Google Analytics

Google adds more flexibility and intelligence to Analytics and Website Optimizer

Friction and Anxiety in your marketing process: defining the difference

Analytics & Testing

We have been quite busy at the labs here, but I wanted to cover a PPC development that blipped on our radar earlier this year. For many of us, PPC is a critical source of traffic, and can be quite the task to manage. Well to add to the list of things to consider, Google is beta-testing the collection of phone lead information directly from SERPs (Search Engine Results Pages).

Google generates roughly 97% of its revenue from online advertising, so it makes sense that they delve into new areas of online marketing – which now seems to include part of the sales process as well.

Given the huge potential (or threat) this represents to you, the Internet marketer, I think this is a vital development to cover on this blog (and even reached out to a search engine marketing firm to get their ideas for you as well.) While this will not affect all verticals, for some niches this might pour some gasoline (or more correctly napalm), on already very competitive areas.

So how does it work?

According to Amber from PPC Hero, “The gist of the beta is if you’re running a PPC ad in Google, and you’re in the top position, you can click on a plus sign next to a call to action (the name you give your contact form) and Google will drop down your contact form to be submitted right there in the Google search results.”

Here is an image from that article to show the lead collection:

It’s important to note that, as with all things in beta, this has the potential for change, as I believe Google does do some testing from time to time.

And since this new feature is currently in beta, limited to businesses appearing in position #1 of PPC results, and on select keywords, it may not be available to you yet. If you’re interested, the best person to contact is your friendly Google rep.

More importantly…how can I make it work for me?

While I think this could have potentially large ripples in certain areas of online marketing, I wanted to have a chat with our friends at ROI Revolution, and get another viewpoint as well.

As a quick background, ROIRevelution is a company whose main area of focus is in pay-per-click management and they are quite good at it. We had the chance to speak with PPC Account Manager Mike O’Rourke. Here are just a couple of points Mike and I discussed:

Upsides

• Simplifies the conversion process.
• No friction (and lead loss) from weak landing pages.
• Great tracking for businesses that use the phone a lot, a traditional weakness of current PPC tracking solutions.  Interesting, how could this be applied to other voice communication/VOIP solutions like Google Voice?
• The feature will be good for certain verticals (e.g., plumbers) that provide local services tied to specific keywords (e.g., “broken pipes”).

Downsides

• Can’t have a conversation in a PPC ad. Landing pages provide a good service for addressing anxiety and questions. So how effective will this lead capture be?
• Opportunities to increase the value of a lead with a well-optimized funnel are lost (e.g. upsells). Without the ability to do that how valuable will the leads really will be?
• Because conversion is expected on the search page, you do not have their undivided attention.  On a landing page, you can guide their thinking.
• Maximum cost-per-click (CPC) rates are applied to leads submitted. This will likely cause your cost per acquisition to rise.
• Since you must in position #1, the bids for the top position might heat up significantly.
• Many people also now use many CRM solutions for keeping track of customers and sales. Will this information be easily integrated with common solutions or sent to advertisers in a standardized way?

Won’t you tell me your name? I love you. Hello.

Personally, I am interested in seeing are how users respond to putting their information directly into a form on a search engine results page. While I think some users are very trusting, others might be turned off by your attempt to get their number so soon.

It might seem like an out-of-sequence conversation. As Flint McGlaughlin, the director of MECLabs Group (our parent company) describes trying to ask for a lead to early in the process, “You don’t ask a girl for a kiss before you have a date with her.”

That might seem kind of corny. But think about how you handle your own phone number. Many of us treasure our phone numbers and can guard them quite fiercely. Will a relatively short ad space be enough to capture users and convince them to give up this information? Are you asking for too much, too soon?

Also, some of the things we talk about in great detail here are anxiety and value proposition. While you can address these with your ad copy in a limited fashion, the fact remains you only have so many characters to work within.

Your pay-per-click ad copy and the space search providers give you is simply to get the conversation going – address why they should click, how you are meeting their motivation, and then get them to a landing page to do the heavy lifting.

In skipping that step, you can end up with spending a lot of money (especially by paying max CPC) with leads that are not as qualified or do not convert.

Also, since this is a new technology, you might be getting a fair amount of people using it because they wanted to see how it works, rather than being genuinely interested in your message.

I have some more thoughts on this that we will post soon, but I want to get your opinion on what impact, if any, you think this new Google innovation will have?

Paid Search Marketing (PPC), Research Topics

Many marketers I talk to are quite interested in optimizing the content of their email messages. They test images, calls to action, subject lines, and the tone of the email. However, how many companies test the timing of email sends and how this affects readership?

Proper timing = greater relevance

To illustrate how timing might affect open and click-through rates, think about how you read email.  In the afternoon when the day is dragging on and you need a break, do you give each email message a little more time than when you first get into the office in the morning and are confronted with 20 hot items bursting from your inbox?

So would an email with a more complex conversion goal (such as signing up for a recurring subscription) do better with you in the afternoon while a simple conversion goal (like signing up for a free web clinic) might have a better chance in the morning when you’re plugging and chugging and not putting as much thought (and perhaps doubt) into your actions?

While you were sleeping

If you subscribe to our informative email, you know that we send it in the middle of the night. By testing, we learned that email messages sent before 9 a.m. EST dramatically lifted click-through rates for our list. Here are the key takeways from our testing:

• Late-nighters in the management level and ‘indiepreneur’ crowds on the West Coast are opening work email up until the midnight hour. East Coast execs are responsive in the ‘early bird’ hours.
• Subscribers based in Asia and Europe respond to email messages that don’t get buried in their inbox during non-work hours.
• Time zone segmentation is worth a test for any marketer with a substantial international list – especially B-to-Bers.

What works for your audience?

Keep in mind, that for every demographic and persona that is part of your readership, their habits and optimal send time might be different. Test sending out at different times to see what affect that has on not only readership, but conversion – because even in these “tight time zones,” people might just glance at the email, (giving you the open metrics) but save the action for later. However, we all know sometimes “later” never comes.

Speaking of testing, it is not just good enough to just try different send times for entire lists. Aggregate testing like this can get you subpar results and hide the real conversions nuggets. Narrowing the scope to particular segments in your list (which you should always be doing…) will help you see how certain segments respond to timing and allow you to make stronger conclusions.

Sometimes the conversion gems that are waiting to be discovered are not only in the message itself, but how and (in this case) when it is being delivered. It is like when you asked your Mom to borrow the car – you knew not to bother her when she was busy if you wanted a good response.

Good luck in testing.

For a deeper discussion about timing and relevance, you can join our Senior Manager of Research Partnerships, Andy Mott, as he explores Increasing Conversion with Right Time, Right Message Strategies on Thursday, March 11 at 2 p.m. This free BtoB Magazine webcast is sponsored by Eloqua.

Email Marketing, Research Topics

In my last blog post, I challenged you (and myself as well) to be more proactive in approaching customers for feedback. I recently found an excellent example on Twitter of an auto detailing supply company tying in the New Year with an offer to give feedback on things they can do better in the coming year.

Finding the right incentive

Notice they also offer a small incentive for providing feedback. However, it is important to note that the incentive is not a brand new car or a Neil Diamond Cruise Trip. It is just enough to pique the interest of followers, but probably not enough to cloud the feedback with nonsense in an effort for a chance at winning the car wax.

Social media caveat

When using social media channels for feedback, please understand the control you have with this medium. Remember that with the advent and popularity of social media channels you can receive a flood of real-time feedback from all over the globe.

But that feedback rolling in can be positive or negative. The negative feedback can now become very visible not only to your past customers but also prospective ones.

Now I say all of this not to dampen your enthusiasm for using emerging social media, but to have you consider what the outcome could be once you hit that update button on Twitter.

So consider questions like these… Did your company have a massive shipping issue this year? Was a bad batch of products sent out? Did you offend Ashton Kutcher in any way?

While the feedback you could gather is still very valuable, you might choose a more controllable channel to gather customer feedback if you have reason to believe some of that feedback may be negative.

Speaking of feedback, we want to hear from you. What is your favorite source of news and information in the advertising and marketing industry? Not a blog or magazine you just like…but something you really love enough to send a Valentine to on this upcoming (and well-marketed) holiday.

Email or share your feedback using social media and we’ll publish our favorites in a future post right here on the blog.

Practical Application