MarketingExperiments Blog: Research-driven optimization, testing, and marketing ideas

Toward the end of last year, Active Network, a technology and media company specializing in online registration and event management software, began a testing and optimization program on the website of one of its brands, RegOnline.

This week’s MarketingSherpa B2B newsletter article is a look at that entire program, but for this blog post I want to highlight an interesting test conducted in the middle of the cycle. This particular test deserves a closer look because it created impressive results, but more importantly, it illustrates why it is important to be flexible with a testing program.

At the beginning of Active Network’s testing cycle, the tests were conducted on the RegOnline homepage. The problem was that tested treatments were not producing positive results — and the homepage accounts for about 90% of RegOnline’s revenue.

Essentially, with the treatments being outperformed by the control, the homepage tests were hurting RegOnline’s bottom line.

This caused concern and discouragement with both the testing team and top-level management at Active Network.

A decision was made to begin testing other channels, such as landing pages, PPC ad copy and email messaging, to hopefully find some learnings that could inform future homepage tests. Another benefit was because these other channels had a more singular purpose than the homepage, it was easier to design a test to get a “win” and ease some of the discouragement.

Lauren Guinn, Director Online Marketing, Active Network, explains, “Because there is so much business risk testing on our homepage, we shifted to smaller tests on other channels and then applied these learnings to big tests on our homepage.” Read more…

Imagine for a second that you’re running a banner ad campaign on a website with a black background. Now, if you were going to choose a color for that banner to get more clicks, would you choose a dark shade of blue or bright yellow?

If you’re like me, you would have guessed yellow. Unfortunately for our marketing intuition, a recent test we ran with a Research Partner proves otherwise.

For this test, we ran two different versions of a banner offering a large financial institution’s branded version of a credit card, alongside a $50 gift card incentive. Essentially, if the visitor applied for the branded credit card (the same brand as the site on which the banner ad appeared), they could get a $50 gift card after making qualifying purchases. The test itself was an A/B sequential test. Read more…

Copy length. It’s been somewhat of an obsession of mine lately. For marketing copywriting. Heck, for content in general.

So I was really looking forward to running a pretty simple copy length test – long vs. short copy.

However, the interpretation was anything but simple. In fact, at first I thought there wasn’t a significant difference between the control and treatment…until we dove down a little deeper.

But I’m getting ahead of myself. Let me set the test up for you…

- Read more…

In college, I had a journalism professor who said, “Make your headline twice as powerful as the event.” This is sage advice if you’re covering the Kardashian beat for a weekly tabloid, but it doesn’t seem to directly apply to today’s marketers.

(Apologies to any members of the Kardashian marketing team who may have been offended by the previous comment. You’ve done a terrific job promoting whatever it is that made them famous.)

But maybe it is applicable, after all. Despite a marketer’s goal of thoroughly conveying value on a landing page through well-crafted body text and use of images, there often remains a need for a powerful “hook” to further motivate a potential customer.

(Just recently, MarketingExperiments held a Web clinic that covered this very subject.)

When it comes to landing pages, one would think that it would be much easier to get users to convert, as they have already expressed a certain amount of interest via search or email clickthrough. But these users still need to be quickly reminded of why their clicks landed them there in the first place.

In the following test, you’ll see that a continuation of the value proposition was deftly handled with the addition of a few short, powerful words, alongside a much-needed change in the way our partner asked for information. Read more…

Professional infiltrator should seriously be added to my job description. Because once again, I gathered some marketing “intel” from somewhere that I wasn’t quite, how do I put it…invited to?

This time, I stepped out of the MECLABS classroom and into a beautiful, oceanfront hotel in Jacksonville Beach for a MarketingExperiments Landing Page Optimization Workshop. There, I blended into a crowd of about 70 marketers; listening to the presenters, Director of Training, Chuck Coker, Senior Optimization Manager, Adam Lapp (Mr. Lapp when class is in session) and MECLABS Managing Director (CEO), Dr. Flint McGlaughlin.

As I sat there taking notes, one subject really stood out to me, and that was testing. It seemed as though many questions were geared towards that. “How big does your sample size have to be in order to know your test results really worked?” “How long should you test for?” “What should you test?” etc… Read more…

As I delve deeper into the abyss of numbers while preparing for my interactive panel on validity at Optimization Summit, I’m coming across more fallacies about validity in marketing tests. Here’s one I recently heard…

“We were told that if we send each treatment to 4,000 people on our list we would have a valid test.”

I changed the number to protect the innocent, but this is a common misconception.

That’s why they play the game

Now I’ll give them credit. The misconception above is thinking along the correct lines. A large enough sample size is necessary to ensure you have validity.

However, while you can take an educated guess, it is impossible to know the minimum sample size before the test is actually run. Just ask a Las Vegas bookie.

Because, an important factor in sample size determination is the difference in results between the treatments. If the treatments return very different results, it’s much easier to confidently say that you really do have two (or however many) emails that will perform differently. You don’t need as many samples to do that.

However, if the treatments have very similar results, you want many more observations to see if there really is a difference.

Think of it this way. I recently went to Disney World, and while waiting in line for a ride the line split.I was  curious to see if more people would go to the left or the right.

If I saw nine people go to the left, and one go to the right, I’d feel pretty confident that people tend to favor the left.

But what if the split was six to the left and four to the right? I would want way more observations to feel confident that there is a real difference, whether people really do favor one side over the other, or if what I’m seeing is just random chance. Maybe for the next ten people, six will go to the right and four to the left.

And that’s why it’s impossible to determine the exact sample size you need for every test. You would, essentially, need to know the response you would get for each treatment before you tested. And, after all, that’s why we run the tests. Because it’s nearly impossible to guess on an outcome. Again, just ask a Las Vegas bookie. The house often wins. But not always.

I asked Phillip Porter, a data analyst here at MECLABS, for a more official sounding explanation than my Mickey Mouse example above; an explanation that you can use verbatim to sound smart and win any internal debate. Here’s what he said…

“Significance is based on sample size and effect size. The larger the sample you have, the smaller effect size you can find to be significant. The larger the effect size present, the smaller sample size you need to find significance. Larger sample sizes are generally better, however any difference, no matter how small, can be found to be significant with a large enough sample size.”

“Imagine the thrill of getting your weight guessed by a professional.”

However, much like Navin R. Johnson’s carnival barker in “The Jerk,” it doesn’t hurt to take a guess as long as you know what you’re doing. And if you truly are a professional, you don’t have anything to lose by guessing (except maybe some Chiclets).

“You can guesstimate before you run the test, but the actual numbers may not match what you were expecting. If you could guesstimate correctly every time before running the test, why would you need to run the test?” Phillip said. Proving not only that great minds think alike, but that it doesn’t hurt to try to get a sense for how much traffic or email recipients you need to get a valid test.

In fact, we include a pre-test estimation tool in our Fundamentals of Online Testing course. There’s one in the MECLABS Test Protocol as well, which our researchers use for all of their experimentation.

But a true professional will still run the final numbers…

How to determine if you have a big enough sample size

I don’t want to get all Matt Damon writing on a chalkboard in “Good Will Hunting” on you, but (and read this in your best Boston accent) it comes down to the math. I had a tough time on how I could truly serve you with this blog post. The last thing I wanted to do was raise a problem and not give you a solution (or only provide a paid solution).

I think, in the end, the best thing to do is just provide you with the equation. The sample size calculation that we use internally can be found in Cochran, W. G. (1977). Sampling Techniques, 3rd ed., Wiley, New York.):

Phillip explained the formula…

“This formula provides us with the minimum sample size needed to detect significant differences when Z is determined by the acceptable likelihood of error (the abscissa of the normal curve). The value of Z is generally set to 1.96, representing a level (likelihood) of error of 5%.  We want the highest accuracy possible, with the smallest sample size.  This level of error, 5%, gives us the best tradeoff between these two goals.

p is the conversion rate we expect to see (estimate of the true conversion rate in the population), and d is the minimum absolute size difference we wish to detect (margin of error, half of the confidence interval).”

We are working on a dead simple validity tool (the iPod of validity tools, if you will) to pass out at Optimization Summit. But for now, you can try putting the above formula in your own Excel spreadsheet.

Even Phillip admitted, “If you are trying to calculate this by hand it can look intimidating, but if you build the formula in Excel it is pretty simple.”

Related Resources

Optimization Summit 2011 – June 1 -3

Email Marketing Tests: What to do when a radical change produces negligible results

Online Testing and Optimization: ROI your test results by considering effect size

Online Marketing Tests: How could you be so sure?

Testing Madness: What the odds of picking a perfect NCAA Tournament bracket can teach us about running valid tests