Live Experiment (Part 1): How many marketers does it take to optimize a webpage?
Last week I had the privilege of being in the world’s tallest hotel in the Western Hemisphere, joined by over 200 other “nerdy” marketers, for what was the first-ever conference hosted by MECLABS on the topic of optimization and testing. Overall, it was a value-packed week.
But what I found most uniquely valuable about the Optimization Summit was a surprise live experiment in which the audience was asked to optimize and test a marketing campaign during the course of the conference. I had a backstage pass to what would become a thriller of an experiment, with many ups and downs, bends and turns. The only thing I could compare it to while in Atlanta was trying to hold two suitcases while free-standing on MARTA (which I successfully did by the way).
All in all, I learned a lot about testing in the process, and in the next two blog posts, I’d like to break out some of the key insights I walked away with.
The Original Page (click to zoom)
The subject of the test was a real offer from the Premier Sponsor of the event, HubSpot. They had partnered with MarketingSherpa to offer a free chapter of the 2011 MarketingSherpa Landing Page Benchmark Report.
The overarching goal for the campaign was to increase HubSpot’s knowledge of their current email list. So, the audience was charged with increasing the form submission rate.
Here’s the catch, they could only optimize five elements:
1. The headline
2. The product image
3. The call-to-action
4. The copy
5. The page layout
If you’ve ever been in a contentious meeting with your team, you know it can be extremely difficult to get two marketers to agree on the most effective approach. So, how do you get more than 200 marketers to agree on any one of these elements? Well, first you wow them with flying pigs (hey, anything is possible with optimization). And then immediately afterward, you break them into small groups of about three to four, give them a limited amount of design options, and then force them to come to consensus in 15 minutes or less.
That’s exactly how this went down (minus any gravity-defying farm animals). The groups’ choices were then tallied up and whatever options got the most group votes made it into the treatment.
For instance, here were the four potential options for the image. For this variable (as circled in red) the audience overwhelmingly chose option #3.
Here were the different layout options and the winning vote:
Making sense? You can see the rest of the variable options (or values) here.
The Treatment Page (click to zoom)
After all the votes were cast and counted, the audience had decided that a single column layout with short copy, a stronger headline and benefit-centric call-to-action would perform better than the control.
From my perspective, the two major optimization advantages behind this new design were:
1. It strengthened the Value Proposition through specificity in the headline and image, highlighting the nature of the free content for download
2. It reduced Friction on the page by shortening the copy, and providing clear singular eye-path.
From here we launched the test, sending out 800,000 email messages, posting four blog posts (including this one here), and tweeting over 140,000 followers.
Overall, we received so much traffic that we could statistically validate a 1% change in conversion. And believe it or not, it would come down to that.
So, do you agree with 200+ marketers?
If there was one thing that I learned from this stage of the process, it’s that great (marketing) minds don’t always think alike. A couple of the test options were selected by a landslide, but most were so evenly split that we didn’t know the winner until the last vote was counted. There were many divided groups, heated discussions, and one marketer even got voted off an island. But overall, this whole experience helped me clearly see why online testing is so desperately needed: it helps marketers live at peace with each other.
With that said, if you are a marketer reading this, I know you have an opinion.
- What do you like or not like about the new design that the audience voted on?
- Which of the options would you have chosen instead? Why?
- Do you think the treatment will win?
On Friday on the MarketingExperiments blog, we will do a deep dive into the results of the experiment and the key insights that I learned from the outcome.
Landing Page Optimization Workshop: Become a Certified Professional in Landing Page Optimization
B2B Marketing Summit: Join us in Boston or San Francisco to learn the key methodologies for improving your conversion rates