MarketingExperiments Blog: Research-driven optimization, testing, and marketing ideas

As I was reading a few LinkedIn discussions about multivariate testing (MVT), I began to wonder if 2010 was going to be the year of multivariate.

1,000,000 monkeys can’t be wrong

Multivariate Testing (MVT) is starting to earn a place in the pantheon of buzzwords like cloud computing, service-oriented architecture, and synergy. But is a test the same thing as an experiment? While I am not a statistician (nor did I stay at the Holiday Inn last night), working at MarketingExperiments with the analytical likes of Bob Kemper (MBA) and Arturo Silva Nava (MBA) has helped me understand the value of a disciplined approach to experimental design.

What I see out there is that a little knowledge is indeed a dangerous thing. Good intentions behind powerful and relatively easy-to-use platforms like Omniture® Test&Target™ and Google® Website Optimizer™ have generated a misleading sense that as long as a multivariate test is large enough (several hundred or more combinations being tested), at least one of the combinations will outperform the control.

This notion has become the value proposition of a growing number of companies offering services around either the big-name or their own (simpler, and often therefore easier to set up) MVT tools. They are ostensibly betting on the technology, and not on a systematic approach to experimental design or any particular UI/UX (user interface/user experience) optimization theory.

Even though, as Bob has pointed out to me, it is reasonable that an MVT setup with a billion combinations may not yield a lift over the control, my contention is that the risk-weighted business cost of a dissatisfied customer is low. Therefore, little stops the burgeoning MVT shops from safely offering a “100% lift guarantee.” Just like the proverbial million monkeys with typewriters, somewhere among thousands of spray-and-pray treatments their MVT tests are expected to produce one that’s better than the rest.

1 monkey with a stick

One major difficulty with testing in general becomes painfully obvious with MVT: the more treatments, the longer the test will run. For most companies, what looks at first like a great test may require a year’s worth of traffic to get statistically valid results.

In response, one emerging MVT service model offers getting to a “lift” faster by using adaptive elimination of likely underperformers, in exchange for the test results providing limited information beyond identifying the winner. Such test results are not as useful as their full-factorial brethren for designing subsequent tests because adaptive elimination of treatments makes it difficult to extrapolate the psychological factors and consumer preferences responsible for the test outcome. The immediate business benefits, however, are more immediate.

So, where exactly is the problem? As marketers, are we in the business of employing the scientific method to design graceful experiments or is our fiduciary duty to get measurable results? I humbly suggest that as marketing professionals, we should neither bet on nor be satisfied with just one test, no matter how successful it is.

The bad news and the good news is that we must design an experimental plan to optimize continually, to learn from preceding test results, and to respond to changes in customer preferences, market conditions, and our ability to segment data and traffic. Expertise in experimental design and understanding how to interpret results simply cannot be replaced by set-it-and-forget-it technology (yet).

Economy of testing

That is not to say that MVT provides incorrect results. The results are mathematically valid, even if they do require a long time to obtain. At the same time, from the business point of view, investment into experimental design expertise is expensive. Understanding volumes of published research consumes valuable time. The 100% guarantee sure sounds good.

And so the “guaranteed lift” offers will appeal to the spendthrift marketers who are yet to delve into the science of optimization. The critical issue in the economy of testing is whether methodical design of experiments is likely to provide greater ROI through an interpretation-driven sequence of test iterations than a successful, but terminal one-off test. Our research supports the former.

2010 may become the year of multivariate, but I hope that it will also quietly set the stage for an upcoming year of ROI-conscious design of experiments.

How do you use multivariate testing? Have you created an experimentation plan or do you rely on a series of one-off tests? Share your triumphs and concerns in the comments section of this post or start a conversation with your peers in the MarketingExperiments Optimization group.

Analytics & Testing

Email design always proves to be a hot topic with marketers. And when you have top agencies competing against each other, the fire just gets hotter as we learned during last week’s live web clinic – Maximize your Agency ROI: How adding science to the creative process reveals a 26% gain.

We received a plethora of questions, most which we could did not have time to address during the hour-long clinic. So, as with every Web Clinic Extra, we have picked a handful of the most common questions to address here on our blog. This week we pulled in Andy Mott, the Senior Manager of Research Partnerships, to answer these questions…

Email marketing is a topic that comes up often in the MarketingExperiments community. In fact, Dr. Flint McGlaughlin is delivering a keynote today at Em@il Summit ’10 in Miami as well as teaching a pre-summit Live Email Optimization Workshop. If you couldn’t make it out there this year to get valuable insights from your peers and industry leaders, come back to the blog on Friday for some key takeaways from this year’s summit.

You can view a replay of the clinic or read the latest issue of MarketingExperiments Journal. Our next live web clinic, The Five Best Ways to Optimize Email Response (Part 2): How to craft effective email messages that drive your customers to action, will be taught on February 3rd from 4 to 5 p.m. EST.

Clinic Notes, Email Marketing, Marketing Q&A

Senior Manager Boris Grinkot recently attended the Eloqua Experience 2009 Global User Conference in San Francisco. He sat down to answer a few questions about his key takeaways for online marketers…

Q: You were the only non-Eloqua employee to have a booth in the Marketing Effectiveness Zone. Users from around the world sought your advice about conversion rate optimization. Was there any pattern to the challenges marketers brought you?

Most websites are not really having a conversation with the customer. They are not guiding the customer through the page. And, most importantly, they are not testing their pages. They may test an email message or a form with Eloqua, but they don’t test the whole page.

There is massive room for improvement to optimize landing pages to ensure they are customer-focused, and then continually improve the results generated by those pages through testing.

Q: Measuring those results was also a major topic of conversation at Eloqua Experience?

Yes. Eloqua is seeking to standardize key performance indicators, or KPIs, that CxOs use to measure results in much the same way financial accounting uses key ratios and cash flow statements, income statements, and balance sheets. Today, most measurements in web analytics – like visitors per month, for example – do not directly translate to a standard accounting metric, such as net revenue.

Q: Based on past statements we’ve received from web clinics and here on the blog, KPIs are a little misunderstood by some marketers. How can readers choose and use KPIs effectively?

In our research, we find marketers most frequently tend to look at numbers like conversion rate, bounce rate or number of visitors. While these numbers can be meaningful within the context of page or process functional performance, they don’t necessarily do a good job of measuring the financial performance.

Revenue per visitor is usually an essential KPI that connects online customer behavior to a financial outcome. While this is a more straightforward KPI to calculate for ecommerce sites, even if you have a lead-generation site, you should understand the value of each lead to determine your revenue per visitor.

Average order size can also be a meaningful KPI that helps you distinguish customer segments (e.g., collectors vs. gift shoppers) or test functional changes like in-cart upsells. Choosing the right KPIs is a big topic, but the short of it is that you need to distinguish between behavioral, demographic, and financial metrics and use them appropriately.

Q: So the important thing to remember is, that while some metrics might be useful in an intermediate step, the overall goal should be a revenue-based number?

Absolutely. For testing and optimization, you need those intermediate numbers.

For example, if you’re optimizing just one step of a shopping cart, conversion rate or clickthrough are important testing metrics. But you don’t want to lose sight of the big picture – which would be overall revenue or revenue per visitor. Let’s say you’ve optimized a step and more visitors are clicking through, but they are less motivated and in the end are buying less. If you don’t have that overall revenue KPI, and you just looked at conversion rate, you would erroneously assume that you have definitively improved your shopping cart.

Q: What are some other caveats when choosing a KPI?

Even if you choose the right KPI, you can still get bad information by not looking at individual channels. You may have good revenue per visitor, but that number is just an average. What if one channel is delivering ten times the revenue per visitor of another channel? That is important information to have, especially if you’re paying for traffic, because you need to understand how that spend converts to revenue.

Q: What is the overall benefit of choosing the right KPI for C-level reporting?

By focusing on dollars instead of traffic, business leaders gain a deep understanding about how the investments they make impact revenue generated through the website. If you invest more in a site, what ROI are you getting?

Clickthroughs and conversion rates only muddy the answer to this question. You need numbers directly related to the spend of each campaign. And not in the aggregate, but specific to a channel – so you know that investing X in PPC, email, TV, or even print will lead to a return of Y.

The other upside, which I’m glad Eloqua is pushing, is that numbers like overall revenue, revenue per visitor, or cost per acquisition bring web metrics much closer to standard accounting. These numbers give CxOs the best indicators of their site’s performance in a format they are used to seeing.

Use the comments section below or post your questions to our MarketingExperiments Optimization group.

Analytics & Testing, Internet Marketing Strategy, Marketing Q&A, Practical Application

Did you catch our free Web Clinic on Wednesday? The topic was Measuring What Matters: How simplifying your metrics can increase Marketing ROI by up to 75% — and if you joined us, chances are you’re already implementing new ideas and tools to improve your analytics.

If you couldn’t make it, you can view the presentation here and download our free MarketingExperiments Essential Metrics Tool here (Excel file).

Metrics isn’t the sexiest topic, yet it’s one that most marketers have been grappling with for years and still don’t have many concrete answers. In our live poll, 74% of the marketers characterized their experience level with Web analytics as moderate to novice.

There’s a lot more to cover with metrics than our one-hour session allowed: Different tools, the type of website and levels of data, your depth of experience with analytics, to name just a few.

Many attendees told us the big takeaway was our blueprint for distilling several data points into just four key measurements — and using that to get beyond numbers and into your customers’ mindset.

To paraphrase Dr. McGlaughlin, too often the focus with analytics is on us: the actions we’re trying to force or entice, the conversion rates we want to see, the transactions and revenue we desperately need to achieve. Those are valid measures, but they obscure the intentions of our prospects and customers when they visit our sites.

The trick is taking all those raw numbers and using them to create a snapshot of what your site visitors are thinking, as well as what they’re doing. That’s what really helps us adapt our processes and content and improve ROI dramatically.

Several attendees requested another Clinic on this topic, so we’ll likely revisit metrics with a new session in the months ahead. In the meantime, please enjoy the complete Clinic and try our Essential Metrics Tool with our compliments.

We’d appreciate any feedback you have on the metrics Cinic or Tool, and invite you to post any other metrics-related comments you’d like to share.

Clinic Notes, Marketing Insights, Practical Application, Site Metrics

Budgets are tight. Bosses are telling their employees that “nice-to-do” conferences and non-essential training and travel are out until further notice.

So here are two things to consider when deciding how best to spend that tight marketing budget.

One: Consider taking the MarketingExperiments online Landing Page Optimization course, and never leave your desk. That’s one way to make your boss happy. Training with no travel required. You can apply what you learn to your company’s site as you go, and once you pass the test, you’ll be MEC-certified in Landing Page Optimization.

Two: Consider the case for a very, very short trip to Ponte Vedra Beach, Fla. during the first week in June, when we’re going to teach a condensed version of the LPO course in just twelve hours over one and-a-half days.

Yes, this workshop will be intense, it will be a lot to learn, and it will not be easy. But when you deliver that first short, sharp, shock to your company’s online performance, believe me, neither you nor your boss will regret it.

Those who have taken our professional certification courses know how substantial the ROI can be for a company that sends the marketing ranks through them. By making carefully selected changes and adjustments to a product’s or service’s Landing Page, the certified experts who come out on the other side of our course know how to help their companies gain double- , triple-, and in some cases even quadruple-digit increases in click-through rates, conversion, leads, and revenue.

Yeah, that’s right: quadruple. Sounds like essential training to me.

If you can’t make it to Ponte Vedra Beach, you can register here for the online courses in Fundamentals of Online Testing, Landing Page Optimization with an emphasis on subscriptions, or on e-commerce, or email marketing.

Internet Marketing Strategy