|Thursday, 29 December 2005|
Topic: Multivariable Testing — How testing multiple changes simultaneously can save you time, speed up your optimization schedule, and increase your profits.
We recently released the audio recording of our clinic on this topic. You can listen to a recording of this clinic here:
Traditional thinking tells us that accurate testing of web pages, email messages, or offline media depends on us changing just one element at a time.
After all, if you change more than one element on a website sales page, and testing shows an increase or decrease in performance, how will you know which element was responsible for the change?
If you are conducting a simple A/B split test, this principle remains true. You need to test just one change at a time.
However, multivariable (or multivariate) testing allows you to test many changes simultaneously – five, ten, or even twenty. You'll still get accurate results, without having to increase your total sample size, and you will be able to identify the impact of each individual change.
But how does multivariate testing work? Is it reliable? How does it stack up next to A/B testing?
In this brief, working with data from a variety of sources, we will show you how multivariate testing works, and how it may be able to help you dramatically and quickly improve the performance of your site pages and email messages.
Beyond the immediate improvements in page performance, multivariable testing offers two other important benefits.
Is multivariable testing the best choice for every company and in every circumstance? Are there times when a simple A/B split test is exactly what you need?
1. How does a multivariable test differ from an A/B split test?
An A/B test isolates a single page variable and tests two variations against each other. Here are the results of a simple A/B test:
What You Need To UNDERSTAND: The optimized page improved the conversion rate of our test site from 6.00% to 6.51%, an increase of 8.5%. The 87 additional orders were worth approximately $2600 in added revenue.
In a recent brief, we covered A/B split testing in great detail. In order to get the most out of the material below, we recommend you review that article if you are unsure of how to implement effective A/B split testing.
A multivariable test transcends the limitations of a simple A/B test in two ways:
With A/B split testing, it can become tedious to isolate and optimize all of the elements on a page, one at a time. In response, a number of companies have developed testing platforms that will simultaneously test a variety of elements such as graphics, background color, headline text, body copy, "call to action," and other page constituents. Each of these elements or variables can be tested with two or more variations. The testing software splits incoming traffic among the variations and shows each visitor only one version of the composite page.
Here are the results of a multivariate micro-test. In the results below, the variables are sorted by conversion rate. This was a rather simple test, using only two variations for each of the six variables, resulting in twelve rows of results:
What You Need To UNDERSTAND: In this multivariate test, six page elements were tested simultaneously. Background color had the largest impact on conversion rate (an increase of 65.6%), followed by call to action (an increase of 56.8%).
The above test was implemented using software provided by Vertster.
Again, this was a rather simple test, using only two variations for each of the six variables, resulting in twelve rows of results.
Multivariate testing, though, can make it practical to test with many more variables and variations of each, although by increasing the variables and values, the number of visitors required for a conclusive test increases exponentially. Section Four, below, discusses how providers of multivariate testing platforms have addressed this problem.
In a subsequent multivariate test, the headline proved to be one of the most important factors that impacted the effectiveness of the page. Here are the results from just the six headline variations:
What You Need To UNDERSTAND: When weighed against the original page, headline 1 resulted in a 20.39% increase in conversion.
You can also view a screenshot showing the actual Vertster interface which generated these results:
So with all of the above, why wouldn't you just convert to 100% multivariable tests? Multivariable testing is not the ultimate tool, but rather one of several you should use to optimize your marketing. There are times when a simple A/B test will prove superior.
A/B split testing offers the following advantages:
Multivariable testing should be used in different circumstances:
2. How has multivariable testing helped companies improve their marketing?
In this section, we will look at two case studies of companies who have successfully used multivariate testing to improve their marketing.
The first company, JoAnn.com, is a website serving millions of arts and crafts enthusiasts. The company used Offermatica to set up multivariate tests in a number of site areas. After one round of testing, they registered the following improvements:
What You Need To UNDERSTAND: This company was very pleased with the results of its multivariate testing, including an overall revenue per visitor improvement of 209%.
But the most memorable lesson was this: the offer that the marketing team thought would be the least viable ("buy two sewing machines and save 10 percent") actually generated the highest return. "People were pulling their friends together and we sold enough ... machines to outperform single purchases," said Linsly Donnelly, JoAnn.com's chief operating officer.
KEY POINT: Even the smartest marketers are often proven wrong by testing. Intuition is no substitute for well-designed experiments.
The second case study we looked at was Monster.com, a leading employment website.
In one of Monster.com's multivariate tests (with Offermatica), they optimized their "jobs" page by testing four page elements, each with three (or two) values. These page elements were:
The default page consisted of the original copy and layout, no displayed savings amount, and no savings calculator:
The winning page included new copy, a stronger headline, savings in dollars from buying in bulk, and a savings calculator:
KEY POINT: The optimized page resulted in an 11.6% increase in performance.
3. What is the Taguchi Method and when should it be used?
Consider a hypothetical multivariate test that has five variables with three values each. There are 243 possible composite pages built from these elements. It may take 100 or more conversions to each of these pages to generate the most trustworthy results. If your conversion rate averages 5%, it would take 486,000 visitors to generate accurate results under these conditions (*1).
The following table illustrates the exponential increase in number of composite pages as variables and values increase:
What You Need to UNDERSTAND: As the number of variables and values increase, the number of composite pages increases at an exponential rate that often defies simple intuitive marketing design.
The Taguchi Method uses "fractional factorial testing" to reduce the number of variations necessary to determine the variables and values with the greatest impact. It was originally implemented 50 years ago and has been used successfully to test automobile and other product manufacturing. More recently, companies have begun to apply the Taguchi Method to direct marketing and the Internet. The method dictates exact combinations of page elements that allow a marketer to determine accurate estimations of the most important variables on the page, and the best values for those variables. The length of the test cycle and the number of visitors required using the Taguchi Method is surprisingly small.
For more on the Taguchi Method, see:
KEY POINT: Several multivariate service providers feature Taguchi-enabled testing, which significantly reduces the amount of traffic necessary to create meaningful test results.
4. What are some of the key insights we have learned through A/B split and multivariate testing?
Keeping the above points in mind should help you get the most out of A/B and multivariate testing. Finally, we have listed a number of vendors of multivariate testing platforms in the "Literature Review" section, below.
As our future experiments reveal what really works, we will continue to share our findings. If you have any suggestions for future topics, please let us know.
(*1) To calculate the number of visitors needed to generate reliable multivariate testing, simply multiply the number of composite pages (see table above) by the historical average number of visitors required to create one conversion and by the number of conversions to create statistically significant results (typically 50-200).
V = P * C * S
V = Total Visitors
RELATED MEC REPORTS:
As part of our research on this topic, we have prepared a review of the best Internet resources on this topic.
These sites were rated for usefulness and clarity, but alas, the rating is purely subjective.
* = Decent ** = Good *** = Excellent **** = Indispensable
Editor — Flint McGlaughlin
Writers — Brian Alt
Contributors — Jalali Hartman
HTML Designer — Cliff Rainer