MarketingExperiments Blog: Research-driven optimization, testing, and marketing ideas

As a presenter on our recent technology-focused web clinic, I had the pleasure of learning about an experiment devised by my colleague, Jon Powell, that illustrates why we must never assume that we test in a vacuum devoid of any external factors that can skew data in our tests (and even looking at external factors that we can create ourselves).

If you’d like to learn most about this experiment in its entirety, you can hear it firsthand from Jon on the web clinic replay. SPOILER ALERT: If you choose to keep reading, be warned that I am now giving away the ending.

According to the testing platform Jon was using, the aggregate results came up inconclusive. None of the treatments outperformed the control with any significance difference.  However, what was interesting is the data indicated a pretty large difference in performance with a couple of the treatments.

So after reanalyzing the data and adjusting the test duration to exclude the results from when an unintended (by our researchers at least) promotional email had been sent out, Jon saw that each of the treatments significantly outperformed the control with conclusive validity.

In other words, if Jon had blindly trusted his testing tool, he would have missed a 31% gain. Even worse, this gain was at the beginning of a six-month-long testing-optimization cycle. If Jon had assumed he had learned something based on inaccurate data that he really hadn’t, this conclusion more than likely would have sent Jon down a path of optimizing under false findings and assumptions.

In other words, to create a simple pre-GPS era analogy, if you make a wrong turn at the beginning of a 600-mile road trip and keep heading in the wrong direction, you will be much farther off the mark than taking the wrong road when you’re just a mile away.  However, in our cases with many businesses, wrong turns and mis-directions can cost from thousands to millions of dollars in lost time and revenue.

Worst of all, this email came from the Research Partner itself. As we run into many times, they unwittingly sabotaged their own tests. With the Internet being a dynamic place, it is next to impossible to avoid every external validity threat to your test, but at the very least we need to make sure that we are not introducing threats with internal campaigns to the same audience.

This is not to say we stop those campaigns, but just be aware of the potential effects on testing. That awareness, at least until computers become sentient beings, requires human involvement. Of course, that’s just one area where a little human curiosity is essential…

Do not let testing tools overshadow the human element of creativity.

Sure, many tools are now evolving to the point they will create “treatments” for you based on combinations, uploading content etc. But what this can create is a “perfect” sub-standard general design. These tools are limited to the inputs we give them so the optimization that can occur is constrained, where a human could take findings and radically change an entire process.

Begin by taking a step back, putting yourself in your customers’ shoes, and taking a human look at the big picture. Ask, “Is this even the proper overall design?” rather than taking the easy shortcut of testing a randomly generated combination of calls to action or headlines.

Multivariate testing (MVT) has its place. In fact, here at MarketingExperiments we use it frequently. But as with any tool, the result is only as good as the craftsman. So, when using MVT, make sure you have not ignored the big picture of what your users want by using the same sub-standard message presentation you’re trying to optimize in the first place as the base that you build your tests off of.

So how do we trust our tests? Here are steps for better setup.

• Sound Test Design – The test you are performing must represent the same environment where you are going to potentially apply the results. Many times we find people stretching the finding to different audiences, and then wondering why the results do not translate. For example, are you taking the lessons learned in email testing and applying them to your PPC ads? Well, they could each have different audiences that react in different ways.

• Research Question – Have you set a clear and specific objective for testing? Without establishing a clear objective, it is possible to get lost looking at a vast array of data points and trying to correlate them all. The research question also provides guidance on what items should be included in a test and what should be reserved for later.
• Proper Execution – Are you selecting the right test audience? Based on this audience, will you be able to apply the results to other aspects of your web communications?  Beyond that, you must ensure you have enough of this audience to reach a statistically valid conclusion, i.e. really learn something not just think you learned something. To do that, you must be recording accurate measurements. Ensure you double check your metrics technology before launching a test (more on that in the checklist).

• Confidence – Establish a standard for your results to uphold. Simply, you are trying to arrive at a finding that you have seen replicated enough times that you can confidently say, “we have sufficient information to make a conclusion on the research question we sought to answer.” The amount of times you need to measure will be a decision based on the volatility of the experimental environment and other factors. At the end of the day, though, it will also boil down to a business decision to continue or move on. This is something that needs to be agreed to and developed in house. Just understand that while setting this mark low carries some risks, some processes with low traffic or time sensitivity necessitate that we move on with lower confidence levels at times.

At MarketingExperiments, we try to stick to a 95% statistical significance as much as possible.  However, there are times where we have to accept a lower mark.

But remember, statistical significance from a piece of software cannot alert you to data that is inherently wrong or warn you that something else has influenced (and perhaps invalidated) a test, it only tells you that the results were unlikely to happen by chance.  Omniture (interesting alert for segmented data) and Google Analytics (GA intelligence) have been dabbling in this area, but still require human interaction and do not cover all aspects.

So make sure that you perform your due diligence with tool setup, test design, and data analysis – because it is very easy to gain confidence in the wrong decision with bad data from a tool that says it is 95% confident. Again, it is so important for us to invest greatly in people along with tools. As Avinash Kaushik says, you should invest 10% in tools and 90% in people.

Technology options/features that can trip you up.

• Metrics calculation process – Know how conversions are calculated (for example, visits vs. absolute visitor vs. page views, etc.). Many tools allow you to change how metrics get calculated, so make sure you are looking or pulling data using the same measure or comparison items throughout the test. Also, realize that individual tools may calculate conversions slightly differently.

• Default validity confidence levels – Understand your testing tool’s default measure of confidence and make sure that it matches your own internal measures.

• Default summaries – One of the most dangerous items in testing is the summary or dashboard view. Most of the juicy test details are hidden so problems that might be occurring in the test are tough to spot. Jon’s experiment is a great example of this. Looking at more specific data (like day-to-day metrics) will give you a better health check of what is happening with the test.

• Uniform sample distribution assumption – Tools assume that the data we are going to receive within tests will be uniformly distributed. However, if you have run your own test you know that this is not always true. As mentioned earlier, testing software has started adding some intelligence tools to try to spot “interesting” data points, but in our experience not many people use these tools. Non-uniform distribution can drastically affect validity and needs to be monitored…which means you need to pay attention to data closely (not in aggregate).

For a five-point testing technology checklist, and to learn more about other technology blind posts and how to address them, view the replay of our latest webclinic.

Related Resources

Online Marketing Optimization Technology: We have ways of making technology talk, Mr. Bond

Optimizing Site Design: How to increase conversion by reducing the technology barrier

Essential Metrics for Online Marketers

Clinic Notes

Today on our web clinic – Technology Blind Spots: How human insight revealed a hidden (and almost missed) 31% gain – we will be releasing never before published research from our laboratory. And you know what we like to do with our audience when we have fresh research that they have never seen before…

That’s right; we like to turn them into guinea pigs.

We like to see if our blog readers, knowing the basic circumstance surrounding a recent test, can predict the outcome. How good is their online marketing radar? Can they spot a good webpage when they see one? How is marketing intuition performing these days?

But honestly, what really matters is the cheese they will be racing for today – one good-ole slice of free online certification course cheese with a little Twitter-love wine to wash it down.

Leave a comment below to enter and let the games begin.

The Experiment

The Research Partner we were working with provides online consumer brokerage services through a subscription-based model. This page, in particular, was aimed at visitors interested in signing up for the foreign exchange trading (FOREX) solutions.

The Control (click to zoom)

After analyzing the current landing page, we concluded that there were some significant factors contributing to confusion on this page. For one, there were  many competing graphical elements and objectives. In almost all cases, this type of layout negatively impacts conversion. We also believed that the value of this offer could be communicated with a little bit more oomph.

So we tested three designs against the control to address some of these issues.

Treatment 1 (click to zoom)

The first treatment is probably the closest to the control. However, there are some strategic changes.

First, we added a headline that better communicated the value of the offer. The copy also has been reorganized in a clearer, easier to read fashion.

And finally, we added a call-to-action button in the main section of copy.

Treatment 2 (click to zoom)

The next treatment used a more long copy approach than the control. It also incorporated a stronger headline and clearer copy layout similar to that of the first treatment.

It is important to note that some of the visual elements from the control have been removed from the bottom of the page. However, the left-hand column remained the same as the previous two designs.

Treatment 3 (click to zoom)

This version of the page is almost identical to treatment 2’s long copy layout.

The one big change for this version was that the elements in the left-hand column were changed into a simple navigation.

(Update) The Results

If you are reading this post now, the contest mentioned above is over. Congrats to @terryrydzynski, a marketer who’s intuition got him a free seat in one of our online certification courses. If I were you, I’d follow this brilliant guy’s twitter account.

Which one was the winner you ask? All of the treatments outperformed the control, but Treatment 3 had the highest conversion rate with a validated 31% increase over the control. Now the results were not too surprising if you read some of the reasoning behind our designs above. Treatment 3 significantly reduced the amount of friction over the control by removing the competing graphical elements and focusing the visitor on one objective.

So what can we learn from this experiment?

If there’s one thing that we can all take away from this case study, it is that many times we are trying to accomplish way too much with our pages, and if we could just simplify our message and make options clear for our visitors, we would potentially see an increase in response.

But this is just scratching the surface, if you would like more information about this case study and some of it’s implications, you can find a more detailed explanation in the replay of yesterday’s web clinic, which will be available next week. To be notified when the replay is available, feel free to sign up for free research updates from MarketingExperiments.

Analytics & Testing, Clinic Notes, Landing Page Optimization

Editor’s Note: Research Manager Adam Lapp is reviewing the battle between common Internet marketing practices to help you determine which optimization strategies are most effective and give you ideas for new tests. On Monday, we published Part 1 in this series. On Wednesday, Part 2. Well get ready fight fans, hold on tight to your Mozilla Firefox, Apple Safari, or Google Chrome browser, because here comes Part 3…

Heat Map vs. Data Analysis

Breakdown:

When Heat Map enters the ring, he’s very intimidating, and looks great on the surface. He always shows up in a flashy, beautiful silk robe and has crazy music playing when he enters. Heat Map was a wizard at getting attention and he values the things that give him attention the most. He points to the camera flashes and banners and professes his love with a big red kiss.

All of Heat Map’s theatrics are fine. It’s part of the show. However, he gets himself into trouble by making outlandish conclusions about his opponents, like being able to know exactly what punch his opponent would land just by looking at the color of his clothes. Many times when Heat Map thinks someone will punch him in the rib cage, they instead throw a cross to his temple.

For his fight against Data Analysis, he trained mercilessly in the gym – lifting weights, doing crunches, and hitting punching bags. When it came time to fight, he strolled into the ring confident that his opponent’s strength was the left hook. Why? Well because he saw a red and yellow tattoo of a python on his left bicep.

Too bad he didn’t actually watch footage of Data Analysis’s previous fights. If he had, Heat Map would have known that most of his punches come from the right. What a shame to see him knocked out in a matter of seconds.

Sometimes Heat Map is right about his opponent though. Sometimes the key to victory is written on one’s sleeve. But it’s very, very dangerous to rely on just “sometimes.”

Data Analysis on the other hand is subdued and quiet. A stark contrast to Heat Map. But under that calm exterior is a supreme confidence because, unlike Heat Map, he has been studying his opponent’s film for hours. He’s learned that Heat Map throws himself a little off balance every time he attempts a left hook. He knows that Heat Map doesn’t bob and weave, but instead focuses his attention on one thing…in this case it’s the tattoo of the python.

Analytics & Testing

Nowadays the Internet is a battle royal. MySpace vs. FaceBook. We all know how that matchup turned out.

Google vs. Yahoo. This one’s scheduled for 12 rounds, but I think it will be a knock out.

And Google Buzz trained for years for a title match but failed a drug test. So for now, Twitter still holds the Light Heavyweight Belt, but eventually everyone meets their match. Anyone remember Buster Douglas knocking out Mike Tyson?

This spectacular entertainment is appreciated by so-called “gurus” of epic proportions. The fight for Internet marketing supremacy is a quality collection of bouts that will satisfy many an Internet guru, but how will it impact the masses in attendance?

Will they follow the hot trends blindly? Will tricks for quick bucks convince them to throw in their monthly budget for an unproven, highly touted secret weapon?

So whether it’s above-the-fold landing pages or live chats, you need to see the results before you invest your wad. There are reasons galore to believe “best practices” or “paradigm-shifting Internet sensation” hype and hope these latest and greatest tactics to generate revenue make a dent into your bottom line, but if you don’t personally see the fight through you are merely gambling. Do you really want to do that?  Before you start paying the piper, make sure you do your own research first so you have a high confidence level to make bets on the entire fight card.

The Undercards

Above “The Fold” Page vs. Sequence of Thought

The Breakdown: The Fold is not new to the fight scene. He originates from a previous generation of fighters who have mostly retired or passed on. The Fold comes from the newspaper ring, where everyone fought for the space on the top half of the page. For people who didn’t read an entire newspaper word for word, the top half of the newspaper page is where you would look for the most important and popular stories. Much of the content below the fold was missed.

The Fold has a new following on the internet. Everywhere I go people are talking about The Fold. He’s the place on the page where a typical visitor’s screen ends. Anything below him takes extra effort to get to either by manually scrolling or using your mouse wheel. Although it only takes a quick twitch of the index finger, many people will not exert this effort.

So what are we supposed to do? Cram every possible image, word, or call-to-action above The Fold? The answer is no, and our tests time and time again prove it.

With over a decade in the game, and 100+ professional fights under his belt, Sequence of Thought enters the ring to take down the champion. At first glance, The Fold appears ready. He’s neatly arranged and he dons a beautiful new robe. The Fold feels this is good matchup for his style. He is typically at his best when he can use his brawn to draw in conversions without much thought. He wants you to press his buttons right away without even doing the pre-fight research.

But consumers are getting smarter, more keen. His tactics have got him this far, but The Fold has met his match.

The reason why Sequence of Thought can defeat The Fold isn’t because of his appearance. The fight for conversions is not won on the page, but in the mind of the user. It’s won before anyone even steps in the ring. It’s won by understanding your visitors’ motivations and anticipating what they will do, what punches they will throw before they arrive on your website.

What’s their thought process? What information are they looking for? Do they need longer copy because the product requires a big decision? Or do you need to utilize the top half of the page to introduce yourself because you’re unproven in the ring?

Not only is it vital to understand the motivations of the user before you build your page, but it’s essential to test your way into an effective page design. Use metrics and data to fully optimize your site. Launch a test for a few weeks then analyze your visitors’ actions in order to identify patterns and find the areas of your site where improvements will produce the biggest ROI gains.

For some product offerings, a short page is effective. But The Fold is not a one-size-fits-all type of fighter. Our research has proven that often a visitor will scroll if you lead them down the page, either through compelling content or product descriptions. If you tap into their motivations and provide them with what they are looking for, page length is of no consequence.

ADAM LAPP’S UNOFFICIAL SCORECARD:

The Fold – 76

Sequence of Thought – 102

Pretty Page vs. Ugly Page

The Breakdown: All of the promoters these days are looking for lean and chiseled with a marketable face. Their goal is to attract attention, put up a visually stimulating facade in order draw viewers to the fight and maximize pay-per-view orders. Pretty Page has lots of colors and images on it. (Do customer service reps really look like the stock photos? Wow!)

There’s lots of flash (along with some Flash). White text on black background. I see at least three twitter icons. Gradients galore. And then there’s that video that I had go out of my way to click pause just so I could focus on reading the actual content of the page.

The Pretty Page looks good. As it should be since you probably paid a small fortune to have it designed and developed.

But does a Pretty Page increase your conversion rate?

Meanwhile in the cellars of the arena, Ugly Page is whaling away at a punching bag. Unlike his opponent, Ugly Page is a grizzled veteran with an acute knowledge of how to win a fight. This comes from years and years of experience. He knows that conversion doesn’t depend on looks, having the biggest biceps, or wearing the most colorful shorts. He knows that conversion depends solely on your strategy. And strategies are not built from the latest trends or by implementing quick hits that you learned at a one-day marketing conference. They are built from testing. From figuring out definitively what works and what does not work.

But the point is not that Ugly Page is a better fighter than Pretty Page. In fact, Pretty Page may eventually become a much better fighter than Ugly Page. The important thing to understand is that strategy is more important than design. You could have the best-looking page in the world, but if it hasn’t been tested, if it hasn’t been optimized based on what you learned from data and results, then it’s nothing more than a pretty page.

The fighter that will prevail is one that tests the layout, tone, copy, and positioning before testing images and color schemes. Ugly Page learns how to communicate his value as a fighter before he adorns himself with a fancy robe. He reduces friction and anxiety before his publicist gives him a makeover.

The bottom line is that the ideal testing sequence is to test key strategies first, figure out which works the best, then add an aesthetic layer.

ADAM LAPP’S UNOFFICIAL SCORECARD:

Pretty Page – 88

Ugly Page – 96

Let’s get ready to rumble

Throughout the week, I’ll be shedding light on more Internet marketing fights right here on the blog. We have Flash Banner vs. Headline, Quantity vs. Quality, Heat Map vs. Data Analysis, Bounce Rate vs. Conversion, and finally…the main card…Tricks vs. Testing.

Related Resources

The Magical Metrics Tour: Demystifying the secrets behind analytical “tricks” to help you drive ROI

A/B Split Testing – How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

Multivariable Testing – How testing multiple changes simultaneously can save you time, speed up your optimization schedule, and increase your profits

Analytics & Testing

So you’ve sat through our webinars or taken our online certification courses. You’ve identified what to test but haven’t yet thought about how you are going to test. How do you get started on the path to triple-digit returns?

LOGISTICS

When it comes to the “how to test” question, what most people usually want to know is “which tool should I use.” But before you delve into testing platforms, you should really ask yourself a few questions:

• Do I have IT or technical support to build out the new page/path to test?
  • If not, who can I contract to build out the HMTL of the page?
• Do I have FTP or Apache Subversion (SVN) access to modify the page code so that I can add the testing platform code?
  • If not, where does this fit into IT’s project queue?

TOOL SELECTION

Once you’ve documented the logistics to build out the test, now it’s time to choose the testing platform that is best for your business. There are quite a few options out there right now. To help you pick the right tool, I put together the following list with Jessica McGraw, our technical manager of research partnerships…

Google Website Optimizer

The biggest benefit is quite obvious – it’s free! So what do you get for nothing?

Let’s start with support. There is a lot of instructional information about GWO available on the GWO site, including step-by-step directions and FAQs. This makes it easy for just about anyone to implement. And it is fairly straightforward and easy to use with little IT involvement necessary.

Yet it has the capabilities to allow you to conduct everything from simple A/B split tests to more complex multivariate tests (testing different combinations of elements).

Paired with Google Analytics e-commerce tracking, you’re able to report at the transaction and product level.

It can take a few hours for data to begin showing in the reports.

Omniture Test&Target

Test&Target also offers A/B and multivariate testing along with transaction- and product-level reports, but it does so with some really impressive in-depth reporting. For example, you have access to not only aggregate data, but daily data as well. This gives you the ability to spot cyclical changes within your test data and validate tests quicker.

You also gain the flexibility to target content to different groups of people based on defined segments. If you don’t know which segment should get which content, you can find out by setting up a test with different segments and watching the results of each segment individually to determine your new content. Test&Target also includes a feature where offers can be automatically targeted to site visitors based on several different variables including observed visitors’ on-site behavior.

In terms of support, you are not on your own. Your contract likely provides access to an account rep along with support hours to gain some insight into reports and help with complex test setup.

You data will likely appear in reports in mere minutes. And you can integrate reporting into a Site Catalyst account (Omniture’s web analytics product).

And while Omniture is a paid tool, the cost difference compared to a free tool might not be clear cut, because your company may already have a contract with Omniture. If it does, find out the additional cost to use Test&Target.

Unbounce

Unbounce bills itself as having “self-served, hosted landing pages for marketers with A/B testing.” It allows you to not only design the page, but build it and set it up all within the same tool.

This is an easy-to-use service which requires little to no IT involvement. You even get a

WYSIWYG (what you see is what you get) landing page editor, so you can build landing pages with no web designer or without even having any HTML experience.

The tradeoff is that you can only run simple A/B split tests.

Drupal

If your site uses a content management system (CMS) like Drupal, there is likely a split testing plug-in that is available. Drupal itself is free and open source, and if it is your CMS and you use it for testing, you are assured instant compatibility with no need to pay for extra testing platforms.

As with other open-source solutions, “free” usually comes at a cost. Not surprisingly, Drupal requires a lot of IT involvement. Although, new platform updates are coming that will require less from IT.

On the flip side, since Drupal is open source, you can add any and all reporting features your particular tests require.

Build your own

You can always build your own split testing tool. Well it’s free, right? And by free, I mean that there are no outside costs or need to engage a consultant on product questions.

And you certainly have flexibility to build your own reports. You also have the support of in-house staff with the full knowledge of tool capabilities and ability to add extra features as IT has time.

And there’s the rub…”as IT has time.” You’ll have to gauge exactly what those words mean for your particular organization.

BUILDING THE TIMELINE

This last section usually doesn’t fall under “how to test” for many marketers, but rather “why I really can’t test even though I’d like to.” In fact, in a poll we conducted in a recent web clinic, time was identified as our audience’s biggest challenge to begin the implementation of the testing-optimization cycle in their organization.

This might seem like an insurmountable challenge up front, (since most marketers feel like they can barely keep up with their deadlines as it is). But as with any excuse (I have no time to exercise, eat right, learn Swahili, etc), it can be overcome by simply planning ahead and getting a firm understanding of the resources you will need.

When building your marketing campaign timelines, keep in mind that depending on what direction/solution you take with testing… you’ll need to build in extra time for:

• Reviewing your site analytics to identify your biggest opportunity and the best elements to test
• Working with a designer to modify your current design and incorporate the test variables
• Fitting into the development queue (if not using Unbounce). In most cases, marketing is the lowest priority in the development world, so ensure that you have your place in the queue before setting an expected launch date

Related Resources

Technology Blind Spots: How human insight revealed a hidden (and almost missed) 31% gain

A/B Split Testing — How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

Cost of Delay: How to win approval for your test and test schedule

What to test (and how) to increase your ROI today

How Can You Make Your Web Site Smarter?

Are You Listening Fast Enough? Using A/B and Multivariate Testing to Accelerate Conversion Improvements

Analytics & Testing

As I was reading a few LinkedIn discussions about multivariate testing (MVT), I began to wonder if 2010 was going to be the year of multivariate.

1,000,000 monkeys can’t be wrong

Multivariate Testing (MVT) is starting to earn a place in the pantheon of buzzwords like cloud computing, service-oriented architecture, and synergy. But is a test the same thing as an experiment? While I am not a statistician (nor did I stay at the Holiday Inn last night), working at MarketingExperiments with the analytical likes of Bob Kemper (MBA) and Arturo Silva Nava (MBA) has helped me understand the value of a disciplined approach to experimental design.

What I see out there is that a little knowledge is indeed a dangerous thing. Good intentions behind powerful and relatively easy-to-use platforms like Omniture® Test&Target™ and Google® Website Optimizer™ have generated a misleading sense that as long as a multivariate test is large enough (several hundred or more combinations being tested), at least one of the combinations will outperform the control.

This notion has become the value proposition of a growing number of companies offering services around either the big-name or their own (simpler, and often therefore easier to set up) MVT tools. They are ostensibly betting on the technology, and not on a systematic approach to experimental design or any particular UI/UX (user interface/user experience) optimization theory.

Even though, as Bob has pointed out to me, it is reasonable that an MVT setup with a billion combinations may not yield a lift over the control, my contention is that the risk-weighted business cost of a dissatisfied customer is low. Therefore, little stops the burgeoning MVT shops from safely offering a “100% lift guarantee.” Just like the proverbial million monkeys with typewriters, somewhere among thousands of spray-and-pray treatments their MVT tests are expected to produce one that’s better than the rest.

1 monkey with a stick

One major difficulty with testing in general becomes painfully obvious with MVT: the more treatments, the longer the test will run. For most companies, what looks at first like a great test may require a year’s worth of traffic to get statistically valid results.

In response, one emerging MVT service model offers getting to a “lift” faster by using adaptive elimination of likely underperformers, in exchange for the test results providing limited information beyond identifying the winner. Such test results are not as useful as their full-factorial brethren for designing subsequent tests because adaptive elimination of treatments makes it difficult to extrapolate the psychological factors and consumer preferences responsible for the test outcome. The immediate business benefits, however, are more immediate.

So, where exactly is the problem? As marketers, are we in the business of employing the scientific method to design graceful experiments or is our fiduciary duty to get measurable results? I humbly suggest that as marketing professionals, we should neither bet on nor be satisfied with just one test, no matter how successful it is.

The bad news and the good news is that we must design an experimental plan to optimize continually, to learn from preceding test results, and to respond to changes in customer preferences, market conditions, and our ability to segment data and traffic. Expertise in experimental design and understanding how to interpret results simply cannot be replaced by set-it-and-forget-it technology (yet).

Economy of testing

That is not to say that MVT provides incorrect results. The results are mathematically valid, even if they do require a long time to obtain. At the same time, from the business point of view, investment into experimental design expertise is expensive. Understanding volumes of published research consumes valuable time. The 100% guarantee sure sounds good.

And so the “guaranteed lift” offers will appeal to the spendthrift marketers who are yet to delve into the science of optimization. The critical issue in the economy of testing is whether methodical design of experiments is likely to provide greater ROI through an interpretation-driven sequence of test iterations than a successful, but terminal one-off test. Our research supports the former.

2010 may become the year of multivariate, but I hope that it will also quietly set the stage for an upcoming year of ROI-conscious design of experiments.

How do you use multivariate testing? Have you created an experimentation plan or do you rely on a series of one-off tests? Share your triumphs and concerns in the comments section of this post or start a conversation with your peers in the MarketingExperiments Optimization group.

Analytics & Testing