Archive for October, 2009

Ask an Optimizer: How to structure and execute better tests

October 22nd, 2009

Editor’s note: During our October 14 web clinic on overcoming testing obstacles, senior researchers Boris Grinkot and Gaby Diaz fielded several audience questions on optimization testing. We’ve distilled the best questions and answers for the latest edition of our Ask an Optimizer column.

Q: How long should we split test a variable or collect data points in order to make a valid decision?

You need to use a statistical formula called the chi-square test that takes into account the number of samples you are going to collect, the amount of variance (between the control and the best performing treatment), and the statistical confidence level you desire before making a business decision (typically 95%).

Since you won’t know the variance before you start the test, you must project it to begin with and then plug the actual numbers into the formula as you conduct the test. The number of samples will be affected by the variance. The larger the variance, the fewer samples you need to collect to reach a statistically significant conclusion.

To learn more about testing, consider taking our The Fundamentals of Online Testing course, which includes our proprietary test protocol.

Q: How many tests can you have going on at the same time without skewing your results?

You can run as many multiple tests at the same time as you want, as long as they are independent and do not impact each other.

Q: What are the benefits of multivariate testing vs. A/B testing in marketing?

Multivariate testing has become quite a buzzword, so instead of focusing on the benefits, let’s discuss some things you have to take into account for a successful test. A multivariate test will be heavily impacted by some obstacles we mentioned before – time, IT resources, availability of analytics, etc.

Mainly, you need a substantial amount of testing to reach statistically significant results. On the testing web clinic, we talked about radical redesigns as a solution to a time crunch. But if you have the time, you can run a multivariate test to reach intermediate conclusions before drilling deeper with A/B split tests.

How much time do you need? As in question one, duration is affected by how many samples you have and the number of treatments. The more treatments, the more samples you have to get and the longer the test needs to run.

For example, in a single factorial A/B split test, your traffic is split just in two ways. So if you have 100,000 visitors, 50,000 see the control and 50,000 see the treatment. But if you have three headlines, three versions of the body copy, and three versions of the call to action, that becomes a fairly large number of combinations and will split your test up many more ways so it will take that much more time to get a statistically valid result.

Once a multivariate test helps you determine which elements had the biggest influence, A/B split tests will help you optimize further. For example, if you see the headline had a big influence in a multivariate test, you could perform single factorial tests to see which headline works best.

But keep in mind, on its own a multivariate test can be fairly hard to interpret. While you may find a certain combination performs well, it can be difficult to determine which of multiple variables to attribute those gains to. It is best to use multivariate testing first, then take your best guess about which elements to drill deeper into and test further.

You’ll also want to review three past research briefs that cover these areas in more depth: Multivariable Testing, A/B Split Testing, Conversion Rate Optimization Tested.

Q: How do I test two different pages with two different channels? For example, two pages with my new customers and two pages with my returning customers?

By using cookies, you can identify which customers are new and which are returning. Since returning visitors are already highly motivated and can overcome high levels of friction, you might not even want to show them a homepage at all. Perhaps show them their latest purchases or a recommendation page suggesting related items based on what they’ve purchased in the past.

Keep in mind, as we saw with the web clinic‘s second case study, it was much harder to achieve gains from the shopping cart traffic since it was already converting so high. You likely have much more room to gain from your new visitors. On the other hand, even small gains in your highest converting channels could provide a major boost to the bottom line. Testing different channels will help you understand what works best for each of them.

Q: I’ve been running a test for over a month and haven’t seen a statistical significance. Should I keep running the test or stop?

You should stop. Month-to-month variations in your traffic, due to seasonality for example, will start to skew the results. Reassess the page and launch a new test.

Q: How do you test for customer loyalty to a website?

You have to get really creative with your data analytics. One of the obstacles is prioritization. First, identify your objective. That may seem obvious ­– “my priority is to increase customer loyalty.”

But you have to drill deeper to understand how to measure customer loyalty to determine your key performance indicator (KPI). For example, one measurement might be return visits to your site. Working with a research partner recently, this was an important performance indicator. So KPI would be the number of visits per month from the same visitors (unique visits divided by total visits per month).

For this example KPI, you can run a test to determine which treatment is going to get people coming back to your site. You could test by sending target emails that remind them of reasons to return. If you have a retail site, perhaps the emails highlight a sales promotion. If it’s a service site, highlight an action they need to take on the site ­– like completing a form.

We teach a whole separate course on email optimization. But at a high level, the goal of an email is to get them to click back to your website. You don’t need all the information in the email itself. Get them to come back. And then you can test and measure the loyalty in terms of return visits.

Q: Can you test different channels with Google Analytics?

Yes. Combined with Google Website Optimizer, you can test different channels. Google Website Optimizer will track the conversion rate and statistical significance. But if you have unique URLs and mark your page with Google Analytics, you can see how they’re working in combination with your website and see the type of visitors you’re getting.

You can also see success metrics such as the conversion rate and where people navigate after that page. You can set up tests for channels such as PPC, banner advertising, or any other channel. Also, see our recent post on how to use the new features in Google Analytics.

Have additional questions? Other things you’d like to Ask an Optimizer? Use the comments section below or post your questions to our
MarketingExperiments Optimization group.

Google adds more flexibility and intelligence to Analytics and Website Optimizer

October 20th, 2009

As I sat in the conference room of building 14 at the Googleplex last week, my excitement about what Google was doing to improve its analytics and testing platforms went through the roof. At the 2009 Partner Summit, Google gave a preview of the new Website Optimizer (GWO) features as well as soon-to-be-launched, feature-packed version 4 of Google Analytics.

So what’s new with GWO? Well, there is not a lot, but the few features Google added to its testing platform are HUGE! For me, two of the most notable updates are:

  1. Management API for the creation and management of experiments outside of the Website Optimizer interface
  2. Daily conversion tracking

The GWO API will allow you to much more easily set up tests and record stats, especially if you are using a content management system or a third-party shopping cart. Basically, you will be able to do almost anything you can do in GWO, but with your own interface.

As for the daily conversion tracking, I have always been one of those people who like to see test results to the minutest detail. Until now, Google has only allowed you to see aggregated stats for the entire testing period.

As of today, we can now see daily conversion rates:

Daily Conversion Rate

This will give us much more intelligence regarding daily conversion swings and subtleties between the performance of experiment combinations. To read more, visit the GWO blog.

Now, I am even more excited about a couple of new features Google Analytics has included in its newest version. My favorites are:

  1. 20 Goals, including brand new engagement goals
  2. Custom alerts
  3. Advanced filters built into report interface

I’m sure many of you have shared my frustration when trying to track more than four goals for one website. Before, we would have to create a duplicate profile of our website just to add goal five, six, seven….

Well, that’s all changed! Google has announced 20 goals for each profile. These will be grouped into four sets of five, but you will be allowed to use the 20 slots however you want.

Screenshot of new 20 goal limit:

More Goals
But it doesn’t stop there. Add on top of that the fact that you will now have the ability to set user engagement goals. And you are no longer limited by a goal being attributed only to a pre-defined conversion page. The new engagement Analytics goals are:

  1. Time on page
  2. Pageviews per visit

Engagement Goals
You will also be able to set custom alerts to be emailed to you. By “custom,” I mean you set the variables however you want. So if I want to know if my website’s New York PPC traffic drops 15%, Google will let me know and I can act accordingly.

Screenshot of custom alerts:

Custom Alerts
Finally, they have added custom filters to the report interface. Sure you could export the data to Excel then sort, filter, and do whatever you wanted to make the data make sense to you. But now, Google has added the flexibility to get a clear and intelligent picture of what’s going on with your website without having to go back and forth between its interface and Excel.

Screenshot of advanced filters:

Advanced Filters
Say I’m looking at my Top Content report and I want to sort by bounce rate to see which pages are performing poorly so I can dedicate new resources for improvements to be made. Previously, I would see many pages with a bounce rate of 100% simply because they just had one or two visits.

Now, I can set the filter to only look at pages with a minimum of 100 visits and a bounce rate of less than 70%. This will allow me to make much more sense out of the heaps of data that Analytics is collecting.

You should be seeing all of these new features and many more appear in your Analytics and Website Optimizer interfaces over the next few weeks. We will be taking advantage of them right away with our research partners.

Check back next month — after we’ve tested some of these new features, we’ll offer suggestions on how you can make the most out of them.

Clinic notes: Ecommerce holiday playbook wrap-up

October 1st, 2009

Editor’s note: Anyone involved in eretailing should know ecommerce analyst Linda Bustos. The driving force behind the award-winning GetElastic blog, Linda is also a MarketingExperiments certified optimization professional and knows our methodology inside-out. So we were delighted to have Linda as a featured guest on our ecommerce clinic and now on our blog, with her wrap-up and takeaway tactics from the session.

Just as shoppers often wait until the last minute to finish their holiday shopping, often online retailers find themselves behind on their holiday optimization.

Procrastinators need to implement ideas that don’t involve long lead times for design, development or approvals. In the Ecommerce Holiday Playbook for Procrastinators web clinic, we shared tips and tactics that online retailers can implement in as little as five minutes to make the most out of the “most wonderful (selling) time of the year.”

The 5 areas covered were:

  • SEM (Search engine optimization and paid search)
  • Shopping engines
  • Email
  • Landing pages
  • Post-holiday (ring in the New Year!)

1. Search Engine Marketing

  • SEO: Add value propositions in meta descriptions and page title tags to increase clickthroughs

using value propositions in SEO tags

If you’re familiar with the MarketingExperiments Conversion Sequence, you’re well versed in the importance of value propositions. They are the key to success in any optimization effort. Are you leveraging your value propositions at every marketing touch point?

If it’s true that 80% of web sales begin with a search engine query, it’s important that you sell yourself in your search listings, whether they be organic results or pay per click. Are you communicating why someone should click on your link, instead of anyone else’s? What is it about your store that is unique?

Make sure searchers can see in your title tags and your meta descriptions what it is that makes shopping with you the best choice.

  • PPC: Bid on relevant holiday keywords, such as …


People are not just searching for products, they are searching for ideas. It’s quick and easy to set up a holiday Ad Group or groups targeted to gifts and gift ideas for different holidays and recipients. Then flip the switch December 26th for after-Christmas sales. Last year Google Trends, which tracks the most popular searches of the day, exploded with searches for after Christmas sales.

2. Shopping Engines

  • Add free shipping, value proposition
  • Plan for increased bids during holiday period
  • Know when to turn down your bids (after shipping cutoff)
  • Sanity check data-feed accuracy
  • Pull non-holiday categories if budget is a concern

We know that shoppers use search engines to hunt for gifts, and often their searches direct them to a comparison shopping engine like Google Products,, BizRate, PriceGrabber etc. Shopping Engine Optimization is the “other other” SEO, and is also called Data Feed Optimization.

Data feeds are the way merchants provide their catalog information to these sites, and the information you include in your data feeds may vary from engine to engine. Some engines allow you to add shipping offers or other value propositions in extra fields. So it’s important that your feeds are tailored to each engine and really take advantage of your options.

On the strategic side, you should also be planning for increased bids to remain competitive. Holiday click prices are often higher than the rest of the year. While you don’t have to increase bids, depending on what other retailers are doing you may have to to keep appearing high enough in results, and it also provides you with a slight advantage if your competitors don’t turn up their bids.

Don’t forget to turn down your bids after your shipping cutoff date. This is different for every retailer, so make sure you have a process to do this.

And make sure your data feeds are accurate – that you’re not listing out of stock product, pre-orders, backorders and non-holiday items, especially when you’re working with a set budget and you’re spending more per click. It may make sense to remove categories that are typically not gifted to others (like computer cables if you’re an electronics store) temporarily for the holiday season.

3. Email

Stress the benefits of online shopping in your subject lines, including:

  • Save time
  • Save gas
  • Avoid lines
  • More selection
  • Hard to find items
  • Gift finder tools

Some of my favorite examples from last year are (emphasis mine):

Easy-to-Make Holiday Cards. We’ll Mail Them.
Avoid the Rush! Get Your Gifts Now, Save $15 & Pay NO Shipping!
Avoid the crowds – Shop from Home and get Free Shipping
The Gift Guide Is Here: The Best Gifts at Even Better Prices
Email Exclusive Free Shipping, No Threshold. Today Only!

Holiday time is both great and gruesome for sending retail email. On one hand, when you deploy retail email you hope that the recipient is interested in shopping that week. During the holidays, you’re nearly guaranteed that he or she is. However, you are also competing for attention against any other retailer that the subscriber has opted in to.

Like Dr. Flint McGlaughlin says, the goal of the email is not to sell but to generate interest in visiting your site. The offer and the creative is important here, but before one sees your email message, they must be persuaded to open your message. Subject lines matter!

Remember that we as ecommerce marketers want customers to use the online channel to shop. It’s great if you have local stores that may still benefit from your demand generation, and converting online can be attributed back to which campaign and email version referred the visit. This gives you better insight into what is and isn’t working. So may I suggest that you really communicate the benefits of shopping online, as well as shopping from you. The above examples do that.

4. Landing Pages

Address the FUDs (Fears, Uncertainties and Doubts, or, Anxiety)

  • Clear link to gift guide (create new category if needed)
  • Clear link to store locator
  • Clear shipping cutoff link
  • Shipping policies, return policies, countries ship to
  • Customer service number (every page is a landing page)
  • Promote e-gift cards (never too late!)

Just as important as driving traffic is having an optimized website prepared to convert it.

If you already have a gift guide feature, make sure you’re flaunting it — not just in navigation as a text link that blends in with the rest of the links, but clearly on your homepage and on product/landing pages.

Have a clear link to your store locator if you have offline stores. It’s fine to have it subtle the rest of the year in the header or footer menu, but around the holidays – especially after your shipping cutoff date — this deserves more prominence.

Customers want to know what your hours of operation are and even telephone numbers, so make sure that info is on your store locator page.

Shipping cutoff information is also very important. Many retailers do put this information front and center on the homepage. I recommend you show it on every page, because every page is a potential landing page. Don’t assume everyone starts at the homepage and absorbs your messaging and remembers all your details (research shows that’s often not the case).

Ditto for links to return policies and shipping policies, including countries you ship to. Even if your shopper is in the US, they may be shipping overseas.

Don’t forget customer service numbers on every page of your site and every step of the checkout process.

And if you do offer electronic gift cards which can be sent instantly, this is important to showcase – especially after your shipping cutoff.

5. Post-holiday

Post shipping cutoff strategies include promoting gift cards, any in-store pick up options and even gift notifications sent immediately to gift recipients, letting them know they weren’t forgotten but that their gift may arrive a bit after the holiday.

For example, is a service for merchants to offer such messages.

Last holiday, Musician’s Friend offered a $20 comeback coupon for the gift buyer, which is a great incentive to come back and purchase again during the typically slow month of January.

Finally, though we are really close to the holiday, it’s not too late to think about your merchandising for post-holiday. Popular approaches include showcasing items for New Year parties…

…and New Year resolutions – like’s emphasis on products to help you lose weight, stop smoking, get fit, look your best, go green or be healthy.

For more ecommerce tips, tactics, ideas and research, be sure to visit (and bookmark) the excellent GetElastic blog.