As we all know, the holiday season is upon us. What does that mean? Shopping! Black Friday, Cyber Monday, Green Monday, Purple Tuesday. … Yes, I made the last one up, but I digress.
This time of year is one of the biggest, and often the biggest, revenue-generating periods for most e-commerce and B2C companies. So one of the most frequent questions we hear around here at The Lab is “To test or not to test?” Deep, I know.
I’m an advocate for testing year round. However, you must be aware of external factors out of your control that may change your prospect’s behavior. While the holiday season may bring a large number of visitors to your site — which is great for testing — you don’t want to risk losing those valuable visitors to an underperforming test.
So what do you do? Well, in my opinion, two factors will make the decision for you.
Factor #1: How risk averse are you?
If you really need to squeeze every dollar out of the holidays, it probably doesn’t make sense for you to do anything too radical during this time of year and open yourself to that sort of risk.
There’s nothing wrong with smaller iterative tests that might give you a few extra conversions here and there, or restricting your more radical tests to a very small fraction of your traffic in order to mitigate a potential for negative impact. You could hit on something big, but are you willing to risk it?
Factor #2: How much does the holiday season affect your yearly traffic and conversion patterns?
Typically, the smaller the price point and more seasonal the product, the more likely the holidays will cause significant swings in your traffic and conversion patterns.
As an example, if you’re selling tablets, you better be prepared to have your best pages in place during the holidays.
On the other hand, if you’re selling office furniture or industrial furnaces, maybe not so much. As long as you know, and take into account, the external factors that could lead to validity threats when setting up tests and interpreting test results, you’ll be fine.
The biggest risk of all?
Pretending the holidays don’t make an impact on your prospects’ motivations. One of my favorite tests I’ve ever run was during my first year at MECLABS. I was working with an e-commerce partner on a shopping cart overhaul. Since its products were commonly sent as holiday gifts, we needed to get the test launched before Christmas to understand its true impact during that timeframe. So what happened?
We launched our test in early-December and through Christmas; the control and all treatments were converting between 31-34%. The differences just weren’t statistically significant at any reasonable level of confidence. Needless to say, everyone involved was a little disappointed.
Then, something interesting happened. We kept running the test for a few weeks, and while the control dropped back down to its typical conversion rate of around 20%, the highest performing treatment only regressed a few percentage points, eventually stabilizing around 30%.
Ultimately, this single test alone made the entire Research Partnership worthwhile for this e-commerce company.
What did we learn? When people are highly motivated, they’re willing to go through almost anything to get what they want. I guess it turns out there’s a reason we put that big 4 in front of motivation in the Conversion Heuristic.
Related Resources:
E-commerce Testing: Redesigned order page, shortened shopping cart drive 13.9% lift in conversion
Online Marketing Tests: A data analyst’s view of balancing risk and reward
Optimization 201: Fix the broken leg before you fix the broken toe
Website Optimization: 1,915 marketers rank the top optimization goals
6 Steps to Optimize Website Conversions: 43% Boost (via MarketingSherpa)
