Testing and Optimization: Looking past conversion rates for customer insights
Performing a radical redesign can be a daunting endeavor.
You really can’t tell what changes are helping or hurting your conversion rates when you look at the data.
But, what if you were to look into your data beyond conversion rates?
What you find might surprise you.
I say this because recently, we tested a couple of radical redesigns with a Research Partner and had not seen any significant lifts.
So, instead of continuing to try and optimize every element on the page, we decided to change our testing approach.
Return to simple ideas and build from there
We wanted a test that would isolate individual variables, so we could gain transferable insights for future tests.
One simple test idea was to reorder the product listings, so in the control, products were ordered by price in descending order.
In the treatment, they were listed by price in ascending order.
Statistically, there was no difference between the two pages based on conversion rates.
Secondary metrics can tell you a lot about your customers
We could have let the test continue to run, but once we reached the required sample size for validation, we decided to shut the test down.
Instead of just declaring the test as a failure, reverting back to the control and moving on, we decided to take a look at the effects the treatment had on revenue.
Our assumption was if the product mix was altered, there would likely be some sort of difference in the revenue also.
Once we looked a little further into the metrics, we discovered a 15% increase in revenue just by starting with the lowest price first, which also validated at a level of confidence of 98%.
Conversion rates are a means to discovery, not an end
Overall, what does this mean?
While there are a few plausible explanations for the increase of revenue, nothing will be conclusive until we run some more tests.
But, this discovery has taught me a valuable lesson about looking past conversion rates to gain customer insights.
So, the next time you have a test and there is no lift in conversion rate, think about discoveries the data offers beyond conversion.
In some cases, that discovery could be a potential ROI you’re leaving on the table.