A/B Testing: Split tests are meaningless without the proper sample size
A/B testing can be extremely influential. Heck, it even convinced the leader of the free world to send some curious emails (“It doesn’t have to be this way”).
This is because numbers are powerful. Seeing one call-to-action button receive a clickthrough of 10% while another button only receives 8% is pretty convincing.
Just know this …
Numbers can be misleading, as well. And one of the most frequent mistakes I see inexperienced testers make is not having a sufficient sample size.
This problem is so counterintuitive. After all, you can clearly see that 10% is more than 8%. How can that button not be better?
A quick story from my own life to explain …
Getting to the heart of testing
So my wife has a (don’t worry, it’s minor) heart thing. And she was telling me how her heart felt. And I said …
“Don’t worry, it’s perfectly normal. I get that all the time, and I’m fine.”
Well, the punch line is, I wasn’t (don’t worry, I’m all fixed now).
And that, my friends, is random chance. That’s why you want to make sure you ask enough people (or, in the case of A/B testing, have enough visits and conversions for your split test) to defeat random chance, and to know with a pretty high level of confidence that your control and treatment are actually different.
Here are a few resources to help you with sample size sufficiency in your online testing: