A/B Testing: Learn about testing hypotheses from a 200% increase in chocolate consumption
Last year around Christmas, I decided to offer Hershey’s Kisses in a bowl at my desk to the people that regularly walked by my office and came in for calls. When the candy didn’t disappear too quickly, I felt like I needed a test to figure out what was going on. When I asked people, they didn’t really know why. They just either wanted it or didn’t.
Customer Theory Question: Why do people not take the candy? After all, it is the season of giving, and I want to give it away!
So, I started doing my research, and I came across our property manager, Mike W. He had asked me where all the Hershey’s Kiss wrappers were coming from because he had been picking them up around the office.
Of course, I didn’t tell him it was me, but I then had a hypothesis:
Hypothesis #1: People do not take my candy because they fear the wrath of our property manager. Either that or there is just something about the Kiss’ packaging that gets on my prospects’ nerves.
Test #1: Hershey’s Kisses vs. Hershey’s Miniatures
Same chocolate — extremely similar servings — but the package of the Miniatures is easier to take off and throw away for some people. Will the packaging have an effect on how much chocolate disappears from my bowl?
FRICTION (difficult of wrapper), ANXIETY (wrath of property manager)
Results: More of my candy started to disappear! However, not a whole lot more (like 20% more). Thus, I could conclude that some people didn’t take candy because of the effect of the packaging (possibly due to anxiety) but there are still more people to reach.
So I then wondered … Now that I have the Miniatures up, why are they not taking more?
Hypothesis #2: People do not take more of my candy because they are being polite (only taking one).
Test #2: Hershey’s Miniatures vs. Hershey’s Nuggets
Again, same chocolate, same packaging type … but different serving sizes. People could now have more chocolate for every one piece they took. Does a significant difference in serving size help me get more chocolate out of my bowl?
ANXIETY (being polite)
Results: Negative in terms of volume. Less of my candy was being taken. If I took the time to do the math, though, it would have been an even wash in terms of serving size, as some were still being taken. So people who were highly motivated would still take one, and get more, but the less motivated decided that double the chocolate wasn’t for them to the point of not taking any at all.
So, I’m back to Miniatures. I’ve had an incremental gain, and an incremental learning (aka loss). How can I earn that big win for which everyone is looking?
I strolled down the aisles of the grocery store, and it hit me.
Hypothesis #3: People do not take more of my candy because not everyone wants just milk chocolate. Different people prefer different candy, probably to the point of not taking any at all.
Test #3: Hershey’s Miniatures vs. Variety pack miniatures (Krackel, Mr. Goodbar, Hershey’s, Special Dark)
Now it’s the same optimal serving size and packaging, but different choices.
Results: 200% increase! I never saw candy disappear so fast off of my desk. More and more people were taking chocolate from my desk, AND coming back for more.
In fact, this learning taught me how to bribe my developers when I was in a test set-up pinch (they loved Mr. Goodbar) as I paid attention to which segments of people took which candy.
Creating customer theory can be as simple as figuring out what candy your people like (without asking them). If you stick to the testing process, and anticipate the right tests through use of our Conversion Index Meta-theory, you’ll likely get the answers for which you are looking.