Jon Powell

A/B Testing: Learn about testing hypotheses from a 200% increase in chocolate consumption

November 16th, 2012

Last year around Christmas, I decided to offer Hershey’s Kisses in a bowl at my desk to the people that regularly walked by my office and came in for calls. When the candy didn’t disappear too quickly, I felt like I needed a test to figure out what was going on. When I asked people, they didn’t really know why. They just either wanted it or didn’t.

 

Customer Theory Question: Why do people not take the candy? After all, it is the season of giving, and I want to give it away!

So, I started doing my research, and I came across our property manager, Mike W.  He had asked me where all the Hershey’s Kiss wrappers were coming from because he had been picking them up around the office.

Of course, I didn’t tell him it was me, but I then had a hypothesis:

 

Hypothesis #1: People do not take my candy because they fear the wrath of our property manager. Either that or there is just something about the Kiss’ packaging that gets on my prospects’ nerves.

Test #1: Hershey’s Kisses vs. Hershey’s Miniatures

 

Same chocolate — extremely similar servings — but the package of the Miniatures is easier to take off and throw away for some people. Will the packaging have an effect on how much chocolate disappears from my bowl?

FRICTION (difficult of wrapper), ANXIETY (wrath of property manager)

Results: More of my candy started to disappear! However, not a whole lot more (like 20% more). Thus, I could conclude that some people didn’t take candy because of the effect of the packaging (possibly due to anxiety) but there are still more people to reach.

So I then wondered … Now that I have the Miniatures up, why are they not taking more?

 

Hypothesis #2: People do not take more of my candy because they are being polite (only taking one).

Test #2: Hershey’s Miniatures vs. Hershey’s Nuggets

 

Again, same chocolate, same packaging type … but different serving sizes. People could now have more chocolate for every one piece they took. Does a significant difference in serving size help me get more chocolate out of my bowl?

ANXIETY (being polite)

Results: Negative in terms of volume. Less of my candy was being taken. If I took the time to do the math, though, it would have been an even wash in terms of serving size, as some were still being taken. So people who were highly motivated would still take one, and get more, but the less motivated decided that double the chocolate wasn’t for them to the point of not taking any at all.

So, I’m back to Miniatures. I’ve had an incremental gain, and an incremental learning (aka loss). How can I earn that big win for which everyone is looking?

I strolled down the aisles of the grocery store, and it hit me.

 

Hypothesis #3: People do not take more of my candy because not everyone wants just milk chocolate. Different people prefer different candy, probably to the point of not taking any at all.

Test #3: Hershey’s Miniatures vs. Variety pack miniatures (Krackel, Mr. Goodbar, Hershey’s, Special Dark)

 

Now it’s the same optimal serving size and packaging, but different choices.

MOTIVATION

Results: 200% increase!  I never saw candy disappear so fast off of my desk. More and more people were taking chocolate from my desk, AND coming back for more.

In fact, this learning taught me how to bribe my developers when I was in a test set-up pinch (they loved Mr. Goodbar) as I paid attention to which segments of people took which candy.

 

The lesson

Creating customer theory can be as simple as figuring out what candy your people like (without asking them). If you stick to the testing process, and anticipate the right tests through use of our Conversion Index Meta-theory, you’ll likely get the answers for which you are looking.

 

Related Resources:

Marketing Optimization: How your peers predict customer behavior

A/B Testing: Think like the customer

Digital Marketing: Understanding customer sentiment

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: Analytics & Testing Tags: , , ,



  1. November 16th, 2012 at 09:51 | #1

    I LOVE the Special Dark Chocolate!

    Milk Chocolate, not so much…

    Show’s the value of thinking outside of the box and overcoming your preconceived notion of your ideal prospects needs.

  2. Lauren Pitchford
    Lauren Maki
    November 16th, 2012 at 10:34 | #2

    Super cool post, Jon!

    Just to add… I’d also say that it may be the original Hershey’s packaging people were more attracted to; perhaps it had more value in people’s eyes. Take Coke for example…

    Imagine a bottle of coke… the Classic Coke plastic bottle vs. an unlabeled plastic bottle…

    They’re both the same stuff, same amounts of liquid, same amount of friction, just presented differently. How would you react? (This one’s like your kiss vs. minis)

    Now, imagine the classic red coke can vs. one that was yellow.

    What if you knew these were, again, the same thing, same amount, etc.? I would bet that a lot more people would take one of they were presented with the classic red vs. the yellow. (Like your minis vs. nuggets example)

    There’s a certain emotion that the classic Coke or the classic Hershey’s bars evoke within people… That red Coca Cola can holds a ton of value, just as I think the original S’mores-makin’ Hershey’s bar does.

    Again, cool concept Jon! Thanks!

    • Jon Powell
      Jon Powell
      November 16th, 2012 at 10:46 | #3

      I actually did that follow up test too (after the last one mentioned) and tested the Variety pack packaging (Christmas version which almost exactly matches the nugget style) and classic (same exact shape and serving size…)

      Results? No noticeable difference in consumption.

      Not to say that it isn’t a factor. :) In this case, I don’t think it’s as huge a factor compared to the wrapping for that small gain. Mostly because the Hershey kiss is arguably as widely known (especially in our office) as the classic Hershey’s logo, that would be my guess as to why it’s not as big as a factor.

      Of course I don’t know, and even if I asked people they wouldn’t be able to tell me the truth. :)

  1. December 28th, 2012 at 03:03 | #1
  2. January 11th, 2013 at 03:01 | #2
  3. March 15th, 2013 at 09:02 | #3
  4. July 15th, 2013 at 08:26 | #4