Boris Grinkot

Focus Groups Vs. Reality: Would you buy a product that doesn’t exist with pretend money you don’t have?

Would you believe a focus group over reality? Of course not. Experienced researchers, including those that regularly work with focus groups, know how difficult it is to orchestrate a focus group study that would produce realistic predictions.

“Calibration” is the name of the game, but there might be little or nothing to calibrate by if what you are studying is radically different from any previous available studies. Or if the study finds a problem with your boss’s favorite squirrel (the name we gave to your boss’s bad idea in Part 1, the idea that you can’t quite change his mind on)…

A cloudy crystal ball

When you are developing a new product, you have little choice. My team was recently involved with a Web design company whose client was a major CPG (consumer packaged goods) manufacturer that was rolling out a wholly new product category.

Extensive focus group research performed by an experienced agency delineated a target market (complete with size, demographics, and psychographics), an acceptable price point, and a sub-set of product benefits that would be received especially favorably as part of a marketing message.

The manufacturer is a highly sophisticated investor in new products, and found the business plan for this new idea sound and valid. Yet, out of the gate, the product didn’t perform. The intended customers simply weren’t buying, even though the focus group predicted they would.

Of course, there are many moving parts here: were the infomercials poorly produced? Was the landing page sub-optimal? Perhaps. However, all of the marketing and sales messaging was developed strictly based on focus group preferences.

Does this mean that the focus group research was wrong? One thing I am certain about is that the research was performed to the highest standards. Why didn’t it predict what would really happen?

Would you buy a product that doesn’t exist with pretend money you don’t have?

This experience underscored the fundamental problem with using focus groups to decide what people are willing to buy. When it comes to purchase decisions, we cannot expect consumers to be able to imagine a buying situation so perfectly that their make-believe decision would be identical to a real-life one.

Again, I didn’t just discover this. Marketers have known this since they started using focus groups. So, what’s new?

“The real voyage of discovery consists not in seeking new landscapes but in having new eyes.” – Marcel Proust

I suggest that with interactive Internet media (Web, email, social media, etc.), focus group testing can be useful to get an approximate idea of sentiment and preferences, but real-life behavioral testing is what produces the greatest return on the time, effort, and money spent on the research.

Based on preliminary focus group research (again, I am not discounting it entirely), a small batch of the physical product can be produced (if the product is entirely digital, this is a no-brainer) and offered online without major investment in ad media.*

Of course, you will still need to drive traffic to the site. Paid Search is a measurable way to do just that, since you have full control of spend and real-time reporting of how that spend performs (or under-performs) to test and manage your ad copy. Likewise, your landing pages provide an opportunity for real-time testing on real incoming traffic of potential customers.

Real time, real-world data

Rather than ask these people to imagine a purchase situation, we can observe them making an actual purchase. While asking them about what they think, feel, or are “likely” to do is impossible, we can measure what they actually do in a rather fine detail. We know whether they buy or not; we know whether they return to the site or not; we know even which step of the shopping cart is the “leak” in the funnel.

There are certainly some blind spots. For example, we can’t tell why a particular customer bought, and another didn’t. We also don’t know (depending on how long the test runs) if a customer is planning to come back to buy later. We don’t know if something outside of our own efforts (ads, website copy, etc.) moved the customer to buy (or not buy).

Do vs. If

Behavioral data is necessarily aggregated above the level of individual preferences and sentiment. However, with some help from statistics, it tells us with the requisite level of confidence what consumers do or do not want (as opposed to would or would not want if…).

Making the business case for testing

On Wednesday, we’re devoting an entire web clinic to not just our latest online marketing optimization discoveries, but a behind-the-scenes look at how one marketing manager implemented a culture of testing across her enterprise organization.

* Note that it is against FTC rules to sell a product that doesn’t exist. The more unscrupulous marketers go as far as testing a product in Paid Search ads before it’s ever been produced to gauge demand, and then refund consumers money if too few buy.

Related research

Never Pull Sofa Duty Again: Stop guessing what your audience wants and start asking

A/B Split Testing

MarketingExperiments Methodology

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: Analytics & Testing Tags: ,



  1. May 12th, 2010 at 12:49 | #1

    Thoughtful post!
    I find one of the biggest sins of focus groups is that people don’t pay attention to what consumers say…which means you can’t discern the truth from the whimsy. Here’s a fun listening game: Focus Group Bingo.
    http://admajoremblog.blogspot.com/2010/05/focus-group-bingo.html

  2. May 13th, 2010 at 14:55 | #2

    @Steve Schildwachter
    Thank you for the kind words! Great point about listening–this is where focus group research is probably much more susceptible to human error, both on the researcher’s and the subject’s end. Of course, looking at behavioral data using statistical tools still requires watching out for validity threats, which can be difficult or entirely impossible to detect.

    Not sure I understood the purpose of that Bingo game idea–sound like it would reinforce preconceived notions (by prompting the researchers to notice the words they had expected to hear), and not correct for them as the premise suggests.

  3. July 15th, 2010 at 07:50 | #3

    Focus schmocus.

    Focus groups are fab for getting *insight* into what some people are thinking and feeling about a product or service.

    But they are useless for getting real actionable data. What people *say* and what they *do* are 2 different things.

  4. July 15th, 2010 at 07:57 | #4

    Use domestic smoke alarms as an example.

    A focus group would all agree 100% that it makes sense to fit smoke detectors all round the house. But we already know that not everyone does this. So if we’re deciding to launch smoke detectors or not then a focus group will help us to make a bad decision.

    A focus group would help to home in on the reasons why people don’t buy/fit them. Convenience? Cost? Ugliness and disruption to decor? Or is it just delay/inertia?

    These would help to try different approaches: nicer-looks or cheaper or ultra-easy installation, or…

  5. Anne T Stone
    July 15th, 2010 at 14:37 | #5

    I like the way Domino’s Pizza worked to turn itself around with the “focus group follow up” campaign – I wonder if they have seen returning customers. The commercial shows the company members engaged in the focus group – not behind glass which I thought was interesting – AND obviously in some pain…Totally authentic?
    http://www.youtube.com/watch?v=nI28ecDy9sE

    as far as dancing squirrels go….KIA SOUL is sold by hamsters, I guess that makes all the difference. I just can’t get enough of that commercial! I can’t see Kia testing that with a focus group, but maybe they did http://www.youtube.com/watch?v=c68V3z2tSjw&feature=related

  1. April 6th, 2010 at 15:46 | #1
  2. May 21st, 2010 at 08:57 | #2
  3. November 23rd, 2010 at 15:08 | #3
  4. December 9th, 2010 at 03:03 | #4
  5. February 25th, 2011 at 09:15 | #5
  6. September 19th, 2011 at 16:49 | #6