Kayla Wainwright

A/B Testing: Cut through your KPIs by knowing your ultimate goal

February 4th, 2016

Marketers often struggle to know what metrics to use when trying to decide on the positioning of their marketing collateral. This can lead to many problems. At MECLABS Institute, the parent company of MarketingExperiments, we have run experiments and tests for over 20 years to help answer this question.

Customers take many actions when moving through the funnel, but what is the ultimate goal the company is trying to achieve with their marketing collateral? By answering this question, companies can best determine what the most important KPI is to measure.

To best illustrate this point, let’s walk through an experiment that was run regarding metrics. By reviewing this experiment we will understand how important it is to have a clearly defined idea of what the ultimate goal is for your marketing collateral.

 

The Experiment:

Background: A large newspaper company offering various subscription options.

Goal: To determine the optimal regular price point after the introductory discounted offer rate.

Research Question: Which price point will generate the greatest financial return?

Test Design: A/B split test

 

Subscription services often offer a discounted introductory rate for new subscribers. This gives potential subscribers a low-risk opportunity to try out the service for a period of time before the cost defaults to the regular full price. In this test, The Boston Globe team hoped to determine the optimal price point for a monthly subscription after the introductory offer rate expired. 

Five variations were tested. The Control and all four treatments used a $.99 introductory price for the first four weeks of subscription. After the introductory price ended, customers incurred the regular weekly rate, which escalated per treatment.

 

 

Not surprisingly, the Control, which had the same $.99 regular weekly rate as the introductory offer, outperformed all four treatments. In fact, with each increase in regular weekly rate, response decreased. 

 

At this point, it is easy to see how a $.99 regular rate would beat out the more expensive treatments and therefore lead to more subscribers.

But was getting more subscribers the ultimate goal the company had? We must keep in mind the goal of the experiment, which was to find the optimal price point after the introductory offer.

In order to continue our search for the optimal price point, the next step in the experiment was to look at which price point had the lowest cancellation rates after the introductory period.  When we looked at the data, we discovered a direct correlation between regular weekly price and cancellation rate.  Simply put, the lower the weekly price, the lower the cancellation rate, and vice versa. 

 

With this knowledge of the cancellation rates, it still looks like the $.99 price point is the best offer for the company. But have we found the most important KPI to measure when it comes to finding the optimal price point for The Boston Globe?

You may want to recall here that the research question was, “Which price point will generate the greatest financial return?”

At this point in the experiment, we still haven’t answered that question yet.

The next step was to look at the total revenue per subscriber. This is where we started to think differently about metrics to understand conversion for The Boston Globe. 

 

While the $.99 weekly price point generated the highest subscription rate and the lowest cancellation rate, it also had the lowest revenue per subscriber during the testing period. It was Treatment 4 that generated the highest revenue per subscriber.

Knowing what we know about conversion at MECLABS, we know that you must look at all metrics before coming to a final conclusion. This entails looking at the test from different angles. It is simply not enough to look at the subscription and cancellation rates, and it is also not enough to only look at the total revenue per subscriber.

We had to take the test one step further, and look at the total revenue from the test traffic.  Interestingly, we found that Treatment 3 had produced the highest total revenue from the test traffic.

 

Even though Treatment 4 had the highest initial revenue per subscriber, Treatment 3 had a 163% relative increase in total revenue over Treatment 4.

What is interesting here is that some companies would have stopped when they saw that the $.99 regular weekly rate had gained the most total subscribers, or even when they saw that the $4.99 price point generated the highest revenue per subscriber. We as marketers have to be aware of what is actually going on. If The Boston Globe’s ultimate desired action was for individuals to subscribe, then they would have gone with the $.99 regular rate.

Instead, The Boston Globe’s ultimate goal was to find the price point that generated the greatest financial return. From our testing, we can see that the greatest revenue is being produced in Treatment 3, at the $3.99 price point.

 

Key Takeaway

At this point we can see why it is so important to know exactly what metrics to use when your company has established a goal.  You must know what the most important KPIs are to measure when looking at the data from your experiment. It is vital to look at all the different elements of a funnel to see how all of the different actions tie together to create an overarching desired action which leads to the goal of the company.

The Boston Globe could have easily looked at the number of subscribers it received from the $.99 weekly price point, and chosen this as its regular rate; however, the company would have left revenue on the table had it not considered all metrics from this test.

 

You might also like

MarketingSherpa Summit 2016 — At the Bellagio in Las Vegas, February 22-24

Site Optimization: 2 strategies to consider when trying to increase conversion

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: General Tags: , ,

Liva LaMontagne

Customer Co-Production: How one furniture company tested self-assembly messaging to enhance value and reduce frustration

February 1st, 2016

Self-service and customer co-production of products is everywhere — customers assemble furniture themselves, follow directions on food packages to prepare meals, scan their own groceries at supermarkets and use online banking. Despite its price-lowering and customization value, co-production has a dark side, requiring effort and time from the customer and potentially causing frustration. Today, we’ll look at a study from the furniture industry comparing the effects of two marketing communication strategies to mitigate customer frustration with the co-production process.

 

The Study

In November 2015, Till Haumann, Pascal Grϋntϋrkϋn, Laura Marie Schons and Jan Wieseke from the Ruhr-University of Bochum, Germany published the results of a field experiment with a multinational furniture company. The company sells furniture that requires co-production from the customer or, in other words, standardized, ready-to-assemble furniture that customers purchase in flat packages and assemble at home.

Depending on customers’ assembly skills, the process can be frustrating to a degree. So the scientists set out to test two ways of alleviating customers’ frustrations with the process (“co-production intensity”) by (1) enhancing the perceived value of the process and (2) reducing the perceived effort and time required in the process.

 

Read more…

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: General Tags: , , , , ,

Andrea Johnson

Discover the Power of the “Only-Factor” to Drive Conversion

January 28th, 2016

According to the CMO Council, digital ads will lead the way for global media growth in the next two years, accounting for 33% of total advertising revenue and growing from $133 billion to $194.5 billion. This month’s MarketingExperiments Web clinic examined how to make the most of this increasing digital investment by focusing on the “Only-Factor” — the point of your value proposition where your product’s exclusivity and appeal meet.

Check it out here.

The two experiments below illustrate the power of the “Only-Factor.”

In the first experiment, a credit card company wanted to see which PPC ad would produce the most applications.

Version A promoted its affiliation with a popular organization.

 

Version B noted that it offered the only credit card that supported this particular organization.

 

Read more…

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: General Tags:

Daniel Burstein

Exclusivity: Does your product and offer stand out from the competition?

January 25th, 2016

“Unicorn.” If you read the business press these days, it is a term you simply can’t avoid. The analogy of a mythical creature to describe a very real phenomenon — the startup that has reached a valuation of at least $1 billion.

So what makes or breaks a startup? Why do some succeed magnificently while so many simply fizzle out?

There are many answers to that question, but one key element that unicorns have in common is an effective value proposition. On the flip side, venture capitalist David Skok wrote that one of the reasons startups fail is, “There is not a compelling enough value proposition, or compelling event, to cause the buyer to actually commit to purchasing.”

To help you create an effective value proposition, MECLABS Institute (parent research organization of MarketingExperiments) has created the Value Proposition heuristic. This patented heuristic is a thought tool, meant to help you optimize the factors that increase the force of your value proposition.

 

One key element that helps create a successful value proposition is the exclusivity of the product or the offer (denoted by “Ex” in the heuristic).

Read more…

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: General Tags:

Paul Cheney

This 1960s Statistician Can Teach You Everything You Need to Know About the Fundamentals of A/B Testing

January 21st, 2016

I did a training on selling training for the sales team today. It was what Millennials call “meta.”

I was talking about how our training uses scientifically valid experiments to back everything we say in our training rather than best practices, anecdotal case studies or just “expert advice.”

The question naturally arose: “What do we mean when we say ‘scientifically valid experiments’?”

When I answered the question in the meeting, I immediately thought it would be a good idea for a blog post. So, with that said, here’s the answer:

In short, it means that we use the scientific method to validate every piece of knowledge we transfer in the training (and also in our Web clinics and on this blog).

I found myself trying to explain what I learned in high school about the scientific method, and while I was able (I think) to get the basic gist across, I don’t think I did it justice.

Fortunately, after doing a little searching online, I found this guy.

His name is J. Stuart Hunter and he is one of the most influential statisticians of the last half of the twentieth century.

Fortunately, back in the 60s, he recorded some rad videos around experimental designs in a business context. If you can extrapolate a little bit from the industrial context and apply this to a marketing context, it should be everything you need to know about the scientific method, or “what we mean when we say ‘scientifically valid.’”

 

 

Read more…

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: Marketing Insights Tags: , ,

Shelby Dorsey

Check It Twice: The importance of editing ads

January 18th, 2016

As the copy editor at MECLABS (parent company of MarketingSherpa and MarketingExperiments), I see a lot of content. Between the four blog posts published every week and the multiple case studies produced, I have an ample amount of exposure to the testing brands are doing and what their ads look like.

That being said, I have very harsh eyes when it comes to examining ads. My job is to fix mistakes in text, so I often find myself looking for mistakes in text.

It turns out, so do your customers.

Few things are as fun to share on social media as when a company messes up. This claim can be backed up by the crazy amount of attention Starbucks gets for spelling people’s names wrong on coffee cups around the world.

Other companies are getting similar attention for failing to follow basic English grammar. Daily email newsletter theSkimm is obtaining unwanted attention from readers because of its frequently found typos. 

 

Read more…

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: General Tags: , ,