Author Archive

3 A/B Testing Case Studies from Smart Brand-Side Marketers

October 24th, 2016

When you’re embarking on a new campaign, or a launching a new CMS, there are a million things that can go wrong. To which I say: GOOD.

Bring it on. That’s what testing is all about, right? Letting the things that will go wrong, go wrong – but tracking it every step of the way.

That, my friends, is the difference between a mistake and a “learning.”

See that positive spin?

As the Managing Editor of MarketingSherpa (MarketingExperiment’s sister site) I have covered hundreds of marketing campaigns by companies, and generally they are focused more on high-level inspirational stories. However, many of them have a lot of specific ideas for testing, and I’ve put together a collection featuring three of those case studies here.

Case Study #1. Extra Space Storage


In this MarketingSherpa Reader’s Choice nominated campaign, the team at Extra Space Storage knew they were missing opportunities with customers using a business-first email program. So, they decided to tear it down to the studs, and rebuild it as a customer-first email program.

The first place they started with was to scrutinize the previous system for lost opportunities – the first of which was that it was completely transactional, and not mobile-friendly.

“It’s a natural progression [that] customers are contacting us on the web. They don’t want our manager calling them to talk about their reservation,” said Jennifer Stamper, Interactive Marketing Manager, Extra Space Storage. “They’d prefer to get an email. They want to be spoken to in the channel that they’ve contacted us in.”

She and her team tied all of the available data together to come up with four different personas to represent the customer base. They then took those four personas and worked through the purchasing patterns of each, to figure out what would convert each of them best.

After utilizing a lot of behavioral research, data, and even utilizing a Myers-Briggs personality-type indicator, they were able to come up with template aspects that spoke to each specific customer group.

“Sixty percent of our customers have actually never used storage before. We did a lot of research around what questions do they have. What information do we need to provide them before they show up at the store? We really focused on answering those questions before the customer even realized they had those questions,” she said.

Read the full case study to see how through three iterations, Stamper and her team reworked templates, personalized sends and ran email frequency and content tests to drive a 50% jump in attributed conversion rate for email – a first in company history.


Case Study #2. BookPal

1-asA better B2B shopping experience is what drove BookPal, a B2B ecommerce firm specializing in bulk book sale, to redesign its website. The idea driving the entire effort boiled down to: while BookPal sells to schools and businesses, its customers are individual consumers.

That’s the principle that drove this entire campaign, starting with an in-depth website audit. The objective of which was discovering what BookPal was doing right and uniquely in the space, and what areas needed greater attention.

“We went through a fairly substantial needs analysis looking at what our existing website technology provided, where the shortcomings were, and then ultimately looking at the unique nature of our B2B business and our school education and corporate customers, what we did uniquely online that we could do better, where could we enhance that B2B experience,” said Tony DiCostanzo, President, BookPal.

Enhancing that experience included integrating B2C practices like social media, and tracking via AdWords, while also reconfiguring the website based on analytics data, to remove a lot of the existing friction blocking the path to conversion.

Read the full case study to show how by creating a simple, persona-focused experience, BookPal was able to drive order volume up 211% in three years.

Case study #3. Skyjet



This Reader’s Choice nominated case study, by private jet charter provider Skyjet, focuses almost entirely on value proposition.

In early 2015, Jonathan Levey, Digital Marketing Manager, Skyjet and his team identified an opportunity to leverage mobile based on an impressive trend in Skyjet web analytics.

“The mobile traffic on Skyjet jumped 50% in Q1 2015, compared to Q4 2014, and quote requests increased 177% on mobile devices,” he said.

Furthermore, the average mobile user spent two minutes and 14 seconds on, which is 29% longer than the typical desktop user.

Skyjet realized the opportunity in supplying its niche group of customers with the value of an app featuring on-demand booking, with the ability of its system to instantly calculate and process an itinerary, evaluate different variables to give customers a price within seconds.

The team is also constantly testing features of the app, such as push notifications, and iterating it on a monthly basis to ensure customers have the best possible experience.

Read the full case study to see how the app’s real-time pricing tool now processes roughly 1,500 user itineraries on a weekly basis, and has more than 22,000 downloads since its launch.

You might also like:

Homepage Optimization: 5 Marketing blind spots that inhibit conversion (and how you can correct them)

Content Marketing Generated a 5,100% ROI for Health Care Innovator Optum [via MarketingSherpa]

Make Your Content Useful: How a simple UI change created 74% more page views

Bet on Horses, Not Customer Assumptions: How the Kentucky Derby tested content for relevance with customers

July 23rd, 2015

Assumptions can be a dangerous territory — especially when it comes to being relevant with your customers.

When a brand has a large gap between purchases, keeping customers engaged becomes a consistent concern.

The team at the Kentucky Derby faced that issue when they decided to use the weekly newsletter to identify and validate customer segments.

“When we look to grow a brand like the Kentucky Derby, that breadth of engagement is really core to our growth path,” Jeff Koleba, Vice President of Marketing and Programming, Kentucky Derby, said in this session from MarketingSherpa Email Summit 2015.

To solve this issue, Jeff and Kate Ellis, Marketing Analyst, Kentucky Derby, decided to begin segmenting and directing content directly towards the customers who wanted it most. Within its established customer personas, the Derby focused testing on three segments:

  • Social content interests
  • Equine enthusiasts
  • Betting/wagering information

Once they set up segmentation and supported it with relevant content, the team began optimizing for maximum engagement.

Read more…

Testing to Find Your Aha Moment for Value Proposition

May 12th, 2014

The issue Jon Ciampi, Vice President of Marketing, Business Development & Corporate Development, CRC Health, faced is a common one – his team was reaching out to their audience from a marketer’s mindset.

It can be so easy to assume you know what your customer wants, and what they are looking for. The key to discovering your value proposition is to ask the right questions, not just the ones that affirm what you already know. The relationship between marketers and consumers needs to constantly evolve.

Once they re-evaluated the message by testing for customer logic, Jon and his team were able to test and affirm the element that resonated most with customers.

In this short clip from Optimization Summit 2013, see how the CRC Health marketing team came to their “aha moment” for their value proposition that drove a 220% increase in total conversion.


“Just giving me more money wasn’t working out, it wasn’t giving me the results we wanted,” Jon said.

That is when the team decided to test for a different approach that unearthed a whole new, smarter approach. When bringing the pre-eminent high-end addiction and mental health rehabilitation facility to market, Jon assumed that people were concerned with facts like the doctor-to-patient ratio, or facility amenities.

“We show them some horses, we show them a pool and nothing about what we do,” Jon said about the original control page.

Jon ended up throwing out the “best practices” they had so religiously stuck by and actually tested them.

For example: Call-to-action above the fold? Why?

In the end, Jon realized that no matter how many tricks of the trade the team put into the page, none of it mattered if they didn’t understand their customers. The “aha moment” was simple – trust was more important to customers than luxury.

Watch the full free session from Optimization Summit 2013 to see how Jon and his team applied that understanding across the board.

Read more…

Marketing Career: Don’t overlook these 4 marketing fundamentals

April 1st, 2013

Technology is becoming more and more essential to marketing – from marketing automation platforms to complex databases. In fact, Gartner analyst Laura McLellan has predicted CMOs will spend more on IT than CIOs by 2017.

While this focus on technology can be helpful to marketers, the danger is that it causes marketers to become technologists instead of, well, marketers and overlook the fundamentals of marketing.

To help get back to the building blocks of marketing, I turned to Bob Kemper, Senior Director of Sciences, MECLABS, to help you understand how some of the fundamental teachings you read about on the MarketingExperiments blog were initially developed.

The mechanics behind how MECLABS (the parent company of MarketingExperiments) teaches marketers may have become more structured since its beginnings, but the philosophy is still the same, according to Bob.

In the earliest days, Bob said, “we were kind of feeling around for what it is that marketers needed to know, and didn’t know.”

Research Partnerships are an important element of today’s MECLABS that took time to develop into their current, structured format. In the beginning, it was a much more informal conversation with Flint McGlaughlin, Managing Director, MECLABS.

While the logistics have changed, Bob said the essence of the approach is the same, and “there’s a continuity of approach from the earliest days,” he said, only with a refined set of processes. 

Bob discussed four fundamentals.


Fundamental #1: Ideal customer

Working with Research Partners, Bob said the team would look at a page together, and identify “what today we would call an ideal customer.”

It wasn’t as well defined at the beginning, he said, and discussions centered on questions like, “tell me about your best customers, those that you are most able to help – who are they, how do they think. When they arrive at this page, what’s in their minds?”

This process evolved over time toward what’s now referred to in the MECLABS Offer/Response-Optimization meta-theory as “customer thought sequence.”

Theory into practice: You should focus on your ideal customer while crafting your value propositions at four levels, and then use them to keep every member of your company focused on communicating and delivering on those value propositions through not only your marketing, but also your sales, customer service and product development organizations.


Fundamental #2: Thought sequences

“It’s principally about the process of decision making — about what to buy, from whom to buy, and how,” Bob said.

Flint has always had an approach that adopts the frame of reference of “a customer arriving at your conversion funnel, in whatever form it might take, and addressing it as a thought sequence,” Bob said.

Theory into practice: By understanding how your customer thinks, you can create more effective landing pages, offers and campaigns.


Fundamental #3: Redefining marketing 

“[Flint] would conduct consultation calls with marketers … reviewing their current landing pages and conversion paths together, and make recommendations based not upon the operational ‘here’s what has worked before,’ or ‘I’ve done this,’ but based upon the philosophical principles of human decision making,” Bob said.

Marketing began to be viewed as a process for gaining insights about what compels people to make the choices we do, instead of the more established method of constantly pushing onto customers. The focus became placing value at the forefront, not “convincing or cajoling, but rather simply revealing the truth,” Bob said.

This revelation of the truth to consumers centers around whether or not your business has a value proposition, he said, and if “you are truly the best for some significant group of people who are definable, who are discernible.”

That shift in thinking became the basis for how MECLABS approaches marketing as “a revolution of thinking, a complete reversal, a dichotomy from even professional marketers coming out of business school,” he said.

Theory into practice: are you conducting your marketing in a rigorous fashion and using A/B testing to learn how and why your customers make the decisions they do? For example, you can review the MECLABS methodology for discovering what really works in optimization.

Read more…

Lead Generation: 4 tips to present value before cost

February 8th, 2013

Some industries require an abundance of information from leads before they can deliver their product or service. It can be extremely difficult to show value without handing out exact price quotes.

However, leaving breadcrumbs of value along the lead generation path can earn conversions later when the price is finally revealed.

A good example of this was discussed in MECLABS’ twice-weekly PRS, or Peer Review Session. A lead generation form for a mortgage lender was brought up among the conversion team – an industry where leads are looking for any excuse to bounce at handing over large amounts of information. It is especially confusing, as one team member voiced, because of new regulations and government safeguards on the industry.

In this and many other industries like it, value needs to be shown along the lead generation path so  people will continue to be motivated, even without knowing the exact cost.


Tip #1. Establish an end goal

Discuss everything from what you want your business to achieve at the end of the day, to what you want this one lead generation form to accomplish. They should work together to serve those goals.

Read more…

Value Proposition: 3 lessons to learn from the Microsoft Surface campaign

November 12th, 2012

In any aspect of marketing, it all comes down to value proposition. A marketer’s job is to communicate the value of a product or service to a potential customer.

Even companies with big advertising budgets can lose sight of the necessity and responsibility of every marketer to convey value – they may actually be more susceptible to it.

For instance, ever since Apple’s renaissance, pretty much everyone wrote Microsoft off as the poor (and old) man’s Apple. Even they seemed to, with its not cool enough to be a Mac ad campaign a few years ago.

With all of the recent talk about Windows 8, and the debut of the company’s first tablet, Surface (set up to be an iPad competitor), a Cinderella-story comeback seemed possible. Maybe not likely – but possible.

When Microsoft announced it wasn’t going with one of its usual three agencies (Crispin Porter & Bogusky, Deutsch or Razorfish) to produce an ad for the Surface — instead choosing the relatively unknown 22-year-old Caleb Slain, who incredibly made Internet Explorer cool again with this ad — I thought Microsoft might pull it off.

However, instead of spotlighting the sleek and cool but extremely product-focused video that Slain created, the one I see on a constant loop is the incredibly distracting (and after the tenth view, annoying) dancing people and clicking noise ad, as seen below:

Let me make this disclaimer before we go any farther: I liked the ad, but I hated the sacrifice of value proposition it made in favor of the cool factor – because it is possible to have both. Just ask Microsoft’s competition, Apple.

“The click” is the highlighted aspect of the ad, an external keyboard that attaches to the tablet. Other than that, I’m just not sure what is supposed to motivate me to purchase the Surface. The krumping school girls? Maybe the ability to leap through mid-air while clicking into place the keyboard that comes with it?

“I think the only people who have a legitimate chance of going to the store and purchasing are people who are sold on Microsoft already,” says Adam Lapp, Associate Director of Optimization, MECLABS, adding, “The ad doesn’t do anything to distinguish itself from the iPad, and you don’t even know cost. It does nothing to motivate you to learn more.”

As you design your own ads, consider these three steps to create an ad with a strong value proposition.

  Read more…