Courtney Eckerle

3 A/B Testing Case Studies from Smart Brand-Side Marketers

October 24th, 2016
Comments Off on 3 A/B Testing Case Studies from Smart Brand-Side Marketers

When you’re embarking on a new campaign, or a launching a new CMS, there are a million things that can go wrong. To which I say: GOOD.

Bring it on. That’s what testing is all about, right? Letting the things that will go wrong, go wrong – but tracking it every step of the way.

That, my friends, is the difference between a mistake and a “learning.”

See that positive spin?

As the Managing Editor of MarketingSherpa (MarketingExperiment’s sister site) I have covered hundreds of marketing campaigns by companies, and generally they are focused more on high-level inspirational stories. However, many of them have a lot of specific ideas for testing, and I’ve put together a collection featuring three of those case studies here.

Case Study #1. Extra Space Storage


In this MarketingSherpa Reader’s Choice nominated campaign, the team at Extra Space Storage knew they were missing opportunities with customers using a business-first email program. So, they decided to tear it down to the studs, and rebuild it as a customer-first email program.

The first place they started with was to scrutinize the previous system for lost opportunities – the first of which was that it was completely transactional, and not mobile-friendly.

“It’s a natural progression [that] customers are contacting us on the web. They don’t want our manager calling them to talk about their reservation,” said Jennifer Stamper, Interactive Marketing Manager, Extra Space Storage. “They’d prefer to get an email. They want to be spoken to in the channel that they’ve contacted us in.”

She and her team tied all of the available data together to come up with four different personas to represent the customer base. They then took those four personas and worked through the purchasing patterns of each, to figure out what would convert each of them best.

After utilizing a lot of behavioral research, data, and even utilizing a Myers-Briggs personality-type indicator, they were able to come up with template aspects that spoke to each specific customer group.

“Sixty percent of our customers have actually never used storage before. We did a lot of research around what questions do they have. What information do we need to provide them before they show up at the store? We really focused on answering those questions before the customer even realized they had those questions,” she said.

Read the full case study to see how through three iterations, Stamper and her team reworked templates, personalized sends and ran email frequency and content tests to drive a 50% jump in attributed conversion rate for email – a first in company history.


Case Study #2. BookPal

1-asA better B2B shopping experience is what drove BookPal, a B2B ecommerce firm specializing in bulk book sale, to redesign its website. The idea driving the entire effort boiled down to: while BookPal sells to schools and businesses, its customers are individual consumers.

That’s the principle that drove this entire campaign, starting with an in-depth website audit. The objective of which was discovering what BookPal was doing right and uniquely in the space, and what areas needed greater attention.

“We went through a fairly substantial needs analysis looking at what our existing website technology provided, where the shortcomings were, and then ultimately looking at the unique nature of our B2B business and our school education and corporate customers, what we did uniquely online that we could do better, where could we enhance that B2B experience,” said Tony DiCostanzo, President, BookPal.

Enhancing that experience included integrating B2C practices like social media, and tracking via AdWords, while also reconfiguring the website based on analytics data, to remove a lot of the existing friction blocking the path to conversion.

Read the full case study to show how by creating a simple, persona-focused experience, BookPal was able to drive order volume up 211% in three years.

Case study #3. Skyjet



This Reader’s Choice nominated case study, by private jet charter provider Skyjet, focuses almost entirely on value proposition.

In early 2015, Jonathan Levey, Digital Marketing Manager, Skyjet and his team identified an opportunity to leverage mobile based on an impressive trend in Skyjet web analytics.

“The mobile traffic on Skyjet jumped 50% in Q1 2015, compared to Q4 2014, and quote requests increased 177% on mobile devices,” he said.

Furthermore, the average mobile user spent two minutes and 14 seconds on, which is 29% longer than the typical desktop user.

Skyjet realized the opportunity in supplying its niche group of customers with the value of an app featuring on-demand booking, with the ability of its system to instantly calculate and process an itinerary, evaluate different variables to give customers a price within seconds.

The team is also constantly testing features of the app, such as push notifications, and iterating it on a monthly basis to ensure customers have the best possible experience.

Read the full case study to see how the app’s real-time pricing tool now processes roughly 1,500 user itineraries on a weekly basis, and has more than 22,000 downloads since its launch.

You might also like:

Homepage Optimization: 5 Marketing blind spots that inhibit conversion (and how you can correct them)

Content Marketing Generated a 5,100% ROI for Health Care Innovator Optum [via MarketingSherpa]

Make Your Content Useful: How a simple UI change created 74% more page views

Paul Cheney

Homepage Optimization: 5 Marketing blind spots that inhibit conversion (and how you can correct them)

October 11th, 2016
Comments Off on Homepage Optimization: 5 Marketing blind spots that inhibit conversion (and how you can correct them)

homepage-optimizationWe all have blind spots. Some of us more than others.

But if we’ve learned anything from the last 100 years or so of marketing and advertising, it’s that marketers have some of the worst blind spots imaginable.

So it’s no surprise that we need a little help seeing our pages with new eyes: the eyes of the customer. For the past 20 years, MarketingExperiments has been researching why people say “yes” to a given offer.

What we’ve found through this research is that it’s often not what we, as marketers, think it is. Our self-interest always gets in the way of seeing why our customers want what we have to offer.

But more importantly than the research we’ve been doing, we’ve also been turning that research into a repeatable method for overcoming our marketing blind spots. We have a series of patented heuristics that can help marketers like you and me see a piece of collateral with the eyes of the customer, rather than our own.

It’s with the full weight of that research and methodology that Flint McGlaughlin, our Managing Director approaches any piece of collateral submitted by a MarketingExperiments audience member. The same is true of the page submitted by in the video below.

In this video, Flint talks about the blind spots inherent in the InnocareHealth homepage, and how to correct them.

You Might Also Like:

15 years of marketing research in 11 minutes

MECLABS/MarketingExperiments Education

Sustainable Competitive Advantage: How customer-first science transforms the way companies achieve a superior position

What 15,000 Commercial Experiments Reveal about Why People Are Chosen

Paul Cheney

How Does Page Load Time Affect Conversion Rate? New Research Shows Significant Correlation

October 4th, 2016
Comments Off on How Does Page Load Time Affect Conversion Rate? New Research Shows Significant Correlation

Coversion Rates Correlated to Page Load TimesMost marketers understand that a slow page produces low conversion rates. But how significant is the correlation? How fast do customers expect sites to load? Are customer expectations for page load time changing?

In a recent interview at Internet Retailer Conference and Exhibition (IRCE), Daniel Burstein sat down with Tammy Everts, Senior Researcher, SOASTA to talk about her recent research on the correlation between page load times and conversion rates across multiple companies and billions of data points.

Everts discusses her research in depth and covers almost everything you could imagine related to page load times in 15 minutes.

Time Stamps:

1:03 – chart of conversion rates correlated to page load time

1:15 – background of how Everts started this project

2:50 – 2014-2015 comparison of conversion rates correlated to page load time

3:38 – the exact load time that produced the highest conversion rate in 2014

3:50 – the exact load time that produced the highest conversion rate in 2015

4:44 – an interesting data point on year-over-year conversion rate

5:10 – how sites can prepare for faster load time expectations in the market

5:28 – guy does unintentional but very cool video bomb

5:42 – number one culprit for most slow page load times

5:53 – quick Nordstrom page load time case study

7:00 – how to improve image load times

9:00 – why and how to audit your site for page load times

11:36 – mobile-specific recommendations for page load times

You might also like:

MarketingSherpa Summit 2017

Ecommerce: How an online retailer achieved a 17% conversion lift with faster page loading

Ecommerce Charts: How conversion rates correlate to page load times

Customer-centric Marketing: How New England Biolabs increased time spent on site more than 74%

Paul Cheney

Make Your Content Useful: How a simple UI change created 74% more page views

September 27th, 2016
Comments Off on Make Your Content Useful: How a simple UI change created 74% more page views

When your digital media business depends largely on page views, anything you can do to increase those views also increases the chance of long-term success. For Car & Driver, one set of pages in particular drives a significant amount of revenue – the vehicle research section.

It was true in 2008, when they ran this A/B test, and it’s still true today. In fact, the discoveries from the A/B test here have been carried over through the years into today’s designs.

Here’s some background:

Car & Driver Test Background

The control widget was a simple block of creative on the homepage that linked to the various pages within the vehicle research section.

Car & Driver Test Control

The treatment widget took a different approach. In order to increase engagement with the widget, and eventually page views to the section, the treatment added an element of interactivity via a simple drop-down functionality. In short, they made the content useful to the customer.

Car & Driver Experiment Treatment

The result was a 74% increase in page views in the vehicle research section of the site.

Car & Driver Experiment Results

But what is truly interesting about this test, and illustrates the true value of testing, was that Car & Driver learned something important about their customer. Something so important, in fact, that today, eight years later, this approach is used to create an even more effective page design.

Car & Driver Current Page Design

By simply making a set of links, more useful to the customer, Car & Driver was able to not only increase engagement for the vehicle research section, but they learned a system to give themselves a long-term edge.

Below is the full, downloadable slide presentation for this test.

You might also like:

Millennials something Snapchat something something

5 Call-to-action examples that increased conversion rate (just by being helpful)

Call-to-Action Button Copy: How to reduce clickthrough rate by 26%

Build a Better Home Page Strategy: How multiple objectives on a homepage increased clickthrough to a single page by 52%

Paul Cheney

Build a Better Home Page Strategy: How multiple objectives on a homepage increased clickthrough to a single page by 52%

September 21st, 2016
Comments Off on Build a Better Home Page Strategy: How multiple objectives on a homepage increased clickthrough to a single page by 52%

Homepages are among the most difficult places for marketers to optimize for a few of reasons:

  1. Internal company politics can create a battle for real estate on the home page that may or may not be in the best interests of the customer or Digital’s P&L.
  2. Even if Digital has full control, it’s difficult to determine what to emphasize on the homepage to maximize ROI.
  3. It seems that one’s really clear on just what a homepage should be in the first place which makes it incredibly difficult to optimize for a KPI.

In a test from the MECLABS Research Library, a large healthcare company was dealing with all of these issues and more. Their homepage was originally focused on a single objective – to get customers onto the “find a treatment center” page further down their funnel.

Here is the control from that test:

Control Homepage Strategy

The team, however hypothesized that due to the different customer personas coming to the page, a large portion of the visitors were not being properly served by the content on the homepage or the “find a treatment center” page.

To remedy this issue, the team created a treatment that served multiple customer personas with separate paths towards the ultimate goal of landing on the “find a treatment center” page.

Here is the treatment homepage strategy:

Treatment Homepage Strategy

When they pitted the two strategies against each other in an a/b split test, the results were significant. The treatment produced a 52% increase in clickthrough to the “find a treatment center” page.

Homepage Strategy Results

By simply making the shift to serve multiple customer personas coming to the page, and creating content to serve those customer personas, the treatment significantly increased the performance of the homepage.


You Might Also Like:

Homepage Optimization: Tips to ensure site banners maximize clickthrough and conversion

B2B Marketing: Homepage segmentation effort increases time spent on site 171%

Homepage Testing: 91% conversion lift from new copy and layout

Paul Cheney

Page Templates Tested: How a few UX tweaks to 45 template pages generated a 52% increase in leads

September 15th, 2016
Comments Off on Page Templates Tested: How a few UX tweaks to 45 template pages generated a 52% increase in leads

Does UX really make that much of a difference?

Besides being an important business question, there’s an entire industry of UX professionals who count on it making a difference. Sure, intuitively, we think it will. And there’s even survey data to support it.

But what about at the company-level?

Has anyone been able to measure the impact of UX on an actual company’s bottom line?

Bryce Miller of MasterControl, an enterprise quality management software company, has measured it. It happened on a template for 45 of their product pages. The team at MasterControl working with MECLABS, found that using the MECLABS conversion heuristic and focusing on friction (the element that is most commonly associated with traditional UX) generated a 52% increase in leads and $1,543,320 in sales pipeline growth.

The reason Miller can be so sure about his increase? He ran a valid test.

Here’s Miller’s control. It’s essentially a page template for content about their products. But it has a significant amount of friction in the process.

Control Sample: Standard Product Page Template

master control control

Treatment Sample: Updated UX and Reduced Friction

master control treatment


By making these small UX tweaks, Miller was able to generate an additional $1.5 million in sales pipeline.


What you need to understand

In this case, reducing friction with simple UX tweaks that helped users get to the right content on the page faster produced a significant result.


You May Also Like:

MarketingSherpa Summit 2017 Speakers – Miller will be talking about this and other tests he’s run in-depth during his session at MarketingSherpa Summit 2017

Website Optimization: How MasterControl ran a 45-page test to achieve a 52% lift – The full MarketingSherpa Case Study for this test

How MasterControl got a 52% increase in leads From Key Pages in 3 months – Miller’s first-hand account of this test on LinkedIn

The Web as a Living Laboratory – Learn How to Use the Web to Experiment Your Way Into a More Profound Understanding of Your Customer