Archive for the ‘Analytics & Testing’ Category

3 A/B Testing Case Studies from Smart Brand-Side Marketers

October 24th, 2016

When you’re embarking on a new campaign, or a launching a new CMS, there are a million things that can go wrong. To which I say: GOOD.

Bring it on. That’s what testing is all about, right? Letting the things that will go wrong, go wrong – but tracking it every step of the way.

That, my friends, is the difference between a mistake and a “learning.”

See that positive spin?

As the Managing Editor of MarketingSherpa (MarketingExperiment’s sister site) I have covered hundreds of marketing campaigns by companies, and generally they are focused more on high-level inspirational stories. However, many of them have a lot of specific ideas for testing, and I’ve put together a collection featuring three of those case studies here.

Case Study #1. Extra Space Storage


In this MarketingSherpa Reader’s Choice nominated campaign, the team at Extra Space Storage knew they were missing opportunities with customers using a business-first email program. So, they decided to tear it down to the studs, and rebuild it as a customer-first email program.

The first place they started with was to scrutinize the previous system for lost opportunities – the first of which was that it was completely transactional, and not mobile-friendly.

“It’s a natural progression [that] customers are contacting us on the web. They don’t want our manager calling them to talk about their reservation,” said Jennifer Stamper, Interactive Marketing Manager, Extra Space Storage. “They’d prefer to get an email. They want to be spoken to in the channel that they’ve contacted us in.”

She and her team tied all of the available data together to come up with four different personas to represent the customer base. They then took those four personas and worked through the purchasing patterns of each, to figure out what would convert each of them best.

After utilizing a lot of behavioral research, data, and even utilizing a Myers-Briggs personality-type indicator, they were able to come up with template aspects that spoke to each specific customer group.

“Sixty percent of our customers have actually never used storage before. We did a lot of research around what questions do they have. What information do we need to provide them before they show up at the store? We really focused on answering those questions before the customer even realized they had those questions,” she said.

Read the full case study to see how through three iterations, Stamper and her team reworked templates, personalized sends and ran email frequency and content tests to drive a 50% jump in attributed conversion rate for email – a first in company history.


Case Study #2. BookPal

1-asA better B2B shopping experience is what drove BookPal, a B2B ecommerce firm specializing in bulk book sale, to redesign its website. The idea driving the entire effort boiled down to: while BookPal sells to schools and businesses, its customers are individual consumers.

That’s the principle that drove this entire campaign, starting with an in-depth website audit. The objective of which was discovering what BookPal was doing right and uniquely in the space, and what areas needed greater attention.

“We went through a fairly substantial needs analysis looking at what our existing website technology provided, where the shortcomings were, and then ultimately looking at the unique nature of our B2B business and our school education and corporate customers, what we did uniquely online that we could do better, where could we enhance that B2B experience,” said Tony DiCostanzo, President, BookPal.

Enhancing that experience included integrating B2C practices like social media, and tracking via AdWords, while also reconfiguring the website based on analytics data, to remove a lot of the existing friction blocking the path to conversion.

Read the full case study to show how by creating a simple, persona-focused experience, BookPal was able to drive order volume up 211% in three years.

Case study #3. Skyjet



This Reader’s Choice nominated case study, by private jet charter provider Skyjet, focuses almost entirely on value proposition.

In early 2015, Jonathan Levey, Digital Marketing Manager, Skyjet and his team identified an opportunity to leverage mobile based on an impressive trend in Skyjet web analytics.

“The mobile traffic on Skyjet jumped 50% in Q1 2015, compared to Q4 2014, and quote requests increased 177% on mobile devices,” he said.

Furthermore, the average mobile user spent two minutes and 14 seconds on, which is 29% longer than the typical desktop user.

Skyjet realized the opportunity in supplying its niche group of customers with the value of an app featuring on-demand booking, with the ability of its system to instantly calculate and process an itinerary, evaluate different variables to give customers a price within seconds.

The team is also constantly testing features of the app, such as push notifications, and iterating it on a monthly basis to ensure customers have the best possible experience.

Read the full case study to see how the app’s real-time pricing tool now processes roughly 1,500 user itineraries on a weekly basis, and has more than 22,000 downloads since its launch.

You might also like:

Homepage Optimization: 5 Marketing blind spots that inhibit conversion (and how you can correct them)

Content Marketing Generated a 5,100% ROI for Health Care Innovator Optum [via MarketingSherpa]

Make Your Content Useful: How a simple UI change created 74% more page views

Make Your Content Useful: How a simple UI change created 74% more page views

September 27th, 2016

When your digital media business depends largely on page views, anything you can do to increase those views also increases the chance of long-term success. For Car & Driver, one set of pages in particular drives a significant amount of revenue – the vehicle research section.

It was true in 2008, when they ran this A/B test, and it’s still true today. In fact, the discoveries from the A/B test here have been carried over through the years into today’s designs.

Here’s some background:

Car & Driver Test Background

The control widget was a simple block of creative on the homepage that linked to the various pages within the vehicle research section.

Car & Driver Test Control

The treatment widget took a different approach. In order to increase engagement with the widget, and eventually page views to the section, the treatment added an element of interactivity via a simple drop-down functionality. In short, they made the content useful to the customer.

Car & Driver Experiment Treatment

The result was a 74% increase in page views in the vehicle research section of the site.

Car & Driver Experiment Results

But what is truly interesting about this test, and illustrates the true value of testing, was that Car & Driver learned something important about their customer. Something so important, in fact, that today, eight years later, this approach is used to create an even more effective page design.

Car & Driver Current Page Design

By simply making a set of links, more useful to the customer, Car & Driver was able to not only increase engagement for the vehicle research section, but they learned a system to give themselves a long-term edge.

Below is the full, downloadable slide presentation for this test.

You might also like:

Millennials something Snapchat something something

5 Call-to-action examples that increased conversion rate (just by being helpful)

Call-to-Action Button Copy: How to reduce clickthrough rate by 26%

Build a Better Home Page Strategy: How multiple objectives on a homepage increased clickthrough to a single page by 52%

Build a Better Home Page Strategy: How multiple objectives on a homepage increased clickthrough to a single page by 52%

September 21st, 2016

Homepages are among the most difficult places for marketers to optimize for a few of reasons:

  1. Internal company politics can create a battle for real estate on the home page that may or may not be in the best interests of the customer or Digital’s P&L.
  2. Even if Digital has full control, it’s difficult to determine what to emphasize on the homepage to maximize ROI.
  3. It seems that one’s really clear on just what a homepage should be in the first place which makes it incredibly difficult to optimize for a KPI.

In a test from the MECLABS Research Library, a large healthcare company was dealing with all of these issues and more. Their homepage was originally focused on a single objective – to get customers onto the “find a treatment center” page further down their funnel.

Here is the control from that test:

Control Homepage Strategy

The team, however hypothesized that due to the different customer personas coming to the page, a large portion of the visitors were not being properly served by the content on the homepage or the “find a treatment center” page.

To remedy this issue, the team created a treatment that served multiple customer personas with separate paths towards the ultimate goal of landing on the “find a treatment center” page.

Here is the treatment homepage strategy:

Treatment Homepage Strategy

When they pitted the two strategies against each other in an a/b split test, the results were significant. The treatment produced a 52% increase in clickthrough to the “find a treatment center” page.

Homepage Strategy Results

By simply making the shift to serve multiple customer personas coming to the page, and creating content to serve those customer personas, the treatment significantly increased the performance of the homepage.


You Might Also Like:

Homepage Optimization: Tips to ensure site banners maximize clickthrough and conversion

B2B Marketing: Homepage segmentation effort increases time spent on site 171%

Homepage Testing: 91% conversion lift from new copy and layout

Page Templates Tested: How a few UX tweaks to 45 template pages generated a 52% increase in leads

September 15th, 2016

Does UX really make that much of a difference?

Besides being an important business question, there’s an entire industry of UX professionals who count on it making a difference. Sure, intuitively, we think it will. And there’s even survey data to support it.

But what about at the company-level?

Has anyone been able to measure the impact of UX on an actual company’s bottom line?

Bryce Miller of MasterControl, an enterprise quality management software company, has measured it. It happened on a template for 45 of their product pages. The team at MasterControl working with MECLABS, found that using the MECLABS conversion heuristic and focusing on friction (the element that is most commonly associated with traditional UX) generated a 52% increase in leads and $1,543,320 in sales pipeline growth.

The reason Miller can be so sure about his increase? He ran a valid test.

Here’s Miller’s control. It’s essentially a page template for content about their products. But it has a significant amount of friction in the process.

Control Sample: Standard Product Page Template

master control control

Treatment Sample: Updated UX and Reduced Friction

master control treatment


By making these small UX tweaks, Miller was able to generate an additional $1.5 million in sales pipeline.


What you need to understand

In this case, reducing friction with simple UX tweaks that helped users get to the right content on the page faster produced a significant result.


You May Also Like:

MarketingSherpa Summit 2017 Speakers – Miller will be talking about this and other tests he’s run in-depth during his session at MarketingSherpa Summit 2017

Website Optimization: How MasterControl ran a 45-page test to achieve a 52% lift – The full MarketingSherpa Case Study for this test

How MasterControl got a 52% increase in leads From Key Pages in 3 months – Miller’s first-hand account of this test on LinkedIn

The Web as a Living Laboratory – Learn How to Use the Web to Experiment Your Way Into a More Profound Understanding of Your Customer

[Infographic] How to Create an Effective Testing and Optimization Plan

August 30th, 2016

Editor’s Note: This is the third step in last week’s post by Adam Lapp. Here’s Adam…

Paul Cheney
Managing Editor

If you’re entrenched in Conversion Rate Optimization, A/B testing, or have read articles before from MarketingExperiments or MECLABS, you’re probably been inundated with the concepts of friction, anxiety, and value proposition.

But how do you take those concepts and build an effective test plan so you can start increasing your bottom line?

To help you with that, our team of amazing designers (shout out to Lauren, Charlie, Leah, James, and Chelsea) designed this infographic for you. Hope you find it useful. 🙂

MECLABS Institute - Conversion Heuristic Infographic

Share this Image On Your Site

You might also like:
The Web as a Living Laboratory
Test Plan: Build better marketing tests with the Metrics Pyramid
The Best Conversion Rate optimizers do NOT make changes to webpages…

The Best Conversion Rate optimizers do NOT make changes to webpages…

August 23rd, 2016

Editor’s Note: For anyone new to this blog, Adam Lapp has been MarketingExperiments’ head of optimization for around three years. He’s been optimizing web paths for much longer, though – somewhere in the ballpark of 10 years. I’ve personally worked with Adam for five years now, and he has, hands down, the most brilliant optimization mind I’ve ever seen.

So naturally, I was thrilled when he sent me a draft of this post for the blog. It’s been a while since Adam took some time out of his busy schedule to write for our blog, but his posts are full of real-world optimization wisdom that many of our readers have found invaluable in the past.

The casual tone of this post may be a little different from what you might be used to on this blog. That’s because I’ve left Adam’s personal writing style, for the most part, intact. This post is written by a true expert and I wanted it to be as directly from the source as possible.

I hope you enjoy. Here’s Adam…


Paul Cheney

Managing Editor


Best Conversion Rate optimizers_Adam


I remember I once I designed and ran four tests in a row — two product page tests and two homepage tests — for a Fortune 500 industrial supply company, and lost every time. The designs were solid – better navigation, easier to find buttons, improved copy and value proposition – but they all lost.

When I look back at it, these four tests lost because I was trying to optimize webpages.

So, what the heck am I talking about?

Well fortunately and unfortunately, the probability of a prospect converting begins increasing or decreasing long before they get your website.

Read more…