Posts Tagged ‘KPI’

A/B Testing: Cut through your KPIs by knowing your ultimate goal

February 4th, 2016

Marketers often struggle to know what metrics to use when trying to decide on the positioning of their marketing collateral. This can lead to many problems. At MECLABS Institute, the parent company of MarketingExperiments, we have run experiments and tests for over 20 years to help answer this question.

Customers take many actions when moving through the funnel, but what is the ultimate goal the company is trying to achieve with their marketing collateral? By answering this question, companies can best determine what the most important KPI is to measure.

To best illustrate this point, let’s walk through an experiment that was run regarding metrics. By reviewing this experiment we will understand how important it is to have a clearly defined idea of what the ultimate goal is for your marketing collateral.


The Experiment:

Background: A large newspaper company offering various subscription options.

Goal: To determine the optimal regular price point after the introductory discounted offer rate.

Research Question: Which price point will generate the greatest financial return?

Test Design: A/B split test


Subscription services often offer a discounted introductory rate for new subscribers. This gives potential subscribers a low-risk opportunity to try out the service for a period of time before the cost defaults to the regular full price. In this test, The Boston Globe team hoped to determine the optimal price point for a monthly subscription after the introductory offer rate expired. 

Read more…

A/B Testing: 3 steps to help you test smarter, not harder

January 20th, 2014

2014 is here and with it opens a new year full of many opportunities to test. We are presented with a clean slate.

Any missed opportunities or “we should’ve tested that” are in the past. This post will help you with the basic steps to learn how to approach testing in order to test smarter, not harder, when setting up your tests in 2014.


Step #1. Develop key performance indicators first

Every test begins with an idea on how to make a webpage or email better by deciding the key performance indicators (KPI). In other words, what are we trying to learn from the test?

A few examples of questions we might ask ourselves when developing KPIs are:

  • Are we trying to increase clickthrough rate to another page in the funnel?
  • Are we trying to increase the conversion rate on a product page?
  • Are we trying to determine user motivation by changing the value proposition?

Setting a clear objective for your key performance indicators in the beginning helps to focus on the main learning of the test.

For example, if you change the copy of the call-to-action (CTA) in one treatment versus the control, your key performance indicator is clickthrough rate. Your change of the copy on the CTA button might affect clickthrough rate.

It is important to have a clear understanding of what you are trying to learn from your test based on the key performance indicators you have developed. It is easy to lose sight of this once the data starts coming in, but this is the basis of the test so stay focused on what you are trying to learn.


Step #2. Hold a strategy session

Once you have your key performance indicators, get a group of people together to strategize.

Strategy sessions give you the opportunity to bring ideas to the table, but more importantly, the brainstorming with others helps keep your test plans on track.

People who are not as familiar with your project offer the advantage of seeing these webpages or emails for the first time and can point out parts that may have been overlooked, or offer a different perspective.

In the end it is worth it to take the time for collaboration to build a better test – a foundation that can, and hopefully will, make all the difference when the results start coming in.


Step #3. Win or lose, learn something!

The results make all the hard work and thought put into building a test worth it. This is the point in the process when you sit back and watch the data.

Like a sporting event, you are rooting for your treatment to outperform the control. When your treatment achieves a lift over the control, it feels just like your team scored a touchdown – except better, because you’re the one on the marketing field running the ball in for the score.

It is important to remember that even if your treatment(s) did not win during the test, there are still valuable lessons to be learned.

You should always dig into the results and walk away with some type of learning, even if it is not what you originally intended to learn from the test.

Read more…

Marketing Metrics: Can you have one number to rule them all?

June 6th, 2013

One of the common questions I receive from Research Partners focuses on what metric they should use to track and evaluate tests. The tendency is often to want a single metric that defines the measure of success.

While it is important to gather consensus on which key performance indicators, or KPIs, will be used to evaluate tests early, there should never be a reliance solely on a single metric as the gatekeeper of success given your secondary metrics can provide just as much – if not more – insight into your visitors behavior.


In the land of testing, the marketer with one metric is not king…

If you are only using one metric, you are not seeing a full picture. Each of your KPIs tells a part of the story of performance. Only relying on one alone can mislead marketers to make poorly informed decisions.

For example, let’s say you’re testing a PPC ad. As you know, the sole purpose of an ad is to get the click and let the landing page do the selling. For this reason, you determine  your KPI is clickthrough rate since that is what the ad directly affects.

Makes sense, right?

Now let’s say that your results come back and show that both ads receive the same number of clicks and that there is no statistically significant difference in clickthrough rate.

So what happens now?

Since clickthrough rate was the only metric measured, then you may draw the conclusion that both ads perform the same and that either could be used to achieve the same result and in some cases you may be right…

However, making this assumption is a big risk that flirts heavily with a similar risk of assumption derived from artificial optimization.

Read more…

Marketing Analytics: 6 simple steps for interpreting your data

November 7th, 2012

You’ve finally set up tracking on your site and have gathered weeks of information. You are now staring at your data saying, “Now what?”

Objectively interpreting your data can be extremely overwhelming and very difficult to do correctly … but it is essential.

The only thing worse than having no insights is having incorrect insights. The latter can be extremely costly to your business.

Use these six simple steps to help you effectively and correctly interpret your data.

Read more…

Eloqua Experience 2009: Why choosing the right KPI is essential to gaining ROI

November 16th, 2009

Senior Manager Boris Grinkot recently attended the Eloqua Experience 2009 Global User Conference in San Francisco. He sat down to answer a few questions about his key takeaways for online marketers…

Q: You were the only non-Eloqua employee to have a booth in the Marketing Effectiveness Zone. Users from around the world sought your advice about conversion rate optimization. Was there any pattern to the challenges marketers brought you?

Most websites are not really having a conversation with the customer. They are not guiding the customer through the page. And, most importantly, they are not testing their pages. They may test an email message or a form with Eloqua, but they don’t test the whole page.

There is massive room for improvement to optimize landing pages to ensure they are customer-focused, and then continually improve the results generated by those pages through testing.

Q: Measuring those results was also a major topic of conversation at Eloqua Experience?

Eloqua Experience 2009Yes. Eloqua is seeking to standardize key performance indicators, or KPIs, that CxOs use to measure results in much the same way financial accounting uses key ratios and cash flow statements, income statements, and balance sheets. Today, most measurements in web analytics – like visitors per month, for example – do not directly translate to a standard accounting metric, such as net revenue.

Q: Based on past statements we’ve received from web clinics and here on the blog, KPIs are a little misunderstood by some marketers. How can readers choose and use KPIs effectively?

In our research, we find marketers most frequently tend to look at numbers like conversion rate, bounce rate or number of visitors. While these numbers can be meaningful within the context of page or process functional performance, they don’t necessarily do a good job of measuring the financial performance.

Revenue per visitor is usually an essential KPI that connects online customer behavior to a financial outcome. While this is a more straightforward KPI to calculate for ecommerce sites, even if you have a lead-generation site, you should understand the value of each lead to determine your revenue per visitor.

Average order size can also be a meaningful KPI that helps you distinguish customer segments (e.g., collectors vs. gift shoppers) or test functional changes like in-cart upsells. Choosing the right KPIs is a big topic, but the short of it is that you need to distinguish between behavioral, demographic, and financial metrics and use them appropriately.

Q: So the important thing to remember is, that while some metrics might be useful in an intermediate step, the overall goal should be a revenue-based number?

Absolutely. For testing and optimization, you need those intermediate numbers.

For example, if you’re optimizing just one step of a shopping cart, conversion rate or clickthrough are important testing metrics. But you don’t want to lose sight of the big picture – which would be overall revenue or revenue per visitor. Let’s say you’ve optimized a step and more visitors are clicking through, but they are less motivated and in the end are buying less. If you don’t have that overall revenue KPI, and you just looked at conversion rate, you would erroneously assume that you have definitively improved your shopping cart.

Q: What are some other caveats when choosing a KPI?

Even if you choose the right KPI, you can still get bad information by not looking at individual channels. You may have good revenue per visitor, but that number is just an average. What if one channel is delivering ten times the revenue per visitor of another channel? That is important information to have, especially if you’re paying for traffic, because you need to understand how that spend converts to revenue.

Q: What is the overall benefit of choosing the right KPI for C-level reporting?

By focusing on dollars instead of traffic, business leaders gain a deep understanding about how the investments they make impact revenue generated through the website. If you invest more in a site, what ROI are you getting?

Clickthroughs and conversion rates only muddy the answer to this question. You need numbers directly related to the spend of each campaign. And not in the aggregate, but specific to a channel – so you know that investing X in PPC, email, TV, or even print will lead to a return of Y.

The other upside, which I’m glad Eloqua is pushing, is that numbers like overall revenue, revenue per visitor, or cost per acquisition bring web metrics much closer to standard accounting. These numbers give CxOs the best indicators of their site’s performance in a format they are used to seeing.

Use the comments section below or post your questions to our MarketingExperiments Optimization group.