MarketingExperiments Blog: Research-driven optimization, testing, and marketing ideas

What do you get when you divide Jacksonville Beach, Fla. by Arden Hills, MN? I’m sure there’s a punch line in there somewhere. However, if you were tracking your customers’ ZIP codes in a database you would have 32250/55112, or 0.585.

Never mind that it doesn’t make any sense to you and me to divide one ZIP code by another, but a statistical software package is happy to do exactly that for us. Most software just isn’t smart enough to realize that each ZIP code holds a discrete meaning from the next. It sees them as numbers: values which can be sorted in order and used in any type of calculation.

That is why researchers and statistical software packages classify variables into four main types: Nominal, Ordinal, Interval and Ratio.

In this post, I’m going to describe each type of variable to help you understand how they should be used, let you know how this can help improve your data collection…and, while we’re at it, help you sound sharp the next time you’re chatting with your data analyst at the water cooler.

-

Nominal Variables: Used to describe categories

Variables are classified by the structure of what they represent. For example, ZIP codes are an example of a Nominal variable, a categorical name which simply allows us to differentiate between groups.

Gender and Ethnic group are other common examples of this type. Only a limited number of statistical analyses are valid for this type of variable. We can count how many customers have each ZIP code, and compare the counts to see what is most common (Statisticians call this most frequent value the Mode).

We cannot “average” their ZIP codes to determine a population center, or calculate correlations between ZIP code and a customer satisfaction index because there is no real meaning to a “higher” or “lower” numerical ZIP code.

If we wanted to know about geographic patterns in customer satisfaction, we would have to take the average satisfaction index for each ZIP code and compare those averages to one another. Browser type and operating system are two other common Nominal variables.

Word of Caution – This first one seems obvious, but keep in mind it is an easy oversight to have a number in a spreadsheet or database inadvertently become part of a calculation.

-

Ordinal Variables: Used to rank preference

The next level of complexity is represented by the Ordinal variable. Ordinal variables are sequential; they advance in a direction but the increments on the scale are unknown or uneven.

For example, the organizational chart of a company might show that the mailroom attendant is below the marketing analyst, and he in turn is below the vice president, who is below the president. There is a clear direction, but the relationship between ranks is not consistent.

In marketing research, consumers sometimes rank new products in order of preference. They do not necessarily like product 1 twice as much as product 2, or 3 twice as much as 4. So when analyzing the data from the test, a researcher can find the Mode, or calculate the middle ranked item (the Median), but it is not valid to calculate the “average rating” given to an item. Because the distance between items on the scale is unknown it is not possible to really tell an average value.

Calculations such as addition and multiplication can be done with ordinal data, however any calculation made on one must be consistently made on all items in the data set, in order to maintain the proportions and order of all members of the data set.

Word of caution – One common survey scale is the Likert scale, which allows respondents to rate their agreement with statements on a 5- or 7-point scale from “Strongly Agree” to “Strongly Disagree.” Because there is no way to know the difference between “Strongly Agree” and “Agree” in the mind of each respondent, or to ensure that each respondent is consistent in their judgments, these results are Ordinal data.

Many research studies treat Ordinal data as Interval data (more on that next), making a basic and sometimes flawed assumption that the scale represents a consistent interval between one ranking and the next. While each individual will be relatively consistent in their ratings, there is no consistency between individuals. This creates a limitation on the generalization of the results of the calculations, but this type of analysis may still offer significant insights into your data. It is important to understand that the results from such an analysis are imprecise and should only be interpreted generally, rather than by comparisons of small differences.

-

Interval and Ratio Data: Now we can get into the valuable number crunching

Both Interval and Ratio variables possess not only a sequence, but an even interval. Here’s where it gets tricky: the difference between the two types is zero. Yes, 0.

Interval variables may have a point which we designate “zero,” however negative numbers are theoretically possible.

A Ratio variable has a real zero point, a point which nothing can be below.

For example, an item’s price can be zero, or “free,” but price is not a Ratio value. Why? Because -$1.99, or a negative price, is conceptually possible. Take German government bonds. In a recent auction, the bonds yielded negative 0.0122%.

We try never to pay our customers to purchase our products, but theoretically, negative price has meaning. Therefore, price is an Interval variable.

Many true Ratio variables are found in marketing research. “Number of Page Visits” and “Time on Page” are common Ratio variables. The good news is that almost all statistical techniques used in marketing research can be applied to both Interval and Ratio data. Mean, Median, Mode, Correlation, Standard Deviation and ANOVA are all equally valid with both types of data.

-

So what does this mean for you?

When you design your experiments, think about the type of variables you will be collecting data for. Interval and Ratio variables allow the most flexibility in statistical analysis, so whenever possible try to use them rather than Ordinal or Nominal data. A survey question could ask “which of the following tasks have you undertaken in the last 24 hours?” which produces a multiple choice, Nominal, answer.

It could also ask, “Please rank these tasks from most to least recently undertaken,” which produces Ordinal data and allows some additional analysis.

Finally, the survey could ask, “At what time and date did you last undertake these tasks?” producing concrete Interval data which will allow you to compare between respondents and run in depth statistical functions.

In the design phase of your marketing tests, think about the statistical data you would like to produce, and what variable types are required to calculate the results you need in order to answer your research questions. When you enter your data into a statistical software package, be careful to designate the correct variable type in the software so that the program can prevent you from dividing Florida by Minnesota.

-

Related Resources:

Marketing Optimization: You can’t find the true answer without the right question

Research Update: The state of email marketing testing and optimization

Marketing Optimization: What your peers learned this year about Adwords, the inbox, and telling the truth

Evidence-based Marketing: How your peers protect against bad marketing data

This time of year, many marketers are beginning to execute on their new marketing plans. However, sometimes you have to deviate from the plan and just need a sale or lead generation lift… RIGHT NOW!

When your boss or client challenges you to gain a quick conversion increase on your landing pages, what tools do you turn to in your marketing toolbox?

In Wednesday’s Web clinic – Rapidly Maximizing Conversion: How one company quickly achieved a 53.9% lift with a radical redesign – MECLABS Managing Director Flint McGlaughlin will share our top discoveries around how to quickly improve your conversion rate.

But before we share what we learned, we wanted to hear from you. Here are a few of our favorite “quick hit” tips from your peers …

-

Headline optimization

I have found that headline and subhead optimization works well for the B2B SaaS clients I typically work with. Even after I think I have tested my way to the perfect headline, I run more tests and get more lift. I regularly get 10% lifts from this tactic. If I have more time to gather data, I will multivariate test headline, CTA button and benefit/bulleted text.

Finally, if you haven’t already, make sure there is just one key CTA button which is huge and obvious. I’m always surprised at how many sites don’t do this.

– Chris Neumann, General Manager, TextMarks

-

5 Quick Tips

A few quick things come to mind:

1) Drop prices and provide free shipping: This one is pretty obvious, but nothing converts like low prices and free shipping.

2) Add security/trust logos and other “credibility” links (security policy, etc.) to checkout process: These types of additions have proven to immediately bump the conversion rate by providing a visual feel of safety and security, even if the users never do anything besides see the presence of the icons or links.

3) Simplify checkout process, including NOT requiring users to create an account in order to checkout: A simple checkout process reduces the likelihood users will drop-off.

4) Increase frequency of targeted email campaigns: There is so much email going around these days, from so many different sources, for so many different purposes. Research and testing has shown that sending a single email campaign up to 9 times can continue to provide incremental benefit in sales, with very little subsequent downside in customer satisfaction. The truth is, most people don’t see a very high percentage of their email.

5) Implement abandoned cart targeted emails: Enticing users to complete the checkout process can be very effective because you are targeting shoppers that you know are already interested in some of your products.

The above items are all proven to increase conversion – some are more quickly implemented than others.

– Todd Stalter, Senior Visualization Analyst, OneSpring

-

Contests and chatting

For quick results I would implement the following:

1. An online contest where all the visitor needs to do is provide a name and email address, Facebook “like,” and/or Twitter follow, depending on what kind of lead capture you want. Online contests with enticing prizes can go viral and get you many followers quickly.

2. Implement a live chat feature on the site to make it easier to interact with visitors. However, I have found that live chat software with the standard popup window do not convert as well as the newer live chat programs such as Zopim and Olark that have a more social feel to them. Another option is to implement a video live chat program so customers can see the site representative on video, which helps even more with building trust in your company.

– Shai Atanelov, CEO & Founder, BigtimeWireless.com

-

Related Resources:

Rapidly Maximizing Conversion: How one company quickly achieved a 53.9% lift with a radical redesign – Web clinic

Most-Tweeted Posts of 2011: Social media marketing, copywriting, email testing and more …

Landing Page Optimization: How to plan a radical redesign so you get a lift AND a learning

Marketing Campaign: Landing page optimization can help improve the return on your media spend

As I prepare to wade through hundreds of submissions for the MarketingSherpa 2012 Marketing Wisdom Report, (sponsored by HubSpot) I was compelled to take a final glance at the 2011 edition.

While combing through the pages, many of last year’s submissions evoked some forward-thinking thoughts for 2012. Here are just a few of the standouts…

- Read more…

The majority of B2B organizations are increasing their marketing budgets for inbound marketing tactics. One of the most popular of those inbound tactics is virtual events and webinars, with 60% of marketers increasing their investment according to the MarketingSherpa 2011 B2B Marketing Benchmark Report.

“It is essential for organizations to gain the trust of their buyers before they can hope to convert them,” said Jen Doyle, the Benchmark Report’s lead author. “Webinars offer an effective platform to improve thought leadership and reputation, both essential components to winning trust. The cost effectiveness of webinars is just the icing on the cake, so many organizations are shifting to include webinars as part of their marketing plans.”

Of course, a webinar isn’t very effective if no one attends. So in today’s MarketingExperiments blog post, Gaby Paez and I will give you some test ideas for those all important webinar invite emails (and if you’d like to see how we craft our own webinar invite emails, just sign up) by reviewing a live optimization submission from The Chronicle of Philanthropy.

Gaby is associate director of Research at MECLABS, and you can hear more of her test ideas in the Web clinic replay, Email Messaging: How overcoming 3 common errors increased clickthrough 104%, along with some of the audience’s optimization advice for this submission.

Here’s the submission (and you can view it online as well) …

Click to enlarge

BACKGROUND

Email – Invitation to a paid social media webinar, “Going Mobile: How Nonprofits Succeed,” which features a bonus opportunity to gain access to “an exclusive discussion group” and three speakers:

• Felicia Carr, Senior Director of Online Communications, National Parks Conservation Association
• Jason Wood, Director of Digital Communications Services, Salvation Army National Headquarters
• Cody Switzer, Assistant Web Editor, The Chronicle of Philanthropy

Audience – Nonprofit professionals in fund-raising, marketing, social media and development

Objective – To get registrants for a paid webinar

  Read more…

It’s the holiday time, so let me talk about a holiday. Passover, naturally (hey, if you want to succeed in marketing, don’t follow the crowds).

In the classic seder, there are The Four Sons. For this blog post, I’m going to focus on “the one who does not know how to ask a question” because I think that’s a perfect explanation of where many marketers are right now with their testing practices. For example, according to Jeff Rice’s just released 2012 Email Marketing Benchmark Report, he found that 85% of marketers don’t even know why they’re running every test they run!

-

Click to enlarge

-

Take a look at the second field in this chart. Only 15% of marketers routinely define the question, objective, and key metric when running a test. Why bother taking the time to set up a test if you don’t know what you’re looking for in the first place?

My guess is, that like the fourth son referenced above, they simply do not know how to ask. So in today’s blog post, I want to briefly discuss how to write a research question. And in this Thursday’s MarketingSherpa webinar – Negative Lifts: Turning a 25% loss into a 141% increase in conversion – Junior Editorial Analyst Paul Cheney and I will be discussing in more depth how you can learn about your customers from tests, along with Tina Hou, the director of product marketing for webinar sponsor TRUSTe.

-

How to create a clear research question

While all of the elements in Jeff’s chart are important for running a valuable test (i.e. truly learning what really works with your customers), the research question will play the biggest role in guiding your test design.

That said, the research question is just one part of step #11 (“Test Comps”) in the MarketingExperiments Optimization process. Before our analysts begin to design tests using the MECLABS Test Protocol (in which they define the research question) and begin the iterative testing process, they do everything from determining the page objective to submitting comps for peer review (you can see our full landing page optimization process in session 7 of our paid Landing Page Optimization Online Course.)

While I obviously can’t cover the entire process in this blog post, if I can help you write a true research question, I can set you on the path to learning about your customers from your tests. This is a complex process, but if I had to simplify it into three steps, I would say…

-

1. Start with asking “what” you want to know

Clearly you’re running a test for a reason. Write that question down on a piece of paper. Go ahead, do it, I’ll wait.

Now pass that piece of paper (or email) around. Are all of the key players aligned that this is, in fact, what are you trying to learn from your tests?

For example, you may want to know “What is the best price for product X?” This is the variable you will test.

-

2. Turn this into a question of “which”

Good start. Except the only problem is, to truly answer the above question, you would have to test an infinite number of prices. And I’m guessing your time, resources, traffic, and patience are not infinite.

So to narrow your focus, you want to ask a question of “which.” Not only will this force you to think about exactly how you’re designing your test, it helps you create a testing-optimization cycle to continually learn about your customers from your tests and improve your marketing performance.

A year from now, when you’re been promoted three times for driving such impressive results, and the new hotshot your direct report’s direct report hired sees a test that asks, “Which of these three price points – $1, $2, or $3 – is best for product X?” he will know exactly what you tested. And exactly what you learned about the customer.

-

3. Add in your KPI

So now you know exactly which prices (technically speaking, which values of your variable) you need to design a test around. The next question you need to ask is – how do I pick a winner?

You certainly don’t want to write the rules after the fact. “My favorite analogy for this is throwing a rock in the forest and saying, ‘look, I hit that tree,’” said Phillip Porter, Data Analyst, MECLABS. “If you aren’t aiming for something before you start, how do you know if you hit what you aimed for?”

What KPI (key performance indicator) will help you determine which value is the winner? To reformulate our example question, you would say “Which of these three price points – $1, $2, or $3 – will generate the most revenue for product X?”

Now everyone on your team (and everyone on your team a year from now) knows exactly how you define “best.” If you don’t think through and define the question beforehand, you might just try to come up with an answer based on whatever metrics you had on hand after the test is run. For example, choosing sales instead of revenue, and picking a winner that sells more product but generates less money in your pocket.

You might also not even have the chance to redefine the rules after the test is run since, since depending on the metric, the testing platform, and your transactional data system, you might not have captured the KPI that you later determine would have been most effective to know.

-

“If you don’t know where you are going, any road will get you there.” – Lewis Carroll

-

In the end, the value of the research question is that it helps ensure all the effort and resources you invest in testing and optimization gets you to where you want to go. Or, as Phillip related, you might as well be testing through the looking glass…

Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to
Alice: I don’t much care where.
The Cheshire Cat: Then it doesn’t much matter which way you go.
Alice: …so long as I get somewhere.
The Cheshire Cat: Oh, you’re sure to do that, if only you walk long enough.

-

Related Resources:

Research Update: The state of email marketing testing and optimization

Negative Lifts: Turning a 25% loss into a 141% increase in conversion – Thursday, December 1, 1 p.m. EST

MarketingSherpa 2012 Email Marketing Benchmark Report

Landing Page Optimization: How IBM applied homepage redesign learnings to landing page testing

What marketing errors are easiest to avoid? And how do we avoid making them?

My answer would be…those related to overconfidence. And, as to the second question, I’ll take the rest of this blog post to attempt to answer that.

-

Are you too confident?

In the business world, as in marketing, we usually look at confidence as a good thing. But the “overconfidence bias” can seriously harm your performance.

Here’s how Jonah Lehrer, an American journalist who writes on the topics of psychology and neuroscience, describes this overconfidence bias in The Science of Irrationality: A Nobelist explains our fondness for not thinking…

-

Consider the overconfidence bias, which drives many of our mistakes in decision-making. The best demonstration of the bias comes from the world of investing. Although many fund managers charge high fees to oversee stock portfolios, they routinely fail a basic test of skill: persistent achievement. As Mr. [Nobel Laureate, and professor of psychology at Princeton] Kahneman notes, the year-to-year correlation between the performance of the vast majority of funds is barely above zero, which suggests that most successful managers are banking on luck, not talent.

This shouldn’t be too surprising. The stock market is a case study in randomness, a system so complex that it’s impossible to predict. Nevertheless, professional investors routinely believe that they can see what others can’t. The end result is that they make far too many trades, with costly consequences.

-

Sound familiar? Is marketing a product any less complex than trading on the stock market?

-

How to bank on talent, not luck

I disagree with one aspect of Lehrer’s article. Not to put words into his mouth, but he seems to imply that there is no way to overcome the overconfidence bias. In marketing, I believe there is a way to do so (of course, perhaps that’s just me being …ahem … overconfident).

Let me explain what I mean, and let you be the judge… Read more…