MarketingExperiments Blog: Research-driven optimization, testing, and marketing ideas

I’ve got a research question. Now what do I do with it?

A few weeks ago, Daniel Burstein wrote a blog about writing research questions. In that blog post, we emphasized the importance of asking “which” rather than “what” questions because a “which” question is clearly testable.

You might ask, “Which page format results in the most lead submissions?” or “Which price point generates the most revenue?” Both questions are clearly stated and include two key pieces of information:

• An independent variable you are going to test
• The dependent variable you will use to measure your results

To know if something is better, first you must know if it is different

With the research question on paper, we can easily create a hypothesis. For the former question: “All page formats will result in the same number of lead submissions.” This type of hypothesis is so famous in research circles that it has a name: “The Null Hypothesis.”

In general terms, the null hypothesis states that varying the independent variable will result in no change to the dependent variable.

In other words, you’re testing to see if changing the page (the independent variable) will change the number of leads (the dependent variable). After all, if there is no change, one cannot be any better than the other.

Why not “The new layout will result in the most lead submissions,” you ask. Because there is no concrete reason to know that there will be a change. Besides, if you already knew the effect of A on B, why would you need to test it?

Control vs. Treatment(s)

In most cases, there will be an existing page that all new versions will be compared to. This page is termed the “Control,” and all new pages are dubbed “Treatments” to guide comparisons later.

The next step in testing your research question is to decide on the most appropriate test structure. This will depend on the number of variations you will be testing, and on the amount of traffic your site receives. At MECLABS, our research analysts do this visually using a small flowchart to represent the flow of traffic to the control and treatment pages.

Take your latest research question and write it down. Below it, write out the following until you have listed all the variations to be tested.

Click to enlarge

At the right hand side of the page, write “All Traffic.” At this point, you need to determine if your traffic should be evenly split between all the tests or if you will pull only a small portion of  traffic into the treatment pages and maintain most of the flow to the existing Control page.

At MECLABS, our analysts use the Test Protocol document to determine how many site visits are required to achieve valid results given a set of treatments and typical conversion rates on the existing page. This process is covered in our Online Testing Course.

Split tests

Draw lines between “All Traffic” and the pages to the left showing the split and mark each with a percentage of traffic to be sent in that path (See below). This design is called a split test. It is very important that traffic is randomly split between the treatments and control. In a high traffic site, the percentage sent to the control can be higher than what is sent to the treatments, as long as you will easily meet the required minimum sample size.

Click to enlarge

Multi-factorial tests

The split test design works for tests of only one step, but sometimes we need to test more than one step in a process. We have two independent variables that we will manipulate separately. For example, if your research question is, “Which checkout process generates the most revenue?” you might want to test several variations of cart layout and payment page layout at the same time.

If you were to test [Cart and Payment Treatment 1] against [Cart and Payment Treatment 2], your results might tell you that [CT and PT 1] produced 15% more revenue than [CT and PT 2], but you would never learn that Cart Treatment 1 paired with Payment Treatment 2 would have yielded an even higher lift!

Essentially, you have two research questions: “Which cart design will generate the most revenue?” and “Which payment design will generate the most revenue?” This means you have two independent variables and one dependent variable.

To test multi-step processes, researchers use a research design called a factorial test. Each variation in each independent variable is tested together so that all combinations are tested. A typical factorial design is represented below.

Click to enlarge

Because the traffic is sent evenly to each pairing, the factorial research design accounts for the natural dependency between steps 1 and 2. If a viewer does not like Cart Treatment 1, they will not proceed to the Payment step, but since you have also tested other combinations of Cart and Payment, you can assume the effect is balanced out.

A factorial test requires a lot more traffic than a split test to achieve validity, but it also gathers a lot more insight. From the results of a factorial test, you can infer not only the winning combination but also which treatment of each step was most successful. This subtle distinction comes in handy if you then wanted to test further refinements of the process.

Click to enlarge

There are some situations that cause problems with research design. It may not always make sense to pair all the possible combinations together, in which case a factorial design is not possible and a split test should be used instead.

Don’t make the mistake of forming all but one or two pairs of the factorial design. An asymmetrical design does not neutralize the dependency of the second step on the first. In other words, if every factor isn’t matched with every possible other factor, you could overlook a potentially big lift.

Traffic volume is crucial for factorial tests

One common reason some marketers don’t run multi-factorial tests is a low-traffic page. For example, with only 3,000 hits a month, a 7% historical conversion rate, and six treatment pairs (2 payment designs x 3 cart designs), it could take as much as three years to validate the factorial design shown above!

When faced with an unreasonable completion time, you have a few choices to make. You can test fewer treatments, resulting in quicker accumulation of hits on each treatment, or you can test one step of the checkout process at a time.

You also have the option to test pairs of pages in a split test, losing the additional insights given by the factorial design. All of those options will reduce the time needed to validate the test.

Sequential tests

Some marketers try to learn about which treatment works best through sequential tests. Essentially, one page was live, or one email was sent, and then the page was changed, or another email was sent. One treatment is left online for a set period, followed by the next treatment, and so forth. This is usually because there was no test design to begin with, and marketers are comparing results after the fact.

This could also be because marketers do have a test design but are unable to split traffic. After all, if you can only direct traffic to a single page design at a time, you can only test pages sequentially. (However, with the wide availability of both free and paid optimization tools, this situation has become quite rare.)

Sequential tests are extremely prone to history effects, where an outside event or phenomenon affects the viewers’ behaviors on the site from one moment in time to another (see our Online Testing Course for more information on History Effects).

For example, an email sent out to the mailing list will increase traffic to whatever homepage treatment is currently online, distorting the actual effect of the design changes. This effect is usually noticeable as a sudden rise on an analytics traffic or conversion chart. Although it is not an optimal research design, this type of study can distinguish between a control and a treatment page. Results should only be interpreted if the possibility of history effect has been considered and found insignificant.

Related Resources:

Marketing Optimization: You can’t find the true answer without the right question

Artificial Optimization: Why at least 40% of marketers shouldn’t test

Marketing Optimization: How to determine the proper sample size

Categories: Analytics & Testing Tags: A/B split, multi-factorial testing, online testing, sequential tests, traffic volume

Great things happen … when you extend your manufacturer’s protection right away!

I recently wrote a blog post about the audacity of hype – how companies can overreach with their advertising claims … and the potential customers who just don’t believe them.

So today, on the flip side, let me address the copywriting that doesn’t say anything at all. Take the above headline, for example. For lack of a better word, let’s call this …

Blandvertising

Blandvertising is a wishy-washy marketing claim. Like the italicized headline above, it wants to mean something … but it just doesn’t mean anything.

Maybe because the marketer didn’t want to have to deal with Legal. Or maybe because the marketing manager or copywriter had an empty text box in InDesign and just had to throw something in there.

This background noise, this elevator music copywriting is a total waste of your marketing budget. If you’re paying for the opportunity to say something, whether with a direct mail piece, a PPC ad, on product packaging, or just on your website … then actually say something.

But what exactly? You’re crazy busy. Perhaps you’re not a writer. And you have an empty text box staring you in the face. What do you put in there?

Through our testing, we have found that …

Categories: Copywriting Tags: advertising, copywriting, headlines, specificity

Your customers may be flat out ignoring your latest news, offers, and ads. Don’t blame them. It’s simple human nature.

Take a quick look at your surroundings – your cubicle, your office, your solarium – wherever you’re reading this. How much do you notice what’s around you? I mean…really notice?

Not as much as you think you do, I’m guessing. Take a recent experiment run here in the labs. And by “experiment” I mean “practical joke run by our Associate Director of Optimization, Adam Lapp.”

Adam Photoshopped a picture of one of our Research Analysts, Ashley, posing with a friend. It’s the picture in this blog post. Perhaps it looks normal at first glance, but if you take a closer look, you can see that the blonde woman on the right looks a little, well, masculine.

That’s because Adam Photoshopped the face of a male Research Analyst over the face of Ashley’s female friend. He then replaced the photo she had hanging in her cubicle with this photo.

And, Ashley didn’t even notice her friend’s metamorphosis until someone pointed it out to her. Even though it was right in front of her face all day. Why?

-

People don’t notice subtle changes in familiar environments

Now I don’t want to throw Ashley under the bus. I fully admit, I’m no better (and neither are you…or our customers).

For example, we recently moved the official offices of MarketingSherpa from Rhode Island to right here in Jacksonville Beach, Florida. I’ve been very cognizant of the need to look for where changes of that address need to be made on our many Web properties.

But I didn’t notice that the change wasn’t made on the MarketingSherpa Twitter page – even though I look at it at least five times a day.

However, when I was interviewing Ryan Amirault, Digital Marketing Manager, Whole Foods Market, for his case study at Email Summit 2012, he instantly noticed the location and started talking about Rhode Island.

So while being new to, say, a landing page, makes the customer more likely to notice the discreet marketing message, even novelty may not help…

-

Invisible Gorilla Test

What we’re really talking about when we say “banner blindness” is a phenomenon scientists refer to as “inattentional blindness” or “selective attention.” The typical person is overloaded with visual stimuli and inputs of all sorts, making it simply impossible to focus on everything. So, people often overlook things that are right in front of their face.

One of the most famous examples of this is the Invisible Gorilla Test…

-

Now, I may have already primed you to see the gorilla in the above video. However, when Daniel Simons of the University of Illinois at Urbana-Champaign, and Christopher Chabris of Harvard University ran this test, they found that, in most test subject groups, 50% of the subjects did not report seeing the gorilla.

-

How can you get your customers to see your ads?

So if a gorilla doesn’t work, you might be thinking , “I need to go one step further. From now on, every ad I run will have a tap dancing chimpanzee with a neon tracksuit.”

Easy, big fella. You need not become a carnival barker to grab your audience’s attention. All I want to draw your attention to is the fact that what is obvious to you (since you likely eat, sleep, and breathe your marketing message) is not always readily apparent to your audience. And don’t take for granted that your message got across just because you put it at the top of your homepage.

Here are a few common sense thoughts to keep in mind as you seek to overcome banner blindness:

-

Contrast: The more an object can stick out, due to bright colors, crazy patterns, or motion, the more people are likely to notice it. (Keep in mind, if you are running a pay-per-click ad, you want more than attention and curiosity clicks … you want quality clicks.)

Here is an example from the MarketingSherpa site. There is a clear contrast between the ad and its surroundings, thanks to the different color, the bright visual, and even in how “Reserve Your Seat” interrupts the “Limited seating still available” rectangle (please note, it comes out a little brighter on the screen than it does on the screen capture below).

-

Click to enlarge

-

A Multichannel Approach: Don’t think that just placing a message at the top of your homepage, smack dab in the middle (especially in a similar font as its surroundings), is enough to get your message across. I’ve made this mistake myself before. “What do you mean we didn’t tell people? It’s right there at the top of the homepage.”Most people won’t even see it. Especially if they visit your site often.

That doesn’t mean you can’t put your message there, it just means to reach your potential customers in as many way as possible with the message – email, social media, dedicated landing pages, offline communications – and not take for granted that the message was received just by placing it on your homepage.

-

Get in the Middle: There are certain areas of webpages that most people usually relegate to background noise – the top and bottom headers, the right and left columns. When placing an ad, or putting information on your own site, try to get right into the middle of the content.

If you aren’t able to, try to make sure your information is at least at a natural stopping point for the content – for example, just to the right of or below the end of a blog post or article.

-

Break out of the Box: Many marketers place information in a box on their homepages or landing pages that doesn’t necessarily need to be in that box…and therefore their audience is overlooking it.

From a headline on a homepage that is placed in a box (and therefore ignored) to testimonials that look like text-based ads, this mistake is all too common. When you’re on your own website, make sure you are not inadvertently making important information look like a banner ad that will be — you guessed it — totally overlooked by your visitors.

-

Related Resources:

The Ultimate Click: How to get what you pay for with pay-per-click advertising

Banner Blindness: Optimize your online display advertising to stick out (or blend in)

Online Advertising: The 3 obstacles you must overcome to create an effective banner ad

Banner Ad Design: The 3 key banner objectives that drove a 285% lift

Banner Design Tested: How a 35% decrease in clicks caused an 88% increase in conversion

Categories: Internet Marketing Strategy Tags: banner, blindness, contrast, invisible gorilla test, marketing, message

What do you get when you divide Jacksonville Beach, Fla. by Arden Hills, MN? I’m sure there’s a punch line in there somewhere. However, if you were tracking your customers’ ZIP codes in a database you would have 32250/55112, or 0.585.

Never mind that it doesn’t make any sense to you and me to divide one ZIP code by another, but a statistical software package is happy to do exactly that for us. Most software just isn’t smart enough to realize that each ZIP code holds a discrete meaning from the next. It sees them as numbers: values which can be sorted in order and used in any type of calculation.

That is why researchers and statistical software packages classify variables into four main types: Nominal, Ordinal, Interval and Ratio.

In this post, I’m going to describe each type of variable to help you understand how they should be used, let you know how this can help improve your data collection…and, while we’re at it, help you sound sharp the next time you’re chatting with your data analyst at the water cooler.

-

Nominal Variables: Used to describe categories

Variables are classified by the structure of what they represent. For example, ZIP codes are an example of a Nominal variable, a categorical name which simply allows us to differentiate between groups.

Gender and Ethnic group are other common examples of this type. Only a limited number of statistical analyses are valid for this type of variable. We can count how many customers have each ZIP code, and compare the counts to see what is most common (Statisticians call this most frequent value the Mode).

We cannot “average” their ZIP codes to determine a population center, or calculate correlations between ZIP code and a customer satisfaction index because there is no real meaning to a “higher” or “lower” numerical ZIP code.

If we wanted to know about geographic patterns in customer satisfaction, we would have to take the average satisfaction index for each ZIP code and compare those averages to one another. Browser type and operating system are two other common Nominal variables.

Word of Caution – This first one seems obvious, but keep in mind it is an easy oversight to have a number in a spreadsheet or database inadvertently become part of a calculation.

-

Ordinal Variables: Used to rank preference

The next level of complexity is represented by the Ordinal variable. Ordinal variables are sequential; they advance in a direction but the increments on the scale are unknown or uneven.

For example, the organizational chart of a company might show that the mailroom attendant is below the marketing analyst, and he in turn is below the vice president, who is below the president. There is a clear direction, but the relationship between ranks is not consistent.

In marketing research, consumers sometimes rank new products in order of preference. They do not necessarily like product 1 twice as much as product 2, or 3 twice as much as 4. So when analyzing the data from the test, a researcher can find the Mode, or calculate the middle ranked item (the Median), but it is not valid to calculate the “average rating” given to an item. Because the distance between items on the scale is unknown it is not possible to really tell an average value.

Calculations such as addition and multiplication can be done with ordinal data, however any calculation made on one must be consistently made on all items in the data set, in order to maintain the proportions and order of all members of the data set.

Word of caution – One common survey scale is the Likert scale, which allows respondents to rate their agreement with statements on a 5- or 7-point scale from “Strongly Agree” to “Strongly Disagree.” Because there is no way to know the difference between “Strongly Agree” and “Agree” in the mind of each respondent, or to ensure that each respondent is consistent in their judgments, these results are Ordinal data.

Many research studies treat Ordinal data as Interval data (more on that next), making a basic and sometimes flawed assumption that the scale represents a consistent interval between one ranking and the next. While each individual will be relatively consistent in their ratings, there is no consistency between individuals. This creates a limitation on the generalization of the results of the calculations, but this type of analysis may still offer significant insights into your data. It is important to understand that the results from such an analysis are imprecise and should only be interpreted generally, rather than by comparisons of small differences.

-

Interval and Ratio Data: Now we can get into the valuable number crunching

Both Interval and Ratio variables possess not only a sequence, but an even interval. Here’s where it gets tricky: the difference between the two types is zero. Yes, 0.

Interval variables may have a point which we designate “zero,” however negative numbers are theoretically possible.

A Ratio variable has a real zero point, a point which nothing can be below.

For example, an item’s price can be zero, or “free,” but price is not a Ratio value. Why? Because -$1.99, or a negative price, is conceptually possible. Take German government bonds. In a recent auction, the bonds yielded negative 0.0122%.

We try never to pay our customers to purchase our products, but theoretically, negative price has meaning. Therefore, price is an Interval variable.

Many true Ratio variables are found in marketing research. “Number of Page Visits” and “Time on Page” are common Ratio variables. The good news is that almost all statistical techniques used in marketing research can be applied to both Interval and Ratio data. Mean, Median, Mode, Correlation, Standard Deviation and ANOVA are all equally valid with both types of data.

-

So what does this mean for you?

When you design your experiments, think about the type of variables you will be collecting data for. Interval and Ratio variables allow the most flexibility in statistical analysis, so whenever possible try to use them rather than Ordinal or Nominal data. A survey question could ask “which of the following tasks have you undertaken in the last 24 hours?” which produces a multiple choice, Nominal, answer.

It could also ask, “Please rank these tasks from most to least recently undertaken,” which produces Ordinal data and allows some additional analysis.

Finally, the survey could ask, “At what time and date did you last undertake these tasks?” producing concrete Interval data which will allow you to compare between respondents and run in depth statistical functions.

In the design phase of your marketing tests, think about the statistical data you would like to produce, and what variable types are required to calculate the results you need in order to answer your research questions. When you enter your data into a statistical software package, be careful to designate the correct variable type in the software so that the program can prevent you from dividing Florida by Minnesota.

-

Related Resources:

Marketing Optimization: You can’t find the true answer without the right question

Research Update: The state of email marketing testing and optimization

Marketing Optimization: What your peers learned this year about Adwords, the inbox, and telling the truth

Evidence-based Marketing: How your peers protect against bad marketing data

Categories: Analytics & Testing Tags: interval, marketing, metrics, nominal, ordinal, ratio, software, testing, variable

@veronica Thanks for the response!

Editor’s Note: You’ll never find the right answers if you don’t ask the right questions. So my hat’s off to Veronica Cisneros, lead Web designer and developer at websonlized.com, for continuing to push us to dive deeper into the best use of search engine marketing.

After answering her initial question in “PPC Ads: What is search engine marketing best used for?” Paul Cheney takes our exploration of the most effective use of pay-per-click advertising one level deeper today …

In the post, Daniel points out that search engine marketing (PPC Ads) are best utilized in communicating “the value of a click to your landing page, not to get a sale.”

That is his main point. And he’s absolutely right.

What he didn’t mention (probably for the sake of brevity) was the idea that “the value of a click to your landing page” should be a derivative of the “value of the ultimate sale.”

That is what I mean by “the ultimate click.” The ultimate click is the sale. And in many cases, the sale comes after a series of micro-yeses.

So in other words, it makes more business sense to run an ad for toothbrushes when you are selling toothbrushes, than to run an ad for a free car when you are selling toothbrushes.

This is because in the toothbrush ad, the value of the click to the landing page is to get more information about the toothbrushes your company offers.

The toothbrush ad is a derivative of the ultimate value of buying a toothbrush. The free car ad is not.

That is what I mean when I say it’s important to get “the correct clicks” rather than simply as many clicks as possible. If the goal was to get as many clicks as I could, I would obviously want to run an ad for a free car.

But because the goal is sales, not clicks, I need to run an ad for a toothbrush.

Now, while I’d be open to testing it (especially if I’m selling toothbrushes), the copy of that ad probably wouldn’t be:

Buy Our Toothbrushes

They’re really great

Only $45 each!

I’d most likely run an ad along the lines of:

Designer Toothbrushes

Explore our catalogue of

50 brands used by celebs

In the first ad, I tried to sell in the ad. I made it seem like the reader should click on the ad and buy a toothbrush for $45.

In the second ad, I made the value of the click about being able to browse high-quality designer toothbrushes. And hopefully, that’s exactly what they’ll be able to do when they click the ad.

Daniel, correct me if I’m wrong, but I think this is the point you were trying to get across:

Selling in the ad is usually bad. The goal of an ad should be to get a click.

I’m simply adding that the click should also be as relevant as possible to the ultimate offer.

I hope that clears things up.

Editor’s Note: Spot on, Paul. And might I add that, this is not simply an academic discussion. Remember, these are pay-per-click ads. Why pay for traffic that will not convert?

So while Paul’s examples are purposefully extreme to make a point (although, I’ll admit, he’s got me seriously Jonesing to find out which toothbrush Brangelina uses), it would help you to take a second look at your AdWords account to determine whether your aim is to get a click, or get a click that will convert.

Related Resources:

Banner Ad Design: The 3 key banner objectives that drove a 285% lift

Banner Design Tested: How a 35% decrease in clicks caused an 88% increase in conversion

Converting PPC Traffic: How clarifying value generated 99.4% more conversions on a PPC landing page

Categories: Paid Search Marketing (PPC) Tags: conversion rates, pay-per-click, PPC, relevance, search marketing, value

In this world, there are systems that underperform. It is a fact of life. A quick look at the world’s distribution of wealth is all anyone needs for proof of that. It happens all the time on a macro level. And when a system doesn’t just underperform but is truly broken, it usually means you need to tear it down and start from scratch.

And while it may not be humanly possible to do that for the world’s economic system, it’s very doable with your website.

Our websites are simply little systems that should present enough pieces of our value proposition in the right sequence to our ideal customer so that they take the desired action. You can make many tweaks to your site to improve how well it does that … and in so doing, improve conversion.

But for some websites, the system is broken. A new approach is needed. At MECLABS, we call this a category shift.

How can I implement a category shift for my website?

To implement this category shift, you need a radical redesign.

A radical redesign is simply an experimental approach in which the experimental treatments are “radically” or “categorically” different from the control.

While definitions are certainly interesting, it’s probably easier to give you an example of a radical redesign. So here’s a radical redesign experiment we recently ran with one of our research partners to flesh out that definition. It also happens to be the same experiment we’ll study in-depth for today’s free Web clinic at 4:00 pm EST: Rapidly Maximizing Conversion: How one company quickly achieved a 53.9% lift with a radical redesign.

Experiment Details:

Background: The company is a leading automotive head gasket repair solution

Goal: To increase total orders on cart page

Primary Research Question: Which landing page/cart will result in a higher conversion rate?

Approach: Radical redesign of cart page through a variable cluster A/B split test

Control:

The control is a single product shopping cart process with six steps. Get ready to look at a lot of creative samples:

Click to enlarge	Click to enlarge
Click to enlarge	Click to enlarge
Click to enlarge	Click to enlarge

Treatment:

The treatment attempted to fix the obvious problem with the control by shortening the number of steps down to two. But besides that, it was a completely different approach in design and layout.

Click to enlarge

Click to enlarge

Results:

Now if you read the title of today’s Web clinic earlier in the post, you probably know which of these treatments won and by how much. But what’s really interesting about this test is WHY the treatment won.

Once you can figure out why a treatment performed the way it did, you can apply the underlying principle to your own pages and likely get better success than simply copy/pasting the specific tactics in the treatment.

So to test your marketing intuition, tell us why you think the treatment outperformed the control in this experiment.

The marketer who comes closest to our Sciences team’s expert speculation will win a mention on this blog with a link and the adoration and envy of their peers.

And once you’ve left your comment, be sure to sign up for today’s free clinic to learn more about radical redesigns.

 UPDATE: 

Congratulations to Kai for having the closest reason (in our opinion) for why the treatment outperformed the control. To get a better idea of what went into this experiment, checkout the replay for this Web clinic.

Related Resources:

Web clinic replay – Rapidly Maximizing Conversion: How one company quickly achieved a 53.9% lift with a radical redesign

Landing Page Optimization: How to plan a radical redesign so you get a lift AND a learning

E-commerce Shopping Carts: How a redesigned checkout process led to 13% increase in conversion rate

Homepage Optimization: Radical redesign ideas for multivariable testing

Categories: Site Design Tags: category shift, increased conversion rate, marketing intuition, radical redesign, site design