I’ve got a research question. Now what do I do with it?

A few weeks ago, Daniel Burstein wrote a blog about writing research questions. In that blog post, we emphasized the importance of asking “which” rather than “what” questions because a “which” question is clearly testable.

You might ask, “Which page format results in the most lead submissions?” or “Which price point generates the most revenue?” Both questions are clearly stated and include two key pieces of information:

• An independent variable you are going to test
• The dependent variable you will use to measure your results

To know if something is better, first you must know if it is different

With the research question on paper, we can easily create a hypothesis. For the former question: “All page formats will result in the same number of lead submissions.” This type of hypothesis is so famous in research circles that it has a name: “The Null Hypothesis.”

In general terms, the null hypothesis states that varying the independent variable will result in no change to the dependent variable.

In other words, you’re testing to see if changing the page (the independent variable) will change the number of leads (the dependent variable). After all, if there is no change, one cannot be any better than the other.

Why not “The new layout will result in the most lead submissions,” you ask. Because there is no concrete reason to know that there will be a change. Besides, if you already knew the effect of A on B, why would you need to test it?

Control vs. Treatment(s)

In most cases, there will be an existing page that all new versions will be compared to. This page is termed the “Control,” and all new pages are dubbed “Treatments” to guide comparisons later.

The next step in testing your research question is to decide on the most appropriate test structure. This will depend on the number of variations you will be testing, and on the amount of traffic your site receives. At MECLABS, our research analysts do this visually using a small flowchart to represent the flow of traffic to the control and treatment pages.

Take your latest research question and write it down. Below it, write out the following until you have listed all the variations to be tested.

Click to enlarge

At the right hand side of the page, write “All Traffic.” At this point, you need to determine if your traffic should be evenly split between all the tests or if you will pull only a small portion of  traffic into the treatment pages and maintain most of the flow to the existing Control page.

At MECLABS, our analysts use the Test Protocol document to determine how many site visits are required to achieve valid results given a set of treatments and typical conversion rates on the existing page. This process is covered in our Online Testing Course.

Split tests

Draw lines between “All Traffic” and the pages to the left showing the split and mark each with a percentage of traffic to be sent in that path (See below). This design is called a split test. It is very important that traffic is randomly split between the treatments and control. In a high traffic site, the percentage sent to the control can be higher than what is sent to the treatments, as long as you will easily meet the required minimum sample size.

Click to enlarge

Multi-factorial tests

The split test design works for tests of only one step, but sometimes we need to test more than one step in a process. We have two independent variables that we will manipulate separately. For example, if your research question is, “Which checkout process generates the most revenue?” you might want to test several variations of cart layout and payment page layout at the same time.

If you were to test [Cart and Payment Treatment 1] against [Cart and Payment Treatment 2], your results might tell you that [CT and PT 1] produced 15% more revenue than [CT and PT 2], but you would never learn that Cart Treatment 1 paired with Payment Treatment 2 would have yielded an even higher lift!

Essentially, you have two research questions: “Which cart design will generate the most revenue?” and “Which payment design will generate the most revenue?” This means you have two independent variables and one dependent variable.

To test multi-step processes, researchers use a research design called a factorial test. Each variation in each independent variable is tested together so that all combinations are tested. A typical factorial design is represented below.

Click to enlarge

Because the traffic is sent evenly to each pairing, the factorial research design accounts for the natural dependency between steps 1 and 2. If a viewer does not like Cart Treatment 1, they will not proceed to the Payment step, but since you have also tested other combinations of Cart and Payment, you can assume the effect is balanced out.

A factorial test requires a lot more traffic than a split test to achieve validity, but it also gathers a lot more insight. From the results of a factorial test, you can infer not only the winning combination but also which treatment of each step was most successful. This subtle distinction comes in handy if you then wanted to test further refinements of the process.

Click to enlarge

There are some situations that cause problems with research design. It may not always make sense to pair all the possible combinations together, in which case a factorial design is not possible and a split test should be used instead.

Don’t make the mistake of forming all but one or two pairs of the factorial design. An asymmetrical design does not neutralize the dependency of the second step on the first. In other words, if every factor isn’t matched with every possible other factor, you could overlook a potentially big lift.

Traffic volume is crucial for factorial tests

One common reason some marketers don’t run multi-factorial tests is a low-traffic page. For example, with only 3,000 hits a month, a 7% historical conversion rate, and six treatment pairs (2 payment designs x 3 cart designs), it could take as much as three years to validate the factorial design shown above!

When faced with an unreasonable completion time, you have a few choices to make. You can test fewer treatments, resulting in quicker accumulation of hits on each treatment, or you can test one step of the checkout process at a time.

You also have the option to test pairs of pages in a split test, losing the additional insights given by the factorial design. All of those options will reduce the time needed to validate the test.

Sequential tests

Some marketers try to learn about which treatment works best through sequential tests. Essentially, one page was live, or one email was sent, and then the page was changed, or another email was sent. One treatment is left online for a set period, followed by the next treatment, and so forth. This is usually because there was no test design to begin with, and marketers are comparing results after the fact.

This could also be because marketers do have a test design but are unable to split traffic. After all, if you can only direct traffic to a single page design at a time, you can only test pages sequentially. (However, with the wide availability of both free and paid optimization tools, this situation has become quite rare.)

Sequential tests are extremely prone to history effects, where an outside event or phenomenon affects the viewers’ behaviors on the site from one moment in time to another (see our Online Testing Course for more information on History Effects).

For example, an email sent out to the mailing list will increase traffic to whatever homepage treatment is currently online, distorting the actual effect of the design changes. This effect is usually noticeable as a sudden rise on an analytics traffic or conversion chart. Although it is not an optimal research design, this type of study can distinguish between a control and a treatment page. Results should only be interpreted if the possibility of history effect has been considered and found insignificant.

Related Resources:

Marketing Optimization: You can’t find the true answer without the right question

Artificial Optimization: Why at least 40% of marketers shouldn’t test

Marketing Optimization: How to determine the proper sample size

What do you get when you divide Jacksonville Beach, Fla. by Arden Hills, MN? I’m sure there’s a punch line in there somewhere. However, if you were tracking your customers’ ZIP codes in a database you would have 32250/55112, or 0.585.

Never mind that it doesn’t make any sense to you and me to divide one ZIP code by another, but a statistical software package is happy to do exactly that for us. Most software just isn’t smart enough to realize that each ZIP code holds a discrete meaning from the next. It sees them as numbers: values which can be sorted in order and used in any type of calculation.

That is why researchers and statistical software packages classify variables into four main types: Nominal, Ordinal, Interval and Ratio.

In this post, I’m going to describe each type of variable to help you understand how they should be used, let you know how this can help improve your data collection…and, while we’re at it, help you sound sharp the next time you’re chatting with your data analyst at the water cooler.

-

Nominal Variables: Used to describe categories

Variables are classified by the structure of what they represent. For example, ZIP codes are an example of a Nominal variable, a categorical name which simply allows us to differentiate between groups.

Gender and Ethnic group are other common examples of this type. Only a limited number of statistical analyses are valid for this type of variable. We can count how many customers have each ZIP code, and compare the counts to see what is most common (Statisticians call this most frequent value the Mode).

We cannot “average” their ZIP codes to determine a population center, or calculate correlations between ZIP code and a customer satisfaction index because there is no real meaning to a “higher” or “lower” numerical ZIP code.

If we wanted to know about geographic patterns in customer satisfaction, we would have to take the average satisfaction index for each ZIP code and compare those averages to one another. Browser type and operating system are two other common Nominal variables.

Word of Caution – This first one seems obvious, but keep in mind it is an easy oversight to have a number in a spreadsheet or database inadvertently become part of a calculation.

-

Ordinal Variables: Used to rank preference

The next level of complexity is represented by the Ordinal variable. Ordinal variables are sequential; they advance in a direction but the increments on the scale are unknown or uneven.

For example, the organizational chart of a company might show that the mailroom attendant is below the marketing analyst, and he in turn is below the vice president, who is below the president. There is a clear direction, but the relationship between ranks is not consistent.

In marketing research, consumers sometimes rank new products in order of preference. They do not necessarily like product 1 twice as much as product 2, or 3 twice as much as 4. So when analyzing the data from the test, a researcher can find the Mode, or calculate the middle ranked item (the Median), but it is not valid to calculate the “average rating” given to an item. Because the distance between items on the scale is unknown it is not possible to really tell an average value.

Calculations such as addition and multiplication can be done with ordinal data, however any calculation made on one must be consistently made on all items in the data set, in order to maintain the proportions and order of all members of the data set.

Word of caution – One common survey scale is the Likert scale, which allows respondents to rate their agreement with statements on a 5- or 7-point scale from “Strongly Agree” to “Strongly Disagree.” Because there is no way to know the difference between “Strongly Agree” and “Agree” in the mind of each respondent, or to ensure that each respondent is consistent in their judgments, these results are Ordinal data.

Many research studies treat Ordinal data as Interval data (more on that next), making a basic and sometimes flawed assumption that the scale represents a consistent interval between one ranking and the next. While each individual will be relatively consistent in their ratings, there is no consistency between individuals. This creates a limitation on the generalization of the results of the calculations, but this type of analysis may still offer significant insights into your data. It is important to understand that the results from such an analysis are imprecise and should only be interpreted generally, rather than by comparisons of small differences.

-

Interval and Ratio Data: Now we can get into the valuable number crunching

Both Interval and Ratio variables possess not only a sequence, but an even interval. Here’s where it gets tricky: the difference between the two types is zero. Yes, 0.

Interval variables may have a point which we designate “zero,” however negative numbers are theoretically possible.

A Ratio variable has a real zero point, a point which nothing can be below.

For example, an item’s price can be zero, or “free,” but price is not a Ratio value. Why? Because -$1.99, or a negative price, is conceptually possible. Take German government bonds. In a recent auction, the bonds yielded negative 0.0122%.

We try never to pay our customers to purchase our products, but theoretically, negative price has meaning. Therefore, price is an Interval variable.

Many true Ratio variables are found in marketing research. “Number of Page Visits” and “Time on Page” are common Ratio variables. The good news is that almost all statistical techniques used in marketing research can be applied to both Interval and Ratio data. Mean, Median, Mode, Correlation, Standard Deviation and ANOVA are all equally valid with both types of data.

-

So what does this mean for you?

When you design your experiments, think about the type of variables you will be collecting data for. Interval and Ratio variables allow the most flexibility in statistical analysis, so whenever possible try to use them rather than Ordinal or Nominal data. A survey question could ask “which of the following tasks have you undertaken in the last 24 hours?” which produces a multiple choice, Nominal, answer.

It could also ask, “Please rank these tasks from most to least recently undertaken,” which produces Ordinal data and allows some additional analysis.

Finally, the survey could ask, “At what time and date did you last undertake these tasks?” producing concrete Interval data which will allow you to compare between respondents and run in depth statistical functions.

In the design phase of your marketing tests, think about the statistical data you would like to produce, and what variable types are required to calculate the results you need in order to answer your research questions. When you enter your data into a statistical software package, be careful to designate the correct variable type in the software so that the program can prevent you from dividing Florida by Minnesota.

-

Related Resources:

Marketing Optimization: You can’t find the true answer without the right question

Research Update: The state of email marketing testing and optimization

Marketing Optimization: What your peers learned this year about Adwords, the inbox, and telling the truth

Evidence-based Marketing: How your peers protect against bad marketing data

Optimizing your webpages and marketing campaigns is a daunting task for any marketer. We need all the help we can get. Free help is even better (as long as it’s actually, well, helpful). So to assist our audience of marketers with daily optimization tasks, our researchers have created 11 free marketing tools over the years, and I compiled the below list.

You can think of this as our idea of Halloween candy.

Please try them out and let us know how they work for you in the comments.

Value Proposition Worksheet

Getting to the bottom of your company’s value proposition is one of the most difficult tasks marketers face when working on messaging. This worksheet should help guide you through the process of effectively communicating your value proposition.

Compounding Effect Spreadsheet

When testing multiple steps in a conversion path, it’s important to track where you started and how optimizing each step helped improve your bottom line. The compounding effect spreadsheet gives you a place to do this so you can easily see the impact of incremental changes on your profit margins.

Price Testing Analysis

It can be tricky to test the ideal price of your product. Testing different price points is often counter-intuitive. You might charge less for a product and actually generate more revenue for example. This tool was made to help you easily track your price testing campaigns.

Subscription Site Metrics Tool

Subscription sites with recurring charges over time can’t simply expect to measure profit based on a single conversion at sign-up. This tool can help subscription and membership sites measure the profitability of PPC campaigns over the lifetime value of the subscriber.

Max Bid Analysis Spreadsheet

There are times when you just need to quickly know how much you can bid per click on a particular ad group to remain profitable. The Max Bid Analysis Spreadsheet helps you do just that.

Customer Service Analysis Tool

There are a lot of factors and small expenses to take into account when calculating the ROI of your customer service representatives. This tool helps simplify that calculation by guiding you through the process of plugging in those expenses and then calculating the ROI for you.

A/B Testing Template Doc

If you’re just starting out with A/B testing, you need to condense the basic process into a single document. The A/B Testing Template provided will help guide you in the right direction if you’re worried about missing something in the process.

Competitive Analysis Spreadsheet

Analyzing who your competitors are and how they are either beating you or falling behind can give you huge insight into the minds of your customers. Many times, by simply doing a formal competitive analysis, our researchers have actually found entirely new markets to compete in with the same product. This spreadsheet will help you keep track of your research into the competition.

Sample Size Calculator

To determine the proper sample size you’ll need in a test is the first step in getting a valid result. This stripped down version of the sample size calculator we use in our Test Protocols should help you get a feel for the amount of traffic you need for a test.

Metrics Dashboard Tool

Keeping track of key metrics while you are testing can be overwhelming. This simple spreadsheet tool will help you navigate and monitor the metrics you need to track.

Optimization Scenario Tool

Before you start testing, you need to pitch the idea to your boss for buy-in. Of course, unless you can show him or her projected numbers for how much you can bring in, it’s going to be a tough sell. Enter the Optimization Scenario Tool. This tool makes it simple to plug in your predicted data and get a projected impact on the bottom line.

Of course, it’s one thing to have the tools, it’s another thing to know how to use them. Sign up for the free MarketingExperiments Journal to be notified of our latest research with real-world results on what does and doesn’t work in online marketing.

Related Resources:

Landing Page Optimization: 36 articles and resources to help you complete your next LPO project

Landing Page Optimization: 5 questions every marketer should ask before choosing a testing tool

PPC Marketing: A look at analytic and monitoring tools

This week at the San Francisco leg of MarketingSherpa’s B2B Summit 2011, Brian Carroll, Executive Director of Applied Research, MECLABS, and Nicolette Dease, Program Manager, MECLABS Leads Group, provided tactical training on optimizing lead generation.

Part of this presentation was a case study on finding the most efficient list source based on a test that looked at several different lead sources.

The objective of the test was to determine if higher-cost/higher-quality data can drive down overall cost-per-lead, and the primary research question was, “Which campaign data source will drive the most efficient value?”

The test design looked at six list segments with 300 accounts and 80 hours of calling per segment. Let’s first look at how much each lead costs for discovery:

• Multi-source 1, validated by phone based on role – record cost $24
• Multi-source 2, validated by phone based on title – record cost $14.50
• Multi-source 3, validated by phone – record cost $6
• Multi-source 4, validated by email – record cost $3
• User-generated, validated by business cards – record cost $1
• Single-source, no validation – record cost $0.49

Results of the test

After discovery, every lead (even leads not validated by phone in discovery) in each segment needed the human touch – a phone call to validate the lead. This table looks at the validation calls for each lead segment:

-

Click to enlarge

-

As you can see – the highlighted green boxes illustrate a dramatic difference in list quality – the segments with the higher up-front record costs required many more calls to disqualify and fewer calls to find sales-ready leads. This tells us the higher cost lists are also higher-quality lists.

Nicolette says, “Not only are you paying for the data, but you are also paying someone to find out that lead is no longer with the company.”

So let’s take a look at how the numbers add up when you consider the cost of the human touch. This table looks at the cost-per-lead for each segment:

-

Click to enlarge

-

Looking at these results, Nicolette points out that the role-based discovery (called “multi-source 1 segment” in the bulleted list of lead lists tested above) actually saved money over every other segment, and was dramatically less expensive than the user-generated segment once you figure in the human resources that must be devoted to that list.

List efficiency

“Three to five percent of people change at least one thing on their business card each year,” Brian said. Once you start adding it up, if the list is based on data a few years old, up to half of your list could have incorrect data. With that in mind, those “cheap” lists can become really expensive once you factor in all the time spent validating the leads and uncovering the leads that are stale in cheaper lists.

This table provides the big picture overall results of the test:

-

-Click to enlarge

-

Nicolette provides the three key takeaways from this cost-per-lead test:

“Cheap” data in terms of record cost is actually expensive in the long run

• The less you spend on data, the more you will spend on teleprospecting
• View list acquisition as strategic, not simply transactional

These are Nicolette’s learnings. I’d challenge you to start your own lead optimization and testing process to discover what works best for your company.

Related Resources

B2B Summit 2011: 5 takeaways on social media, lead generation, building a customer-centric approach, and more

Have a minute? Brian Carroll reveals how sales teams pay dearly for cheap data

B2B Lead Optimization: Why cheap leads can be so expensive

B2B Lead Generation: Four experts’ advice on generating higher-level leads

Lead Generation: A closer look at a B2B company’s cost-per-lead and prospect generation

-

Ahhhhhh, Fall. There’s a chill in the air. Football is back. And, if you’re a marketer, it’s time once again for MarketingSherpa’s annual B2B Summit, currently happening in Boston, and soon to invade San Francisco.

We’re live onsite, and Dr. Flint McGlaughlin, Managing Director and CEO, MECLABS, opened the proceedings this morning by teaching this enthusiastic audience that most marketers can’t even define “value proposition,” much less tell you their own. Dr. McGlaughlin was out to change this, by helping the audience define the very questions necessary to establish a strong value prop, and differentiate them from competitors.

The fourth step of Dr. McGlaughlin’s six-step process involves a strategic checkpoint in which you measure the potential force of your value proposition(s).

As Dr. McGlaughlin has taught before, “clarity trumps persuasion.” This applies particularly well to value proposition, as we learned value propositions are discovered, not determined. They don’t come about from something that is predetermined; they grow out of what you are as a company or product. This sincerity needs to come through in a value proposition.

Dr. McGlaughlin showed how creating a value prop, measuring its force, and clearly communicating it to the world, let some marketers off the hook, in a sense. You don’t need to be an expert copywriter. You are simply communicating the unique value that your audience is looking for.

But how do you do it? The force of a value proposition can be measured by four essential elements of the offer:

• Appeal – How much do I desire this offer?
• Exclusivity – Where else can I get this offer?
• Credibility – Can I trust your claims?
• Clarity – What are you actually offering?

For the purposes of this presentation, we asked some attendees to submit what they feel their best value propositions are, and measured each for the above four elements.

-

Audience Submission #1:

-

Click to enlarge

-

Dr. McGlaughlin started off the submission review with a seemingly direct attempt at a value proposition that, in reality, doesn’t address two of the four necessary elements for measuring force. While terms like “help,” “passion” and “purpose” carry some appeal, the lack of specificity and context ultimately lowers that score.

Likewise, there is nothing in the above statement that indicates there is any inherent exclusivity. “Helping mid-life women … ” (a term I’m not entirely comfortable with, to be honest; but we’ll save that for another blog post) –  “… find their passion and purpose” could apply to any number of products, services or outlets, which also affects the clarity score.

At no point do I feel comfortable that this particular company is offering something I cannot find anywhere else, prompting me to conduct more research … likely at another website.

-

Audience Submission #2:

-

Click to enlarge

-

The second submission, as you can see, scored considerably higher in most categories. There is a noticeable improvement in appeal from the first submission, as this value proposition addresses the company’s products from a number of angles – “cute,” “soft,” “durable,” “sun-protective,” etc. This sequence of adjectives moves logically, from subjective to objective, ending with a descriptor that would likely appeal to someone searching for such specialized clothing items.

This submission also scored somewhat well on credibility and clarity, as the statement is fairly clear in expressing the company’s value. However, while the descriptors could interest a particular type of buyer, it is nowhere near specific enough to seal the proverbial deal. How many types of sun-protected items do they sell? Do they cater to children and adults? Is there proof of the products’ effectiveness?

These questions seem minor, considering the answers are likely found on a company homepage. But this level of vagueness – and requiring the audience to dig deeper to find value – could cost the company a considerable amount of business.

-

Audience Submission #3:

-

Click to enlarge

-

The final audience submission proved to be relatively strong in some areas, while deceptively weak in others. The submission’s appeal comes from the inclusion of “up-to-date” and “highly actionable” – terms which speak well to marketers, and play well with the company’s claim of exclusivity as the only provider of these reports.

However, once the excitement of this bold statement wears off, this value proposition does nothing to reinforce its claims. There is no proof offered to support the company’s claim that this is the only way to read this research, nor is there any evidence that this company is known for exclusive products.

Likewise, how does this proposition support the idea that these visuals and case studies are actionable? Do they contain exclusive, relevant tips and how-to information? It’s unclear of what separates this product from any other in the category. As such, this lack of credibility ultimately exposes a lack of clarity as well.

I could go on for another few pages, but let me get back to the sessions at B2B Summit 2011. There’s plenty more information and detail to be had by following the conversation at #B2Bsummit, and by attending the second of our two B2B Summits in San Francisco.

-

Related Resources:

MarketingSherpa B2B Summit 2011 in San Francisco

Value Proposition: A free worksheet to help you win arguments in any meeting

Do You Have the Right Value Proposition? – Web Clinic replay

Online Reviews: How to start discovering your value proposition with Yelp

Value Proposition: A free worksheet to help you win arguments in any meeting

Value Prop: Is there true value in your marketing proposition?

-

If you’ve been reading this blog for a while, then, chances are, you’re probably testing. That’s good …

… at least some of the time.

Marketers who aren’t testing may actually be better off than the ones that are

If you’re not careful, you could be running tests that tell you one thing when, in fact, the situation is completely different. You could be making critical decisions based on bad data. And these are the worst decisions you could make, because you’ve got the data to confirm that you’re right, when you’re actually doing things incorrectly.

This is why we were so surprised when MarketingSherpa’s Landing Page Optimization Benchmark Report came out with the following chart in it:

-

Click to enlarge

-

At least 40% of marketers who test don’t even account for statistical significance.

This means at least 40% of marketers who test are worse off than the ones that don’t test at all. Because, if you make business decisions based on unreliable data, not only could you make a bad choice, but you’ll have made that choice with the confidence of numbers.  The surety of the data would have led you to this bad decision.

So how do we make sure our tests are telling the truth?

In a nutshell, we need to make sure they are valid. To do so, we must be able to identify and guard against four basic validity threats:

1. Sample distortion effect
2. History effect
3. Instrumentation effect
4. Selection effect

How do you identify and guard against these validity threats?

Well, here are two resources that should help you in your testing to guard against these four validity threats.

The first resource is to help you with the first validity threat … sample distortion effect. It’s a blog post to help you figure out when you have a large enough sample size for statistically significant test results. Daniel Burstein wrote it a while ago, and it was especially eye-opening . Here’s a link to the sample size blog post.

The second resource is a Web clinic replay titled Bad Data: The 3 validity threats that make your tests look conclusive (when they are deeply flawed) – educational funding provided by HubSpot. In it, Dr. Flint McGlaughlin walked us through the lesser known validity threats, the last three in my list above.

Hopefully these resources are helpful. And if you use other resources to guard against validity threats, feel free to share them in the comments section.

Related Resources:

Bad Data: The 3 validity threats that make your tests look conclusive (when they are deeply flawed) – Web clinic replay

Marketing Optimization: How to determine the proper sample size

Online Marketing Tests: How could you be so sure?

Email Optimization and Testing: 800,000 emails sent, 1 major lesson learned

-