Internet Marketing Journal -

FREE subscription to more than $20 million in marketing research

Enter your email below to receive the latest in marketing research and exclusives offers from MarketingExperiments.

create Lab ID
We value your privacy. View details on how to contact us.

You can also get our latest research delivered to you via our social media feeds below.


Follow us on Twitter Join our Optimization Group Visit the MarketingExperiments Blog
Listen to our MP3 Podcast Listen to our podcast on iTunes Watch videos on our YouTube Channel

Research Topics

Site Optimization

Email Optimization

Search Optimization

Marketing Optimization



"Your emails and website provide the common sense that we often leave at the door to the office."

David Jones
Senior Practice Leader
Kaiser Permanente

"I often change small things across many pages as MarketingExperiments gets me thinking about how to increase the straighforwardness of my pages for the visitor."

Amanda Schaner
Marketing Coordinator
Home Science Tools

"The live web clinics are very humbling. I think we're doing a pretty good job. But every time I attend, I realize we could do things much better for our clients. I consider MarketingExperiments my encyclopedia, and parrot the principles I learn every chance I get."

Troy O'Bryan
Co-Founder and Chief Response Officer
Response Capture, Inc

Bad Data
Wednesday, 21 September 2011

Read Read     Watch Watch     Listen Listen

Topic: Bad Data: The 3 validity threats that make your tests look conclusive (when they are deeply flawed)

If you’ve tuned into past MarketingExperiments Web clinics, you’ve seen some pretty impressive gains:

  • 141% increase in conversion
  • 12% more revenue across an entire ecommerce website
  • $4.9 million in additional sales pipeline growth (in only 8 months)

And we’ve shared transferable principles to help you achieve similar gains yourself.

But for this Web clinic, the MarketingExperiments team took a different approach and showed you not just what we did to get these types of results, but gave you an inside look at how we did it.

We didn’t get these results by only hiring marketing geniuses. We did it by developing a process focused on determining “what really works.”

Doing that takes much more than just plopping two landing pages into a platform with a splitter. You must know the results you’re seeing truly represent what is going on in the real world.

According to a survey in MarketingSherpa’s 2011 Landing Page Optimization Benchmark Report, at least 40% of marketers don’t even calculate statistical significance when they test.

This means at least 40% of marketers are probably making confident business and marketing decisions based on incomplete data that may be the result of random chance, not a true representation of buyer intent.

But beyond the math, there are three other validity threats that can make your tests look conclusive when they really aren’t.

So how can you be sure you’re making sound marketing decisions based on your tests?

In this Web clinic, Dr. Flint McGlaughlin explained the three validity threats that many marketers overlook when running tests, triggers to identify if they are threatening your tests, and how to mitigate the risks to the data you collect. Here are what some of the audience members had to say about the Web clinic:

The quality of the information and the presenter's style – Superb!! –Simon

Presenter's superb diction – we could understand every thought in spite of below par sound quality.–Cameron

I found this webinar to be very valuable.  More often than not, these types of sessions are high-level and have very little substance.  This webinar was a definite exception to that rule!  Thank you for all the great info.–Trace

Download a full transcript of this Web clinic:

Optimization Researched Transcript

View the clinic replay, or listen to the audio recording (mp3), to learn from our researchers as they discuss how to improve the data you collect from multivariate and split testing.

Webinar Video 2011-09-14
Presentation will open in new window

Listen to an audio replay of the presentation:

iTuns Subscribe to our podcast on iTunes

Audio Subscribe to our mp3 podcast

Watch the next Web Clinic in real-time

Enter your email below for invites to our Web Clinics featuring:

  • First access* invites to Web clinics
  • Latest discoveries in online testing and optimization research
  • Interactive Q&A with the MECLABS research team
  • Live analysis of audience-submitted landing pages and wireframes

You will also receive updates on the latest in marketing research and exclusives offers from MarketingExperiments.

Subscribe for FREE

* Seating is limited to 1,000 attendees and fills quickly

We value your privacy. View details on how to contact us.

About This Brief


Dr. Flint McGlaughlin
Austin McCraw
Phillip Porter

Paul Cheney
Austin McCraw

Daniel Burstein
Selena Blue
Brad Bortone

Technical Production
Luke Thorpe
Steven Beger
Jessica McGraw

Additional Contributors
Beth Caudell

Educational funding provided by



::Top Of Page::