Internet Marketing Journal -

FREE subscription to more than $20 million in marketing research

Enter your email below to receive the latest in marketing research and exclusives offers from MarketingExperiments.

create Lab ID
We value your privacy. View details on how to contact us.

You can also get our latest research delivered to you via our social media feeds below.


Follow us on Twitter Join our Optimization Group Visit the MarketingExperiments Blog
Listen to our MP3 Podcast Listen to our podcast on iTunes Watch videos on our YouTube Channel

Research Topics

Site Optimization

Email Optimization

Search Optimization

Marketing Optimization



"Your emails and website provide the common sense that we often leave at the door to the office."

David Jones
Senior Practice Leader
Kaiser Permanente

"I often change small things across many pages as MarketingExperiments gets me thinking about how to increase the straighforwardness of my pages for the visitor."

Amanda Schaner
Marketing Coordinator
Home Science Tools

"The live web clinics are very humbling. I think we're doing a pretty good job. But every time I attend, I realize we could do things much better for our clients. I consider MarketingExperiments my encyclopedia, and parrot the principles I learn every chance I get."

Troy O'Bryan
Co-Founder and Chief Response Officer
Response Capture, Inc


How do I know if my test data is valid?

Determining if a Data Sample is Statistically Valid When testing, the validity of the data is a function of the how much a difference there is between your results, and the sample size. In simple terms we could say that validity can be described as a function of the size of the data sample, and the variance between 2 or more sets of results.

There are other factors to validity, but this allows us to understand the factors that impact whether a data set is valid or not. There are obvious technical issues that impact test results when we are testing online, and often reporting can be crude and inaccurate.

Still, we can learn a lot from testing and it should be a strong part of any marketers daily practice. Testing is not the goal of marketing, rather a tactic that can be used to save money and improve results.

In a practical world of revenue targets, it can be difficult to always slow down long enough to stop and test and then carefully analyze your results. Understanding the validity of your data can help you to quickly make decisions and truly understand what a test is telling you.

Simply put, if you have a larger variance between two results, then you will need a smaller sample size to achieve a strong degree of confidence.

Imagine these are the results of a ficticious landing page optimization test:

Treatment Unique Visits Leads Conversion
Landing Page A 4,203 32 0.76%
Landing Page B 3,454 534 15.46%

In this particular example, the difference between the number of leads is significant. Using our intuition, we can see that Landing Page B outperformed Landing Page A. However the sample size for Landing Page A Leads is still relatively small, so there is a high amount of room for error caused from sampling. There are obviously very complex algorithms for calculating the statistical relevance of a given data sample.

Where you would like us to send you information on how to calculate validity...

Determining if a Data Sample is Statistically Valid