How do I know if my test data is valid?

When testing, the validity of the data is a function of the how much a difference there is between your results, and the sample size. In simple terms we could say that validity can be described as a function of the size of the data sample, and the variance between 2 or more sets of results.

There are other factors to validity, but this allows us to understand the factors that impact whether a data set is valid or not. There are obvious technical issues that impact test results when we are testing online, and often reporting can be crude and inaccurate.

Still, we can learn a lot from testing and it should be a strong part of any marketers daily practice. Testing is not the goal of marketing, rather a tactic that can be used to save money and improve results.

Simply put, if you have a larger variance between two results, then you will need a smaller sample size to achieve a strong degree of confidence.

Imagine these are the results of a ficticious landing page optimization test:

In this particular example, the difference between the number of leads is significant. Using our intuition, we can see that Landing Page B outperformed Landing Page A. However the sample size for Landing Page A Leads is still relatively small, so there is a high amount of room for error caused from sampling. There are obviously very complex algorithms for calculating the statistical relevance of a given data sample.

Where you would like us to send you information on how to calculate validity...