To test emails, you just send out two versions of the same email. The one with the most opens is the best one, right?
“There are way too many validity threats that can affect outcomes,” explained Matthew Hertzman, Senior Research Manager, MECLABS.
A validity threat is anything that can cause researchers to draw a wrong conclusion. Conducting marketing tests without taking them into account can easily result in costly marketing mistakes.
In fact, it’s far more dangerous than not testing at all.
“Those who neglect to test know the risk they’re taking and market their changes cautiously and with healthy trepidation,” explains Flint McGlaughlin, Managing Director and CEO, MECLABS, in his Online Testing Course. “Those who conduct invalid tests are blind to the risk they take and make their changes boldly and with an unhealthy sense of confidence.”
These are the validity threats that are most likely to impact marketing tests:
- Instrumentation effects — The effect on a test variable caused by an external variable, which is associated with a change in the measurement instrument. In essence, how your software platform can skew results.
- An example: 10,000 emails don’t get delivered because of a server malfunction.
- History effects — The effect on a test variable made by an extraneous variable associated with the passing of time. In essence, how an event can affect tests outcomes.
- An example: There’s unexpected publicity around the product at the exact time you’re running the test.
- Selection effects — An effect on a test variable by extraneous variables associated with the different types of subjects not being evenly distributed between treatments. In essence, there’s a fresh source of traffic that skews results.
- An example: Another division runs a pay-per-click ad that directs traffic to your email’s landing page at the same time you’re running your test.
- Sampling distortion effects — Failure to collect a sufficient sample size. Not enough people have participated in the test to provide a valid result. In essence, the more data you collect, the better.
- An example: Determining that a test is valid based on 100 responses when you have a list with 100,000 contacts.