Email Marketing: Testing subject lines
At the end of March, HubSpot, the marketing software company, held a webinar on “The Science of Timing.” The event drew almost 25,000 registrants and ended up with very close to 10,000 attendees. Driving this success was a fast-paced and extensive email campaign sent to different segments of HubSpot’s list, prospects and new leads.
“The Science of Timing” was hosted by Dan Zarrella, HubSpot’s Social Media Scientist, and was built on a previous series of webinars over the last two years covering topics such as “The Science of Twitter,” “The Science of Blogging,” “The Science of Facebook,” and more. “The Science of Timing” covered when to do these social activities — when to tweet when seeking a retweet, when to publish new blog posts, when to update a Facebook status.
Subject line testing
HubSpot regularly conducts testing and optimization, and for this send it tested the subject line.
Eric Vreeland, Inbound Marketing Manager, HubSpot, explains the reason for this test, “Our goal is to generate the most amount of leads possible. Because our email list is so large, a 0.1% increase in open or clickthrough can result in a large number of form conversions and leads. We opted to test subject lines in order to maximize the total conversions we got for all our promotional efforts.”
To create a mailing list for the test, HubSpot randomly pulled about 50,000 addresses from its total list of around 500,000. This subgroup was randomly broken into five groups, and each group was sent mail with the exact same copy:
And all five groups were sent to the landing page after clicking through:
All five sends went out simultaneously at 10:00 a.m. The total list size was 54,987.
- Subject line: [New Webinar] What time should you blog, tweet, and email?
- Total delivered: 11,037
- Total opens: 1,534
- Total clicks: 245
- Open rate: 13.90%
- Clickthrough: 2.22%
- Subject line: [New Data] The Science of Timing. When you should tweet, blog, email, and more
- Total delivered: 10,996
- Total opens: 1,562
- Total clicks: 247
- Open rate: 14.21%
- Clickthrough: 2.25%
- Subject line: [New Webinar] The best times to do your online marketing
- Total delivered: 10,924
- Total opens: 1,506
- Total clicks: 214
- Open rate: 13.79%
- Clickthrough: 1.96%
- Subject line: [New Webinar] Learn the science of timing your blog posts, tweets, emails and more
- Total delivered: 11,102
- Total opens: 1,488
- Total clicks: 247
- Open rate: 13.40%
- Clickthrough: 2.22%
- Subject line: Learn the best days and times to blog, email, tweet and more
- Total delivered: 10,928
- Total opens: 1,414
- Total clicks: 226
- Open rate: 12.94%
- Clickthrough: 2.07%
We ran the numbers through the MECLABS Test Validation Tool (part of the MECLABS Test Protocol that our research analysts use with Research Partners). Since the tests weren’t performed in our own labs, we weren’t able to check for possible validity threats and had to assume the data was normally distributed. After making those assumptions, we found that the two best performing treatments showed a statistically significant difference for open rate from the worst-performing treatment at a 95% Confidence Level – Group 2 and Group 1 outperformed Group 5.
Group 2 provided a 9.8% life and Group 1 provided a 7.4% lift over the control.
Why? Well, obviously this is only one test. To truly find the answer would require analyzing HubSpot’s entire testing-optimization cycle But if one were to speculate, these were the only two treatments with a solid promise (of data or a webinar) AND used one of the Five Ws as we call them in journalism (in this case, What and When) to entice the subscriber and let them know that opening this email will begin them on a path that ends with the answer to one of their questions.
That’s just my take. I would love to hear your analysis as well. And here’s what Vreeland had to say about the top performer …
That was the only subject line that opened with “[New Data],” and Vreeland thought that could explain the better result.
He says, “Dan (Zarrella) is known for his interesting analysis of a wide variety of data. It makes sense that those interested in one of Dan’s webinars would be someone who is also excited by the prospect of new data.”
Vreeland adds, “The purpose of the test was to determine the best performing subject line so we could use it on the remainder of our 500,000 email list.”
Each of the top two performing subject lines were used in later email sends during the campaign.