David Kirkpatrick

Email Marketing: Testing subject lines

At the end of March, HubSpot, the marketing software company, held a webinar on “The Science of Timing.” The event drew almost 25,000 registrants and ended up with very close to 10,000 attendees. Driving this success was a fast-paced and extensive email campaign sent to different segments of HubSpot’s list, prospects and new leads.

“The Science of Timing” was hosted by Dan Zarrella, HubSpot’s Social Media Scientist, and was built on a previous series of webinars over the last two years covering topics such as “The Science of Twitter,” “The Science of Blogging,” “The Science of Facebook,” and more. “The Science of Timing” covered when to do these social activities — when to tweet when seeking a retweet, when to publish new blog posts, when to update a Facebook status.

Subject line testing

HubSpot regularly conducts testing and optimization, and for this send it tested the subject line.

Eric Vreeland, Inbound Marketing Manager, HubSpot, explains the reason for this test, “Our goal is to generate the most amount of leads possible. Because our email list is so large, a 0.1% increase in open or clickthrough can result in a large number of form conversions and leads. We opted to test subject lines in order to maximize the total conversions we got for all our promotional efforts.”

To create a mailing list for the test, HubSpot randomly pulled about 50,000 addresses from its total list of around 500,000. This subgroup was randomly broken into five groups, and each group was sent mail with the exact same copy:

Click to enlarge

And all five groups were sent to the landing page after clicking through:

Click to enlarge

All five sends went out simultaneously at 10:00 a.m. The total list size was 54,987.

Results

Group 1

  • Subject line: [New Webinar] What time should you blog, tweet, and email?
  • Total delivered: 11,037
  • Total opens: 1,534
  • Total clicks: 245
  • Open rate: 13.90%
  • Clickthrough: 2.22%

Group 2

  • Subject line: [New Data] The Science of Timing. When you should tweet, blog, email, and more
  • Total delivered: 10,996
  • Total opens: 1,562
  • Total clicks: 247
  • Open rate: 14.21%
  • Clickthrough: 2.25%

Group 3

  • Subject line: [New Webinar] The best times to do your online marketing
  • Total delivered: 10,924
  • Total opens: 1,506
  • Total clicks: 214
  • Open rate: 13.79%
  • Clickthrough: 1.96%

Group 4

  • Subject line: [New Webinar] Learn the science of timing your blog posts, tweets, emails and more
  • Total delivered: 11,102
  • Total opens: 1,488
  • Total clicks: 247
  • Open rate: 13.40%
  • Clickthrough: 2.22%

Group 5

  • Subject line: Learn the best days and times to blog, email, tweet and more
  • Total delivered: 10,928
  • Total opens: 1,414
  • Total clicks: 226
  • Open rate: 12.94%
  • Clickthrough: 2.07%

Analysis

We ran the numbers through the MECLABS Test Validation Tool (part of the MECLABS Test Protocol that our research analysts use with Research Partners). Since the tests weren’t performed in our own labs, we weren’t able to check for possible validity threats and had to assume the data was normally distributed. After making those assumptions, we found that the two best performing treatments showed a statistically significant difference for open rate from the worst-performing treatment at a 95% Confidence Level –  Group 2 and Group 1 outperformed Group 5.

Click to enlarge

Group 2 provided a 9.8% life and Group 1 provided a 7.4% lift over the control.

Why? Well, obviously this is only one test. To truly find the answer would require analyzing HubSpot’s entire testing-optimization cycle But if one were to speculate, these were the only two treatments with a solid promise (of data or a webinar) AND used one of the Five Ws as we call them in journalism (in this case, What and When) to entice the subscriber and let them know that opening this email will begin them on a path that ends with the answer to one of their questions.

That’s just my take. I would love to hear your analysis as well. And here’s what Vreeland had to say about the top performer …

That was the only subject line that opened with “[New Data],” and Vreeland thought that could explain the better result.

He says, “Dan (Zarrella) is known for his interesting analysis of a wide variety of data. It makes sense that those interested in one of Dan’s webinars would be someone who is also excited by the prospect of new data.”

Vreeland adds, “The purpose of the test was to determine the best performing subject line so we could use it on the remainder of our 500,000 email list.”

Each of the top two performing subject lines were used in later email sends during the campaign.

Related Resources

HubSpot

B2B Webinars: How HubSpot drew 25,000 sign-ups, almost 10,000 attendees, and more than 3,500 new leads

New to B2B Webinars? Learn 6 steps for creating an effective webinar strategy

Webinar Promotion that Delivers: Use Email, Social, Viral Referrals and Video to Boost Attendance, Drive Lead Gen

Optimization Summit 2011

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: Analytics & Testing, Internet Marketing Strategy Tags: , , ,



  1. April 15th, 2011 at 10:02 | #1

    Whilst there was a clear winner, the difference in performance was respectable but not huge.

    I would say the subject lines were just all too much the same. Very often bigger differences in performance need a totally different approach. I’ve found this can give 30%+ differences.

  2. David Kirkpatrick
    David Kirkpatrick
    April 15th, 2011 at 10:16 | #2

    Tim,

    Thanks for the comment. In this case HubSpot was looking for small, incremental increases in opens and clickthroughs. Because its list is huge, any increase creates a large number of leads.

    Here’s HubSpot’s Vreeland from the post, “Because our email list is so large, a 0.1% increase in open or clickthrough can result in a large number of form conversions and leads”

  3. April 18th, 2011 at 09:22 | #3

    Can you control for people like me who open any and all emails having to do with MECLABS and Hubspot?

  4. April 18th, 2011 at 11:13 | #4

    The testing aspect of subject lines is great, but I’d really like to see analysis of the types of offers that truly drive traffic. Are white papers more effective than webinars? How do informational incentives compare to finacial offers such as discounts and free samples? I would assume the latter would be more effective, but that may not be true in email behavior.

  5. April 18th, 2011 at 16:26 | #5

    Just wondering is #1 would have been higher had you included the “and more…” at the end of the subject line (which are on Groups 2, 4 and 5).

    Tim Watson made a point about the subject lines being too similar. But there’s still value in using a test. Haven’t you ever carefully crafted a subject line – only to have a client or other colleagues try to wordsmith it to death? It’s frustrating when you just innately know that the changes have less power. Running a test to show the winner is more cost-effective, saves face and wins friends!

  6. April 18th, 2011 at 16:36 | #6

    Amy,
    I posed your question to our Director of Sciences, Bob Kemper, and here’s what he had to say…

    Thanks for your question, Amy. You’ve raised an important element of what drives Email Open rates. Since your (gratifying) comment suggested that you’re already familiar with MECLABS material and principles, you may remember that the factors that affect Open rate include such items as Relevance (to a recipient’s innate and current motivations), attributes of the Subject-line copy such as Urgency and Specificity, etc. Your question touches on the element of Relationship. The probability that an email recipient will open a message is strongly influenced by her perception of the email sender. That’s a key reason that the highest Open-rates by far typically come from a business’ own House-list.

    So, instead of something to be ‘controlled’ for per-se, ‘relationship’ is one of the attributes you should use to characterize and manage your email lists and your sends, so you can craft and customize your messages and calls-to-action accordingly.

    Thanks again for your insightful question, Amy.

    All the best,
    Bob

  7. David Kirkpatrick
    David Kirkpatrick
    April 19th, 2011 at 14:53 | #7

    Kevin,

    Wouldn’t it be great if there were one right answer to those questions? The truth is, we each have to test to see what works best for our audience right here and right now. If you have a complex B2B sale, for example, and a very well-respected brand, information may be extremely valuable to that audience. However, if you have a B2B site selling a very straightforward, basic product — the discount may work better.

    We’re constantly trying to explore what really works for other marketers to give you ideas for your own tests. Here are two examples from the paid membership library at our sister site, MarketingSherpa, that you may find helpful …

    https://www.marketingsherpa.com/article.php?ident=31672

    “Webinar Promotion that Delivers: Use Email, Social, Viral Referrals and Video to Boost Attendance, Drive Lead Gen”

    http://www.marketingsherpa.com/content/?q=node/2677

    “How to Get Thousands More White Paper Readers & Webinar Attendees: Red Hat’s Initial Test Results”

  8. April 19th, 2011 at 19:37 | #8

    Very interesting article. Thanks for publishing. I agree with Tim Watson about the subject lines. They are too similar in format, structure, voice, curiosity level, hook, etc.

    I like to test completely different angles in my subject lines. It’s tough to find out what will capture someone’s attention and interest in today’s over-crowded in-boxes.

    Speaking of curiosity, I wonder how other, less obvious, variables might perform in a test for Hubspot beyond the simple subject line modifications.

    Is list degradation a factor for them? Does the average length of time a subscriber has been on their list affecting engagement?

    Is the time of day, 10am vs. 6pm, an factor for Hubspot’s to consider?

    As usual, there’s no universal rule for email. Sometimes it matters and other times not at all. And then it changes in 6 months. Testing rocks.

  9. April 20th, 2011 at 02:46 | #9

    David, how good is that to send emails unanimously to email addresses we have in database? Secondly, mails sent from Hubspot or any other do they land in spam box? If they do land don’t you think emails will reach to only small number. I am not sure If i have expressed my views correctly, but I am sure you must have got it.
    Thanks for this great analyzed report.

  10. David Kirkpatrick
    David Kirkpatrick
    April 21st, 2011 at 17:10 | #10

    Vinaykyaa,

    Thank you for the question. I’m not certain I follow what you are asking, but I believe it is, “does sending email to the entire list result in more mail getting caught in a spam filter?”

    If you have good data hygiene and keep your list updated by regularly responding to unsubscribe requests and checking for bounces, an email send to an entire list shouldn’t in and of itself lead to more email being trapped by spam filters. Now the subject line or some aspect of the mail content could certainly contribute to that issue.

    Data hygiene in the database is always important, and obviously the larger the list — like HubSpot’s — the bigger the challenge in keeping that email list up-to-date and clean.

  11. May 5th, 2011 at 05:29 | #11

    Great post about a good test! The devil is in the detail when it comes to people making microsecond decisions about whether to click to open and then to click through.

    Looking at timing across the globe is also fascinating.

  12. Michael
    May 5th, 2011 at 07:58 | #12

    Hi, I want to note that link 2 and 3 are not working in the related resourse section of this page.

    Best Regards,

    Michael

  13. May 5th, 2011 at 09:04 | #13

    Same here…would have liked to see more variation in the subject lines, in addition to their “looking for small, incremental increases in opens and clickthroughs. “. It would have given more credibility to their test.

  14. May 5th, 2011 at 09:10 | #14

    Interesting post.

    I think the reason Group 1 and Group 2 did the best is because of the use of the phrases “when should” and “what time should”. Those phrases are very direct and simplify an understanding of the new data being offered. Compared to “learn the science of timing” which could mean “when to” but could also mean “learn the process of discovering when to…” I bet HubSpot’s audience would prefer to just have the guideline to follow (as in ‘tweet between 4:30 and 5:30′) than a research process for which they have to do actual work.

  15. May 5th, 2011 at 10:32 | #15

    Fascinating stats. I’ve always found that [New Details] and other bracketed text at the beginning gave a marginally better performance, too.

  16. May 5th, 2011 at 11:40 | #16

    Hi Michael,

    Thanks for pointing that out. The links have now been fixed. :)

  17. March 11th, 2013 at 09:02 | #17

    I’ll try to use this as a model: “The Science of Timing. When you should tweet, blog, email, and more” not exactly that but similar, it’s now in my swipe file :) . First a statement. Then the question. Also if you have a very big list, say 40,000 a 1% increase in CTR is quite significant still.

  1. April 15th, 2011 at 15:59 | #1
  2. April 16th, 2011 at 08:55 | #2
  3. July 22nd, 2011 at 08:57 | #3