Daniel Burstein

Subject Line Test: 125% more unique clickthroughs

The results are in. A few weeks ago, we asked the MarketingExperiments blog community and the Copyblogger community for help on a subject line test. Today we’ll look at the process we went through, the results, and what you can learn from it all.

 

Background

Through a blog post on the MarketingExperiments blog and a blog post on Copyblogger, we asked for your subject line test ideas for an email that promoted Optimization Summit 2012 in Denver. We received 409 entries through the MarketingExperiments blog and 492 entries through Copyblogger.

Then came the culling.

Nathan Thompson, Senior Manager, Research and Strategy, MECLABS Conversion Group, was the lead judge for the entries on our blog, and Sonia Simone, CMO, Copyblogger Media, LLC, led her team’s effort (Sonia wrote about their process today on Copyblogger).

“Some were clever, some were text-book marketing, and others were, well … weird,” Nathan said.

“Our process for cutting down the list involved removing those that were factually incorrect first as we didn’t want to incite a riot among our readers once they opened the email and realized we had tricked them. Then we removed those that were too long, too confusing or unclear.”

Once the list was narrowed down to the acceptable subject lines, Nathan and the MECLABS Conversion Group team began grouping them into categories based on what the subject lines were attempting to convey:

  • Incentive
  • Curiosity
  • Value exchange
  • Newsworthiness
  • Etc.

“Then we voted on the subject lines we felt were the best representation of each of these categories,” Nathan said.

 

The 6 Treatments

The team’s testing strategy ended up focusing on how the incentive held up compared to a more value-oriented subject line without the incentive. Here are the subject lines they selected, along with the categories they were testing for with each subject line:

1. [Optimization Summit] 3 Days to a Better Website ($300 Off Coupon Inside!) – (Value Exchange, Incentive)

2. Learn 3 tips that made 10,000 landing pages extremely successful – (Value Exchange, No Incentive)

3. Optimization Summit 2012 – Speakers List Now Up! + Save $300 Today – (Newsworthy, Incentive)

Three subject lines from the Copyblogger audience were also tested. Here are the subject lines Sonia and her team selected, along with the categories they were testing:

4. Quarterbacks aren’t the only changes being tested in Denver. – (Curiosity)

5. A scientific way to increase your conversions – (Benefits)

6. Do your landing pages pass this test? – (Fear)

Here is the body of the tested email. The body of the email stayed the same for each test treatment; the only variable that changed was the subject line.

 

Click to enlarge

 

The six treatments were then sent to subscribers of the MarketingExperiments email list as well as MarketingSherpa email lists. The lists were de-duped and split into six equal segments, and each subscriber received only one treatment.

 

Primary Research Question

The primary research question was, “Which motivation represented in the subject line will result in the highest clickthrough (Delivered/Clicks) rate?”

The KPI was unique clickthrough rate of overall emails delivered. We didn’t choose open rate as the KPI, because we wanted to avoid subject lines that were click bait but didn’t tie into the motivations of customers who could benefit from attending Optimization Summit 2012. Click bait can increase unsubscribes and decrease the result we really care about: unique clickthroughs. People don’t like to be tricked.

 

The Results

Drumroll, please …

And the winner is — “Do your landing pages pass this test?” written by Christine Parizo.

As you can see in the below screenshot from the MECLABS Test Protocol, Christine’s subject line generated a 125.9% higher unique clickthrough rate than the worst-performing subject line of the test.

 

Click to enlarge

 

 

Congratulations, Christine! You won a free ticket to Optimization Summit 2012, plus a free MarketingExperiments Landing Page Optimization Online Course.

Let’s put the magnitude of this into perspective. By simply changing a few words in one line, Christine helped drive more than twice as many visitors to our landing page. Talk about return on investment. Christine, I look forward to high-fiving you in person in Denver at Optimization Summit.

But wait, let’s take a look at the results from a different angle …

 

Click to enlarge

 

 

While Christine’s subject line was significantly different than most of the other subject lines, there was one subject line which it was not significantly different from at a 95% Level of Confidence – “Learn 3 tips that made 10,000 landing pages extremely successful.”

(Note: If you’re not familiar with statistical significant, you can find more information at Online Marketing Tests: How do you know you’re really learning anything? and Online Marketing Tests: How could you be so sure?)

That subject line was written by PaulM. We had only planned on giving one prize, but since there was no significant difference between the top two subject lines, we decided both copywriters should be rewarded. So Paul also wins a free ticket to Optimization Summit 2012 and a seat in our on-demand LPO course.

Plus … all six copywriters whose subject lines were selected to be tested will receive a 90-day free membership to Third Tribe Marketing, Copyblogger’s private community of online businesspeople.

“It should also be noted that the two top performing subject lines in clickthrough were also the top performing in terms of open rate,” Nathan said.

 

What can you learn from this test?

Of course, here’s the most important part of this blog post — how you can use this information to improve your own marketing efforts?

At a high level, the whole reason we ran this test was to show you just how powerful testing and optimization can be. As I said, the winning subject line drove more than twice as many visitors to the landing page. This isn’t some massive marketing automation or e-commerce cart redesign effort, it is simply changing a few words.

Second, test don’t guess. Some of the tops minds in marketing — like Sonia Simone and Nathan Thompson — were involved in this test, and while I didn’t put anyone on the spot, in working with them on setting up this test, I can tell you there was no clear winner in our minds.

Also, be open. Turn to those outside your normal cadre when crafting tests. That is the beauty of testing: you can always have one or two safe ones, plus one that really challenges the model. And when that outlier wins, it can open your eyes to a whole new approach.

For us, we had help from the savvy marketing professionals who read the MarketingExperiments blog and Copyblogger. While you might not have that access, you can still ask your customers for ideas through your content marketing, social media channels or surveys.

Also, you can ask an expanded peer group. Reach outside of the marketing department and ask for opinions from Sales, Customer Service, Product Development, IT, and even front-line, customer-facing employees. Beyond your company, ask peers in other companies, your husband or wife, the people you play basketball or squash with … anyone who can expand your horizons.

 

As for interpreting the results of this test itself …

The two top performers both used the specific phrase “landing pages.” The people that are interested in landing pages are interested in Optimization Summit because they understand the power of landing page optimization.

Keywords are important for more than just SEO. Google has trained us to conjure up short phrases when we want information. People think in keywords now, any time they’re looking for info, not just when they’re staring at a blank search box. When your recipients scan the subject lines in their inbox, what keywords can you provide to tap into their interests and motivation?

“The results, interesting as always, tend to indicate that incentive was not as big a player as we had guessed it would be,” Nathan said. “In fact, the two top performing subject lines didn’t mention the discount incentive at all. And even the subject line that paired the incentive with a bit of newsworthiness (Speaker’s List) was unable to outperform ‘Learn 3 tips that made 10,000 landing pages extremely successful.’”

Why wasn’t incentive effective in the subject lines? You need to sell the value of a product before anyone will find that incentive valuable. “$300 off” is not valuable, especially in an inbox cluttered with other incentive-laden subject lines. “$300 off” something your audience wants and values, now that is worthy of a clickthrough. I learned that we seriously underestimated how well our audience understands the value of Optimization Summit 2012 … or even knows of it at all.

This is a common problem many marketers make. We’ve been sleeping, eating and dreaming Optimization Summit as we have been putting it together, so it’s all too easy to overestimate how much our audience has learned about this event.

This is a solidly specific headline – the readers know that the message will be about landing pages,” Sonia said, in analyzing the winning subject line. “Passing (or failing) tests is a common anxiety, and in a competitive economy, no one wants to have the landing pages that fail the test.”

“The word this is important (in fact, there was another entry for ‘Do your landing pages pass the test?’). That slight added element of specificity enhances reader curiosity to generate those clickthroughs.”

 

Related Resources:

April Book and Online Training Sale — Save 33% off more than 38 books and online training courses

Copywriting Contest: Write the best-performing subject line and win

Marketing Research: Top email elements to test

Email Marketing: 17.36% higher average clickthrough rate in 7 personalized subject line tests

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: Email Marketing Tags: , ,



  1. Diana Huang
    April 18th, 2012 at 10:48 | #1

    Congrats Christine & Paul!

    I’m a little surprised about “Do your landing pages pass this test?” as it seems a bit disconnected with the email content. Someone who opens the email probably expect to find a test they can apply to their landing pages to judge the quality of these pages, but that’s why we test right? My feeling about the theme of this headline is that it’s less about fear and more about curiosity. It conjures the question of what is this test I can use to judge the quality of my pages. It’s a common question but pretty hard to answer in most landing page professional’s mind so the promise of a simple test would seem enticing. That’s why I was concerned about the potential disconnect once the email is opened.

    DanielB, would it be possible to share the email open rates for each of the subject line as well? Very curious to see how the consideration rate (opens/clicks) plays out for each headline! For example, does “Do your landing pages pass this test?” have a much higher open rate and maybe slightly lower click thru rate (lower consideration rate)?

    Again, congrats guys, I love surprises in testing! That’s when we get to learn something new! :)

    • April 18th, 2012 at 13:57 | #2

      Thanks for the comments Diana.

      And excellent point about the possible disconnect of the promise. Friday’s MarketingExperiments blog post is actually about a test where the winning subject line (in terms of unique open rate) did not have the highest unique clickthrough rate, due to that disconnect once the email is opened.

      These unique open rates are listed in the same order as the first chart…

      13.41%
      14.85%
      16.67%
      12.53%
      15.61%
      13.43%

  2. April 18th, 2012 at 11:20 | #3

    Firstly I would like to say BIG thank you to all the guys and girls at Marketing Experiments, your research has helped us to optimize our emails and landing pages in a way we never thought possible.

    Regarding this subject line test, I was running a similar test on email (Prospects database, members who registered but still have not ordered)

    A – Top Spring Offers – up to 60% off best sellers – sent to 623k
    B – Top Spring Offers – create something special with your photos 623k

    Everyone in the marketing and trading department thought it was a no brainer, subject line A will win by miles. However,…

    - The A test subject only got a 2% increase in open-rates
    - The A test got a -5% in clicks to open and – 2% on overall click-rate (clicks to emails sent)
    - The A test had a 17% increase in opt-out click-rate (opt-out clicks to opens)

  3. Diana Huang
    April 18th, 2012 at 14:37 | #4

    @Daniel Burstein

    Thanks Daniel!

    Backed into the numbers and calculated the consideration rate and the performance of “…pass this test?” turned out to be more amazing than I thought. It had a 24% lift on open rate AND an 84% lift on consideration (click/open) rate. Wow, I really wouldn’t have thought that…cool!

    consideration rate Pct Lift vs. Control
    2.4%
    3.4% 44%
    4.4% 84%
    3.0% 24%
    4.2% 77%
    3.0% 25%
    * all lifts above are >95% significance

  4. PaulM
    April 24th, 2012 at 02:17 | #5

    Thanks Daniel, great to see my subject line made the Top 2 and readers responding to good old fashioned “specificity”.

    As we encourage our clients @stage4solutions.com; be specific, promote/sell the value [not the feature (in this case, Optimization summit)] and provide proof point to validate claim…which holds true across all content that buyers engage with.

  5. April 25th, 2012 at 05:34 | #6

    Interesting. Customers bahaviour and interests differ a lot today when most people get tons of email, advertisement etcetera every day.
    I thought things like distribution time/date, trustworthiness and realtions with the company sending the information would have a bigger impact.
    I will definitely start asking friends/ coworkers and other people what they think before sending out my next marketing email and newsletters.
    BR

  6. Cheryl Kissell
    May 11th, 2012 at 13:12 | #7

    @Daniel Burstein

    I appreciate seeing the open rate…but you said it reflects the 1st chart, I’m reading this page…where is the chart?
    13.41%
    14.85%
    16.67%
    12.53%
    15.61%
    13.43%

    I saw this:
    1. [Optimization Summit] 3 Days to a Better Website ($300 Off Coupon Inside!) – (Value Exchange, Incentive)

    2. Learn 3 tips that made 10,000 landing pages extremely successful – (Value Exchange, No Incentive)

    3. Optimization Summit 2012 – Speakers List Now Up! + Save $300 Today – (Newsworthy, Incentive)

    Three subject lines from the Copyblogger audience were also tested. Here are the subject lines Sonia and her team selected, along with the categories they were testing:

    4. Quarterbacks aren’t the only changes being tested in Denver. – (Curiosity)

    5. A scientific way to increase your conversions – (Benefits)

    6. Do your landing pages pass this test? – (Fear)

    So it looks like 3 and 5 got the best open rate.

  7. Jon
    May 22nd, 2012 at 22:12 | #8

    I couldn’t help but notice that the actual email itself seems to break a few (ok… a LOT) of the tips discussed in one of the ME seminars on crafting an engaging email message.

    No headline. Takes more than 30 seconds to read. Looks like a web page. Multiple (6) calls to action. Different calls to action. No value exchange in the calls for action. No message/benefit in the video screenshot. And very busy in general as far as sight flow.

    Maybe it’s just me but I’m noticing some major conflation issues!

    Just giving you a hard time… Love the site.

  8. Jay
    June 1st, 2012 at 08:17 | #9

    Have you measured the conversion rate yet for the different subject lines?

  9. June 3rd, 2012 at 13:04 | #10

    @Jay

    Great question Jay. No, we have not measured the conversion rate for the different subject lines.

    Thanks.

  1. April 18th, 2012 at 09:43 | #1
  2. April 20th, 2012 at 03:02 | #2
  3. May 16th, 2012 at 03:03 | #3
  4. May 18th, 2012 at 10:45 | #4
  5. June 1st, 2012 at 03:03 | #5
  6. March 20th, 2013 at 19:34 | #6
  7. June 19th, 2013 at 10:25 | #7
  8. October 30th, 2013 at 06:12 | #8
  9. December 3rd, 2013 at 08:00 | #9
  10. February 19th, 2014 at 22:05 | #10
  11. April 15th, 2014 at 11:37 | #11