Daniel Burstein

Email Testing: More specific subject line improves open rate by more than 35%

“She was here on earth to make sense of its wild enchantment and to call each thing by its right name.” – Boris Pasternak (Doctor Zhivago)

Sometimes, as marketers, this is one of our biggest challenges. We must make sense of the “wild enchantment” inherent in our audience so we can call each offering we have by its right name. After all, the way we think about our products can be vastly different from the way our audience thinks about them. This is why specific words matter, and for more than just SEO.

Let’s take a look at a recent test conducted by MarketingSherpa (sister company to MarketingExperiments), to determine which words best tap into the audience’s motivations.


MarketingSherpa conducts several series of workshops across the country. Senior Marketing Manager Justin Bridegan launched a geotargeted microtest in the Denver area for the July 19th stop of the B2B Marketing FUEL Advanced Practices Workshop. The control used the traditional “B2B” wording.

Control Subject Line: Coming to Denver: New Live B2B Marketing Certification Workshop

Control Headline: Accelerate Your B2B Marketing Program Today & Get Certified

Course Title: B2B Marketing FUEL Advanced Practices Workshop

Control Email:

Control Email

Click to enlarge


In the treatment email, Justin changed the name of the workshop to focus on lead generation.

Treatment Subject Line: Coming to Denver: New Lead Generation Certification Workshop

Treatment Headline: Power Your Marketing from Lead Generation to Sales Conversion

Treatment Course Title: Lead Generation Certification Workshop

Treatment Email:

Treatment Email

Click to enlarge


Lead Generation vs. B2B Marketing. Which is the “right name” that resonates with the MarketingSherpa audience?

It appears that the more specific title did increase the open rate in this test.

The measured level of difference in the sample was 38.4%.

Now, please note that this was a highly targeted microtest. For that reason, the difference in open rates was significant only at the 80% level of confidence. The difference in unique click-through rate was not significant at that level, which is why I did not include it in this blog post.

For our tests, we usually require a 95% level of confidence, and frankly 80% is the bare minimum to even share with you on this blog.

But, with a small traffic test such as this, we did not launch the test because we sought to get a definitive answer to a question. We were merely sending out a trial balloon to a very small, very targeted segment of our list.

Before we launch future tests, let’s hypothesize as to why the more specific messaging seemed to work better. This test idea actually came directly from our Managing Director, Flint McGlaughlin, so I asked him for his analysis. Here’s what he had to say…

“B2B is expressing a category and Lead Generation is expressing a solution. B2B only tells someone the category of the enterprise. Lead Gen tells someone the nature of the solution. People are more interested in solutions than they are in titles. When we said that it is going to be Lead Generation, we gave them a tangible benefit that is easy to visualize in their minds. It makes them say, ‘this is what I’m going to get, I’m going to learn how to get better at Lead Generation.’”

“When you tell them B2B, it has one particular power, and that is that it suggests that this might be particular to their niche. The word B2B belonged in the sub-header or in the first paragraph, but not as the main thought in the headline. The main thought in the headline is ‘Lead Generation,’ and then you can intensify by saying that this is especially for the B2B business space. Now that the visitor has his thought sequence right, tell him what he’s going to get, and then intensify its value.”

And, as you can see, Flint already has ideas for future experimentation. So, on to more testing. Or, as Justin Bridegan wrote to me when he sent these test results…

“With more testing and validation we could be on to something. More to come…”

Related Resources:

Email Marketing: Testing subject lines

Email Marketing Tests: What to do when a radical change produces negligible results

Crafting an Engaging Email Message: How a properly focused email message can increase conversion by 85% (a special live optimization clinic)

B2B Summit: Optimize your marketing funnel by mapping what works in Lead Generation

Share and Enjoy:
  • LinkedIn
  • StumbleUpon
  • Facebook
  • del.icio.us
  • Digg

Categories: Analytics & Testing Tags: , , , , , , ,

  1. October 18th, 2011 at 15:57 | #1

    Nice article! As a statistician, I was a bit concerned when I saw how small those population sizes where, but I was immediately satisfied when you pointed out that the confidence level is only 80%.

  2. November 17th, 2012 at 19:36 | #2

    Interesting. Do you think we should change the from labels too? When they see the emails in the inbox they see from who it is coming. Should the from be more descriptive of company brand name?

  1. July 22nd, 2011 at 08:56 | #1
  2. July 26th, 2011 at 04:02 | #2
  3. August 2nd, 2011 at 04:02 | #3
  4. March 21st, 2012 at 09:40 | #4
  5. March 22nd, 2012 at 04:07 | #5
  6. April 17th, 2012 at 13:35 | #6
  7. January 8th, 2013 at 03:03 | #7
  8. February 6th, 2013 at 16:00 | #8