MarketingExperiments Blog: Research-driven optimization, testing, and marketing ideas

As a presenter on our recent technology-focused web clinic, I had the pleasure of learning about an experiment devised by my colleague, Jon Powell, that illustrates why we must never assume that we test in a vacuum devoid of any external factors that can skew data in our tests (and even looking at external factors that we can create ourselves).

If you’d like to learn most about this experiment in its entirety, you can hear it firsthand from Jon on the web clinic replay. SPOILER ALERT: If you choose to keep reading, be warned that I am now giving away the ending.

According to the testing platform Jon was using, the aggregate results came up inconclusive. None of the treatments outperformed the control with any significance difference.  However, what was interesting is the data indicated a pretty large difference in performance with a couple of the treatments.

So after reanalyzing the data and adjusting the test duration to exclude the results from when an unintended (by our researchers at least) promotional email had been sent out, Jon saw that each of the treatments significantly outperformed the control with conclusive validity.

In other words, if Jon had blindly trusted his testing tool, he would have missed a 31% gain. Even worse, this gain was at the beginning of a six-month-long testing-optimization cycle. If Jon had assumed he had learned something based on inaccurate data that he really hadn’t, this conclusion more than likely would have sent Jon down a path of optimizing under false findings and assumptions.

In other words, to create a simple pre-GPS era analogy, if you make a wrong turn at the beginning of a 600-mile road trip and keep heading in the wrong direction, you will be much farther off the mark than taking the wrong road when you’re just a mile away.  However, in our cases with many businesses, wrong turns and mis-directions can cost from thousands to millions of dollars in lost time and revenue.

Worst of all, this email came from the Research Partner itself. As we run into many times, they unwittingly sabotaged their own tests. With the Internet being a dynamic place, it is next to impossible to avoid every external validity threat to your test, but at the very least we need to make sure that we are not introducing threats with internal campaigns to the same audience.

This is not to say we stop those campaigns, but just be aware of the potential effects on testing. That awareness, at least until computers become sentient beings, requires human involvement. Of course, that’s just one area where a little human curiosity is essential…

Do not let testing tools overshadow the human element of creativity.

Sure, many tools are now evolving to the point they will create “treatments” for you based on combinations, uploading content etc. But what this can create is a “perfect” sub-standard general design. These tools are limited to the inputs we give them so the optimization that can occur is constrained, where a human could take findings and radically change an entire process.

Begin by taking a step back, putting yourself in your customers’ shoes, and taking a human look at the big picture. Ask, “Is this even the proper overall design?” rather than taking the easy shortcut of testing a randomly generated combination of calls to action or headlines.

Multivariate testing (MVT) has its place. In fact, here at MarketingExperiments we use it frequently. But as with any tool, the result is only as good as the craftsman. So, when using MVT, make sure you have not ignored the big picture of what your users want by using the same sub-standard message presentation you’re trying to optimize in the first place as the base that you build your tests off of.

So how do we trust our tests? Here are steps for better setup.

• Sound Test Design – The test you are performing must represent the same environment where you are going to potentially apply the results. Many times we find people stretching the finding to different audiences, and then wondering why the results do not translate. For example, are you taking the lessons learned in email testing and applying them to your PPC ads? Well, they could each have different audiences that react in different ways.

• Research Question – Have you set a clear and specific objective for testing? Without establishing a clear objective, it is possible to get lost looking at a vast array of data points and trying to correlate them all. The research question also provides guidance on what items should be included in a test and what should be reserved for later.
• Proper Execution – Are you selecting the right test audience? Based on this audience, will you be able to apply the results to other aspects of your web communications?  Beyond that, you must ensure you have enough of this audience to reach a statistically valid conclusion, i.e. really learn something not just think you learned something. To do that, you must be recording accurate measurements. Ensure you double check your metrics technology before launching a test (more on that in the checklist).

• Confidence – Establish a standard for your results to uphold. Simply, you are trying to arrive at a finding that you have seen replicated enough times that you can confidently say, “we have sufficient information to make a conclusion on the research question we sought to answer.” The amount of times you need to measure will be a decision based on the volatility of the experimental environment and other factors. At the end of the day, though, it will also boil down to a business decision to continue or move on. This is something that needs to be agreed to and developed in house. Just understand that while setting this mark low carries some risks, some processes with low traffic or time sensitivity necessitate that we move on with lower confidence levels at times.

At MarketingExperiments, we try to stick to a 95% statistical significance as much as possible.  However, there are times where we have to accept a lower mark.

But remember, statistical significance from a piece of software cannot alert you to data that is inherently wrong or warn you that something else has influenced (and perhaps invalidated) a test, it only tells you that the results were unlikely to happen by chance.  Omniture (interesting alert for segmented data) and Google Analytics (GA intelligence) have been dabbling in this area, but still require human interaction and do not cover all aspects.

So make sure that you perform your due diligence with tool setup, test design, and data analysis – because it is very easy to gain confidence in the wrong decision with bad data from a tool that says it is 95% confident. Again, it is so important for us to invest greatly in people along with tools. As Avinash Kaushik says, you should invest 10% in tools and 90% in people.

Technology options/features that can trip you up.

• Metrics calculation process – Know how conversions are calculated (for example, visits vs. absolute visitor vs. page views, etc.). Many tools allow you to change how metrics get calculated, so make sure you are looking or pulling data using the same measure or comparison items throughout the test. Also, realize that individual tools may calculate conversions slightly differently.

• Default validity confidence levels – Understand your testing tool’s default measure of confidence and make sure that it matches your own internal measures.

• Default summaries – One of the most dangerous items in testing is the summary or dashboard view. Most of the juicy test details are hidden so problems that might be occurring in the test are tough to spot. Jon’s experiment is a great example of this. Looking at more specific data (like day-to-day metrics) will give you a better health check of what is happening with the test.

• Uniform sample distribution assumption – Tools assume that the data we are going to receive within tests will be uniformly distributed. However, if you have run your own test you know that this is not always true. As mentioned earlier, testing software has started adding some intelligence tools to try to spot “interesting” data points, but in our experience not many people use these tools. Non-uniform distribution can drastically affect validity and needs to be monitored…which means you need to pay attention to data closely (not in aggregate).

For a five-point testing technology checklist, and to learn more about other technology blind posts and how to address them, view the replay of our latest webclinic.

Related Resources

Online Marketing Optimization Technology: We have ways of making technology talk, Mr. Bond

Optimizing Site Design: How to increase conversion by reducing the technology barrier

Essential Metrics for Online Marketers

Clinic Notes

What is it about lead nurturing that makes it so hard to get it right?

Is it the all-too-familiar disconnect between marketing and sales? The hassles of CRM systems — or lack of them? Or the gray areas surrounding lead definitions, scoring, nurturing strategies and who owns what customer touchpoints?

For many organizations, it’s a combination of all the above (and more). And the need for a shift in perspective, if not in the organizational culture, can be a tough sell when monthly and quarterly sales numbers aren’t exactly rosy. That is, unless you can point to proven results and budget-friendly methods you can implement quickly.

If you’re looking for ways to start, or get more from, your lead nurturing process, I’d recommend two new, free resources:

1. In his webinar, Putting the Human Touch into Lead Generation, B2B lead nurturing expert (and colleague) Brian Carroll broke down a case study and several action steps that show how you can use multiple channels — social media, email, webinars, phone and mail — together more effectively. Most organizations are reaching out to prospects in these ways, but they’re not consistently doing it in an optimal way. This one-hour webinar can help you get there.
2. The Definitive Guide to Lead Nurturing, a new eworkbook from our friends at Marketo, not only offers several best practices, but provides the worksheets and real-world examples you can adapt or adopt (i.e., ideas you can steal). What’s best about this guide? It’s not just another whitepaper pitching services; this is information and research that marketers and sales teams can literally plug into their existing processes, or build those processes if they don’t already exist. You can download the guide here.

You’ve heard the MarketingExperiments team say many times that people buy from people, not websites. Invest some time in these lead nurturing resources and you’re sure to find new ways to cultivate the business relationships that ultimately generate more sales.

Lead Generation, Marketing Insights, Practical Application

If you’re joining us for this afternoon’s SEO live optimization web clinic, you already know the topic is way too broad for one hour.

Learning the fundamentals of search engine optimization is only step one. Keeping up with the frequent changes, learning and testing the latest best practices, and steering clear of the mountains of misinformation? That’s a full-time effort.

If you’ve been around the block with SEO, you’re already a regular reader of the following sites and tools. Still, when it comes to reliable SEO info, these resources consistently rise to the top of my shortlist.

10 SEO resources you’ll want to bookmark

• SEOMOZ
• SEOBook
• Search Engine Roundtable
• SearchEngineWatch
• SearchEngineLand
• SearchEngineGuide
  
• TopRank Blog’s Resources list and Big List of SEO Blogs
• HubSpot and Website Grader tool

That’s it? Why not an exhaustive list of 400+ SEO sites?

A few reasons: First, TopRank already has a megalist; it’s right up there (thanks, Lee!). Second, from the sources above, you can branch out to any number of free and paid tools and augment your own list based on your experience level, needs, and preferences. And third, if you really have time to regularly read more than a dozen sites on SEO, more power to you and your Google Reader and/or RSS feeds.

Feel free to add your own favorite SEO resources in the comments section.

And check back with the blog as we’ll be following up today’s SEO clinic with responses to the live audience Q&A, additional resources and specific articles, plus our clinic contest winners — some lucky marketers will not only have their pages optimized, they’ll also win seats at our Landing Page Optimization Training Tour.

Clinic Notes, Search Engine Optimization (SEO), Site Design

With our June 10 web clinic, Twitter Experiments: Getting beyond the “now what?” featuring special guests Jason Breed and Marc Meyer, we knew we had a full house going into the session.

But we never expected to break into Twitter’s top 10 trending topics by taking down a chunk of the interwebs.

What our crew and 1,001 clinic participants also didn’t expect was that the webinar hosting service would go down three times due to a “service disruption … a result of unscheduled maintenance in our network.” Nice.

Despite all that, we managed to plow through the webinar and we’d like to send a special thank you to the 862 folks who stayed with us. This post includes some answers to your questions from the web clinic.

You can now access the presentation: Twitter Experiments: Getting beyond the “now what?”

Pamela Jesseau, our razor-sharp new content queen, tackled the following clinic questions to help you set up your Twitter account, build your following, and expand your business. (Also check out her post: Twitter for businesses: 7 articles + tools you don’t want to miss.)

Q: How do I search Twitter for mentions of my company? What do I do with the results?

The simplest way is to use the search box in the right-hand sidebar of your Twitter page, or Twitter’s search page at search.twitter.com.  These will find mentions of your keyword within all Tweets.

One tool that was featured in the web clinic is Twazzup, a third-party application that can help you track mentions of your brand. Just enter your company name or search string to find out who is Tweeting about you. For example, the image below shows a search for a company that was recently in the news in North Florida.

You can also set up a search string within an application like TweetDeck, which will alert you when someone sends a Tweet with your name. Then what? Listen to what they are saying, address their concerns, thank them for their feedback, and follow them to engage the conversation and build your community.

Q: What if your company name is too long for a good Twitter name?

Twitter only allows 15-character usernames, so there are many companies which have to improvise. We ran into this issue ourselves and abbreviated MarketingExperiments to @MktgExperiments.

Your Twitter handle should still make sense and be fairly easy for people to remember.  Other examples of organizations that got around this issue using abbreviations, shortened forms, or acronyms include:

• @pdxtrailblazers – Portland Trailblazers
• @SBoSM – SmartBrief on Social Media
• @TheHSF – The Heart & Stroke Foundation
• @MarriottIntl – Marriott International

Q: If you are not tweeting with a big following, what is the point?

Everyone has to start somewhere.  You can build on a small following by first engaging and bringing value to those that do follow you.

Add yourself to the user-generated Twitter directory WeFollow and choose three tags that describe your business and industry — such as marketing, SEO, and software for example.  Others with similar profiles will find you when they are looking to expand their networks.  In turn, take the time to search for people in your target group and follow them.  But don’t stop there — start a conversation.  Retweet an article you found valuable, or start a discussion on a topic relevant to your sector.

Fridays provide the opportunity for you to recommend a colleague or partner by Tweeting their name to others and tag it with #followfriday. Some may return the favor, or at least tweet a thanks with your name to their followers. (Learn more about #followfriday and other hashtags at … wait for it … Mashable.)

Build and nurture your following, the same way you might do with other leads. You wouldn’t try to hard sell a new prospect in person, would you? So don’t do it on Twitter. Build the relationship, get your brand and yourself out there, and be there to respond to feedback.

Q: I understand why someone would have a lot of followers. But how can you follow thousands of people? It’s not manageable. It doesn’t make sense to me. Any advice on this would be great.

One of the most valuable tools that we have found for smart Twittering is TweetDeck. As mentioned above, this free application allows you to manage the tweets you receive with columns of separate, customizable feeds and filters.

You can group certain people, and create search strings to monitor mentions of your company, industry and even competitors. That way you can skip all the weather updates and lunch reports, without missing the feedback on your product, or opportunities to connect.

Q: Any results on running price specials via Twitter?

Dell Computer made news earlier this year when it revealed that $1 million in sales could be attributed to their Twitter feeds. Dell has several feeds, each targeting different segments. As we discussed in the web clinic, the company used @DellOutlet to Tweet discounts with a link to purchase the product from the Outlet Store. Shortly afterward, Dell started to offer exclusive deals to its followers. Twitter is still an emerging channel, but results like these show it does have real value for businesses.

Twitter training: Putting practical know-how into action to drive business results

In addition to the questions above, the live audience polls from our web clinic indicated that many marketers still aren’t sure how to use Twitter to improve their bottom line.

That’s why we are pleased to be able to team up again with Jason Breed, Senior Director of Business Development of Neighborhood America, and Marc Meyer, CEO of Digital Response Marketing Group, for a new training eWorkshop on using Twitter to boost your business results.

While our free web clinics provide new research findings, case studies and actionable ideas, our training eWorkshops are designed to help you advance from the “what” and “why” to the “how” and “how-to” with detailed, hands-on guidance and tools.

In this interactive eWorkshop, you’ll learn specifics on:

• Where to start (or restart) and how to map out your Twitter presence/s
• How to develop the most effective Twitter business strategy for your organization
• Ways to build and engage a following that’s more than just a numbers game
• Tools that will help you save time and maximize your efforts

eWorkshop: How to use Twitter to boost your business results
Tuesday, June 30
4:00 to 5:00 p.m. EDT
Training session: $129

Join us if you want to accelerate your knowledge, cut down on the learning curve, and start using Twitter to drive revenue, build equity and decrease costs.

Twitter web clinic attendees can save $30 on this eWorkshop when registering with the special savings code. Please check your clinic follow-up email for the code and secure your spot for this special training eWorkshop. (We’ll keep the interwebs intact this time.)

Clinic Notes, Marketing Q&A, Social Media

It’s not often that I get jazzed about the topic of media relations. In fact, it’s usually just the opposite.

As an editor, ducking clumsy pitches (in all media) and screening calls are part of the job. But while working in the marcom trenches, I learned to sympathize with PR folks who have to deal with editors and reporters, and the often unfair bad rap that PR has to overcome.

The eye-opener? Working with a media relations pro who set the gold standard for the job.

PR 2.0 or Relationships 101?

It was never about cranking out banal press releases or heavy-handed pitches. Instead, she did hours of research, became a subject expert in her clients’ fields, provided compelling, relevant story ideas to the right journalists at the right times — and did all the right things to build relationships. She asked incisive questions, knew her stuff, and in turn, earned enormous respect. On a tight budget, too.

Prominent coverage followed. Major papers, tv news, trade pubs. Clip files that many organizations only dream about. And all pre-Twitter.

Social media doesn’t change everything

These days, with social media, Twitter and Facebook generating so much buzz, things are getting simultaneously easier and harder. Sure, these avenues can help you get your foot in the door easily with some media types, but others have already barred the social media door to PR. Are you linking in — or spamming? How do you figure out not only whom to approach, but how?

That brings us right back to the gold standard: relationship building.

Let’s face it, with or without social media, for many of us not named Godin, Kawasaki, Brogan or Battelle, cultivating relationships can still be time-consuming, hard work. It’s not always fun and there are no guaranteed shortcuts.

But there are tools that can help you get and stay organized, and make the process of building your media relationships much more efficient. And that’s what I’m jazzed about.

A special media relations eWorkshop and toolkit

Today at 4:00 p.m., we’re kicking off our new eWorkshop series with a session on building an effective, sustainable media relations program, including a half-dozen tools plus a campaign playbook.

After reviewing the toolkit, and seeing the components in action, I’m convinced that pros and novices alike will benefit from the tools and strategies in this special eWorkshop.

So no matter what your experience level, with the stakes getting higher every day, why not carve out an hour this afternoon to join us and raise your media relations game?

[Links updated to point to completed eWorkshop presentation.]

Events, Marketing Insights, Practical Application

The underlying theme of our 2009 Marketing Blueprint clinic boiled down to this: Be prepared to break some rules this year.

That advice might seem odd at a time when many companies are scrambling to find solid ground by tightening expenditures, laying people off, and clinging tightly to marketing practices that have already shown results.

In this economy, most organizations aren’t willing to stick their necks out too far. And that’s exactly why it makes sense for the bold to do so.

After all, if your rivals are playing it safe, and you’re forging ahead with new ideas and initiatives, you’re more likely to stand apart. Aside from those competitive advantages, having the gumption to branch out with your testing, optimization and marketing efforts is what can lead to surprising gains. Failure is certainly a risk, but so is inertia; that’s why we test and learn and test again.

Best practices vs. common practices

One of the keys is distinguishing between best practices and common practices — and understanding that neither are fail-safe shortcuts to conversion gains. In fact, our case studies from this clinic showed how applying the same practices that got a 162% increase for one site yielded a 48% decrease for another site. Keep that in mind if you’re using templates or similar elements across several sites or landing pages.

You can now access the Dec. 17 clinic in three formats for your convenience — note that the research brief includes several links to relevant clinics and free tools:

• View the complete presentation (slides and audio)
• Read the print-friendly research brief with additional links
• Listen to the MP3 version

We can learn from both negative and positive results, but we don’t grow and advance our knowledge by simply repeating the same practices and methods. So even though this 2009 Marketing Blueprint lays out a framework for optimization, it’s up to you (and us) to mix things up a bit in order to discover what really works this year.

Clinic Notes, General, Internet Marketing Strategy, Marketing Insights, Practical Application