Your website may never be everything to everyone. But with the right testing and targeting strategy, you’ll come a lot closer than you think. For quick review, last week we looked at 5 key audience segments you should target to answer the first question in developing a great campaign: Who am I targeting?

It goes without saying, of course, that a mismatch between ‘Who’ you target and ‘What’ you test can be fatal or, at best, fall flat. (Imagine promoting a sale on Yankees gear to users in Boston.) But before we dive into examples of great Who/What combinations, let’s be clear about the difference between testing and targeting. Testing refers to the general concept of showing different content to different visitors—without regard for who those visitors are. By contrast, your target refers to a specific user audience that you identify, and segment, for the purposes of running a test.

Without a target, testing makes little sense because your web traffic consists of many different visitor types, who have a variety of needs, intentions, and likelihoods to purchase. For example, a visitor’s familiarity with your site (new vs. returning), search query intent (“iPad sales” vs. “iPad reviews”), or geography (Anchorage vs. Honolulu) can all indicate either different needs or stages of the buying cycle.

Many marketers fall into the trap of just “testing” (with no segmentation) because they’re attracted to the larger sample size and the possibility of achieving statistical significance quickly. But what good is 95% significance when your sample grouped radically different audiences into the same experiment? How valid can your conclusions really be? And dare I say it: How much potential improvement have you left on the table?

Consider the following (hypothetical) scenario of an apparel retailer whose traffic is split 50-50 between those shopping for men’s vs. women’s clothing:

  • The retailer wants to test a new “Add to Cart” button.
  • The test group’s “Add to Cart” button will be a different color (A vs. Control).
  • After one week, the test group’s “Add to Cart” rate has increased 5%, with 95% confidence.

Sounds good, doesn’t it? But what if that 5% increase hid the following details:

  • All improvement was driven by cart adds for female clothing. There was no net improvement on pages selling men’s clothing.
  • Or even worse, the “Add to Cart” rate for women’s clothing increased 10%, but was offset by a decrease in the “Add to Cart” rate for men’s clothing.

Economists refer to this second point as Pareto Inefficiency (i.e., improving the performance of one group by making another group worse off), and in the world of website testing, we solve it through segmentation or targeting.

In Part 2 of this topic, we’ll look at some great Who/What combinations, including bleeding-edge segments such as iPad users. Stay tuned for “Strategies for Successful Testing: Segmenting Your Way to Great Results (Part 2).”