I’m going to be completely honest. Yes, I wrote a blog post titled “Website Testing as We Once Knew It Is Dead” expecting to get the reaction that it did.

While some people seemed annoyed at my post, others were left with a feeling of emptiness. First, to reiterate, testing is not dead. Rather, testing in the enterprise is evolving to another place. I absolutely agree that you will always need to test, and that you’ll always need hypotheses.

So let me expand more on the methodology that replaces the part of testing that is dead. Let’s start with a very basic assumption that, on average, four out of five of your tests don’t have a winner (either they lose or simply yield no significant lift). You then latch on to the one winner, and assume that more of your traffic will convert based on that single winning test that you rolled out to all of your traffic.

What if four out of five of your tests could have positive results? You would be much closer to the holy grail of online marketing: delivering the right experience, to the right person, at the right place, and at the right time.

This becomes a reality with automated segment discovery, which uncovers the customer groups that an experience does work for, and those for which it does not, and then allows the marketer to run educated validation tests on the results. Think of these tests as relevant experiences, not random ones. The marketer’s hunch―at this stage―is replaced by a truly data-driven hypothesis.

Segment discovery is about uncovering revenue opportunities that are valuable, but which would be too difficult to discover on your own. But for it to work, it relies on being able to achieve statistical significance for each of the discovered segments, which obviously is much easier for larger enterprise websites. So for smaller websites, traditional testing remains as alive as it ever was.

Consider that when a website with low traffic runs a test targeted to returning customers, it could take weeks to reach statistical significance. When you sub-segment that test into additional groups, such as returning visitors by geography, returning visitors by demographics, etc., the number of visitors that fit each group are so small that significance wouldn’t be reached for quite some time, if ever.

But for enterprise businesses equipped with the right tools, this new testing methodology has a very short adoption path. I would break it into three parts:

1) Start by running a campaign targeted to all visitors (no segmentation).

2) Use automated segment discovery to determine the customer segments for which the campaign did work, and those for whom it did not.

3) Turn each of the surfaced segments into validation tests.

Suppose you discover that visitors from North Dakota responded well to a “Free Shipping” campaign. You might conclude that this is a value-driven segment: Test additional offers with them, and subsequently redirect acquisition spend to attract more visitors from North Dakota.

Conversely, suppose you discover that a high-income demographic responded well to a brand affinity campaign. You might conclude that this is a relevance-driven segment, so rather than showing them “Top Sellers,” you would test additional campaigns that highlight other products from the preferred brand.

To be clear, segment discovery in no way changes the goal of every marketer: To deliver the right message, to the right person, at the right place, and at the right time. Rather, it enables getting to this end point faster and easier than was previously possible with traditional testing.