This is the second post in my blog series about busting myths of a/b testing and segmentation strategy. In my first post I talked about how change is normal, and what it means for test results that are premised on reliable data and static external factors. So, if behavior is changing all of the time, how can you use that knowledge to determine the right experience for each individual? I’ll dig into that question in this post. 

Promoting a winner is the best way to improve your goal metric. Definitely not. Selecting averages is actively costing you money.

The last post showed how marketing teams can be held back by a limited understanding of change. A similar (and common) mistake is to believe that we can produce segments of our customer populations based on only a few dimensions.

What if your segments are too simple to drive deep customer understanding? Marketers often rely on coarse segments like loyalty members vs. non-program participants, full-price vs. discount buyers, or people who buy within a specific product category. These segments can fall short when they are defined by only a few data points and leave out lots of potentially influential factors. They may help you generate some useful insights, but this method doesn’t scale very well because of the great diversity in your customer base.

Imagine a segment like “Women’s Urban Hipster.” It may seem like it reflects data along several variables, but individual customers within that segment aren’t uniform at all: they may have different preferences for colors, styles, purchasing frequencies, price brackets and categories. What’s more, those behaviors may all change across different sessions based on their geography and time of browsing. Not only are customers in one segment not consistent across the group; they aren’t even consistent with themselves–and treating this group as a homogenous entity may prevent you from delivering a truly personalized experience.

Segments may help improve results in the short-term, but they will reach a ceiling. So, what’s the solution?

We need a much more nuanced view of our customer population, one that may consist of hundreds of data points, not only a select few.

Are you using legacy approaches in the new era of automation and machine learning? Get your copy of the Closing the insights loop (perhaps for good) whitepaper to find out.