An article in Sunday’s New York Times posed an interesting question that comes up often: Is using information that you know about an online customer to personalize their experience on your website helpful or intrusive?

As more ecommerce companies attempt to leverage their customer data to create highly relevant experiences, it’s a debate that will only continue. I’m pleased to see this discussion taking place, because it’s an important topic as ecommerce generates more and more revenue.

The best website personalization is really good, targeted marketing. Personalization isn’t easy, but if you do it correctly, it can be an extremely effective way to delight visitors. To do it well, you have to understand your data and your customer, and find out where the line is for your business. That’s the challenge.

Let’s look at personalization as a spectrum with two ends. At one end, using too much of the information you know about your customers can unsettle them. If you went onto a website and the entire experience was based on the news and magazine articles you’d read on different sites, that would be very surprising to you. You wouldn’t expect that website to have, let alone act on, that information. In that case, using information that a consumer doesn’t expect you to have can result in a negative impact.

At the other end of the spectrum, if you don’t use the information you have about your customers, they’ll likely be disappointed. A customer who already purchased a pair of rainboots from you doesn’t want the same pair of boots recommended to them on their next visit. Instead, using the information that you have to recommend a related product could be what delights them. Consumers expect you to know about their prior experiences on your website, and they expect you to use this information well. So it’s a tough balance.

But there is one good yardstick for online retailers here, and that’s thinking about things from an in-store perspective. As pointed out by Mahender Nathan, Vice President for E-commerce and Digital Marketing at Godiva, it can be really useful for people exploring personalization, customization, and relevance for the first time to think about it in terms of people having a conversation. Would it be surprising if you mentioned the information you’re using in a face-to-face conversation with one of your customers? If so, using that piece of information to make changes to your website could make visitors feel uncomfortable.

Picture this: You walk into a brick-and-mortar store where you have a good relationship with a personal shopper. As soon as you walk in, the personal shopper says “Oh, I set this shirt aside for you because I think it goes really well with the sport coat you bought from me last month.” That’s not surprising, it’s good personalization. Even if you decide not to buy the shirt, it makes you feel good about your experience with the store, because it used what it knew about you to be highly relevant.

Now, imagine walking into the same store. But this time, the same personal shopper says “Oh, I set this shirt aside for you because I know that on your flight back from LA this week, you saw it in Esquire and spent some time looking at it.” And you would say “What are you talking about? How on earth could you possibly know what magazine I was looking at on the plane, let alone that I was on a plane, and that I looked at this shirt?” You would probably never go back to that store.

So there’s a line with personalization. When does it cross over from effective and helpful to creepy? That line is different for every business and every customer. And that’s where testing comes in. Testing, personalization, and relevance are inextricably tied together. Without testing and measurement of everything you do to try to create a more relevant experience, you don’t really know whether you’ve crossed the line from what’s effective to what’s turning customers off.

By running iterative tests, by trying and tweaking lots of ideas, and having an environment where you can do that very quickly and inexpensively, you can really explore what relevance means for your customers. Continuing to test and monitor your results will give you enough information to discover the right personalization for your visitors specifically. And that will allow you to delight consumers with a relevant, targeted online experience that doesn’t step over any boundaries.