According to Econsultancy’s 2011 Conversion Rate Optimization Report, 53% of companies use A/B testing to improve conversion rates compared to just 24% using multivariate testing (MVT). And compared to all methods for improving conversion rate, A/B testing ranked #1 by current usage, with MVT registering at a distant #10. Want another sobering statistic? While 24% of companies actually used MVT in 2011, 44% of companies reported in 2010 that they were planning to use MVT this year.

So why the gap between planned and actual MVT usage? Econsultancy doesn’t offer an explanation, but for some companies, multivariate testing just sounds better than it really is. Its appeal is sometimes offset by practical limitations once a site is ready to test. To be sure, though, MVT does have its place—and an important one, as well. Let’s look at eight scenarios for when MVT may be more or less appropriate for you.

Multivariate testing may be inappropriate for you:

1. When the required sample size greatly exceeds your level of traffic: Remember, the ideal length for most tests is about two weeks. Anything longer and you risk introducing noise into the test (holidays, different promotions, etc.) which will invalidate your results. Many sites simply don’t have enough traffic to test multiple variations of multiple variables simultaneously. Here’s a simple calculation. If:

• Your test page has 5,000 visits/day;
• converts at 2%; and you expect the conversion rate to increase to 2.5% (a 25% increase); and
• want to test three variations of three variables (3 x 3 x 3 = 27 test combinations);
• running all 100% of the traffic through the test; then
• your test would last 88 days before achieving 80% significance.

2. If your homepage is the only page on which you can run a valid test: If you have a familiar brand and domain name, chances are that more than 50% of your site traffic enters through your homepage. From there, though, your traffic splits off into many different paths. Perhaps one or two sub-pages are popular paths, but the rest will spread out across a very large number of pages, on which a multivariate test isn’t reasonable. There’s simply not enough traffic to conduct a valid test.

3. When targeting your MVT cuts your sample size to a dramatically small number: In the calculation above, we ran all 5,000 visitors through the test. These consist of new/returning visitors, visitors from various traffic sources, etc. You have every reason to segment these audiences into different tests, but doing so can quickly turn a 5,000-visitor sample size into 50—and an 88-day test into 88 weeks. The problem is, you need targeting. Without it, a statistically significant MVT will tell you what variations had the greatest impact on conversions. But the “best-performing variation” still represents the average of how all your audiences performed in aggregate. What conclusions can you really draw from that?

4. When the low-hanging fruit is easy and obvious: You may have many variables that you want to test, but it’s often a small few that yield the greatest impact. Start with the obvious (to achieve significance sooner), then work your way down to less important elements on your page. From experience, your account manager will be able to identify the elements most likely to drive conversion rate improvements for you.

So, is MVT ever appropriate? Absolutely, especially if any of the following four best use cases apply to your website. MVT may be your better choice when:

1. You have the traffic to support it: The higher your traffic is, the more multivariate tests you can run. Remember, though, there’s an inverse relationship between test duration and the number of elements (variations per variable) that you test. Strive to achieve significance within two weeks.

2. Your offsite traffic acquisition efforts are small in scope: A big driver of on-site behavior involves where prospects come from and the messaging that caused them to click. The more your traffic is spread across many different channels (organic search, paid search, direct, email, display, referral, affiliate, Facebook, etc.), the more difficult multivariate testing becomes. In a perfect world, you’d segment along these lines, but you might end up with too small a sample size. However, if your traffic is concentrated around a smaller number of channels, you’re in much better shape.

3. You offer a fungible product or service: Users in Miami and Maine wear very different clothing in the winter, but would they shop for DVDs in fundamentally different ways? Probably not. If your business allows you to ignore geography (i.e., neither your products nor offers differ by region), you have one less influence you need to worry about.

4. Your tool makes it easy: Let’s face it—setting up a multivariate test could take you minutes, hours, or days. Does your tool require additional code around every test element? Advanced knowledge of JavaScript? Multiple server calls to display each element? Or do you have a one-tag solution that takes care of everything? Consider not only how long your test will need to run, but also how long you’ll need just to get it running.

The appeal of multivariate testing may be high, but just because you can do it doesn’t mean that you always should. An important component of the planning process is understanding the business, resource, and statistical contexts of your test. If you determine that a multivariate test offers the quickest and best way to answer your questions, then leverage MVT to its fullest extent and capabilities. Otherwise, a simple A/B test may be just what the doctor ordered.