More than 50 years ago, IBM’s George Fuechsel coined the familiar adage, “Garbage in, garbage out,” to underscore the relationship between inputs and outputs in computer programming. The same could be said of website testing, with the quality (and reliability and relevance) of your test results being directly related to how well your test was planned. A good testing strategy aims to answer four basic questions:
- Who am I targeting?
- What will I show them?
- When will I show it to them?
- Why?
But you shouldn’t necessarily answer these in this (or any) particular order. One of the challenges of sequential planning is that your “Who” becomes too narrowly targeted — or your “What” has too many variations compared to the traffic “Who” will bring — that your test will take too long to achieve significance, thereby making it difficult to reach valid conclusions.
Holistic planning, by contrast, can encourage you to tackle these questions jointly so that the results of your test — your outputs — are driven by the appropriate inputs. (A 2-week test is generally a good rule-of-thumb, but your business context may merit something shorter or longer.)
But if your site traffic isn’t high enough for hyper-targeted testing, fret not! As my colleague Peter Borden reminded me, low traffic sites benefit by beginning with high-level targets–perhaps just New vs. Returning Visitors. And as you gather data to support or disprove your hypothesis, you can plan increasingly granular tests that piggyback off of these learnings.
In my next set of posts, I’ll tackle each of these four questions in greater detail (holistically, of course). Stay tuned next week for “Strategies for Successful Testing: 5 Key Audience Segments You Should Target”.