An A/B test, when used properly, is a great way to enhance your customers’ experience and optimize your marketing funnels by testing various content and user experiences before expanding to your full audience. When used incorrectly, an A/B can eat up resources, take too long to reach significance and lead to misinterpretations of the success and failure of your test.
As a marketer in the travel industry, or in any industry heavy on first-hand user data, there is a common issue of choosing a sample that gets you to statistical significance but is not an accurate representation of the intricacies and diversities within your audience. This leads to a new feature, content change or functionality that ends up having a negative impact on your conversion rate. Even worse, it may lead to customer frustration or a disassociation from your brand.
While tools like Optimizely can make it easy to determine sample size and statistical significance, it doesn’t take in to account (or at least not out of the box) whether the sample is reflecting dips and spikes relevant to your traffic sources, what day of the week it is (as it relates to sales trends) or who’s likely to be shopping at that time of year. By using features such as Optimizely Dynamic Customer Profiles combined with our data warehouse, we are able to be more selective in who we are testing as well as analyzing who engaged in what way with the test we are running to ensure we aren’t creating an unintentionally biased result.
In thinking this way, you create a solid framework for selecting your sample make-up. Before any experiment, I recommend asking yourself the following:
1. What are the different sources of traffic to your testing environment? This can be your list source in email or your traffic sources for web. Think about how these different sources impact the guests engagement with your brand. How would this influence their decisions?
2. How is your testing period representative of your business? Be sure to account not just for who is typically shopping on days of the week and at what times, but who is typically shopping in that time of the year as it relates to your business. Does your test give you an accurate sampling of that?
3. What are your segments and is this test intended to work for all of them or some of them? Does your sample reflect that?
By asking yourselves these questions and using what you find to select a representative sample and, of course, using a tool that gives you this flexibility of audience selection, you can more easily define the goals of your test, the time you need to run to reach significance and accurate representation while preparing yourself beforehand for any anomalies that may lead to a misreading of results.
If you’re interested in ways to visualize results, check out my previous post: Intro to Data Visualization.