A/B Testing During Holidays or Offers

Would you like to run more tests in less time with equally reliable results? Or maybe you just want to get more sales from the holidays? You can do both by taking advantage of the natural ups and downs in traffic caused by holidays and (offline) promotions.

Instead of stopping your tests during these times, you can keep them running. Even better, prepare for such traffic spikes and ensure specific testing is set up for each holiday, event, promotion or campaign!

Why test during promotions or offers

One of the most important factors that determine the estimated time an A/B test should run is the daily amount of visitors. By operating during holidays and promotions you will be assured of more than enough visitors. You can also use this traffic to shorten the turnaround time of your tests. You can even run additional tests.

Testing on specific holidays

Apart from using the extra traffic to run more tests, you could also consider setting up tests during specific holidays. For example: Does your website convert better when you use a red-and-green theme around Christmas? Or maybe a red Santa in the top right corner of your website can convince visitors to buy more? While the results of this test may, and may not, apply to other holidays (more on this later), they will almost certainly apply to the same holiday next year. To make sure, you can always run the test again for a small percentage of your traffic the following year to see if the results are still the same.

Predict high visitor amounts

Some traffic spikes (such as those caused by viral content hits) are almost impossible to predict. However, annual events such as Black Friday or Christmas can be anticipated well in advance. Try using the following techniques to predict time periods when you can expect high volumes of traffic and sales. Knowing well in advance when there will be high visitor numbers will enable you to set up experiments and ensure that they run smoothly.

Holidays and events

Holidays and events refer not only to (inter)national holidays but also to specific events such as Black Friday and Cyber Monday. Apart from these generic holidays and events, you can also keep an eye on industry-related matters. Examples of this include the Football World Cup or the Eurovision Song Contest.

Marketing calendar

Your company's marketing department will be able to provide you with this information. By using marketing data, you will know when major marketing campaigns are being launched by your company, and you will be able to predict peaks in traffic with great accuracy.

Previous years

By analyzing existing data such as demographics of returning customers or sales data in Google Analytics, you should be able to see cyclical spikes in traffic. While most of these spikes are likely tied to a specific event (such as Black Friday, Mother's Day, Easter) or a promotion, some spikes may be unexpected. Apart from running additional A/B tests during this period, you could also consider setting up additional promotions based on which event causes the peaks in traffic.

How to minize the risk?

In these times of high visitor numbers, a lot is at stake. How can you minimize the risks that visitors will see a poorly functioning site or that your website will experience a decline in conversions?

Double quality assurance

Of course, you should always do proper quality control before your testing. However, with the risks associated with conversion testing during the holidays, I recommend that you add an extra round of quality control to your usual workflow. Make sure you view the variations on as many different devices and browsers as possible, make sure that different people approve the variations, etc.

Coordinate between teams

Also make sure that you coordinate your testing with different departments in your organization. For example, you don't want a specific campaign to go live that directly contradicts any changes in a test variation.

Spread out the experiments

By ensuring that you spread the experiments over time and divide them over the different pages of your website, you can kill two birds with one stone. First of all, you will maximize the number of tests you can run independently. As is known, you should normally perform multiple experiments running on the same page as a multi-variate test (MVT). Second, you spread the risk associated with such testing. Because not all of your visitors are viewing every type of page, parts of them won't be bothered by a broken or poorly performing variation in an experiment. Make sure that different experiments do not contradict each other (consider, for example, contradictory messages that are added).

Start small

Try running your experiment on a small portion of your traffic first. For example, assuming you have enough traffic on your website, you could start with 10%. After you have ~100 conversions across all variants you can decide to increase your traffic allocation. A noticeably high (negative) difference in conversion rates between variations (or in certain segments such as devices or browsers) could indicate additional quality control.

Don't stop too soon

However, be careful not to stop testing too early, as this could lead to an increase in Type 1 and Type 2 errors. Also, although it is technically possible to change an ongoing experiment, it is strongly advised not to do so.

Empower others

Another way to minimize risk is to empower other employees in your organization. For example, you could give a junior conversion optimization specialist access to your testing tool. He/she would then only have access to pause an experiment. If something goes wrong with an experiment and you are unable to pause it yourself (for example because you have a long flight or have become ill), your colleague could temporarily stop the test.

Keep an eye on the results

Keep a close eye on the conversions of different segments on your site. Make sure you actively monitor both the results of your test and the website as a whole. For example, try creating custom dashboards based on your web analytics information so that you can monitor conversions of different segments on your site. This will allow you to spot problems as quickly as possible and intervene if things go wrong.

Conclusion

During the holidays, both the opportunities and risks of conversion testing are high. By ensuring that risks are minimized, for example by using the techniques described, you can benefit from the benefits that testing brings during these fun days.