Using A/B Testing Goals Properly

Do you use the A/B testing tool or a web analysis tool to analyze your experiments? What types of goals are there and how do you deal with them? Learn how to determine which events on a website are suitable as conversion goals and how to store them as smartly as possible.

What is a conversion goal?

Visitors exhibit behavior on a website. For example, consider views of certain pages, clicks on elements, adding items to a shopping cart, etc. In order to determine whether this behavior changes as a result of the change in the A/B testing variant, these actions must be saved. When you analyze the A/B test using the testing tool, you save these actions as a conversion goal. However, if you analyze the experiment using a web analytics tool such as Google Analytics, you will save these actions as page views, goals, transactions, or events.

How do goals differ from KPIs?

Key Performance Indicators (KPI) are measurable values that show progress towards business goals. Some of the actions that visitors perform on a website can be seen as (key) performance indicators. It is important that these actions are saved, otherwise they are not measurable. In the context of an A/B test, this is saved in conversion goals or in the web analysis tool.

How do you determine relevant A/B test goals?

Before starting an experiment, you need to determine what relevant goals are for that experiment. Three important issues play a role here, namely the Overall Evaluation Criterion, the hypothesis, and so-called guardrail metrics.

Overall Evaluation Critereon

The concept of an Overall Evaluation Critereon (OEC) was introduced by Ronny Kohavi, who is one of the founders of online conversion optimization through online experiments. Ronny describes an OEC as a "suitable (often composite) quantitative value for experiments that is in line with the strategic goals". He further states that drawing up an OEC "may seem to be a simple process, but it is difficult to determine which short-term values are the best predictors of long-term outcomes". In practice, this means that many organizations choose to choose metrics such as 'revenue per user' as their OEC, which is already a lot better thought out than a choice for conversion rate or average order value. It is therefore important to measure the chosen value(s) that make up the OEC as testing goals.

Hypothesis

Another important point to keep in mind is the hypothesis. This sets out the expected impact of the change in the experiment on the behavior of visitors. If the number of items added to the shopping cart is part of the hypothesis, then the 'Call to action' button must of course also be measured as a goal. By taking the hypothesis into account when determining the testing goals, you also reduce the chance of cherry picking. This is a process where you search endlessly through the results looking for a target where a statistical difference has occurred.

Guardrail metrics

Finally, guardrail metrics are also important. You keep track of these values to determine whether the experiment is proceeding as expected. The most important, and one of the most practical, guardrail metric is the Sample Ratio Mismatch (SRM). However, in the context of this article, the SRM should be viewed as an analysis of stored values, rather than as a standalone value in itself. For example, you can do an SRM analysis to see whether the number of users is distributed evenly enough across the variants. However, it is also possible to perform an SRM analysis on the number of times a video is viewed, a thumbnail is clicked, a size chart is viewed, etc. In order to perform these analyses, it is necessary that such values are stored during the duration of the experiment.

Save goals in a testing tool or web analysis tool?

It is important to determine in which tool you will analyze the results before starting the experiment. After all, the testing goals must be properly measured in the chosen tool from the first visitor who becomes part of the experiment. Both solutions have their own advantages and disadvantages, which will be discussed below.

Testing tool

You can choose to save your goals in a testing tool such as Convert, VWO, Adobe Target, or Optimizely. However, this poses some potential challenges.

• Mandatory pre-setting: It often happens that after completing the test there is still a need for additional data. For example, consider the clicks on a certain element. If you have not set this in advance in the testing tool, this data will no longer be available in most cases.
• Limited number of targets: Keep in mind that you can usually only save a limited number of testing targets. For example, this makes it difficult to set all relevant values that can be used as guardrail metrics.
• Differences with web analysis: In almost all cases, in addition to the testing tool, a web analysis tool will also be used within the organization. Because the data in the testing tool deviates from that in the web analysis tool (due to different definitions of 'users' or 'sessions', for example), you are potentially making things very difficult for yourself. You will then have to explain why these values differ, and I can tell you from experience that this is actually impossible.
• No filtering possible: During the analysis of an A/B test you will regularly have the need to apply filtering. Consider, for example, solving outliers. Such filtering is not available in virtually any testing tool, limiting your ability to perform a solid analysis.
• Sometimes no segmentation: In some testing tools it is not possible to segment the results afterwards. In other tools this is possible, but sometimes as part of an expensive package option or only when you have defined the segments before starting the test. If you cannot segment the test results, you will not be able to make statements per device group or per browser, for example.

Web analytics tool

An alternative is to save the goals in a web analytics tool such as Google Analytics, Adobe Analytics, or Matomo. However, this also brings its own peculiarities.

• Mandatory pre-setting: Just like when saving via the testing tool, it must also be ensured that the goals are pre-set. The advantage of saving and analyzing with a web analysis tool is that many things will already be set in advance. This way (if all goes well) all page views are already saved correctly and, if all goes well, this also applies to important events.
• Data quality often low: This is not a challenge specific to online experiments, but one that you will have to deal with when you analyze your tests via a web analysis tool. In that situation, it is more important than before that things such as duplicate transactions, bot filtering, and tracking settings are configured correctly.
• Link testing tool: This applies when you are not using two components from the same suite. Consider, for example, VWO + Google Analytics or Adobe Target + Google Analytics, instead of Adobe Target + Adobe Analytics. An incorrect link between the tools can ensure that the data is not stored correctly in the web analysis tool.

What types of A/B test goals are there?

Whether you save the goals in the testing tool or in the web analysis tool, you will have to deal with different types of goals. The most commonly used ones will be explained below.

Page visit goals

The most commonly used form is visit (page view) goals. Practically speaking, these are pages that are visited by the visitor. When saving these goals, keep in mind that you choose the correct matching. For example, a simple match usually compares the URL with any parameters after a ? are ignored, while a starts with grabs all URLs that start with a certain string of characters. In more complex cases (such as matching all product pages) it is often necessary to work with regular expressions. A useful tool to check whether you have created such a regular expression correctly is Debuggex. When determining the URLs for your visit purposes, also make sure that you preferably use the website itself as a source. Web analytics platforms sometimes display pre-filtered data on behalf of you, which may cause you to overlook things such as parameters.

Clicks goals

This may involve clicks on a wide range of elements. For example, think of buttons, links, icons, images, sliders, etc. To be able to select such elements in a targeted manner, it is important to use the correct CSS selector. Don't forget to test these selectors thoroughly. I regularly encounter selectors that are set too specifically (or not specifically enough), causing the clicks to be measured incorrectly. If you want to analyze the clicks in your web analysis tool, it is probably the best solution to save them using your tag manager. Finally, keep in mind that the element whose clicks you want to measure must already be present in the DOM when you create the eventListener.

Revenue goals

In many cases, sales goals will already be measured in the web analysis tool, for example using Enhanced Ecommerce. However, to measure a turnover goal in the testing tool, extra code often needs to be added to the snippet on the thank you page. When using the web analytics tool to look at revenue targets, keep in mind that no duplicate transactions are stored, and if applicable, that items such as returns, shipping costs, and taxes are also processed properly.

Conclusie

Saving testing targets is an important condition for good experiments. Make a conscious and well-founded choice between saving in the testing tool or in the web analysis tool. Finally, ensure that the correct types of goals are selected, and that the goals are set correctly.