4.11. A/B Testing

Owned by Karol Schönsee
Last updated: Oct 20, 2022
4 min read

A/B testing is a method of comparing two versions of a webpage or app against each other to determine which one performs better. A/B testing is essentially an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal.

 

A/B Testing vs Magento

In order to be able to render server-side different variants of the pages for different users, we would have to multiply the amount items being cached, which brings up performance and infrastructure cost topics and complexity which is apparently not worth it. Therefore, the only alternative seems to be client-side-based testing, which relies on loading additional scripts/CSS for the users, to alter their view to the changes defined in the experiment variants. Because our customers use Google solutions for tracking, like Google Analytics and Google Tag Manager, the natural choice seems to be the Google Optimize platform, especially since it is automatically sending events to Google Analytics property and its basic version is free.

Google Optimize

In order to set up Google Optimize, the following prerequisites are needed.

  1. A Google Analytics account and property.

  2. Google Analytics integrated into your project (via GTM)

  3. Google Optimize Account https://optimize.google.com/

There are many tutorials on how to create a Google Optimize account and connect it to Google Analytics, and you can follow one of these:

https://www.youtube.com/watch?v=6RMpCNrc7Ds

https://www.youtube.com/watch?v=ufczfAZGUjM

What’s important - please skip the script placement tutorials, as we have our own module providing this setup by default.

MageSuite module for Google Optimize integration

In order to integrate Google Optimize properly, we’ve prepared a module, that simplifies configuration and automatically puts initialization and anti-flickering scripts in proper order and place of the DOM.

Add this to your project’s composer.json:
"creativestyle/magesuite-google-optimize-integration": ^1.0.0

The module requires free Google Tag Manager for Magento 2 by MagePal, because the most of our projects use it and we have to place Optimize scripts in relation to dataLayer initialization.

This module allows integration of the Google Optimize script, as described in the official documentation:

It also offers additional features which are described below.

After the module installation, the following options appear in the admin panel:
Stores → Configuration → MageSuite → Google Optimize

It consists of two separate sections. The first one allows integrating Google Optimize script directly in your project (without GTM involvement). It is as simple as enabling it and defining your GO container name taken from the GO account. Async script is optional but not recommended - enabling this option will increase performance, but also delay script execution and your experiments will be loaded later, which might not be intended. See attached documentation from the current section for details.

 

The anti-flicker script is responsible for hiding the page content until the GO experiment is fully loaded and ready to be shown. Optimize Container ID defines what the anti-flicker script should wait for. If GO is integrated directly, using “Integration” settings, then here we have to pass GO container ID. If GO setup is based on Google Tag Manager, here we need to pass GTM container ID instead, to make it wait for GTM to be loaded.

Use anti-flicker wisely, as it is not the best impression for the users to see the loading process before page starts. Script should be used only if you use asynchronous GO setup (direct async integration or GTM integration) and your experiments are changing the layout significantly, what might be noticeable for the users.

By default, the anti-flicker script shows a blank page before the content is loaded. We’ve extended that to make it “more friendly”, displaying a progress indicator instead.
Also, we’ve introduced a possibility to restrict the anti-flicker script execution only to specific pages and media queries.

Include in paths option stands for part of the URLs, like “/cms-page” or Magento controllers, like “catalog_product_view” to be able to target specific page types.

Include for provided match media stands for media query, that will allow the script to run if matches. For example: “(max-width: 600px)” or “(min-width: 768px) and (max-width: 1024px)” what would stand for tablets only.

Info about GDPR/Consent Management involvement
Google Optimize stores data in cookie and uses it to get proper experiments and remember it, that probably should be permitted by the customer. Our module does not provide integration, but the file structure allows overwriting a part of the code, to “connect” third parties like User Centrics and block the script execution until consent is given. This should be done by dev team.

Reading Experiments data in Google Analytics

Creating Reports that involve Google Optimize experiments data requires knowledge about Google Analytics in general - creating reports, custom segments, dimensions, explorations. However, the following videos cover at least basic setup, to be able to read the data:

When using Google Analytics Universal Analytics:

When using Google Analytics 4:

Additional resources