Maximizing engagement, conversions and revenues is part of our shared credos with Kameleoon! We invited this A/B testing expert who reminds us how this technique can optimize the performance of a site, how to deploy it and measure its effectiveness. Meeting with Rémi Raffard, Head of Customer Success & Professional Services.
For what purpose can a brand be led to carry out an A/B test? What are the tests focused on, and with which KPIs should the results be measured?
A/B testing is the first step towards a digital optimization strategy. It allows brands to base their decisions on concrete statistical facts by relying on visitor behavior to improve site usability and overall navigation.
Tests can be very varied and must above all be adapted to the business objectives of the company, and to the expectations of the target audience:
- visitor engagement,
- conversion (purchase, request for information, etc.)
- loyalty and retention,
- optimizing campaign costs…
All of these goals can be tracked by a large number of indicators : add-to-cart rate, increase in conversions, time spent on pages, pages viewed, clicks on CTAs, etc. It is even possible, with some solutions, to create custom data to very precisely track the benefits brought by the test throughout the conversion funnel. Finally, it is also necessary to track KPIs such as cart abandonment or bounce rate to ensure that the test does not deteriorate performance in the event of a bad hypothesis.
More broadly, A/B testing helps establish a culture of optimization within the company and arbitrate important decisions: changing wording, visual identity or managing product sorting. Relying on statistical facts helps avoid sterile debates and make sometimes bold decisions that can significantly boost the company’s turnover.
Following on from A/B testing, personalization makes it possible to contextualize the experience of each visitor in a segmented way, or individually with AI.
How to ensure that the conclusion of an A/B test is reliable and based on a sufficiently large population?
To calculate the level of traffic required for a test, 4 indicators must be taken into account:
- the conversion rate of your reference (e.g. 3%),
- the minimum expected effect on the conversion rate by the variation (the higher the percentage, the fewer visits will be required, because it is easier to analyze a big change than a small variation),
- the confidence threshold for which an effect is detected,
- the confidence threshold for which an effect is detected when it does not exist (in other words, a false positive. But this remains quite rare, around a 5% chance).
With these 4 parameters, it is possible to perform a calculation to define the minimum traffic necessary for a test.
Let’s take an example. Your baseline conversion rate is 30% and the minimum expected effect is 20%. These are high rates. Therefore, the number of visits needed to evaluate the performance of your variations will not need to be very large, because you will be able to quickly see the variations compared to your baseline page.
So, low traffic sites can also do A/B testing, provided that the tests are focused on modifications for which the conversion gains are potentially substantial. For example: banners, a newsletter subscription, a form to fill out for B2B sites with a qualified audience, etc. We can also consider testing the conversion funnel provided that the potential for improvement is significant.
On the other hand, for tests with potentially more limited gains, such as those on the conversion funnel or adding to the basket, the traffic needed to validate a result requires a significant number of visits. They are reserved for e-retailers or high-traffic media.
More broadly, what are the best practices to apply and the pitfalls to avoid for a successful A/B test?
First of all, you need to install a culture of experimentation and enter into a logic of iteration. Indeed, by navigating blindly without determining objectives or setting up a roadmap, there is a risk that the tests will be counterproductive and degrade the experience on your site as well as performance.
To do this, several elements must be taken into account:
- Rely on the analysis of visitor behavior and your site’s performance KPIs . Each test must meet an identified need and must validate your hypothesis.
- Start with simple tests based on specific hypotheses derived from your site analysis.
- Establish processes and prioritize tests to create a coherent roadmap, and eliminate tests that have no impact on your business.
- Define the KPIs to track in order to analyze the performance of your test and achieve your objectives.
- Knowing how to communicate results , whether positive or negative, to enable the entire company to progress and feel involved in the continuous improvement process.
- And above all, accept failure , which does not mean that the teams made bad choices, on the contrary. This is essential to learn from and steer the strategy in the right direction.
Once this dynamic is anchored, teams can launch tests that are relevant to their needs and develop use cases that truly match their audience. It will then be enough to continue these processes rigorously to maintain a strategy consistent with your business objectives.
You should know that Kameleoon provides solutions to address technical difficulties that may arise when using the results of an A/B test, in particular:
- a “ history reconciliation ” API, to be able to count all visits made to the same variation by a user from different devices;
- the implementation of a server-side cookie to bypass the drawbacks of Apple’s Intelligent Tracking Prevention (technology that can disrupt analytics tools because it restricts the use of client-side storage resources).
How to implement A/B testing, and what should you pay attention to to maintain loading speed?
To get the most out of the benefits of A/B testing and ensure an optimal user experience, there are a few technical considerations.
It is important not to interfere with the visitor’s journey when assigning the variation , as this could cause a significant bias in behaviors, and therefore disadvantage one of the variations. A well-known example of interference is the flicker effect , a visual effect that changes the page a few seconds after it loads, and which Kameleoon helps to avoid.
Technically, since an A/B test script determines what content should be displayed in the browser, it needs to run early but without blocking other resources. We recommend placing it at the very top of the code , and we favor blocking asynchronous loading for fast page loading without flickering . Also, only elements used by the client are included in the script (e.g. if the client doesn’t do any customization, it’s not in the script).
Hosting the script as well as caching the page’s static resources saves time on DNS resolution, further improving loading speed.
Finally, still with the aim of an optimal user experience, Kameleoon offers the possibility of applying server-side customization . This hybrid technique allows you to optimize loading speed while managing triggers and analytics on the client side.
What are some of the most impressive conversion increases you’ve seen? Are there any optimizations that “win every time”?
Our clients in the E-commerce sector use A/B testing to achieve “quick wins” as well as more structuring projects across their entire conversion funnel. For example, we see tests on CTAs that can lead to an improvement of more than 60% in conversions !
The important thing to remember is that there is no “winning model”, since objectives and use cases vary from one brand to another. The goal of experimentation and personalization is above all to adapt to visitor behavior, and each use case must be designed for the audience in question.
The key to success actually lies in test preparation:
- analysis of visitor behavior ,
- formulation of hypotheses ,
- prioritization of experiences , etc.
These are all elements that make it possible to define a framework conducive to achieving the set objectives.
However, e-commerce sites have several common specificities and it is essential to continuously optimize:
- The homepage : this is the showcase of your e-commerce site. The challenge here is to highlight the wide variety of products present on the site while offering a page that makes you want to continue the visit, and taking into account everyone’s preferences. To do this, there are many elements to test: presentation of the different offers, product categories, wording, CTA, images , etc.
- Product pages : This is where visitors add products to their cart. You need to make sure the product is highlighted and all the reassurance elements are present to maximize conversion rates. It is important to regularly test product information, price, delivery, payment facilities, CTA visibility, photos, etc.
- Category pages : These pages allow visitors to browse and discover new products. It is important to optimize these pages so that customers can easily find the item they are looking for and add it to their cart. To do this, it is important to optimize the display of products, but also the product sorting algorithms. For example, we can imagine offering a sort by price, or rather by recency, etc.
More generally, e-commerce sites must test all elements of the purchasing journey and optimize the experience to encourage as many visitors as possible to convert and return to the site regularly! These optimizations must focus on choices in terms of interface design as well as on the fluidity of the customer journey and on loading speed – a subject on which Fasterize’s expertise is well known.
Rémi Raffard – Head of Customer Success & Professional Services
With over 15 years of experience in digital innovation, new technologies and customer relationship and success management, Rémi has developed dual business and technical skills. At Kameleoon, Rémi manages the Customer Success and Professional Services teams and ensures the quality of the service delivered, the continuous improvement of processes and organization as well as the development of consultants’ skills on a daily basis, so that Kameleoon offers an incomparable customer experience.