Did you know that, while your website is loading content, every second of blank page = 1000 points deducted on the Speed Index? During an A/B test, it’s inevitable that a website will take longer to display. We spoke with independent web performance expert Jean-Pierre Vincent, who highlighted that, even though these tests are often necessary and beneficial to marketing, they ought to be handled with care because of their impact on loading times.
Fasterize: How does A/B testing affect web performance?
Jean-Pierre Vincent: Just like any third-party application, an A/B test will have an impact on the performance budget. Throughout my tests in ADSL on WebPageTest, I found that every single A/B test increased loading times by a minimum of 1 second. What this means is that, when you decide to launch the test, it’s important to bear in mind that you risk of losing traffic and conversions (Editor’s note: every second = 12% fewer conversions). It’s certainly worth it – but just because an A/B test is often essential to improving a website, it doesn’t mean you should take it lightly.
It’s also risky to rely on a service provider’s server to display your webpage: even though it guarantees high availability, any potential downtime is added to your own.
Fz: Why are loading times higher during A/B tests?
J-P. V.: To avoid flickering, the user will see a blank page while the page defined as part of the A/B test executes and displays. In other words, the loading time is purposely increased to avoid serving a first version of the page that would then disappear and be replaced by the version selected for the test. Failing this, users might perceive this flickering effect as an issue with the website.
Thanks to the development of turnkey SaaS solutions over the last ten years, A/B tests have become highly straightforward to execute. This saves time for the technical teams – who no longer have to intervene in the deployment of these tests; and for the business teams (marketing, e-commerce, content, UX, UI, etc.) – who can run the tests they need without going through IT. The downside is that loading times often suffer as a result.
Fz: How can this be rectified?
J-P. V.: In terms of actually designing an A/B test, you would have to incorporate best practices in web performance when writing the code itself. Today, these assets are sourced externally through third-party solutions, so they don’t go through the usual checks and validation processes of the IT department.
A/B testing should ideally be performed server-side – as it used to be ten years ago. The server was able to see the user arrive with their cookies and decide whether to display version A or B. As I mentioned earlier, however, this procedure places other constraints on marketing and IT.
To avoid relying on the provider’s server, a less heavy option is to host your own script using a reverse-proxy solution with cache, for example. Finally, a good A/B testing solution must provide a timeout option to set maximum loading times for certain users.
Fz: Have you ever encountered extreme slowdown?
J-P. V : Yes, while performing an A/B test for a client’s website called lesfurets.com, it took 3 seconds for pages to display instead of 1! The designs of the A and B versions were so different that we couldn’t just switch pages live: we needed to redirect. The user had to load the page twice – the “normal” version (which was never displayed) and the “test” version – so web performance took a massive hit, and SEO traffic fell by 50%.
Fz: Are there any workarounds?
J-P. V.: Seeing as that particular test involved two radically different UIs, we ultimately performed it server-side with the help of software that was specially developed in house, as opposed to a turnkey JavaScript solution. This required a significant development budget, but my client had the resources and wanted to go down this route.
In this type of scenario, it would have been difficult to automate the front-end optimisations. On top of the development budget for the A/B test, we needed to set a performance budget with its own resource allocation, since the optimisations had to be performed “by hand”, in particular:
- image compression (with these different UIs, we couldn’t adopt a responsive design approach and resize them, we had to provide separate UIs for different screen sizes)
- font-display
- font conversion to WOFF2
- font lazy-loading and subsetting, etc.
Fz: In such cases, what’s the best way to manage your budget?
J-P. V.: Given that the A/B test was performed server-side as opposed to client-side, the IT teams made the decision to invest heavily in training, adopt continuous monitoring, implement supervision and ensure knowledge transfer. In this case, the company could afford it. Otherwise, even as part of an A/B test, a web performance automation tool is still a viable and cost-effective solution to improving loading times – so long as it is deployed client-side.
Would you like to learn more about “web performance and digital strategy”?
