Our SaaS solution automatically improves the your website’s loading speed. Our engine works like a proxy in the Cloud, rewriting your page code on the fly to optimise your website’s loading time on desktop and mobile, for all your users, whatever the browsing context. But in practice, apart from improving the speed perceived by your visitors, how can you measure the impact of optimising loading speed on your conversions? In this article, we look in detail at how to test the effects of Fasterize to evaluate the ROI of our solution.
Measuring the technical and business impact of Fasterize
Do you tend towards Doubting Thomas or Lord Kelvin? You’re quite right! The effects of the tools you deploy and the techniques you apply must be measurable. Here are two scenarios for assessing the impact of our SaaS solution:
IF YOUR TEST DOES NOT INCLUDE BUSINESS KPIS
You don’t need to do anything.
We automatically observe a non-optimised part of your website’s traffic and compare the technical metrics obtained from the 2 populations with and without optimisation. We then make these comparisons available to you in the form of dashboard graphs.
IF YOUR TEST INCLUDES BUSINESS KPIS
If your A/B test includes business measurements, we always make this separation between optimised and non-optimised traffic, and you need to set up tracking in addition.
This is usually done via your Tag Management System (Google Tag Manager, Commanders Act, etc.). On our Support page, you will find all the information you need to easily implement an A/B test on Google Analytics with Google Tag Manager, for business tracking once Fasterize has been deployed.
You can also do this without Google Tag Manager, for example by directly modifying the Google Analytics code as described here. It’s just as easy to activate, taking less than 10 minutes.
Once tracking is in place, all you have to do is use the customised dimension that has just been created to add new segments and compare users with and without Fasterize.
Expert tip: an A/B business test requires a number of parameters to be taken into account. Here are a few things to bear in mind when analysing the results:
Parameters to take into account for an A/B business test
Sampling
Sampling involves taking only a portion of the data points to establish metrics. In our case, it is useful in two ways:
- to measure loading times in Google Analytics, Google measures the loading times of just a few of the pages viewed by your users. During A/B testing, you may have measurements on a page for optimised users and no measurements on the same page for non-optimised users, which renders the comparison void;
- if your audience is large, Google only keeps a fraction of the measurements to calculate the conversion rate (the limit is 250,000 sessions), which can lead to aberrations. The easiest way to do this is to aggregate the data yourself.
MEDIAN VS AVERAGE TO ASSESS LOADING TIMES
When analysing your loading time, focus on the median rather than the average. The median is not influenced by extreme and minority values, whereas the mean is to a large extent. The median is therefore more representative of the reality of most Internet users.
You can also study the 80th, 90th and 95th percentiles (the median is the 50th percentile). To illustrate: let’s imagine that 9 people load a site in 5 seconds, but that one person loads the same site in 100 seconds: the average would then be 14.5 seconds, while the median remains at 5 seconds. You can now see why the median is more relevant than the average.
THE DURATION OF YOUR TEST
Your test must be carried out over a sufficient period of time. This will enable us to stabilise it and collect data from all the populations:
- new users receiving the optimised website;
- new users receiving the non-optimised website;
- returning visitors receiving the optimised website;
- returning visitors receiving the non-optimised website.
At the start of the test, some visitors are placed in the ‘optimised’ category even though they have had a previous experience on a non-optimised version. It therefore takes some time for these populations to stabilise and become clearly separated.
This time varies depending on the website being monitored. It depends on the audience and how long it takes to make a decision before a user converts, because we don’t spend same amount of time thinking about buying clothes or a computer.
SAMPLE SIZE
To find out whether your A/B test is relevant and to determine whether there is a correlation between website acceleration and an increase in conversions, for example, you need to perform the Chi2 test. The question is whether you have enough data to check that the A/B test is reliable and that the result correlates well with the change you have made to your website (improving page speed). You can easily perform the Chi2 test with this online calculator.
You now know everything you need to do to carry out an A/B test that will enable you to assess your ROI in real terms after improving the loading speed of your web pages; and some of our customers have seen their conversions increase by more than 30%!
Do you need help deploying an A/B test,
or would you like to find out more about our solution?
