Have you tried to measure the speed of your website with Test My Site?
When Google released a new version of its web performance testing tool, one of our web performance experts, Anthony Barré, looked into it to decipher the testing conditions and analyze the results.
Let’s observe why it is mainly a tool for marketing, with limitations when it comes to the web performance community.
How to understand the test conditions
In an article previously published on our blog (in French), we explained how to measure the business impact of mobile web performance with Test My Site. How are the tests done? Where does the data presented in the reports come from?
First of all, here below is a reminder of what Test My Site is in the Google environment.
LIGHTHOUSE AND CRUX DATA
Test My Site uses the same tools as Page Speed Insight:
- Lighthouse (Synthetic Monitoring). This is the Chrome extension which analyzes in detail the web pages to establish audits for performance, PWA, accessibility, SEO ... The generated reports are intended for a well-informed audience, they are established on the basis of two successive navigations (run).
- The actual data of page loading collected anonymously by Google via Chrome (Real User Monitoring).
This information is grouped together in a database called CrUX (ChRome User eXperience). It currently includes more than 5 million high traffic websites.
If your website is not in this database, Test My Site (alike Page Speed Insight) can still access you website when executed.
TEST MY SITE LOCALISATION AND CONNECTION CONDITIONS
We tested the tool from our own computers (in France), and requested results for different countries (Spain, France, Mexico), and we noticed that all the tests were carried out from the Google data center in Germany. Here below is an example using bikeshop.es website (note that the proposals vary depending on the tested website):
We discovered this by checking which location was activated via our CDN: it was systematically from Frankfurt. We completed these tests by initiating them from the US, and the results came from the US.
We were able to conclude that the tests are performed from the area where they are activated - unlike WebPageTest which works with agents located in the target countries.
Importantly, these Lighthouse tests are done with a native connection and the results are extrapolated according to an algorithm, which can induce a margin of error and which differs from the Lighthouse tests launched by WebPageTest for example (where the connection is simulated).
As for the number of runs, there were systematically 2 to 3 in our tests. Too little. It would take a dozen to ensure that the loading times are reliable, especially because the unique data displayed in the results does not permit to draw conclusions about the web performance of a site. This would require several web performance indicators (Speed Index, Start Render, Time To First Byte ...).
The drawback of relying on a single indicator is that it offers only a partial result of the performance, ignoring the multiplicity and complexity of the factors involved in the perception of a page’s loading time. In order to avoid this bias, Fasterize takes into account 7 to 8 metrics.
For all these reasons, Test My Site is a friendly tool for performance measures in the context of a SEO, marketing or business strategy, but from a technical point of view the results may be restrictive.
How to understand the results of Test My Site?
TEST MY SITE SPEED AUDIT
If you're part of the Google panel, the first part of the Test My Site results page uses the CrUX data (if you're not, go to the last paragraph of this article). It shows the site speed based on the First Contenftul Paint (FCP), i.e. the time it takes for the page to start appearing in the browser. This indicator is updated once a month and corresponds to the 90th percentile of the data collected by Google. It is an equivalent of the WebPageTest Start Render that we use in the analysis results of our platform.
Note that the first version of Test My Site used the Speed Index as a speed indicator but always under the term "loading time”.
As previously mentioned, CrUX data is collected across multiple countries, which allows the page display speed to be measured according to the location. These are real data reported by Chrome from a user panel, not synthetic tests for which we would have chosen a speed, location and navigation conditions.
Then what about the Test My Site Rating indicator? It measures the website speed based on Google-defined segments: Fast: 0 to 1 second / Average: 1 to 2.5 / Slow: 2.5 seconds or more.
As for the Monthly Trend, it is the comparison with the previous period.
THE “BENCHMARK AND EVALUATE” SECTION
- This section compares the speed of the tested website with that of other sites - up to 10. For this comparison module, Test My Site always relies on:
the real Google CruX database of 5 million websites,
- and the FCP, combined with DOMContentLoaded and onload (this explains why the ranking may not respect the ascending order of the numbers displayed in the Site Speed column).
For a quick competitive analysis by Marketing or Sales teams, this feature is very interesting. However, for methodology reasons (in particular the fact we have only one metric), the tool remains too limited to address technical needs.
In the latter case, more advanced measurement tools such as WebPageTest are preferable.
EVALUATION OF THE BUSINESS IMPACT OF A FASTER WEBSITE
For this module, already mentioned on our blog, the analysis is based on 383,000 unique profiles retrieved via Google Analytics over a year worldwide.
HOW TO OPTIMIZE YOUR WEBSITE PAGES
While the time indicated in the first module at the top of the page is based on real data from the entire website, the one displayed here corresponds to a single page tested by Lighthouse. It is therefore necessary to put things in perspective. As mentioned, very few runs are done, for a single context and only the FCP is taken into account - which is always less representative than the Speed Index or the Time To Interactive to interpret a browsing experience.
Additionally, our tests were carried out from Germany, which is also different to the actual conditions that we have in France, as far as we are concerned.
In short, to estimate the performance of your website, the data announced at the top of the page in the 1st results module is the most relevant.
On the other hand, the optimization recommendations remain interesting even if they are common to all. They are (also) those of Lighthouse and they are a good basis for reflection. Be careful, however, not to follow them to the letter! It's better to have an expert review before you start, to make sure they're right for your case, and how to deploy them to prevent them from being counterproductive if not applied wisely.
IF YOU’RE NOT IN THE PANEL
There are a few limitations of Test My Site for a website that isn't part of the panel of 5 million websites selected by Google: the data will unfortunately be less complete.
The generated report then offers only the 4G data and indicates the page speed only according to Lighthouse (in this case, the data is to be taken into account with caution).
The "small" websites also do not have access to the comparison module, nor to the calculation of business impacts.Thus, for websites that are not part of the Google panel (Internet has billions of websites …), Test My Site will be less interesting.
In conclusion, Test My Site has improved a lot compared to the previous version, notably because of the integration of real navigation data (CrUX), and the consideration of the FCP instead of the loading time (window. onload).
This tool offered by Google has the advantage of being free and easily accessible. It is convenient and recommended for business teams for an overview of web performance KPIs, but for precise technical data intended as a base for optimizing web performance it is preferable to rely on more advanced tools and experts’ recommendations.
What to remember:
- The tests are carried out from the geographical area of the user who initiates them.
- For measuring, Test My Site uses CrUX data if available, and if not it uses a Lighthouse test.
- If the CrUX data is available, the website speed indication is the 90th percentile of the FCP (90% of the users of the CrUX target population have a lower FCP). This information must therefore be put into perspective.
- For Lighthouse, the speed of the website is the FCP of the corresponding test.
- Optimization recommendations are based on Lighthouse data of one-page tests. They are interesting but remain generic.
- If your website is not part of the Google panel (CrUX), the available data is more limited - including no benchmark or impact calculation.