Test My Site: how to use it for your mobile loading times?

Have you ever tried to measure your site speed with Google Test My Site?
Anthony Barré, our CTO and webperf expert, took a look at it to decipher the test conditions and analyze the results.
In this article, we will see in detail how it is an interesting marketing tool, and its limits for an advanced web performance analysis.

Google Test My Site : the test conditions explained

In an article previously published on our blog (in French), we explained how to measure the business impact of mobile web performance with Google Test My Site. How are the tests done? Where does the data presented in the reports come from?
First of all, here below is a reminder of what Test My Site is in the Google environment.

Lighthouse and CrUX data

Google Test My Site uses the same tools as PageSpeed Insights:

Lighthouse (Synthetic Monitoring). This is the Chrome extension which analyzes in detail the web pages to establish audits for performance, PWA, accessibility, SEO ... The generated reports are intended for a well-informed audience, they are established on the basis of two successive navigations (run).
The Field data of page loading collected anonymously by Google via Chrome (CrUX report, Real User Monitoring).
This information is grouped together in a database called CrUX (ChRome User eXperience). CrUX currently includes more than 5 million high traffic websites. If your website is not in this database, Google Test My Site (alike PageSpeed Insights) can still access you website when executed.

Google Test My site localisation and tests conditions

We tested the tool from our own computers (in France), and requested results for different countries (Spain, France, Mexico), and we noticed that all the tests were carried out from the Google data center in Germany, by checking which location was activated via our CDN. It was systematically from Frankfurt. We completed these tests by initiating them from the US, and the results came from the US.

We were able to conclude that the tests are performed from the area where they are activated - unlike WebPageTest which works with agents located in the target countries.

Importantly: these Lighthouse tests are done with a native connection and the results are extrapolated according to an algorithm, which can induce a margin of error, and which differs from the Lighthouse tests launched by WebPageTest for example (where the connection is simulated).
As for the number of runs, there were systematically 2 to 3 in our tests. Too little. It would take a dozen to ensure that the loading times are reliable.

How to understand the results of Google Test My Site?

Once the test is launched, you can obtain the performance results by country, and by network quality:

The unique data displayed in the results does not permit to draw conclusions about the web performance of a site. This would require several web performance KPIs (Speed Index, Start Render, Time To First Byte ...).

The drawback of relying on a single indicator is that it offers only a partial result of the performance, ignoring the multiplicity and complexity of the factors involved in the perception of a page’s loading time. In order to avoid this bias, Fasterize takes into account 7 to 8 web performance metrics.
For all these reasons, Google Test My Site is a friendly tool for performance measures in the context of a SEO, marketing or business strategy, but from a technical point of view the results may be restrictive.

Google Test My Site Speed Audit

If your site has a small audience and is not part of the CrUX panel, you may not have access to all the data, but you can still fill in some of it manually to take advantage of the testing and comparison features we'll see next. In this case, the generated report shows the page speed only according to Lighthouse, without any field data.

The Site Speed data corresponds to the First Contentful Paint (or FCP, equivalent to Start Render) metric, collected via PageSpeed Insights, based on a 4G connection, and not broken down by country. Each test is likely to return different results;
the Fast / Medium / Slow score is assigned according to the speed of the page: Fast = between 0 and 2.5 s, Medium = between 2.5 seconds and 4 s, Slow = 4 s or more;
the quarterly trend compares the speed (Largest Contentful Paint) between the current quarter and the previous quarter, based on CrUX data;
above these results, Google offers optimization tips and best practices to improve performance. Even generic, they are still interesting. They are provided by Lighthouse, and they are a good basis for reflection. For the same reasons you shouldn't apply indiscriminately all PageSpeed Insights or Lighthouse recommendations, it is better get more advanced tools and web performance optimization experts’ recommendations before moving forward.

Test, compare and evaluate

Under the Test My Site results is a tab "Discover other speed measurement tools", to unfold to take advantage of features that will help you understand the business potential of faster pages.

You can :

test the speed of individual pages, and get some basic tips on how to improve their loading speed;
compare your speed with that of your competitors (here the FCP is combined with DOMContentLoaded and Onload, so the ranking may not be in ascending order of the numbers displayed in the Site Speed column);

Evaluate how much more you could generate by optimizing your speed, based on your traffic, your conversion rates and your average basket.

What to remember:

The tests are carried out from the geographical area of the user who initiates them.
For measuring, Test My Site uses CrUX data if available, and if not it uses a Lighthouse test.
If the CrUX data is available, the website speed indication is the 90th percentile of the FCP (90% of the users of the CrUX target population have a lower FCP). This information must therefore be put into perspective.
For Lighthouse, the speed of the website is the FCP of the corresponding test.
Optimization recommendations are based on Lighthouse data of one-page tests. They are interesting but remain generic.
If your site is not part of the CrUX panel, the proposed data are more limited, and they must be filled manually to benefit from the benchmark tools.