Your website was built for users who click, scroll and compare. Now, it also needs to be understood by AI agents capable of interpreting a page and acting on a user’s behalf.
This shift is already happening. Users are beginning to delegate tasks to AI assistants: finding a product, comparing offers, filling in a form, completing a booking. These agents don’t read a page the way a search engine does, they need to understand an interface, identify available actions and, in some cases, act directly.
Many websites have been optimised for the human experience, and rightly so. But they are not always designed with machine comprehension in mind. This is precisely what web.dev explores in its article Build agent-friendly websites. Here is what you need to take away from it, and how to go further.
How AI Agents Actually Read Your Pages
AI agents don’t rely on a single reading mode. Depending on the context, they can draw on four different representations of the same site.
- Screenshots: an agent analyses a screenshot using a vision model, identifying elements by their size, position and colour. This approach is slow, resource-intensive and unreliable whenever the interface is cluttered: pop-ups, cookie banners, components that only appear after user interaction.
- HTML: the agent parses the DOM to understand the hierarchy of headings, links, buttons and fields. A <button> correctly nested inside a product block is understood as an action on that product. A fake button coded as a <div> sends a weaker signal and may be ignored entirely. HTML is not just a technical substrate, it is a comprehension layer for search engines, assistants and agents alike.
- The accessibility tree: this synthesises the roles, names and states of interactive elements, allowing the agent to identify what is actionable without being distracted by visual noise. An accessible, well-structured site is naturally more readable for agents: good accessibility practices are also good practices for machines.
- The llms.txt: this is the fourth layer, and the highest. It goes beyond the scope of the web.dev article referenced above, but belongs to the same movement. Where HTML operates page by page, llms.txt operates at the scale of the entire site. Placed at the root (/llms.txt), this file gives an agent a structured navigation map: who you are, what your site contains, which pages matter most. Without it, the agent must crawl the site to piece together its structure. With it, that understanding is immediate. Lighthouse documentation is clear on this point: without a llms.txt, agents may take longer to explore a site and grasp its overall structure and main content.
- The llms.txt: this is the fourth layer, and the highest. It goes beyond the scope of the web.dev article referenced above, but belongs to the same movement. Where HTML operates page by page, llms.txt operates at the scale of the entire site. Placed at the root (/llms.txt), this file gives an agent a structured navigation map: who you are, what your site contains, which pages matter most. Without it, the agent must crawl the site to piece together its structure. With it, that understanding is immediate. Lighthouse documentation is clear on this point: without a llms.txt, agents may take longer to explore a site and grasp its overall structure and main content.
Worth clarifying. The llms.txt is not currently a signal factored into Google Search or its generative features (AI Overviews, AI Mode). However, it is relevant for automated browsing agents: Lighthouse, Google’s own auditing tool, treats it as a key element in its experimental category dedicated to agentic web. These recommendations come from different teams within Google, which can lead to conflicting messages depending on which documentation you consult. The llms.txt should therefore be treated as a lever for browsing agents, not as a ranking factor, at least for now.
Why High-Traffic Websites Are More at Risk
High-traffic websites typically stack multiple technical layers: legacy CMS, complex JavaScript front-ends, third-party widgets, A/B tests, marketing pop-ups, proprietary design systems. These layers can make a site opaque to agents, even when the experience feels perfectly smooth for a human user.
Some concrete examples: an action button rendered by JavaScript with no semantic markup, a form using placeholders instead of labels, a price displayed visually but absent from the HTML, customer reviews injected client-side and invisible to a machine, important content that only appears after user interaction. And at site level: no llms.txt to guide the agent, no Markdown versions of key pages.
The risk is real: a site that cannot be understood, compared, recommended or acted upon in AI-assisted journeys is a site that loses ground.
Key Optimisations to Become Agent-Friendly
Priority 1: make critical actions explicit. Use <button> for actions, <a> for links, add accessible labels. The “Add to cart” button must be identifiable as an action, associated with the right product and present in the HTML, not just in the visual interface.
Priority 2: expose business information in the HTML. Price, availability, reviews, options, delivery conditions, opening hours, FAQs: this data must not rely solely on a client-side widget or a user interaction to become visible.
Priority 3: strengthen structured data. Deploy or fix markup according to page type: Product, Review, FAQPage, BreadcrumbList, Organization, LocalBusiness, Article. Structured data significantly improves machine readability, without replacing HTML optimisations.
Priority 4: deploy a llms.txt. Create the file at the root of your site: a summary, links to key pages, concise descriptions. For each referenced URL, provide a .md version of the page: clean Markdown, free of navigation and JavaScript noise. That is the version browsing agents will read first. A well-built llms.txt changes how an agent approaches your site: instead of crawling, it understands.
Priority 5: audit the accessibility tree and critical journeys. Chrome DevTools lets you inspect roles, accessible names and states of interactive elements. The journeys to prioritise: product search, add to cart, quote request, booking, contact form. The goal is not only to verify these journeys work for a human user, but that they remain legible for a machine.
Fasterize: Optimise Your Site for Agents
The optimisations described in this article share one thing in common: they all require changes to the site’s code. On a high-traffic website, each change can depend on an IT ticket, a product decision or a release cycle. That is where Fasterize comes in, with two complementary products: EdgeSEO and EdgeSpeed.
- Structured data (EdgeSEO): deploy or fix Product, Review, FAQPage and other schemas directly on the relevant URLs, without touching the source code.
- JS reviews converted to HTML (EdgeSEO): review widgets injected client-side are invisible to agents. EdgeSEO makes this content available in the served HTML, accessible without JavaScript execution.
- llms.txt and html2Markdown (EdgeSEO): pages referenced in your llms.txt need to exist as clean Markdown. EdgeSEO’s html2Markdown feature automatically generates these .md versions, stripped of all navigation and JavaScript noise, and exposes them exclusively to browsing agents — without creating duplicate content for search engines.
- CLS optimisation (EdgeSpeed): layout shifts during page load (Cumulative Layout Shift) disrupt agents relying on screenshots or visual interface stability. EdgeSpeed corrects these instabilities at the Edge layer, without waiting for a front-end fix.
Fasterize does not replace a front-end overhaul or a long-term accessibility strategy. It dramatically shortens time to market on optimisations that today depend on IT, and enables them to be deployed at scale across high-traffic sites.
AI Agents Are Bringing Web Fundamentals Back to the Centre
Becoming agent-friendly does not mean starting from scratch. It means strengthening the fundamentals: semantic HTML, accessibility, structured data, correctly exposed content, clearly identifiable actions, and now a llms.txt that gives agents a readable map of your site.
AI agents are not replacing users. They are becoming a new interface between users and your site. To remain visible, understandable and actionable in these new AI-assisted journeys, your site needs to send clean signals, at page level and at site level. Fasterize is here to support you through this transformation, with one constant objective: a short time to market, measurable performance, and tangible business impact.
Is this your current focus?
