Try this: open your server logs for a single day and search for “bot”. In between Googlebot’s visits, you will find GPTBot, ClaudeBot, PerplexityBot and a handful of robots that nobody on the team can name. Some collect your pages to train a model. Others fetch them at the very moment a shopper asks ChatGPT a question.
Lumping them all together leads to one of two classic mistakes. Block everything, and disappear from assistants’ answers. Or let everything through, and serve thousands of pages without knowing what they are used for. There are in fact three families of AI crawlers, and each one calls for its own decision.
What exactly is an AI crawler?
It is a robot that browses a website’s pages on behalf of an artificial intelligence company: OpenAI, Anthropic, Perplexity, Google, Meta or Apple. Like Googlebot, it downloads the HTML of your pages. What changes is what it does with them: train a model, feed an assistant’s answer engine, or fetch a page live to answer a user.
The first use brings you neither a link nor a visit. The other two decide whether the assistant’s answer cites your site, or a competitor’s.
What are the three families of AI crawlers?
| Family | What the robot does | User agents (the robot’s signature) | What you gain |
|---|---|---|---|
| Training | Collects content that may be used to train models | GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) | Nothing directly: no link, no visit |
| Search | Builds the index from which the assistant picks its sources and citations | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity) | Being cited, with a link, in answers |
| User request | Fetches a page when a user asks a question or pastes a link | ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity) | An up-to-date answer, given to someone interested in your offer |
List checked on 30 October 2026 against the documentation from OpenAI, Anthropic and Perplexity.
What about Google?
Google has no separate AI crawler. AI Overviews and AI Mode draw on Googlebot’s index. Google-Extended, for its part, is not a robot but a robots.txt directive (known as a “token”): it lets you refuse to have your content used to train Gemini models and to ground their answers. Google states it in black and white in its documentation: Google-Extended has no effect on your presence in Google Search, nor on your ranking.
Blocking Google-Extended therefore does not take you out of AI Overviews. And blocking Googlebot to get out of them means getting out of Google altogether.
Do these robots cost more than they bring in?
In volume, they still weigh little. At the end of October 2026, we analysed a week of logs across around a hundred French e-commerce sites supported by Fasterize: fewer than 1 in 1,000 pages served went to an AI assistant (0.085%). Training robots, on the other hand, can spend far more time on the site, especially when navigation filters generate thousands of near-identical URLs.
But this small volume is valuable. A shopper who asks ChatGPT which lamp to choose for a dark office will not visit ten sites: they will read the answer, then perhaps click on one of the sources cited. If your product page could not be read, it will not be among them. And the visitor who arrives this way has already been advised: in the United States, they convert 42% better than other channels (Adobe, March 2026).
Can your tools see these robots?
Your analytics cannot. Robots do not run your tracking code, so they do not exist in GA4. Nor can your SEO crawler: it sees your pages, not who visits them. Only server or CDN logs show who comes by, when, and what they get.
- List the AI user agents found in your logs, and sort them into the three families.
- Weed out the impostors. Some scrapers declare themselves as “GPTBot” to get past protections. If you open your firewall to an AI robot, open it to its official IP addresses, published by OpenAI, Anthropic and Perplexity.
- Look at what they receive. 404s, redirect chains, URLs with parameters: all wasted visits.
It is by cross-referencing these sources that you know what to address. Fasterize matches your CDN logs against the crawl of your site: the console identifies verified robots, measures the share of their visits wasted on errors or URLs with parameters, and relates it to the pages that matter to your business.
Which ones should you let through?
Training robots are a matter of principle, a legal and editorial decision rather than an SEO one. Blocking GPTBot or ClaudeBot is legitimate. At both OpenAI and Anthropic, each robot is configured separately: blocking training does not remove you from search answers.
Search robots should be allowed, unless you have a strong reason not to. Citations come through them. Blocking OAI-SearchBot, for example, removes you from ChatGPT Search answers.
User requests should be allowed too: behind them is a person asking a question right now. Note that OpenAI warns that robots.txt may not apply to ChatGPT-User, and that Perplexity states that Perplexity-User generally ignores it, since the visit is triggered by a user.
For a site that refuses training but wants to be cited, the robots.txt looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Your robots.txt says yes, yet the robot gets a 403?
The robots.txt is not the only door. The site’s protection layer (web application firewall, anti-bot tool, rate limiting) can refuse a robot that the robots.txt allows. Nobody decided it, and nobody notices: a refused robot does not complain, it simply leaves.
This is far from rare. In our Digital Experience Benchmark 2027, 1 in 10 of France’s most-visited sites shuts the door on at least one AI robot, and 4 in 100 also shut it on the robots that feed assistants’ answers. The setting sometimes comes from an anti-bot tool configured by default, with no business trade-off.
To find out, look in your logs for 403 and 429 responses served to verified AI robots. In the Fasterize console, a check does exactly that, testing both doors at once: what the robots.txt says, and what the protection actually lets through.
How do you stay in control over time?
A robots.txt changes with every redesign, new robots appear regularly (Anthropic, for example, split Claude-SearchBot and Claude-User off from ClaudeBot), and protection rules evolve after every incident. A decision that is right in October may be wrong by March. Three checks are enough:
- every month, the list of AI robots seen in the logs, sorted by family;
- every month, the share of their visits that hit errors, redirects and URLs with parameters;
- with every redesign or change to security rules, a robot-by-robot test of the robots.txt and the protection.
And once the robot is in, it still needs to leave with something to read: content that is present without JavaScript, structured data, a page that responds quickly. Our AI Search guide covers each of these points in detail.
This is the work Fasterize does continuously. The platform audits your site non-stop, tells you what to fix first, lets you apply fixes from the console (or send your developers an automatically generated brief), then tracks their impact. One tool to audit, optimise and measure.
A tip to get started today: look at your product pages. In the logs we analysed, nearly 6 in 10 pages read by AI assistants are product pages, ChatGPT alone reads more than four fifths of them, and 1 request in 9 ends in an error or a block. Look for 403 responses served to OAI-SearchBot and ChatGPT-User on your product pages: if there are any, you know where to start.
Which AI robots are really getting into your site?
The AI Search Audit checks, free of charge, AI robots’ access, the readability of your pages and their structure, then sends you an action plan ranked by priority.
Frequently asked questions about AI crawlers
It is a matter of principle. GPTBot collects content to train OpenAI’s models. Blocking it has no effect on your presence in ChatGPT Search, which relies on a different robot, OAI-SearchBot. The two are configured separately in the robots.txt.
No. Google-Extended controls the use of your content to train and ground Gemini models. Google specifies that it has no effect on inclusion in Google Search or on ranking. AI Overviews draw on Googlebot’s index.
No. The robots.txt is an instruction that reputable robots respect, but nothing enforces it, and robots triggered by a user, such as Perplexity-User, generally ignore it. To block effectively, you need to act at the level of the site’s protection.
In your server or CDN logs, look for the GPTBot, OAI-SearchBot and ChatGPT-User user agents, then check that the IP addresses appear in the lists published by OpenAI. Your analytics cannot see these visits.
Most do not. Apart from Google, the main AI robots read the raw HTML without running JavaScript. Content injected after loading may remain invisible to them: text, headings and prices must be in the page’s initial HTML.
