AI crawlers and llms.txt
There are now a dozen or so user agents in this space and they do genuinely different jobs. Treating them as one group is how businesses accidentally remove themselves from answers they wanted to be in.
Who is knocking
| User agent | Operator | What it is for | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Bulk crawling that feeds OpenAI models and their search index. | Blocks the broadest OpenAI use of your content, training included. |
| OAI-SearchBot | OpenAI | Builds the index ChatGPT search draws on when answering. | Blocking this is the one that most directly removes you from ChatGPT answers. |
| ChatGPT-User | OpenAI | Fetches a page live because a user asked about it in that moment. | Blocking means ChatGPT cannot open your page even when someone explicitly asks it to. |
| ClaudeBot | Anthropic | General crawling for Anthropic. | Removes you from what Claude has crawled. |
| PerplexityBot | Perplexity | Indexing for Perplexity answers, which are heavily citation-led. | Perplexity shows its sources prominently, so this is a costly one to block. |
| Google-Extended | A robots.txt token, not a crawler. Controls whether Gemini and Vertex may use your content. | Does not remove you from Google Search or AI Overviews. See the warning below. | |
| Applebot-Extended | Apple | Same idea as Google-Extended, for Apple Intelligence. | Opts you out of Apple model training without affecting Siri or Spotlight results. |
| CCBot | Common Crawl | An open web archive that many models are trained on downstream. | Wide but slow-acting effect, since existing archives already exist. |
This list moves. Operators add and rename agents, and the honest position is that any table like this is a snapshot. Check your own server logs for what is actually hitting you rather than trusting anyone’s list, including ours.
The Google-Extended misunderstanding
A lot of people add Google-Extended to robots.txt believing it keeps them out of AI Overviews. It does not.
AI Overviews are built from Google’s ordinary search index, gathered by Googlebot. Google-Extended governs a different thing: whether your content may be used to improve Gemini and Vertex AI models. Setting it is a reasonable choice on its own terms, but it does not change what appears above the search results.
The only way out of AI Overviews is the ordinary search-snippet controls, and those cost you your normal snippet too. Almost nobody actually wants that trade.
Is llms.txt worth adding?
The idea is a plain-text file at the root of your site that gives a model a curated map of your content in markdown, rather than making it work that out from your navigation. It is a sensible proposal and it costs almost nothing.
Being straight with you: it is a community convention, not a standard, and we have seen no evidence that the major assistants read it. Anyone telling you it is required is ahead of the facts.
Our position
Add one if writing it is cheap for you, because the exercise of listing your genuinely useful pages in one place is worth doing regardless, and the file costs nothing to serve. Do not pay anyone much for it, and do not treat it as a substitute for the pages themselves being clear.
If it ever becomes load-bearing you will already have one.
The accidental blocks we find most often
- Cloudflare’s AI bot toggle. One switch, often flipped by whoever set up the account, blocking every AI crawler at the edge. Your robots.txt can say whatever it likes; the request never arrives.
- A blanket disallow inherited from staging. A
Disallow: /that was correct on a preview site and got copied to production. - Content that needs JavaScript. Some of these crawlers do not run it. If your prices or service areas are rendered client-side, they may as well not be there.
- Geo-blocking. Locking traffic to Australian IPs is understandable and it also blocks crawlers running from overseas data centres.
- Rate limiting that looks like an attack response. Aggressive throttling makes a crawler give up part-way through your site, usually on the deeper pages that hold the specifics.
Check yours
We will request your key pages as each of the agents above and send you what came back against what a browser sees. It is the fastest way to rule out the boring explanation before spending money on the interesting one.
Run the crawler check