Should You Block AI Crawlers Like GPTBot, or Does Blocking Keep You Out of ChatGPT and Perplexity Answers?

Blocking GPTBot does not remove you from ChatGPT's answers. GPTBot is OpenAI's model-training crawler. ChatGPT's search index is fed by a different agent, OAI-SearchBot, and live user queries are fetched by a third, ChatGPT-User. So the honest verdict is: you can block training and stay citable, but only if you know which agent does which job. The line in your robots.txt that actually costs you ChatGPT and Perplexity citations is almost never the one founders think it is.

That distinction did not exist when most of these files were written, which is why so many B2B SaaS sites are quietly invisible today.

What everyone did in 2023 and 2024

A founder read a thread about AI companies scraping content for free, opened /robots.txt, pasted four lines disallowing GPTBot, and moved on. It felt like a free option: keep your content out of the training set, lose nothing.

At the time there was only one crawler per platform to reason about. Blocking it was a single yes-or-no decision about training. Plenty of teams went further and added a blanket User-agent: * rule or copied an "AI bot blocklist" gist that swept up retrieval agents alongside training agents. Cloudflare then started default-blocking AI crawlers for new domains in mid-2025, so a large number of sites now block retrieval bots without anyone on the team ever choosing to.

Nobody went back to re-read the file. It is a thirty-line text document that no dashboard ever complains about.

What changed: every engine now runs separate crawlers

The platforms split their crawlers because publishers demanded the ability to say no to training without saying no to traffic. That split is the whole ballgame now.

• OpenAI: GPTBot trains the model. OAI-SearchBot builds the ChatGPT Search index. ChatGPT-User fetches a page live because a human asked a question right now. Block the first, allow the other two.
• Anthropic: ClaudeBot trains. Claude-SearchBot indexes for Claude's search and citations. Claude-User is the live per-query fetch.
• Perplexity: PerplexityBot is the indexing and citation crawler, and blocking it means no Perplexity citations. Perplexity-ModelFetcher is the training agent. Perplexity-User is user-triggered.
• Google: Googlebot feeds Search and AI Overviews from the same crawl. There is no setting that keeps you in Google results and out of AI Overviews. Google-Extended is a training and grounding control for Gemini, not a crawler, and blocking it does not pull you out of AI Overviews.

Blocking training does not block citations. Blocking retrieval does. Those are two different sentences and most robots.txt files on the internet confuse them.

What a correct robots.txt looks like now

If your position is "do not train on my content, but do cite me," this is the posture:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-ModelFetcher
Disallow: /

User-agent: PerplexityBot
Allow: /

Two things to keep in mind. Robots.txt requests, it does not enforce, so treat it as a policy signal rather than a lock. And it is checked after your edge: a Cloudflare bot rule, a Vercel firewall setting, or a WAF default can refuse an AI crawler before robots.txt is ever read. Fix the file, then confirm the edge agrees with it.

The silent failure that permissive robots.txt does not solve

Allowing retrieval crawlers gets them to the door. It does not guarantee they can read the room.

AI retrieval crawlers generally do not execute JavaScript. If your article text is assembled client-side, a React SPA or a misconfigured headless setup, the crawler fetches a shell and leaves with nothing, no matter how open your robots.txt is. Curl your own blog post with JavaScript off and check whether the body copy is in the initial HTML response. If it is not, crawler access was never your real problem.

This is why crawler access is 30 percent of the AI Readiness Score our free readiness check returns, and content structure is another 30 percent. Structured data carries 25 percent and llms.txt 15 percent. On llms.txt specifically, be realistic: no major engine has confirmed it reads the file. It is cheap, low-risk, and worth having, which is why we check and generate one, but it is not the lever. Access plus server-rendered, answer-first content is the lever.

The lesson: robots.txt is a demand channel, not a crawl-budget file

The 2023 instinct treated AI crawlers as a cost to minimize. That math has flipped. Semrush put LLM-referred conversion at 4.4x organic in 2025, because buyers arrive with intent already formed. It is also why so much of this traffic lands in your reports as unattributed direct visits rather than as an obvious source. One wildcard line can take you out of the only assistant your buyers use, and nothing in your stack will tell you.

So do not fix it and hope. Fix it, then measure. Our site audit returns the full issue list, blocked retrieval agents and render problems included. Daily visibility monitoring across ChatGPT, Claude, Gemini, and Perplexity shows whether those engines start naming you afterward, and keeps the actual answers as receipts. Attribution then ties the leads that follow back to the exact engine and post that earned them. Everyone shows you rank. We show you revenue.

Run the readiness check on your own domain first. If crawler access scores low, you found your quarter's cheapest fix.