AI search & GEO

AI crawlers

AI crawlers are the bots that fetch web pages for AI companies — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended — for training, for retrieval, or for both.

They are distinguishable in robots.txt, and the distinction matters: some fetch for model training, others fetch live to answer a question a user just asked. Blocking the first while allowing the second is a coherent position for a publisher; blocking everything is a coherent position for some business models and a costly accident for most others.

Two practical points. Most of these crawlers do not execute JavaScript, so client-side rendered content is invisible to them regardless of your robots.txt. And a blanket block often arrives by default from a security vendor or CDN rule rather than by decision — check what your infrastructure is actually doing before assuming your policy is what you intended.

Related terms

Guides on this subject