robots.txt is the standard crawl-access file where a site can allow or disallow URL paths for named user-agent tokens. It controls compliant crawler access, not every downstream use of content, and blocking one AI-related token does not automatically remove a page from every AI surface.
In plain terms
GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Googlebot, and Bingbot have different operators and purposes; they should not be treated as one interchangeable AI crawler.
Google AI Overviews and AI Mode are Search features. Google says Search eligibility is governed by Googlebot indexing and snippet controls, while Google-Extended controls training and grounding in some other Google systems.
Allow rules keep a crawl path open for compliant bots. They do not guarantee that a page will be indexed, retrieved, mentioned, or cited.
Why it matters
Crawler rules can open or close specific retrieval paths, so an accidental block is worth detecting.
Correct policy requires mapping each token to its documented purpose; a blanket allow or block can produce consequences different from what the site owner intended.