How to Block AI Crawlers with robots.txt

Start with the crawler’s purpose

“AI crawler” is not one technical category. A company may operate separate bots for model training, AI search discovery, and retrieval triggered by an individual user. Blocking every bot from the same company can therefore remove more access than you intended.

Before editing robots.txt, identify the exact token and decide what outcome you want. For example, OpenAI documents GPTBot for potential model-training use and OAI-SearchBot for ChatGPT search. Anthropic similarly separates ClaudeBot from Claude-SearchBot. User-request agents such as ChatGPT-User and Claude-User are a third case and should be reviewed separately.

Block one crawler with an exact group

To block a compliant crawler across the whole host, create a dedicated group using its published User-agent token and the root path:

User-agent: GPTBot
Disallow: /

Repeat the pattern for each crawler you want to restrict. Do not replace the specific token with * unless your real intention is to block every compliant crawler, including ordinary search engines. A broad wildcard block can remove valuable search access and make important pages harder to discover.

Keep allowed crawler purposes explicit

When the same provider has multiple crawler purposes, write independent groups. The following policy blocks two training-oriented crawlers while keeping the providers’ search crawlers open:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

Specific crawler groups are easier to audit than a mixture of broad wildcard rules and exceptions. They also make your intent clear when a provider adds a new bot or changes one product without changing another.

Protect normal search visibility

Do not block Googlebot, Bingbot, or other search crawlers merely because some search products now use AI features. Search indexing bots, AI-training controls, and product-specific extended tokens are not interchangeable. Review each published token before adding it to a block list.

If you want to allow AI search citations while refusing model-training crawling, use the more specific workflow in Allow AI Search but Block AI Training. The ready-made template provides a starting point that you can adapt to your own policy.

Publish and verify the file

1. Serve the file at the root of the exact host, such as https://example.com/robots.txt.
2. Add a separate file for each subdomain whose crawling policy differs.
3. Open the public URL and confirm that it returns plain text with a successful HTTP response.
4. Run the file through the robots.txt checker and test representative public and blocked URLs.
5. Review server logs later to confirm whether the named crawlers changed their behavior.

Understand the limit of robots.txt

Robots.txt is not authentication, copyright enforcement, a paywall, or a guaranteed anti-scraping mechanism. Well-behaved crawlers may honor it; malicious or unidentified clients may ignore it. Protect private or licensed material with server-side authorization, rate limits, bot controls, and contractual measures appropriate to the content.