Robots.txt Templates for AI and Search Crawlers

Choose a template by policy outcome

The templates in this section cover common decisions: keeping search open while limiting AI training, blocking a particular provider, restricting a broader crawler group, or editing robots.txt safely on WordPress and Shopify. Start with the narrowest template that matches your actual goal.

Three questions to answer before copying

1. Which use are you controlling? Training, AI search discovery, user-triggered retrieval, conventional search, dataset collection, and SEO analysis are different activities.
2. Does the site already generate robots.txt? A CMS, ecommerce platform, SEO plugin, CDN, or hosting layer may own the live response. Edit the active source instead of uploading a second unused file.
3. What must remain discoverable? A broad wildcard block can affect Googlebot, Bingbot, product feeds, images, or other services you did not intend to restrict.

Use every template as a reviewed starting point

Copy the relevant groups, replace example domains and paths, and merge them with rules already required by your platform. Do not publish duplicate or contradictory groups without understanding how the target crawler combines and evaluates them. The examples are intentionally readable so the final policy can be explained later to a client, editor, or developer.

Verify the result after deployment

Open the live /robots.txt URL, confirm HTTP 200 and plain-text content, then test representative URLs with the robots.txt checker. Recheck the file after plugin changes, platform updates, CDN rules, staging-to-production moves, or domain migrations.

If no existing template matches the policy, use the robots.txt generator to create crawler-specific groups. Robots.txt remains a cooperative crawling instruction; it should never be used as the only protection for private or paid content.