About the Robots.txt Generator
The robots.txt file lives at the root of your domain and tells crawlers which paths they may or may not visit. It is the primary lever for controlling crawl budget, blocking staging areas, and — increasingly — refusing consent to AI training scrapers.
Anatomy of a robots.txt
Each block starts with a User-agent line identifying the bot, followed by Allow or Disallow rules describing paths. Wildcards (*) and end-anchors ($) let you match patterns. A Sitemap directive at the top or bottom points crawlers to your XML sitemap. Blank lines separate user-agent groups.
Rules apply from most specific to least specific. If Googlebot has its own block, it ignores the generic User-agent: * block entirely. Design each bot section to be self-contained.
Blocking AI scrapers
As of 2024, OpenAI, Anthropic, Google-Extended, Common Crawl, and other AI vendors respect a robots.txt Disallow. Adding a rule that blocks GPTBot, ClaudeBot, and CCBot is the fastest way to opt your site out of most large-language-model training corpora.
Bear in mind that non-compliant scrapers ignore robots.txt entirely. If you need enforcement, combine robots.txt with server-side bot detection and Cloudflare's AI-scraper block list.
Crawl budget and sitemap
For large sites, robots.txt is your first line of crawl-budget control. Disallow faceted URLs, calendar archives, and internal search pages so Google spends its budget on canonical content. Always declare a Sitemap so new URLs are discovered quickly.
The Crawl-delay directive is respected by Bing, Yandex, and Baidu, but ignored by Google. Use Google Search Console to set Googlebot's rate directly.
Step-by-step usage
1. Pick a preset (WordPress, E-Commerce, or Block AI Scrapers) or start blank. 2. Set the default access for User-agent: *. 3. Add specific rules per bot with Allow/Disallow paths, one per line. 4. Enter your sitemap URL. 5. Copy the result or download as robots.txt and upload it to the root of your domain.
Frequently Asked Questions
At the exact root of your domain: https://example.com/robots.txt. Subdirectories are ignored. Each subdomain needs its own file.