Robots.txt Generator

What robots.txt is and what it is not

A robots.txt file is a plain-text instruction sheet that lives at the root of your domain and tells automated crawlers which parts of your site they are welcome to fetch. It follows the Robots Exclusion Protocol, a convention that dates back to 1994 and is now a published standard. The single most important thing to understand is that robots.txt is advisory, not enforcing. Well-behaved crawlers — Googlebot, Bingbot, and most reputable bots — read it and obey it. A malicious scraper can ignore it entirely, and nothing stops them. So robots.txt is the right tool for steering search engines and saving crawl budget, but it is the wrong tool for hiding private data. Anything that must stay secret needs server-side authentication, not a Disallow line.

How the syntax is built

The file is a list of rule groups. Each group starts with a User-agent line naming the bot it applies to, followed by one or more Allow or Disallow lines giving URL paths. A * user-agent means "every crawler." This generator builds those groups visually so you never have to remember the exact keyword spelling. Add a group, set its user-agent, then stack rules inside it; the output pane rewrites the file live as you type. You can also append a Crawl-delay rule (a hint that asks slower-polite crawling, honoured by Bing and others but ignored by Google) and a single Sitemap line pointing to your XML sitemap.

A worked example

Say you want search engines to index everything except your admin area and a private folder, and you want to point crawlers at your sitemap. You would create one group with user-agent *, add Disallow: /admin/ and Disallow: /private/, leave the rest implicitly allowed, and set the sitemap field. The tool produces:

An empty Disallow: line, by contrast, means "disallow nothing" — that is, allow everything. This is the classic gotcha: Disallow: / blocks the whole site, while Disallow: with nothing after it opens it. The presets handle these edge cases for you.

The four presets, explained

• Allow All — a single group letting every bot reach everything. The friendliest default for a public site.
• Block All — Disallow: / for all bots. Useful on a staging or pre-launch site, but remember to remove it before going live, or you will vanish from search.
• Common Crawlers Only — explicitly allows Googlebot, Bingbot, and Yahoo's Slurp, then disallows everyone else.
• Block AI Bots — keeps the site open to search engines but disallows known AI-training crawlers including GPTBot (OpenAI), ChatGPT-User, CCBot (Common Crawl), Google-Extended, anthropic-ai, Claude-Web, and Omgilibot, so your content is less likely to be ingested into model training sets.

Common mistakes

• Using robots.txt to deindex a page. Disallowing a URL stops crawling, but a blocked page that other sites link to can still appear in results without a snippet. To truly remove a page from the index, allow crawling and add a noindex meta tag instead.
• Wrong location. The file must sit at exactly https://yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored, and each subdomain needs its own.
• Blocking CSS and JS. Disallowing your asset folders can stop Google from rendering the page as users see it, which hurts ranking. Keep stylesheets and scripts crawlable.
• Case sensitivity. Paths are case-sensitive: /Admin/ and /admin/ are different rules.

Saving and testing

When the output looks right, copy it or use the Download button to save a ready-named robots.txt for your server's root directory. After deploying, test it with Google Search Console's robots.txt report, which shows whether a given URL is allowed or blocked under your current rules. This generator runs locally in your browser, so the rules you draft are never sent anywhere until you publish them yourself.