robots.txt is a plain text file placed at the root of your website that instructs web crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol.

Does robots.txt guarantee bots won't access my pages?

No. robots.txt is advisory — well-behaved crawlers (Googlebot, Bingbot) respect it, but malicious bots can ignore it. Use server-side authentication for truly private content.

What is the 'Block AI Bots' preset?

This preset adds Disallow: / rules for known AI training crawlers including GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended, and anthropic-ai, preventing them from training on your content.

Where should I place the robots.txt file?

Place it at the root of your domain: https://yourdomain.com/robots.txt — it must be at this exact path to be recognized by search engines.

Developer Free No signup

Robots.txt Generator

Generate a robots.txt file with presets for common crawlers and custom user-agent rules.

Loading tool…

About this tool

Build a valid robots.txt file visually without memorizing the syntax. Choose from quick presets — Allow All, Block All, Common Crawlers Only, or Block AI Bots — or craft fully custom rules. Add as many User-agent / Allow / Disallow rule groups as you need using the dynamic row builder. Optionally add a Sitemap URL. The generated robots.txt updates live and can be copied to clipboard or downloaded directly as `robots.txt` ready for your server's root directory.

How to use

1 Choose a preset (Allow All, Block All, Common Crawlers, or Block AI Bots) or start custom.
2 Add rule groups by clicking 'Add User-agent Group'.
3 For each group, enter the User-agent and add Allow/Disallow rules.
4 Optionally add your sitemap URL at the bottom.
5 Click Copy or Download to save your robots.txt file.

Frequently Asked Questions

Related tools

Meta Tag Generator

Generate SEO-optimized HTML meta tags with live Google and Twitter card preview.

URL Parser & Builder

Break any URL into its components or assemble a URL from parts — with a live query-param editor.

HTML / CSS / JS Minifier

Minify HTML, CSS, and JavaScript to reduce file size — runs entirely in your browser.

JSON Schema Validator

Validate JSON data against a JSON Schema with detailed path-level error messages.