robots.txt is a plain text file placed at the root of your website that instructs web crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol.

Does robots.txt guarantee bots won't access my pages?

No. robots.txt is advisory — well-behaved crawlers (Googlebot, Bingbot) respect it, but malicious bots can ignore it. Use server-side authentication for truly private content.

What is the 'Block AI Bots' preset?

This preset adds Disallow: / rules for known AI training crawlers including GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended, and anthropic-ai, preventing them from training on your content.

Where should I place the robots.txt file?

Place it at the root of your domain: https://yourdomain.com/robots.txt — it must be at this exact path to be recognized by search engines.

Developer 免费无需注册

Robots.txt Generator

Generate a robots.txt file with presets for common crawlers and custom user-agent rules.

正在加载工具…

关于此工具

Build a valid robots.txt file visually without memorizing the syntax. Choose from quick presets — Allow All, Block All, Common Crawlers Only, or Block AI Bots — or craft fully custom rules. Add as many User-agent / Allow / Disallow rule groups as you need using the dynamic row builder. Optionally add a Sitemap URL. The generated robots.txt updates live and can be copied to clipboard or downloaded directly as `robots.txt` ready for your server's root directory.

使用方法

1 Choose a preset (Allow All, Block All, Common Crawlers, or Block AI Bots) or start custom.
2 Add rule groups by clicking 'Add User-agent Group'.
3 For each group, enter the User-agent and add Allow/Disallow rules.
4 Optionally add your sitemap URL at the bottom.
5 Click Copy or Download to save your robots.txt file.

关于此工具

使用方法

常见问题