SEO
Robots.txt Generator
A clean robots.txt in thirty seconds, with zero syntax errors.
The robots.txt is a tiny file with enormous consequences: one badly written line can deindex an entire site — or let crawlers roam sections that have no business showing up in Google. Compose yours in a few clicks: allow or block all crawlers, choose whether to refuse AI training bots (GPTBot, ClaudeBot, PerplexityBot, CCBot…), list the paths to exclude and add your sitemap's address. The tool generates a file with flawless syntax, ready to drop at the root of your site, and the output updates live as you tick the options. Free, no account required, and explained clearly enough that you won't break anything.
How does it work?
- Choose the base rule: allow or block all robots.
- Optionally tick AI bot blocking and list the paths to exclude.
- Add your sitemap URL.
- Copy the generated file and place it at the site root (/robots.txt).
Frequently asked questions
What exactly does robots.txt do?
It tells crawlers which sections of the site they may explore. It's a crawling directive, not a security measure: sensitive content must be protected by authentication, never by a mere Disallow line.
Should I block AI bots?
It's an editorial choice: blocking GPTBot, ClaudeBot or CCBot prevents your content from being used for model training, but may also reduce your visibility in AI assistants' answers — a fast-growing discovery channel. Classic search engine crawlers are unaffected either way.
Does blocking a page in robots.txt remove it from Google?
Not necessarily: a blocked page can stay indexed if other sites link to it. To keep a page out of results, use a noindex tag instead — and don't block it in robots.txt, or Google will never see the tag.
Where do I put the generated file?
At the domain root, reachable at yoursite.com/robots.txt. Each subdomain has its own robots.txt. Once live, verify it in Search Console.