robots.txt Generator
Generate robots.txt files with custom User-agent rules, Allow/Disallow paths, and Sitemap URLs.
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
When a robots.txt generator is the right tool
Use this page to draft a robots.txt with one or more user-agent groups, Allow and Disallow paths, optional sitemap URLs, and an optional Crawl-delay. The preview updates as you type. For a page-level robots meta tag, use Meta Tags Generator.
Use cases
Starting from the built-in example
The default group targets * and disallows /admin/ and /private/. Change those paths, add Allows, or add another group for a named bot. Sitemap accepts several URLs, one per line.
Splitting rules by crawler
Add Rule Group, set a User-agent such as Googlebot, and list the paths that apply only to that bot. Remove is offered when more than one group exists.
How the file is built
Groups are written in order. Allows are listed before Disallows in the same group. A Crawl-delay value is copied into every group. Sitemap lines follow all groups.
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlWhat this tool does not do
It does not fetch your live robots.txt, test how Googlebot will interpret a rule, or upload the file. Path syntax is not validated. Generation stays in your browser.
How to use robots.txt Generator
- Edit the default group, or add groups for named user-agents.
- Add sitemap URLs and, if you know a crawler that honors it, a Crawl-delay.
- Copy or Download the file and place it at the site root.
Frequently asked questions
How does robots.txt Generator work?
The file on the right updates as you edit rule groups and the global Sitemap / Crawl-delay fields. Each group has a User-agent (blank becomes *), Allow paths, and Disallow paths, one per line. Empty lines are dropped. Copy or Download saves a file named robots.txt. The starting example disallows /admin/ and /private/ for all bots and lists https://example.com/sitemap.xml.
In what order are Allow, Disallow, and Crawl-delay written?
Inside each group the file is User-agent, then every Allow line, then every Disallow line. If Crawl-delay is set, that line is repeated inside every group, even though the input is a single global field. Sitemap lines are written once at the end. There is no Host directive and no comment field.
Do all crawlers honor Crawl-delay?
No. Crawl-delay is a non-standard extension that some crawlers (historically Bing or Yandex) may honor. Google’s crawler does not use Crawl-delay in robots.txt. Treat it as optional and check each crawler’s documentation.
Is this file uploaded or validated against a live site?
No. Generation runs in your browser. Paths are not checked against your site, and the tool does not fetch robots.txt from a URL. You can add more groups; the last remaining group cannot be removed.
Related tools and guides
References
- The formal standard is RFC 9309 (Robots Exclusion Protocol). Sitemap lines and older extensions such as Crawl-delay are documented by individual crawlers, not fully by that RFC.
- Place the downloaded file at
https://your-domain/robots.txt. This generator does not publish it for you.