Robots.txt Generator

Compose robots.txt groups, paths and sitemap lines from a form, then read the parser's report on the emitted file before you copy it — entirely on this device.

0 bytes uploaded

robots.txt builder

File header

Header commentWritten as `#` lines at the top of the file. Crawlers ignore comments.

Groups

Group 1

User-agent — Group 1

Insert a known AI-crawler token

A group holds at most 20 user-agent tokens.

Rule — Group 1
Path — Group 1

A group holds at most 100 rules.

Crawl-delay — Group 1Crawl-delay is not part of RFC 9309. Google ignores it; other crawlers read it in seconds, and some read it as a rate.

Group comment — Group 1

A draft holds at most 20 groups.

Sitemaps

Sitemap lines are written last and must be absolute URLs. They belong to the file, not to a group.

Emitted file

User-agent: *
Allow: /

23 B · 2 lines

VerificationRead back and matched

The emitted file is parsed back by the same engine that powers our robots.txt tester. That proves the file parses as intended — not that any crawler will obey it.

Code N_EMIT_PRESET_APPLIED This draft is still the "allow-all" preset exactly as it ships — it describes a common starting point, not your site. Edit the groups, paths and sitemap lines until the file says what you actually want crawled.
Code N_EMIT_ROUNDTRIP_OK The emitted file was read back with the same parser that powers our robots.txt tester and came out exactly as written: 1 group, 1 rule, 0 sitemap lines — which proves the file parses as intended, not that any crawler will obey it.

The download is the file above, byte for byte, named robots.txt.

Save it as robots.txt at the root of each host and scheme you want it to cover, served as text/plain. This page writes the file; it cannot deploy it.

Press ? for shortcuts

Do it offline:$ python3 -c "import urllib.robotparser as r; p=r.RobotFileParser(); p.parse(open('robots.txt').read().splitlines()); print(p.can_fetch('*','https://example.com/admin/x'))"

How to use it

Pick the preset closest to your intent — allow everything, block everything, close the admin paths, the WordPress baseline, or the AI crawlers named in vendor documentation — then edit it.
Give each group its crawler tokens, one token per line, and add the Allow and Disallow paths that group should carry.
List each sitemap by its full address, and write a header comment if the next person to open the file will need the context.
Read the report beneath the output pane, then copy the file or download it and publish it at the root of your site.

Choose a starting point, name the crawlers you care about, and list the paths each of them may or may not fetch. The file assembles itself as you type, and every version of it goes back through our own parser before you reach the copy button, so the report under the pane is our parser's reading of the file you are about to copy.

Examples

Close the admin area, keep one page in it open

# robots.txt for example.com — updated 2026-08-12

User-agent: *
Disallow: /admin/
Allow: /admin/public

Sitemap: https://example.com/sitemap.xml

One group, two rules, 148 bytes. The header becomes a comment line, the sitemap line lands last because it belongs to the whole file, and the verification well flags that placement even though it is correct.

Block every AI crawler we could verify

User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Amazonbot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
Disallow: /

Eleven tokens share one group, so they share one rule. Each token was read by hand in its own vendor's documentation on 2026-08-12 and the date ships with the list; a vendor whose page could not be opened was left out rather than guessed at.

Edge case: a draft that is legal and still wrong in six ways

# staging and search pages
User-agent: *
Disallow: /search?q=%23top
Disallow: /tmp*
Disallow: /*.pdf$x
Disallow: /tmp*
Disallow: drafts/
Crawl-delay: 10

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Nothing here is a refusal, and every line of it is worth reading twice. The repeated sitemap row was dropped, the hash in the first path was rewritten, and the fifth rule is reported plainly: “The pattern "drafts/" at line 7 starts with neither "/" nor "*" — patterns are compared against a URL path, which always begins with "/", so this rule can never match anything. Add the leading slash.”

Every example on this page is executed against the same engine that powers the tool, on every build.

Gotchas worth knowing

An empty value is never written for you

A rule with nothing after the colon is the 1994 spelling of open everything, and it reads as its own opposite. The builder refuses it and names the modern spelling, which is the line the allow-everything preset already writes.

One crawler reads one group

Name the same token in two groups and the second one is scenery: the rules a crawler follows come from a single group. The report says which two collided, so you can merge them and read what the crawler reads.

Sitemap rows carry the whole address

A row without a scheme and host is refused: the file can be fetched from anywhere, and a bare path resolves against the wrong place. Rows are written once each, at the end, and a repeat is reported and left out.

Crawl-delay is written and rarely obeyed

RFC 9309 does not define the directive, Google ignores it, and the crawlers that do read it disagree about what the number means. It is emitted on request with that caveat attached; a real rate limit belongs in your server or CDN.

Some characters cannot survive a rule line

A space, a control character or a hash would break the record it sits in, so it is written in escaped form and the report names both spellings. Everything else is emitted exactly as typed.

Writing the file is not publishing it

The result belongs at the root of every host and scheme you serve, handed back as text/plain, and one host cannot borrow another's copy. Nothing here uploads or deploys it.

Common errors

The "Disallow" rule in group 1 at line 2 has no path — an empty value is RFC 9309's way of spelling "matches nothing", which reads as the opposite of the line it is written on. Type a path such as "/admin/", or write allow-everything as "Allow: /".: The one place this builder declines to reproduce a legal file, and it says so in the message rather than quietly writing something else.
User-agent token "Googlebot News" contains a space at line 1 — a product token is one unbroken word (RFC 9309 §2.2.1), so a crawler would never match this line. Remove the space, or add the second name as its own "User-agent" line.: Product tokens are single words. Two crawlers in one group means two lines, not one line with a space.
Sitemap row 1, "/sitemap.xml", is not an absolute http or https URL — RFC 9309 §2.2.3 requires the scheme and host on a "Sitemap" line because the file may be fetched from anywhere. Write it as https://example.com/sitemap.xml.: Refused rather than repaired: only you know which host and scheme the sitemap is actually served from.
This draft has 21 groups — this builder writes at most 20, because a crawler obeys exactly one group and long files are read less reliably. Merge the groups that share the same rules.: Twenty groups is the ceiling and the draft is refused whole; groups with identical rules are the usual thing to collapse.
Group 1 has 101 rules — this builder writes at most 100 per group, and RFC 9309 §2.5 only obliges a crawler to read the first 512,000 bytes of the file. Block a parent directory, or cover the set with one "*" pattern.: A hundred rules per group. Past that, a parent directory almost always says the same thing in one line.
This draft holds 1,048,577 characters of text — this tool writes at most 1,048,576 characters (1 MB) in one run, so nothing was emitted. Remove some paths, or build the file in sections and join them locally with the command on this page.: The house one-megabyte ceiling on everything typed into the form.

FAQ

Will a Disallow rule take a page out of Google?

No. These rules govern fetching, not listing: an address nobody may fetch can still appear in results when other sites link to it, and a page that is never fetched can never be re-read to discover an instruction to drop it. Use a noindex rule on the page itself, and leave that page crawlable.

Where does the finished file have to go?

At the root of each host and scheme you serve, returned as text/plain — one file per origin, and a subdomain cannot inherit the parent's copy. This page writes the file and hands it to you; uploading it, and everything that happens afterwards, stays on your side.

Does blocking an AI crawler undo any training?

It does not, and nothing published later can. The token list was read out of each vendor's own documentation on the date shown beside it, which makes it a snapshot rather than a registry; no vendor is bound by appearing on it.

Keyboard shortcuts

⌘/Ctrl + Enter: Write the file again from the current form.
⌘/Ctrl + S: Download the result as robots.txt.
⌘/Ctrl + C: Copy the emitted file when nothing is selected.
⌘/Ctrl + Shift + C: Copy the emitted file over a selection.
? or ⌘/Ctrl + /: Show the list of shortcuts.
t: Step through the colour themes.
Esc: Dismiss whichever dialog is on top.

Nothing you paste here leaves this tab. State only ever travels in links you explicitly create.

RFC 9309 — Robots Exclusion Protocol · Google Search Central — Create and submit a robots.txt file · Google Search Central — How Google interprets the robots.txt specification · Google Search Central — Google crawlers and user-agent strings · Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler? · Apple — About Applebot · Common Crawl — CCBot · Amazon — Amazonbot · Mistral AI — Robots

Last reviewed: 2026-08-12