Technical SEO tool
Free Robots.txt Generator
One per line, starting with /. A * matches anything, and a $ at the end matches the end of the address.
Rules for each crawler
A crawler set to Allow or Disallow gets a group of its own and stops following the default group, so an allowed crawler's group repeats the paths above.
Search engines
Blocking one stops it crawling your pages for its search engine.
AI training
Collect pages that may train AI models, or say whether pages may be used that way.
AI search
Find pages to show and link in AI search answers.
Fetches a user asks for
Visit a page because a person asked an assistant to. Some say they may not follow robots.txt.
Other
Each documented crawler checked against its company's own documentation on Oct 29, 2026; a crawler marked commonly cited has no such page. Sources are listed further down this page.
Names only, one per line or separated by commas: letters, hyphens and underscores.
Full addresses, one per line. Google reads a Sitemap line only when it is a full URL.
robots.txt
User-agent: * Allow: /
Publish it as a plain-text file at the root of the site, such as https://example.com/robots.txt. Each host and subdomain reads its own file.
How do I create a robots.txt file?
Choose what every crawler may do by default, add any paths to keep them out of, set rules for the crawlers you want to treat differently, and add your sitemap's address. The file on the right updates with each change. Save it as robots.txt and publish it at the root of your site, so it answers at an address such as https://example.com/robots.txt.
- Pick a preset, or set the default for User-agent: * yourself.
- List the paths no crawler should fetch, one per line, such as /admin/ or /cart. A * matches anything, and a $ at the end matches the end of the address.
- Set Allow or Disallow for any search engine or AI crawler that needs its own rule, and add other user agents by name.
- Add your sitemaps as full addresses.
- Copy or download the file, or test it first with the robots.txt tester.
The file must be plain text, UTF-8, at the root of the host it is for. Each subdomain and each protocol reads its own file, and Google reads only the first 500 KiB.
robots.txt file generator presets
| Preset | What it writes | Use it when |
|---|---|---|
| Allow everything | User-agent: * with Allow: /, plus your sitemap lines | You want every crawler in and a place to list your sitemaps |
| Block AI training, allow AI search | One group that disallows every AI training crawler on the list, and an open * group | You want to opt out of AI model training but stay in search and AI answers |
| Block everything (staging site) | User-agent: * with Disallow: / | The site is a staging or test copy that should not be crawled |
A preset sets the default and the crawler rules. Your paths, extra user agents and sitemaps stay as you typed them, so you can switch presets to compare.
How do I block AI training crawlers but stay in AI search?
Disallow the training tokens and leave the search and user tokens alone. OpenAI, Anthropic, Google, Apple, Meta and Amazon each use separate names: GPTBot and ClaudeBot collect pages for model training, while OAI-SearchBot and Claude-SearchBot find pages to show and link in answers. Google-Extended and Applebot-Extended never fetch anything; they only tell Google and Apple whether pages their normal crawlers fetched may train their models, and disallowing them has no effect on Google Search or Apple's search results.
The block AI training preset does exactly that, using the crawler list the AI crawler checker draws on. Each documented crawler's purpose comes from its company's own documentation, listed with the date it was checked at the end of this page. Bytespider is included and marked as commonly cited, because ByteDance publishes no page about it that we could find. Common Crawl's CCBot is listed as an open archive rather than a training crawler, because that is how Common Crawl describes it; set it to Disallow yourself if you want it out.
What goes in a robots.txt file?
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules for URL paths. Sitemap lines can go anywhere and apply to the whole file.
- User-agent: the crawler's name, matched without case, or * for every crawler with no group of its own.
- Disallow: a path the crawler should not fetch. Disallow: / covers the whole site; an empty Disallow allows everything.
- Allow: a path the crawler may fetch even inside a disallowed folder. The longest matching rule wins, and Allow wins a tie.
- Sitemap: the full address of a sitemap, such as https://example.com/sitemap.xml.
- Lines starting with # are comments.
A crawler follows only the group that names it, and ignores the * group when it has one. That is why the generator repeats your blocked paths in the group for crawlers you allow: without them, those crawlers could fetch the paths every other crawler is kept out of.
What can't robots.txt do?
robots.txt asks crawlers to stay out; it does not lock anything. Anyone can read the file, so listing a secret folder in it tells people where to look. Crawlers that ignore the file can still fetch every page, and some fetchers that act for a user say they may not follow it.
It also does not remove pages from search. Google says a URL blocked by robots.txt can still appear in results without a description. To keep a page out of Google, let it be crawled and add a noindex tag, or put it behind a password, which is also the right way to protect a staging site.
The crawlers in this generator, and where each comes from
Each documented token below was checked against its company's own crawler documentation on the date shown, and we check them again at least once a year. The presets use these purposes. A token its company does not document is marked as commonly cited.
| User-agent token | Run by | Purpose | Source |
|---|---|---|---|
| Googlebot | Search engine | Google's documentationChecked Oct 29, 2026 | |
| Bingbot | Microsoft | Search engine | Microsoft's documentationChecked Oct 29, 2026 |
| Applebot | Apple | Search engine | Apple's documentationChecked Oct 29, 2026 |
| GPTBot | OpenAI | AI training | OpenAI's documentationChecked Oct 29, 2026 |
| OAI-SearchBot | OpenAI | AI search | OpenAI's documentationChecked Oct 29, 2026 |
| ChatGPT-User | OpenAI | Fetches for a user's request | OpenAI's documentationChecked Oct 29, 2026 |
| ClaudeBot | Anthropic | AI training | Anthropic's documentationChecked Oct 29, 2026 |
| Claude-SearchBot | Anthropic | AI search | Anthropic's documentationChecked Oct 29, 2026 |
| Claude-User | Anthropic | Fetches for a user's request | Anthropic's documentationChecked Oct 29, 2026 |
| PerplexityBot | Perplexity | AI search | Perplexity's documentationChecked Oct 29, 2026 |
| Perplexity-User | Perplexity | Fetches for a user's request | Perplexity's documentationChecked Oct 29, 2026 |
| Google-Extended | AI training | Google's documentationChecked Oct 29, 2026 | |
| Applebot-Extended | Apple | AI training | Apple's documentationChecked Oct 29, 2026 |
| Meta-ExternalAgent | Meta | AI training | Meta's documentationChecked Oct 29, 2026 |
| Meta-WebIndexer | Meta | AI search | Meta's documentationChecked Oct 29, 2026 |
| Meta-ExternalFetcher | Meta | Fetches for a user's request | Meta's documentationChecked Oct 29, 2026 |
| Amazonbot | Amazon | AI training | Amazon's documentationChecked Oct 29, 2026 |
| Amzn-SearchBot | Amazon | AI search | Amazon's documentationChecked Oct 29, 2026 |
| Amzn-User | Amazon | Fetches for a user's request | Amazon's documentationChecked Oct 29, 2026 |
| Bytespider | ByteDance | AI trainingCommonly cited; ByteDance publishes no page about it that we could find. | Secondary sourceChecked Oct 29, 2026 |
| CCBot | Common Crawl | Open web archive | Common Crawl's documentationChecked Oct 29, 2026 |
The file format follows these pages:
Questions about robots.txt files
How do I make a robots.txt file?
Pick a preset or a default policy, add the paths to block, set rules for any crawlers that need their own, and add your sitemaps. Copy or download the file, save it as robots.txt in UTF-8 and publish it at the root of your site.
Where do I put the robots.txt file?
At the root of the host it is for, such as https://example.com/robots.txt. Google says a robots.txt file applies only to the protocol, host and port it is served from, so a subdomain such as shop.example.com needs its own file.
How do I block GPTBot and other AI crawlers?
Use the Block AI training preset, or set each crawler to Disallow. That writes a group such as User-agent: GPTBot with Disallow: /. Leave OAI-SearchBot and the other AI search crawlers allowed if you want your pages to keep appearing in AI answers.
Should I block everything on a staging site with robots.txt?
Use the Block everything preset, and also put the staging site behind a password. robots.txt only asks crawlers to stay out, and a URL that is linked from elsewhere can still appear in search results without being crawled.
Do I need a robots.txt file at all?
No. With no robots.txt, crawlers may fetch every page, which is fine for many sites. A file is useful to keep crawlers out of areas such as carts or internal search, to set rules for AI crawlers, and to list your sitemaps.
How do I check the file works before I publish it?
Press Test this file. The robots.txt tester opens with the file in its paste box and answers, for any paths and crawlers you choose, whether each is allowed and which line decided. Nothing runs until you press Test.
More free tools
See all free tools- Robots.txt TesterTests whether a site's robots.txt, or one you paste, lets Googlebot, Bingbot, AI crawlers or any user agent fetch each URL, and shows the line that decided.
- AI Crawler CheckerReads your site's real robots file and shows what it tells GPTBot, ClaudeBot, PerplexityBot and other AI crawlers, with every syntax problem in the file.
- Sitemap CheckerFinds a site's sitemaps, validates the XML against the sitemap protocol, checks child sitemaps and spot-checks listed URLs for errors, redirects and noindex.
Find every page robots.txt keeps out of search
A new robots.txt can block more than you meant. Azuqe's site audit checks every page of your site for indexing problems, including pages robots.txt blocks, stray noindex tags and broken canonicals, and lists the fixes in order of impact.
Free trial on your first site · Cancel any time