Free Robots.txt Tester

Enter the page's address and run the test. The tester fetches robots.txt from the root of that site once, reads it, and answers for the page's path whether each crawler may fetch it. To test other pages on the same site, list their paths, one per line.

1. Choose URL to read the site's live file, or Paste robots.txt to test a draft before you publish it. Nothing is fetched for a paste, and the file is used for that one test only: it is not cached or saved.
2. List up to 20 paths, such as /blog/post or /search?q=shoes. Leave the box empty to test the path of the address you entered, or the home page for a paste.
3. Name up to 10 user agents, or leave the box empty to test Googlebot, Bingbot and the 17 AI crawlers listed further down this page.
4. Read the answer for each path and crawler, then click a deciding rule to see its line highlighted in the file.

A paste can be up to 60 KB. A live file is read up to 500 KiB, the most Google reads, and a file past that point is flagged.

Before any rule matters, a crawler has to get the file. What the address answers decides whether the rules apply at all, and Google documents what it does for each answer.

Redirects are followed, as Google follows at least five of them for robots.txt, and the tester says where the file really lives. Each host reads its own file: www.example.com and shop.example.com each need their own robots.txt.

A robots.txt file has no strict syntax check: crawlers skip lines they cannot read and carry on. That makes mistakes quiet. The tester lists every line that a crawler skips, or that Google reads in a way other crawlers may not, with the line number and what to change.

• Rules before the first User-agent line, which belong to no group, so every crawler ignores them.
• Misspelled directives that Google's own parser still accepts, such as Disalow or User agent, which other crawlers may skip.
• A missing colon, a rule path that does not start with a slash, or a User-agent line with no crawler name.
• Sitemap lines that are not full URLs, which Google ignores.
• Directives Google does not support, such as Crawl-delay, Noindex and Host, with what to use instead.
• An Allow rule for an index.html page, which Google also reads as allowing the folder itself.

The rule with the longest matching path wins, whatever order the lines are in. When an Allow and a Disallow match with paths of the same length, Google uses the least restrictive one, so Allow wins. RFC 9309, the robots.txt standard, says the same.

A crawler follows only one group: the one that names it, or, if none does, the User-agent: * group. The two are never combined. So a group for GPTBot that says Disallow: /private replaces every rule in the * group for GPTBot, and the tester shows which group each answer came from.

robots.txt controls crawling, not indexing. Google's documentation says a page blocked by robots.txt can still be indexed if other pages link to it, usually shown with no description. To keep a page out of search results, let Google crawl it and add a noindex robots meta tag or header, because Google cannot see a noindex on a page it is not allowed to fetch.

The reverse mistake is common too: a Disallow line meant for a staging site that ships to the live site, or a rule that blocks the CSS and JavaScript files Google needs to render your pages. Test the paths that matter, including a page from each template and the folders your scripts and styles live in.

Search Console used to have a robots.txt Tester that showed which line blocked a URL. Its help page now describes the robots.txt report instead, which lists the robots.txt files Google found for your site's top hosts, when Google last fetched each one, and the problems it found, and lets you ask Google to fetch a file again. The report does not test a given URL against the rules.

This page fills that gap for any site, not only ones you have verified: it answers whether a path is allowed for a crawler and points at the deciding line, the way the old tester did. Use the robots.txt report too, to confirm when Google last read your file.

Each documented token below was checked against its company's own crawler documentation on the date shown, and we check them again at least once a year. A token its company does not document is marked as commonly cited. Any other user agent can be tested by name.

The rules this tester applies come from these pages:

• Google: How Google interprets the robots.txt specification

  Source · Checked Oct 29, 2026

• RFC 9309: Robots Exclusion Protocol

  Source · Checked Oct 29, 2026

• Google: Introduction to robots.txt

  Source · Checked Oct 29, 2026

• Google: Block Search indexing with noindex

  Source · Checked Oct 29, 2026

• Google Search Console Help: robots.txt report

  Source · Checked Oct 29, 2026