Free AI Crawler Checker

The checker fetches /robots.txt from the root of the site you enter, the same file every crawler reads before it fetches a page. It then works out, for each AI crawler on its list, which group of rules applies and which rule decides the page you entered. A bare domain is tested on the home page; a full address is tested on its own path and query string.

Each verdict says what the file tells that crawler, and nothing more. robots.txt is a request, not a lock: a crawler can ignore it, and a crawler the file allows may never visit. Some companies say their user-requested fetchers may not follow robots.txt at all, and the result shows that next to the crawler when the company says so.

How a site answers for /robots.txt matters as much as what the file says. The checker follows Google's documented handling for each kind of answer:

• A plain-text file: its rules apply, and every crawler is graded against them.
• 404, 410 or another 4xx error except 429: Google treats this as if there were no file, so every crawler may fetch every page.
• A 5xx server error: Google treats the whole site as disallowed while it retries, then falls back to its last good copy for up to 30 days.
• A redirect: Google follows it and uses the file it leads to. Google follows at least five redirects, so the checker warns about a longer chain. Other crawlers may handle redirects differently.
• A web page answering 200: that is a soft 404, not a robots.txt file, so it is reported as not found and never read as rules.
• A file over 500 KiB: Google reads only the first 500 KiB and ignores the rest, and so does the checker.

If the site refuses our checker with 401, 403, 429 or a bot challenge, the result says the site blocked our checker and gives no verdict. That answer tells us nothing about the file itself, so it is never reported as robots.txt blocking anyone. You can open the file in your browser and paste it instead.

A crawler first picks one group: the group that names its own user-agent token, or, only when no group names it, the User-agent: * group. The two are never combined, so a GPTBot group with a single rule replaces everything the * group says for GPTBot. Groups that name the same crawler more than once are merged into one.

Inside that group, the rule with the longest matching path wins, and when an Allow and a Disallow match with the same length, the Allow wins. A * in a rule matches any run of characters, and a $ at the end means the path must end there. So Disallow: /*.pdf$ blocks /guide.pdf but not /guide.pdf?download=1. Paths are compared in one percent-encoded form, so a rule written with an accented letter matches the encoded address a crawler requests.

To test a rule before you publish it, paste the draft file (up to 60 KB) and enter the path you care about. Each crawler's row shows the deciding rule and its line number, so you can see exactly which line to change.

Crawlers do not report mistakes in robots.txt. They skip the lines they cannot read and carry on, so a typo can quietly undo a rule you meant to set. Google's parser forgives a few common slips that other crawlers may not, so a file can work for Google and fail elsewhere. The checker reads each line the way Google's open-source parser does and lists every problem with its line number:

• An unknown directive, such as Disalloow, which Google skips.
• A misspelling Google reads anyway, such as Disalow or User agent. Google treats these as Disallow and User-agent, but other crawlers may skip the line, so spell them correctly.
• An Allow or Disallow line before the first User-agent line, which belongs to no group and so applies to no crawler.
• A Sitemap line that is not a full URL with the protocol and domain.
• A line with no colon. Google reads Disallow /private as a rule when the line is exactly two words; other crawlers may not.
• A rule path that does not start with a slash or a *.
• Directives Google does not support, such as Crawl-delay and Noindex, with what to use instead.
• A file larger than Google's 500 KiB limit.

Every token below was checked against the company's own crawler documentation on 2026-10-29. Tokens that no company documents itself are left out, because nothing first-party says what they do.

Google-Extended and Applebot-Extended are not crawlers. They never fetch a page. Google and Apple crawl with Googlebot and Applebot as usual, and these tokens only tell them whether the pages they fetched may be used to train their AI models. Disallowing them has no effect on Google Search or Apple's search results.

Most AI companies run separate crawlers for training and for search. OpenAI's GPTBot collects pages that may train its models, while OAI-SearchBot finds pages to show and link in ChatGPT search answers. Anthropic splits ClaudeBot and Claude-SearchBot the same way. Blocking a training crawler does not remove you from that company's search answers, and blocking a search crawler can keep you out of them even if you allow training.

Whether to allow training is your call. When the checker finds AI search or assistant crawlers held back from a page only by your User-agent: * rules, it builds a group you can add to your file to let them in, and leaves crawlers you named, and every training crawler, as you set them. None of these tokens is Googlebot, so none of these rules changes how Google ranks your pages.