Robots.txt Tester
Paste a robots.txt file or fetch one from any public site, pick a crawler, and check whether a path is allowed or blocked.
Test how Googlebot is handled by sample crawl rules. Load the example to see the tool in action.
Paste a file to test it locally, or fetch the public robots.txt file for a website. Results are an interpretation of common robots.txt rules, not a guarantee of crawler behavior.
User-agent groups
- *3 Allow/Disallow rule(s)
- googlebot1 Allow/Disallow rule(s)
Sitemaps
https://example.com/sitemap.xml
Parser warnings
No parser warnings.
Rule matching supports * wildcards, end anchors ($), and the longest matching rule. If Allow and Disallow rules tie, Allow takes precedence.
What is Robots.txt Tester?
A robots.txt file tells crawlers which parts of a site they may request. This tester reads the rules you give it, finds the user-agent group that applies to the crawler you choose, and shows whether a path is allowed or blocked and which rule decided it. Use it before you deploy a change, or when a page is not being crawled and you suspect a rule is the cause. robots.txt controls crawling, not indexing, and it is not a security feature. Crawlers can interpret edge cases differently, so confirm important changes in Google Search Console.
Why use this tool?
- See which user-agent group applies to a crawler and which Allow or Disallow rule decided the result, including its line number.
- Test a rule change before you deploy it, so an important section is not blocked by accident.
- Paste a draft file or fetch a site's live robots.txt, then compare how Googlebot, Bingbot, GPTBot and other crawlers are treated.
- Spot syntax problems, such as a rule that appears before any User-agent line or a line with no colon, in the parser warnings.
How to use Robots.txt Tester
- Step 1. Paste your rules into the robots.txt content box, or enter a website URL and select Fetch live robots.txt to load the file from that site.
- Step 2. Choose a crawler such as Googlebot, Bingbot, GPTBot or DuckDuckBot, or select Custom and type a user-agent name.
- Step 3. Enter the path or full URL you want to test, for example /private/report.html.
- Step 4. Select Test path to see whether it is allowed or blocked, the matching rule and its line, the applicable groups, any sitemaps and parser warnings.
Common use cases
Why Googlebot can reach a path that other crawlers cannot
A crawler follows only the most specific user-agent group that matches it, so a Googlebot group replaces the * group for Googlebot.
User-agent: *
Disallow: /private/
Allow: /private/public/
User-agent: Googlebot
Disallow: /no-google/
# Results
# /private/report.html Bingbot -> Blocked (Disallow: /private/)
# /private/public/guide.html Bingbot -> Allowed (longer Allow rule wins)
# /private/report.html Googlebot -> Allowed (own group, * group ignored)
# /no-google/page.html Googlebot -> Blocked (Disallow: /no-google/)Tools to use next
Continue the task with related utilities for the next step in your workflow.
Important notes
- A crawler follows the most specific user-agent group that matches it and ignores the * group. If you add a Googlebot group, repeat any rules from the * group that should still apply to Googlebot.
- Disallow stops crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. Use a noindex tag and keep the page crawlable so the tag can be read.
- The file must be at the root of the host, such as https://example.com/robots.txt. Each subdomain and protocol needs its own file.
- Paths are case-sensitive, so /Private/ and /private/ are different rules.
- Fetch live robots.txt reads only /robots.txt from public websites. If the file returns an error such as HTTP 404, the tool reports it instead of testing.
- robots.txt is public and is not a security feature. Do not rely on it to keep content confidential.
More SEO Tools
Robots.txt Tester FAQ
How does the tester choose between Allow and Disallow?
Among the rules that match the path, the one with the longest pattern wins. If an Allow and a Disallow rule are equally specific, Allow wins. The tool supports * wildcards and the $ end anchor.
How does it choose which user-agent group applies?
It uses the group whose user-agent name best matches the crawler you selected, and falls back to the * group when no named group matches. Rules from less specific groups are not combined with it.
Why is Googlebot allowed when the * group blocks the path?
Googlebot reads only its own group when one exists. For example, with User-agent: * followed by Disallow: /private/ and a separate Googlebot group that does not mention /private/, Googlebot can crawl /private/ pages.
Does Disallow remove a page from Google?
No. Disallow only stops crawling. To keep a page out of search results, use a noindex tag and leave the page crawlable so search engines can see it.
Can I block AI crawlers such as GPTBot?
You can add a User-agent: GPTBot group with Disallow: / and test it here. Compliance is voluntary, so check your server logs to see whether a crawler follows the rule.
Does Google respect Crawl-delay?
No. Google ignores the Crawl-delay directive, although some other crawlers support it.
Is this the same as Google's own tester?
No. Google replaced its old robots.txt Tester with a robots.txt report in Search Console. This tool interprets common rules and does not guarantee crawler behavior, so confirm important changes with the URL Inspection tool.