How to Test robots.txt Rules Before You Publish

Step-by-step

1. Paste your draft robots.txt, or fetch the live file from your site with the Robots.txt Tester.
2. Choose the crawler, such as Googlebot or Bingbot.
3. Enter a path you care about, for example /products/ or /private/report.html.
4. Select Test path and read whether it is allowed, which rule matched and any warnings.
5. Test several important URLs, including ones that should be blocked, then publish the file at the root of the site.

How rules are chosen

A crawler follows the most specific user-agent group that matches it, and ignores the * group when a more specific one exists. Among the rules in that group, the one with the longest matching path wins. If an Allow and a Disallow rule are equally specific, Allow wins.

Disallow is not noindex

Blocking a URL stops crawling, but it can still appear in results if other pages link to it. To keep a page out of search results, use a noindex tag and allow crawling so the tag can be read. A blocked page cannot be seen to carry noindex.

Mistakes that cost traffic

A leftover Disallow: / from a staging site is the most damaging one. Others include blocking CSS or JavaScript files that pages need to render, putting the file in a subfolder instead of the root, and forgetting that each subdomain needs its own file.

After publishing

Confirm the file loads at https://yourdomain.com/robots.txt, and use the robots.txt report in Google Search Console to check how Google reads it. Search engines may take time to refetch the file.