01
Choose the crawler token
The default is Googlebot. A specific group replaces the wildcard group; it does not inherit wildcard rules. Enter the product token, not a full browser User-Agent header.
Crawl control
robots.txt tester— build and check one file
Edit a local robots.txt draft, then test one crawler and URL path. No website is fetched.
robots.txt tester
Path verdict
Allowed to crawl
- Selected group
- *
- Deciding rule
- Allow: /private/public/ · Line 3
The longest matching rule decides; Allow wins a tie.
Sitemap lines
- https://example.com/sitemap.xml · Line 4
Rules stay in this browser tab. No website request is sent.
Rule group builder
Current draft tested.
Before replacing a live robots file, a robots.txt tester can reveal whether a public page is caught by a broad block. The example starts with /private/ blocked but /private/public/ allowed.
Our robots.txt generator adds a reviewed rule group to the same text you test. It keeps comments and unknown lines untouched; a separate allow disallow checker is unnecessary when the deciding line is visible here.
Why use this robots.txt tester
See the exact rule that changes a crawl decision before a file goes live.
A narrow Allow can beat a broad block
The robots.txt tester compares every applicable rule. For /private/public/page, the longer Allow wins over Disallow: /private/. It names the winning line, so a permitted page is not mistaken for a blocked one.
Build, test, and export one source
Draft a User-agent group, inspect its exact text, then append it to the editor. The next test uses that same file, and the downloaded robots.txt contains the same bytes. Sitemap lines are shown separately because they do not decide access.
Test a draft without guessing
02
Check the path that matters
Use /private/public/page or paste a full URL; the tester uses only its path and query. Automatic testing waits 350 ms after changes. Turn it off to use the Test button or Ctrl/Cmd + Enter.
03
Read the deciding line before export
The result names the selected group, winning directive and source line. Any ignored syntax has its own line-numbered notice; copy and download remain disabled until the current draft is tested.
What the local rule engine shows
Specific group selection
It merges repeated groups with the same matching token, but a named Googlebot group does not inherit User-agent: * rules. That distinction explains many unexpected crawl results.
Longest path wins
Allow and Disallow paths use prefix matching, * wildcards and a final $ anchor. The longest matching path wins; an equally specific Allow beats Disallow.
Sitemap and diagnostics
Valid absolute Sitemap URLs appear with their source lines. Malformed rules and unsupported directives remain in the editor and are reported instead of being silently rewritten.
What this test cannot prove
- Only pasted text is tested
- This page never fetches the live /robots.txt, follows redirects or checks HTTP status. A local allowed verdict says nothing about the file currently served by a website.
- One crawler and one path at a time
- This robots.txt tester does not crawl the site or run a bot matrix. Re-test important paths and crawler tokens individually before publishing.
- Crawl permission is not privacy
- A Disallow rule asks compliant crawlers not to fetch a path. It cannot remove a linked URL from search results or protect private content; use noindex or server authorization as appropriate.
- Extensions do not change the verdict
- Crawl-delay, Host and unknown fields are preserved as text but ignored for local Allow/Disallow matching. A Sitemap URL is a discovery hint, not a permission rule.
- Existing rules are never repaired silently
- The builder appends a reviewed group after preview. It does not reorder comments, fix typos, merge existing groups in the exported file, or deploy the file for you.
robots.txt tester FAQ
Does robots.txt stop a page being indexed?
No. It controls fetching. A blocked URL can still appear in search when other pages link to it. Use an accessible noindex tag or header to request removal from the index.
Where does robots.txt go?
Put the downloaded file at the root of the relevant host, such as https://example.com/robots.txt. A copy inside a subdirectory does not govern the whole host.
How do I block AI crawlers in robots.txt?
Add a named User-agent group for each crawler you mean to address and a Disallow path. Preview the group and test the token here. Compliance depends on the crawler; robots.txt is not access control.
Why did Google retire its robots.txt Tester?
The old standalone Search Console tester was removed. This local rules check helps before deployment; Google Search Console remains the place to inspect what Google has fetched from your verified property.
Can I test full URLs?
Yes. Enter an HTTP or HTTPS URL and only its path and query are matched. The host is not requested or checked against the pasted file, so confirm you are testing the right site yourself.
Is a rule outcome unclear?
Use the robots.txt tester on your draft
Load the sample, change one rule, and inspect the deciding line. The robots.txt tester exports only the current, tested text.