01
Choose the crawler
Enter a crawler token such as Googlebot. The most specific matching user-agent group is selected.
ALL FREE SEO TOOLS
CRAWLER PATH TEST
Robots.txt tester
Fetch or paste a robots.txt file, choose a crawler, and test whether a specific path matches an Allow or Disallow rule.
HOW TO USE THE RESULT
Test the exact crawler and path you care about
Robots rules depend on the crawler name and URL path. Test representative public, private, parameter, and asset paths before publishing a change.
02
Enter a path
Test the path and query string beginning with a slash. Wildcards and end-of-path dollar signs are evaluated.
03
Read the matched line
The result shows the Allow or Disallow directive that decided the answer and its line number.
CHECK MORE THAN ONE PAGE
Run the free site check
The site check crawls several pages and returns a prioritized list of technical and content findings.
What this check covers
- You can fetch a public robots.txt file or paste proposed contents for a browser-only check.
- The parser reports groups, sitemap directives, unknown directives, and lines without a colon.
- This tool models common crawler matching behavior and does not contact a search engine.
- A crawl block does not guarantee that the URL will stay out of search results.
Frequently asked questions
Does robots.txt prevent indexing?
Robots.txt controls crawling. A blocked URL can still appear in search results when a search engine discovers it through links. Use an indexable response with a noindex directive when removal from search is the goal.
Which rule wins when Allow and Disallow both match?
This tester uses the longest matching rule. When matching rules have the same length, Allow wins.
Is Crawl-delay supported?
The parser recognizes Crawl-delay as a known directive, but it does not calculate request timing because major crawlers handle that directive differently.