robots.txt Tester

Fetch and test robots.txt rules.

Indexation and crawl Last reviewed Nov 11, 2026 Free, no signup

Input source

Fetch from URL Paste robots.txt

Site URL or domain

robots.txt content

Paste the full file to parse groups, sitemaps, and test a path.

Test path

User-agent

Custom token

Matches the User-agent: token in robots.txt. Default is *.

Enter a site URL or paste robots.txt, then run the check.

Results

→

Longest matching Allow/Disallow for this crawler (fallback *). Crawler implementations can still differ on wildcards.

Groups

Sitemaps

Raw robots.txt

Yes. Fair-use rate limits keep the suite available for everyone.

Yes. CDNs, registry redaction, and heuristics mean you should verify critical findings in your own environment.

Runs are processed to return results. Avoid pasting secrets or personal data you would not put in a public tool.