Robots.txt & Meta Robots Tester

Check whether any URL is crawlable and indexable for a given crawler — and get a plain-English explanation of every robots.txt, meta robots and X-Robots-Tag directive.

How to use

Paste your robots.txt

Copy it from yoursite.com/robots.txt, and enter the URL you want to test.

Add any robots tags

Optionally paste the page meta robots tag and X-Robots-Tag header to check indexability too.

Read the verdict

See whether the URL is crawlable and indexable, which rule applied, and what every directive means.

URL to test

Crawler (user-agent)

robots.txt contents filled by ‘Fetch live’, or paste your own

Meta robots optional · the page’s <meta name=robots> content

X-Robots-Tag optional · HTTP header value

Result

Crawlable

—

Enter a URL and your robots.txt.

Indexable

—

Directives explained

Directives you enter will be explained here.

What this tool does

Fetches your site’s live robots.txt (or accepts a pasted one) and evaluates a specific URL against it for a chosen crawler — Googlebot, Googlebot-Image, Bingbot or any (*) — showing exactly which rule wins and why.
Combines that crawlability result with any meta robots tag or X-Robots-Tag header you add, to give a single Indexable verdict, not just a crawl verdict.
Explains every directive found in plain English, including less-common ones like noarchive, max-snippet and unavailable_after.

Related tools

AI Crawler Access Auditor — The same robots.txt logic, applied specifically to 11 named AI crawlers
llms.txt Generator + Validator — robots.txt controls crawling; llms.txt is a separate, complementary AI-discovery file
SERP Snippet Preview — Once a page is confirmed crawlable and indexable, check how it will actually appear in results

Last updated: 19 August 2026 · Built by the CWA Europe PPC team.