Robots.txt & AI Crawler Checker MCP Server

by workers-quietworklabs-robots-check Your server? Claim it

answering

Robots.txt & AI Crawler Checker is answering right now. Last checked 3 min ago. It exposes 3 tools.

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Uptime history 8 days of history

8 days agonow

100.0%

Uptime 24h

92 of 92 checks

Tools

read from the server

198 ms

Response time

average over 24h

open, no key

Access

streamable-http

Nothing serious here today

Today is the operative word: we check Robots.txt & AI Crawler Checker every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 3 min ago.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add robots-check --transport http https://robots-check.quietworklabs.workers.dev/mcp

Settings → Connectors → Add custom connector, then paste this address

https://robots-check.quietworklabs.workers.dev/mcp

~/.codex/config.toml

[mcp_servers.robots-check]
url = "https://robots-check.quietworklabs.workers.dev/mcp"

.cursor/mcp.json

{
  "mcpServers": {
    "robots-check": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

.vscode/mcp.json

{
  "servers": {
    "robots-check": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

Available tools 3

Read directly from the server with tools/list, grouped by what they act on. If a tool disappears, we record the date.

crawler

ai_crawler_access

AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

robots

robots_check

Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.

sitemap

sitemap_check

Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Endpoints

URL	Transport	State	Latency	Checked
https://robots-check.quietworklabs.workers.dev/mcp	streamable-http	answering	142 ms	3 min ago

Alternatives to Robots.txt & AI Crawler Checker

same job, measured the same way

SEO Records

by seorecords

Free SEO check of any website, AI crawler access test, and the full Audit SEO Records.

5 tools answering

Sap Adt MCP

by yzonur

Give Claude live access to SAP ABAP via ADT: read, search, check, test, and edit objects.

54 installs/wk local only

Skillproof

by skillproofdev

Check whether a Claude Code skill was tested and whether it works before installing it.

14 installs/wk local only

Vellium & Co Testing Lab

by velliumandco

Independent checks of paid x402 data sellers and of AI-agent access to shopping and travel sites

9 tools answering

ProdPoke

by prodpoke

AI QA tester — real browsers scan sites for bugs, SEO, perf, and accessibility issues via chat.

answering

rowstile

by rowstile

Check, test, prove and review a rowstile policy: access rules for Postgres row-level security

218 installs/wk local only

Wcag Accessibility

by aditya-ariosity

A WCAG accessibility testing MCP for AI agents, Codex, Claude Code, and automated UI audits.

16 installs/wk local only

fline

by fline

Hosting for apps you build with AI. Tested with Claude Code and Codex. Private beta: ask for access.

40 tools answering

Robots.txt & AI Crawler Checker — questions

Answers built from our own checks of this server.

What can Robots.txt & AI Crawler Checker do?

It exposes 3 tools, read directly from the server on our last check. Among them: ai_crawler_access, robots_check, sitemap_check. The full list with descriptions is on this page — we take it from the server itself via tools/list, not from a README. How MCP servers expose tools in the first place →

Is Robots.txt & AI Crawler Checker working right now?

We send a real MCP handshake every 15 minutes. Over the last 24 hours 92 of 92 checks got a reply (100.0%), average response time 198 ms. The bar chart above shows every period we have measured.

How do I connect Robots.txt & AI Crawler Checker?

Copy the ready config from this page. We generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each in the form that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address.

Does Robots.txt & AI Crawler Checker need an API key?

No. Robots.txt & AI Crawler Checker completed a full MCP handshake with us as an anonymous client and listed its tools without asking for anything. All 3 of them are readable on this page. This is what we observed, not what the docs claim.

How fast is Robots.txt & AI Crawler Checker?

It answers our handshake in 198 ms on average, which is faster than 79% of all working MCP servers we measure. That puts it in the quick quarter of the ecosystem. The comparison comes from our own checks across the whole registry, every 15 minutes.