AI Crawler Readiness MCP Server

by lumnika Your server? Claim it

answering

AI Crawler Readiness is answering right now. Last checked 7 min ago. It exposes 5 tools.

Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

Uptime history 4 days of history · worst day 98%

4 days agonow

98.9%

Uptime 24h

90 of 91 checks

Tools

read from the server

560 ms

Response time

average over 24h

authentication required

Access

streamable-http

AI Crawler Readiness does not always answer

Over the last week it answered 98.6% of our checks. We check every 15 minutes, so you hear about the next outage within the hour — not from your users.

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 7 min ago.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add ai-crawler-readiness --transport http https://shop.lumnika.com/ai-readiness/mcp

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "ai-crawler-readiness": {
      "url": "https://shop.lumnika.com/ai-readiness/mcp"
    }
  }
}

~/.codex/config.toml

[mcp_servers.ai-crawler-readiness]
url = "https://shop.lumnika.com/ai-readiness/mcp"

.cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-readiness": {
      "url": "https://shop.lumnika.com/ai-readiness/mcp"
    }
  }
}

.vscode/mcp.json

{
  "mcpServers": {
    "ai-crawler-readiness": {
      "url": "https://shop.lumnika.com/ai-readiness/mcp"
    }
  }
}

This endpoint answered with an authorization challenge. The server is running, but it did not say what kind of credentials it expects.

Available tools 5

Read directly from the server with tools/list, grouped by what they act on. If a tool disappears, we record the date.

domain

domain_readiness

Can AI assistants actually read this domain? Returns the measured readiness score, the per-crawler verdict (served / blocked in robots.txt / refused by the edge despite robots.txt allowing it), and which edge vendor answers for the host. Measured with real HTTP requests, not guessed from robots.txt alone. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: they are flagged robotsOnly, carry no server verdict, and are excluded from every count about what the server did. Also returns `fixes`: the ordered, domain-specific list of what to change to let the blocked crawlers in, separating what is written in the site own robots.txt from what the CDN/WAF applies on top of it.

edge

edge_blocking

Who is actually doing the blocking: for each CDN/WAF vendor, the share of (domain x crawler) pairs that robots.txt ALLOWS and the server refuses anyway — i.e. how much of the blocking is an infrastructure default rather than a decision the site owner made. Each vendor also comes broken down per crawler, which separates a blanket wall (same rate for every crawler) from a managed block list that names some AI user-agents and not others.

measure

measure_domain

Measure a domain RIGHT NOW instead of reading the archive: 9 real HTTP requests, one from a browser user-agent and one per AI crawler user-agent, plus robots.txt / llms.txt / sitemap. Use it for any site the index has not reached yet, or when the caller wants a fresh verdict after changing robots.txt or a WAF rule. Returns the same `fixes` list as domain_readiness, derived from the fresh measurement.

recent

recent_changes

What CHANGED: the domains that recently started or stopped blocking a specific AI crawler, with the day the flip was observed and whether it happened in robots.txt or at the edge. This cannot be reconstructed after the fact from any public source — it exists only because the index made the same requests the day before and the day after. Use it to answer 'who just blocked/unblocked ChatGPT, Claude, Perplexity...' or to watch one domain over time.

web

web_openness

How much of the most-visited web is readable by AI assistants right now: the share of measured domains that block at least one AI crawler, and the served / robots-blocked / edge-blocked breakdown per crawler.

Endpoints

URL	Transport	State	Latency	Checked
https://shop.lumnika.com/ai-readiness/mcp	streamable-http	needs auth	255 ms	7 min ago

Alternatives to AI Crawler Readiness

same job, measured the same way

Robotstxt AI

by sharozdawa

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

25 installs/wk local only

AI Visibility Checker

by krisdiallo

AI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.

local only

llms.txt Checker - AI Readiness Audit

by apifymcpfactory-droid

Checks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.

answering

Crawl Readiness

by grubbomatic

Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.

728 installs/wk local only

Momentic

by momenticmarketing

Free SEO, GEO, and AEO audits: analyze any page or domain, AI-crawler access, agent readiness.

11 tools answering

SEO Audit Console

by houtini-ai

Technical SEO in Claude: GSC + crawl + DataForSEO - audit, market sizing, content recon.

183 installs/wk local only

Robots Txt AI Crawler Audit

by jmshinhwa

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

365 installs/wk local only

ActableSite AI Crawler Monitor

by actablesite

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

5 tools answering

AI Crawler Readiness — questions

Answers built from our own checks of this server.

What can AI Crawler Readiness do?

It exposes 5 tools, read directly from the server on our last check. Among them: domain_readiness, edge_blocking, measure_domain, recent_changes, web_openness. The full list with descriptions is on this page — we take it from the server itself via tools/list, not from a README. How MCP servers expose tools in the first place →

Is AI Crawler Readiness working right now?

We send a real MCP handshake every 15 minutes. Over the last 24 hours 90 of 91 checks got a reply (98.9%), average response time 560 ms. The bar chart above shows every period we have measured.

How do I connect AI Crawler Readiness?

Copy the ready config from this page — we generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each with the file path that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address. You will need your own credentials: it refuses anonymous clients.

Does AI Crawler Readiness need an API key?

Yes. Every time we knock, AI Crawler Readiness answers with an authorization challenge instead of its tool list — that is how we know it is running and gated rather than broken. Bring your own credentials and it will talk.

How fast is AI Crawler Readiness?

It answers our handshake in 560 ms on average, which is faster than 26% of all working MCP servers we measure. The comparison comes from our own checks across the whole registry, every 15 minutes.