AI Crawler Readiness MCP Server

by lumnika Your server? Claim it

answering

AI Crawler Readiness is answering right now. Last checked 14 min ago. It exposes 5 tools.

Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

Uptime history 52 days of history · worst day 93%

52 days agonow

100.0%

Uptime 24h

91 of 91 checks

Tools

read from the server

387 ms

Response time

average over 24h

authentication required

Access

streamable-http

AI Crawler Readiness does not always answer

Over the last week it answered 99.1% of our checks. We check every 15 minutes, so you hear about the next outage within the hour — not from your users.

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 14 min ago.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add ai-crawler-readiness --transport http https://shop.lumnika.com/ai-readiness/mcp

Settings → Connectors → Add custom connector, then paste this address

https://shop.lumnika.com/ai-readiness/mcp

~/.codex/config.toml

[mcp_servers.ai-crawler-readiness]
url = "https://shop.lumnika.com/ai-readiness/mcp"

.cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-readiness": {
      "url": "https://shop.lumnika.com/ai-readiness/mcp"
    }
  }
}

.vscode/mcp.json

{
  "servers": {
    "ai-crawler-readiness": {
      "type": "http",
      "url": "https://shop.lumnika.com/ai-readiness/mcp"
    }
  }
}

This endpoint answered with an authorization challenge. The server is running, but it did not say what kind of credentials it expects.

Available tools 5

Read directly from the server with tools/list, grouped by what they act on. If a tool disappears, we record the date.

domain

domain_readiness

Can AI assistants actually read this domain? Returns the measured readiness score, the per-crawler verdict (served / blocked in robots.txt / refused by the edge despite robots.txt allowing it), and which edge vendor answers for the host. Measured with real HTTP requests, not guessed from robots.txt alone. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: they are flagged robotsOnly, carry no server verdict, and are excluded from every count about what the server did. Also returns `fixes`: the ordered, domain-specific list of what to change to let the blocked crawlers in, separating what is written in the site own robots.txt from what the CDN/WAF applies on top of it.

edge

edge_blocking

Who is actually doing the blocking: for each CDN/WAF vendor, the share of (domain x crawler) pairs that robots.txt ALLOWS and the server refuses anyway — i.e. how much of the blocking is an infrastructure default rather than a decision the site owner made. Each vendor also comes broken down per crawler, which separates a blanket wall (same rate for every crawler) from a managed block list that names some AI user-agents and not others.

measure

measure_domain

Measure a domain RIGHT NOW instead of reading the archive: 9 real HTTP requests, one from a browser user-agent and one per AI crawler user-agent, plus robots.txt / llms.txt / sitemap. Use it for any site the index has not reached yet, or when the caller wants a fresh verdict after changing robots.txt or a WAF rule. Returns the same `fixes` list as domain_readiness, derived from the fresh measurement.

recent

recent_changes

What CHANGED: the domains that recently started or stopped blocking a specific AI crawler, with the day the flip was observed and whether it happened in robots.txt or at the edge. This cannot be reconstructed after the fact from any public source — it exists only because the index made the same requests the day before and the day after. Use it to answer 'who just blocked/unblocked ChatGPT, Claude, Perplexity...' or to watch one domain over time.

web

web_openness

How much of the most-visited web is readable by AI assistants right now: the share of measured domains that block at least one AI crawler, and the served / robots-blocked / edge-blocked breakdown per crawler.

Endpoints

URL	Transport	State	Latency	Checked
https://shop.lumnika.com/ai-readiness/mcp	streamable-http	needs auth	179 ms	14 min ago

Alternatives to AI Crawler Readiness

same job, measured the same way

AI Crawler Access Checker

by nero-engine

Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.

answering

Robots.txt & AI Crawler Checker

by workers-quietworklabs-robots-check

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

3 tools answering

Why Won't AI Mention My Product

by whywontai

See what GPTBot, ClaudeBot and other crawlers get from a site: robots.txt checks and AI scans.

6 tools answering

Crawlerpeek

by crawlerpeek

Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.

147 installs/wk local only

Robotstxt AI

by sharozdawa

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

55 installs/wk local only

Unrendered: AI Crawler Checker

by missiongrowth

AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

97 installs/wk local only

Unrendered: AI Crawler Checker

by imfurkana

AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

local only

AI Visibility Checker

by krisdiallo

AI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.

local only

AI Crawler Readiness — questions

Answers built from our own checks of this server.

What can AI Crawler Readiness do?

It exposes 5 tools, read directly from the server on our last check. Among them: domain_readiness, edge_blocking, measure_domain, recent_changes, web_openness. The full list with descriptions is on this page — we take it from the server itself via tools/list, not from a README. How MCP servers expose tools in the first place →

Is AI Crawler Readiness working right now?

We send a real MCP handshake every 15 minutes. Over the last 24 hours 91 of 91 checks got a reply (100.0%), average response time 387 ms. The bar chart above shows every period we have measured.

How do I connect AI Crawler Readiness?

Copy the ready config from this page. We generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each in the form that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address. You will need your own credentials: it refuses anonymous clients.

Does AI Crawler Readiness need an API key?

Yes. Every time we knock, AI Crawler Readiness answers with an authorization challenge instead of its tool list — that is how we know it is running and gated rather than broken. Bring your own credentials and it will talk.

How fast is AI Crawler Readiness?

It answers our handshake in 387 ms on average, which is faster than 33% of all working MCP servers we measure. The comparison comes from our own checks across the whole registry, every 15 minutes.