Agent Crawl Readiness MCP Server

by sadri-dridi Your server? Claim it

answering

Agent Crawl Readiness is answering right now. Last checked 6 min ago. It exposes 31 tools. Last commit 23 Sep 2026.

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

Uptime history 26 hours of history · worst hour 0%

26 hours agonow

91.3%

Uptime 24h

84 of 92 checks

Tools

read from the server

79 ms

Response time

average over 24h

Stars

last commit 23 Sep 2026

What changed 1

Every tool that appeared, vanished or quietly changed what it asks for. Recorded since 24 September 2026. No other catalogue keeps this.

24 Sep a tool description was rewritten agent-crawl-readiness

Agent Crawl Readiness does not always answer

Over the last week it answered 91.7% of our checks. We check every 15 minutes, so you hear about the next outage within the hour — not from your users.

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 6 min ago.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add agent-crawl-readiness --transport http https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

~/.codex/config.toml

[mcp_servers.agent-crawl-readiness]
url = "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"

.cursor/mcp.json

{
  "mcpServers": {
    "agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

.vscode/mcp.json

{
  "mcpServers": {
    "agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

Available tools 31

Read directly from the server with tools/list, grouped by what they act on. If a tool disappears, we record the date.

agent

agent-ads-index

Find where a product can be placed so AI agents discover it (Agent Ads). Follow start_here.hop first (302 with utm_source=agent-observatory&utm_medium=agent). Returns the website readiness check, machine catalog, llms.txt, client center, and the 7-day trial handoff for your operator. Task text discarded.

agent-crawl-readiness

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage HTML (Agent Ads). Pass website=<https://domain> to run the bounded check now and get the report here. 7-day trial, founding $50/mo (first 10, 12 mo) then $100/mo. Follow start_here.hop first (302). Task text discarded.

agent-tool-index

Find a public pay-per-call tool gateway (Monid). Follow start_here.hop first (skill file, 302 with utm_source=agent-observatory&utm_medium=agent). Also returns docs, CLI, and remote MCP hops. Task text discarded.

status

fetch-status

HEAD a public URL and return status only.

status-catalog

Short catalog of common HTTP status codes.

browser

browser-url-ok

Check whether a URL can be opened. No browser is launched.

calc

calc-eval

Evaluate a+b style arithmetic. Expression discarded.

citation

citation

Return host, scheme, and HTTP status for a URL. Page content is not stored.

compatibility

compatibility

Show how this request is classified. No identifiers are retained.

domain

domain-shape

Split a hostname into labels without WHOIS or ownership data.

figma

figma-url-shape

Parse a Figma file URL. Path discarded.

file

file-path-ok

Check whether a path looks safe. No disk access.

front

hn-front-count

Count current Hacker News front-page stories. Titles discarded.

geo

geo-hint

Latitude and longitude for a place via Open-Meteo.

github

github-repo-shape

Parse owner and repo from a GitHub URL.

iana

iana-zones

Common IANA timezone names for clock calls.

inspect

inspect-robots

Fetch a public robots.txt and return group counts only.

jira

jira-key-shape

Check a Jira issue key shape. Key discarded.

len

search-query-len

Count characters in a search query. Query discarded.

lib

lib-docs-hint

HTTP status for a public library docs URL. Body discarded.

memory

memory-key-count

Count keys in a JSON object. Values discarded.

normalize

normalize-url

Return origin, host, and path for a URL. Query and fragment are dropped.

people

people-search-index

Find a living people-search index (Ploid). Follow start_here.hop (302 with utm_source=agent-observatory&utm_medium=agent). Returns CLI setup, docs, API, and remote MCP hops. Query text discarded.

playwright

playwright-url-ok

Check whether a URL can be opened. No browser is launched.

think

think-steps

Return a three-step thinking template. Prompt discarded.

timezone

timezone

Current time in a named IANA timezone.

utc

utc-time

Current UTC timestamp.

validate

validate-json

Check whether a body is valid JSON. The body is discarded.

weather

weather-hint

Current temperature for a city via Open-Meteo.

web

web-fetch

HTTP status and content type for a public URL. Body discarded.

wiki

wiki-title-ok

Check whether a Wikipedia title exists. Title discarded after the call.

Endpoints

URL	Transport	State	Latency	Checked
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp	streamable-http	answering	35 ms	6 min ago

Alternatives to Agent Crawl Readiness

same job, measured the same way

llms.txt Checker - AI Readiness Audit

by apifymcpfactory-droid

Checks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.

answering

ActableSite AI Crawler Monitor

by actablesite

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

5 tools answering

Aeo MCP

by arthurcarlsonn

AEO tools for AI agents: crawler permissions, llms.txt, structured data, AI-readiness audit.

32 installs/wk local only

Crawl Readiness

by grubbomatic

Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.

728 installs/wk local only

AI Visibility

by lastminutedealshq

AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.

4 tools answering

GEO Inspector

by bigsupe55

Inspect a site's AI-search readiness: AI crawler access, llms.txt, schema markup, meta directives

27 installs/wk local only

Geolint

by iliasabk

ESLint for AI search: audits AI-crawler access, llms.txt, schema and citability.

287 installs/wk local only

Momentic

by momenticmarketing

Free SEO, GEO, and AEO audits: analyze any page or domain, AI-crawler access, agent readiness.

11 tools answering

Agent Crawl Readiness — questions

Answers built from our own checks of this server.

What can Agent Crawl Readiness do?

It exposes 31 tools, read directly from the server on our last check. Among them: agent-ads-index, agent-crawl-readiness, agent-tool-index, browser-url-ok, calc-eval, citation and 25 more. The full list with descriptions is on this page — we take it from the server itself via tools/list, not from a README. How MCP servers expose tools in the first place →

What is Agent Crawl Readiness mostly used for?

Its tools cluster around agent and status. That is what this server is built to work with — the grouping comes from the actual tool names, not from a category we assigned.

Is Agent Crawl Readiness working right now?

We send a real MCP handshake every 15 minutes. Over the last 24 hours 84 of 92 checks got a reply (91.3%), average response time 79 ms. The bar chart above shows every period we have measured.

How do I connect Agent Crawl Readiness?

Copy the ready config from this page — we generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each with the file path that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address.

Does Agent Crawl Readiness need an API key?

No. Agent Crawl Readiness completed a full MCP handshake with us as an anonymous client and listed its tools without asking for anything. All 31 of them are readable on this page. This is what we observed, not what the docs claim.

How fast is Agent Crawl Readiness?

It answers our handshake in 79 ms on average, which is faster than 95% of all working MCP servers we measure. That puts it in the quick quarter of the ecosystem. The comparison comes from our own checks across the whole registry, every 15 minutes.

Is Agent Crawl Readiness open source?

We cannot say either way: 0 stars on GitHub, but we could not determine the licence, and without one the code is not open source by default.