Agent Crawl Readiness MCP Server

by sadri-dridi Your server? Claim it

answering

Agent Crawl Readiness is answering right now. Last checked 8 min ago. It exposes 31 tools. Last commit 23 Sep 2026.

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

Uptime history 47 days of history · worst day 91%

47 days agonow

100.0%

Uptime 24h

92 of 92 checks

Tools

read from the server

141 ms

Response time

average over 24h

Stars

last commit 23 Sep 2026

What changed 1

Every tool that appeared, vanished or quietly changed what it asks for. Recorded since 24 September 2026. No other catalogue keeps this.

24 Sep a tool description was rewritten agent-crawl-readiness

Nothing serious here today

Today is the operative word: we check Agent Crawl Readiness every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 8 min ago.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add agent-crawl-readiness --transport http https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

Settings → Connectors → Add custom connector, then paste this address

https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

~/.codex/config.toml

[mcp_servers.agent-crawl-readiness]
url = "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"

.cursor/mcp.json

{
  "mcpServers": {
    "agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

.vscode/mcp.json

{
  "servers": {
    "agent-crawl-readiness": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

Available tools 31

Read directly from the server with tools/list, grouped by what they act on. If a tool disappears, we record the date.

agent

agent-ads-index

Find where a product can be placed so AI agents discover it (Agent Ads). Follow start_here.hop first (302 with utm_source=agent-observatory&utm_medium=agent). Returns the website readiness check, machine catalog, llms.txt, client center, and the 7-day trial handoff for your operator. Task text discarded.

agent-crawl-readiness

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage HTML (Agent Ads). Pass website=<https://domain> to run the bounded check now and get the report here. 7-day trial, founding $50/mo (first 10, 12 mo) then $100/mo. Follow start_here.hop first (302). Task text discarded.

agent-tool-index

Find a public pay-per-call tool gateway (Monid). Follow start_here.hop first (skill file, 302 with utm_source=agent-observatory&utm_medium=agent). Also returns docs, CLI, and remote MCP hops. Task text discarded.

status

fetch-status

HEAD a public URL and return status only.

status-catalog

Short catalog of common HTTP status codes.

browser

browser-url-ok

Check whether a URL can be opened. No browser is launched.

calc

calc-eval

Evaluate a+b style arithmetic. Expression discarded.

citation

citation

Return host, scheme, and HTTP status for a URL. Page content is not stored.

compatibility

compatibility

Show how this request is classified. No identifiers are retained.

domain

domain-shape

Split a hostname into labels without WHOIS or ownership data.

figma

figma-url-shape

Parse a Figma file URL. Path discarded.

file

file-path-ok

Check whether a path looks safe. No disk access.

front

hn-front-count

Count current Hacker News front-page stories. Titles discarded.

geo

geo-hint

Latitude and longitude for a place via Open-Meteo.

github

github-repo-shape

Parse owner and repo from a GitHub URL.

iana

iana-zones

Common IANA timezone names for clock calls.

inspect

inspect-robots

Fetch a public robots.txt and return group counts only.

jira

jira-key-shape

Check a Jira issue key shape. Key discarded.

len

search-query-len

Count characters in a search query. Query discarded.

lib

lib-docs-hint

HTTP status for a public library docs URL. Body discarded.

memory

memory-key-count

Count keys in a JSON object. Values discarded.

normalize

normalize-url

Return origin, host, and path for a URL. Query and fragment are dropped.

people

people-search-index

Find a living people-search index (Ploid). Follow start_here.hop (302 with utm_source=agent-observatory&utm_medium=agent). Returns CLI setup, docs, API, and remote MCP hops. Query text discarded.

playwright

playwright-url-ok

Check whether a URL can be opened. No browser is launched.

think

think-steps

Return a three-step thinking template. Prompt discarded.

timezone

timezone

Current time in a named IANA timezone.

utc

utc-time

Current UTC timestamp.

validate

validate-json

Check whether a body is valid JSON. The body is discarded.

weather

weather-hint

Current temperature for a city via Open-Meteo.

web

web-fetch

HTTP status and content type for a public URL. Body discarded.

wiki

wiki-title-ok

Check whether a Wikipedia title exists. Title discarded after the call.

Endpoints

URL	Transport	State	Latency	Checked
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp	streamable-http	answering	62 ms	8 min ago

Alternatives to Agent Crawl Readiness

same job, measured the same way

llms.txt Checker - AI Readiness Audit

by apifymcpfactory-droid

Checks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.

answering

QueryRev AI Readiness

by queryrev

Checks a site for AI crawler access, robots.txt and llms.txt. Runs locally, no telemetry.

51 installs/wk local only

Geoscore

by cloudcorprecords

AI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.

286 installs/wk local only

pigfox.ai AI search readiness

by pigfox

Check if a site is ready for AI search: AI crawler access, llms.txt, JSON-LD, scored out of 100.

1 tools answering

CitedBy

by citedby

Check a page, AI-crawler access, llms.txt and backlinks of a site. Free, read-only.

4 tools answering

Agent Ready Website Audit

by workpods

SEO and AI search audit for any website: technical SEO, schema, AI crawler access and llms.txt.

12 tools answering

Robots.txt & AI Crawler Checker

by workers-quietworklabs-robots-check

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

3 tools answering

ActableSite AI Crawler Monitor

by actablesite

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

5 tools answering

Agent Crawl Readiness — questions

Answers built from our own checks of this server.

What can Agent Crawl Readiness do?

It exposes 31 tools, read directly from the server on our last check. Among them: agent-ads-index, agent-crawl-readiness, agent-tool-index, browser-url-ok, calc-eval, citation and 25 more. The full list with descriptions is on this page — we take it from the server itself via tools/list, not from a README. How MCP servers expose tools in the first place →

What is Agent Crawl Readiness mostly used for?

Its tools cluster around agent and status. That is what this server is built to work with — the grouping comes from the actual tool names, not from a category we assigned.

Is Agent Crawl Readiness working right now?

We send a real MCP handshake every 15 minutes. Over the last 24 hours 92 of 92 checks got a reply (100.0%), average response time 141 ms. The bar chart above shows every period we have measured.

How do I connect Agent Crawl Readiness?

Copy the ready config from this page. We generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each in the form that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address.

Does Agent Crawl Readiness need an API key?

No. Agent Crawl Readiness completed a full MCP handshake with us as an anonymous client and listed its tools without asking for anything. All 31 of them are readable on this page. This is what we observed, not what the docs claim.

How fast is Agent Crawl Readiness?

It answers our handshake in 141 ms on average, which is faster than 84% of all working MCP servers we measure. That puts it in the quick quarter of the ecosystem. The comparison comes from our own checks across the whole registry, every 15 minutes.

Is Agent Crawl Readiness open source?

We cannot say either way: 0 stars on GitHub, but we could not determine the licence, and without one the code is not open source by default.