AI Crawler Policy MCP Server

by lazymac2x Your server? Claim it

answering

AI Crawler Policy is answering right now. Last checked 11 min ago. Last commit 17 May 2026.

Cloudflare Workers MCP server: ai-crawler-policy

Uptime history 47 days of history

47 days agonow

100.0%

Uptime 24h

91 of 91 checks

—

Tools

hidden behind auth

166 ms

Response time

average over 24h

Stars

last commit 17 May 2026

Nothing serious here today

Today is the operative word: we check AI Crawler Policy every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 11 min ago.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add ai-crawler-policy --transport http https://api.lazy-mac.com/ai-crawler-policy/mcp

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "ai-crawler-policy": {
      "url": "https://api.lazy-mac.com/ai-crawler-policy/mcp"
    }
  }
}

~/.codex/config.toml

[mcp_servers.ai-crawler-policy]
url = "https://api.lazy-mac.com/ai-crawler-policy/mcp"

.cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-policy": {
      "url": "https://api.lazy-mac.com/ai-crawler-policy/mcp"
    }
  }
}

.vscode/mcp.json

{
  "mcpServers": {
    "ai-crawler-policy": {
      "url": "https://api.lazy-mac.com/ai-crawler-policy/mcp"
    }
  }
}

Endpoints

URL	Transport	State	Latency	Checked
https://api.lazy-mac.com/ai-crawler-policy/mcp	streamable-http	answering	170 ms	11 min ago

Alternatives to AI Crawler Policy

same job, measured the same way

Scrapling MCP Server

by d4vinci

Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.

169 037 installs/wk local only

Olostep MCP Server

by olostep

Search, scrape, and crawl the web for AI agents. Batch scraping and answers with citations.

71 installs/wk local only

Domduck

by domduck

Domain reputation over time: Domain Rating, Tranco rank, TLS, and AI crawler policy.

3 tools answering

Cloudflare Solver

by jojapi

Cloudflare Solver: Scraping API designed to bypass Cloudflare protection.

1 tools answering

MaxAEO AI Visibility MCP

by maxaeo

Local-first MCP server for AI visibility, GEO/AEO, llms.txt, and AI crawler readiness audits.

72 installs/wk local only

ActableSite AI Crawler Monitor

by actablesite

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

5 tools answering

Firecrawl MCP Server

by firecrawl

MCP server for Firecrawl — web search, scraping, and biomedical/arXiv paper search.

30 339 installs/wk local only

Evocrawl MCP Server

by superaihuman

MCP server for Evocrawl — search, scrape, and interact with the web.

27 installs/wk local only

AI Crawler Policy — questions

Answers built from our own checks of this server.

Is AI Crawler Policy working right now?

We send a real MCP handshake every 15 minutes. Over the last 24 hours 91 of 91 checks got a reply (100.0%), average response time 166 ms. The bar chart above shows every period we have measured.

How do I connect AI Crawler Policy?

Copy the ready config from this page — we generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each with the file path that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address.

Does AI Crawler Policy need an API key?

No. AI Crawler Policy completed a full MCP handshake with us as an anonymous client and listed its tools without asking for anything. This is what we observed, not what the docs claim.

How fast is AI Crawler Policy?

It answers our handshake in 166 ms on average, which is faster than 74% of all working MCP servers we measure. The comparison comes from our own checks across the whole registry, every 15 minutes.

Is AI Crawler Policy open source?

We cannot say either way: 0 stars on GitHub, but we could not determine the licence, and without one the code is not open source by default.