AI Crawler Checker — Robots.txt Policy for Six Bots MCP Server

by kriton-dev Your server? Claim it

answering

AI Crawler Checker — Robots.txt Policy for Six Bots is answering right now. Last checked 5 min ago.

Check declared robots.txt permissions for six AI crawlers at URL paths

Uptime history 6 days of history

6 days agonow

100.0%

Uptime 24h

92 of 92 checks

—

Tools

hidden behind auth

364 ms

Response time

average over 24h

API key

Access

streamable-http

Nothing serious here today

Today is the operative word: we check AI Crawler Checker — Robots.txt Policy for Six Bots every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 5 min ago.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add ai-crawler-access-check --transport http 'https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check'

Settings → Connectors → Add custom connector, then paste this address

https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check

~/.codex/config.toml

[mcp_servers.ai-crawler-access-check]
url = "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check"

.cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-access-check": {
      "url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check"
    }
  }
}

.vscode/mcp.json

{
  "servers": {
    "ai-crawler-access-check": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check"
    }
  }
}

This endpoint answered with an authorization challenge. The server is running, but you need an API key from its owner to call it.

Endpoints

URL	Transport	State	Latency	Checked
https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check	streamable-http	needs key	358 ms	5 min ago

Alternatives to AI Crawler Checker — Robots.txt Policy for Six Bots

same job, measured the same way

ActableSite AI Crawler Monitor

by actablesite

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

5 tools answering

Aeo MCP

by arthurcarlsonn

AEO tools for AI agents: crawler permissions, llms.txt, structured data, AI-readiness audit.

50 installs/wk local only

Matchstick

by astralchemist

Checks a site build: sitemaps, structured data, robots.txt and what crawlers see.

76 installs/wk local only

RobotsGate

by workers-mike-tusa-robotsgate

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

4 tools answering

Robots.txt & AI Crawler Checker

by workers-quietworklabs-robots-check

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

3 tools answering

Botcard

by whosthatbot

Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.

4 tools answering

Crawl Readiness

by grubbomatic

Check what AI crawlers may read and what a page permits, generate llms.txt and schema.

208 installs/wk local only

Robotstxt AI

by sharozdawa

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

52 installs/wk local only

AI Crawler Checker — Robots.txt Policy for Six Bots — questions

Answers built from our own checks of this server.

Why is there no tool list for AI Crawler Checker — Robots.txt Policy for Six Bots?

The server answered our handshake with an authorization challenge, so it is running — but it will not describe its tools to an anonymous client. To see them you need to connect with your own credentials. We record it as alive, not as broken: 100.0% of checks in the last 24 hours got a reply.

Is AI Crawler Checker — Robots.txt Policy for Six Bots working right now?

We send a real MCP handshake every 15 minutes. Over the last 24 hours 92 of 92 checks got a reply (100.0%), average response time 364 ms. The bar chart above shows every period we have measured.

How do I connect AI Crawler Checker — Robots.txt Policy for Six Bots?

Copy the ready config from this page. We generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each in the form that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address. You will need your own key: it refuses anonymous clients.

Does AI Crawler Checker — Robots.txt Policy for Six Bots need an API key?

Yes. Every time we knock, AI Crawler Checker — Robots.txt Policy for Six Bots answers with an authorization challenge instead of its tool list — that is how we know it is running and gated rather than broken. Bring your own credentials and it will talk.

How fast is AI Crawler Checker — Robots.txt Policy for Six Bots?

It answers our handshake in 364 ms on average, which is faster than 36% of all working MCP servers we measure. The comparison comes from our own checks across the whole registry, every 15 minutes.