mcpbeat Sign in

RobotsGate MCP Server

by workers-mike-tusa-robotsgate Your server? Claim it
answering

RobotsGate is answering right now. Last checked 10 min ago. It exposes 4 tools.

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Uptime history 9 days of history
9 days agonow
100.0%
Uptime 24h
91 of 91 checks
4
Tools
read from the server
175 ms
Response time
average over 24h
open, no key
Access
streamable-http

What changed 1

Every tool that appeared, vanished or quietly changed what it asks for. Recorded since 19 October 2026. No other catalogue keeps this.

19 Oct a tool changed version

Nothing serious here today

Today is the operative word: we check RobotsGate every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.

Three servers free · no card

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 10 min ago.

run in your terminal
claude mcp add robotsgate --transport http https://robotsgate.mike-tusa.workers.dev/mcp
Settings → Connectors → Add custom connector, then paste this address
https://robotsgate.mike-tusa.workers.dev/mcp
~/.codex/config.toml
[mcp_servers.robotsgate]
url = "https://robotsgate.mike-tusa.workers.dev/mcp"
.cursor/mcp.json
{
  "mcpServers": {
    "robotsgate": {
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}
.vscode/mcp.json
{
  "servers": {
    "robotsgate": {
      "type": "http",
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}

Available tools 4

Read directly from the server with tools/list, grouped by what they act on. If a tool disappears, we record the date.

crawlers
list_crawlers
Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
generate
generate_robots
Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
site
check_site
Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
validate
validate_robots
Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.

Endpoints

URLTransportStateLatencyChecked
https://robotsgate.mike-tusa.workers.dev/mcp streamable-http answering 100 ms 10 min ago

Alternatives to RobotsGate

same job, measured the same way
Crawl Readiness
by grubbomatic

Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.

63 installs/wk local only
Robotstxt AI
by sharozdawa

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

22 installs/wk local only
Citevana AI visibility tools
by citevana

Free MCP tools for AI-search visibility: crawler checks, page audits, and llms.txt generation.

4 tools answering
AI Visibility
by lastminutedealshq

AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.

4 tools answering
ActableSite AI Crawler Monitor
by actablesite

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

5 tools answering
Site SEO & AI Crawler Audit
by openkrill

Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.

10 tools answering
Matchstick
by astralchemist

Checks a site build: sitemaps, structured data, robots.txt and what crawlers see.

189 installs/wk local only
polite-crawler
by mark1russell7

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

27 installs/wk local only

RobotsGate — questions

Answers built from our own checks of this server.

What can RobotsGate do?
It exposes 4 tools, read directly from the server on our last check. Among them: check_site, generate_robots, list_crawlers, validate_robots. The full list with descriptions is on this page — we take it from the server itself via tools/list, not from a README. How MCP servers expose tools in the first place →
Is RobotsGate working right now?
We send a real MCP handshake every 15 minutes. Over the last 24 hours 91 of 91 checks got a reply (100.0%), average response time 175 ms. The bar chart above shows every period we have measured.
How do I connect RobotsGate?
Copy the ready config from this page. We generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each in the form that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address.
Does RobotsGate need an API key?
No. RobotsGate completed a full MCP handshake with us as an anonymous client and listed its tools without asking for anything. All 4 of them are readable on this page. This is what we observed, not what the docs claim.
How fast is RobotsGate?
It answers our handshake in 175 ms on average, which is faster than 84% of all working MCP servers we measure. That puts it in the quick quarter of the ecosystem. The comparison comes from our own checks across the whole registry, every 15 minutes.