Robotstxt AI MCP Server

by sharozdawa Your server? Claim it

local only

Robotstxt AI runs on your own machine — the client starts it, so there is no endpoint to ping. 33 installs a week from npm. Last commit 22 Mar 2026.

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Installs per day peak 10 · avg 5 · +3% w/w

a month agotoday

Installs / week

npm · robotstxt-ai-mcp

Stars

0 open issues

22 Mar 2026

Last commit

no releases in 90 days

MIT

License

TypeScript

Nothing serious here today

Today is the operative word: we check Robotstxt AI every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.

Connect this server

This server runs on your own machine — install it with the package manager and the client starts it for you. Package name taken from the official registry entry.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}

~/.codex/config.toml

[mcp_servers.robotstxt-ai]
command = "npx"
args = ["-y", "robotstxt-ai-mcp"]

.cursor/mcp.json

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}

.vscode/mcp.json

{
  "servers": {
    "robotstxt-ai": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}

Alternatives to Robotstxt AI

same job, measured the same way

RobotsGate

by workers-mike-tusa-robotsgate

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

4 tools answering

Robots.txt & AI Crawler Checker

by workers-quietworklabs-robots-check

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

3 tools answering

Citevana AI visibility tools

by citevana

Free MCP tools for AI-search visibility: crawler checks, page audits, and llms.txt generation.

4 tools answering

AI Visibility

by lastminutedealshq

AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.

4 tools answering

PageCrawl

by pagecrawl

Monitor web pages for changes with AI-powered change detection, diffs, and alerts.

answering

AI Crawler Readiness

by lumnika

Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

5 tools answering

Crawl Readiness

by grubbomatic

Check what AI crawlers may read and what a page permits, generate llms.txt and schema.

744 installs/wk local only

Website to Markdown — Bounded Crawls for Agents

by kriton-dev

Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay

answering

Robotstxt AI — questions

Answers built from our own checks of this server.

Why is there no uptime for Robotstxt AI?

Robotstxt AI runs on your own machine over stdio — there is no network address to reach, so uptime cannot be measured for it by anyone. What can be measured is adoption: the npm package robotstxt-ai-mcp was installed 33 times last week.

How do I connect Robotstxt AI?

Copy the ready config from this page. We generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each in the form that client actually reads. It runs locally, so the command pulls robotstxt-ai-mcp straight from npm; nothing to host, nothing to sign up for.

How many people use Robotstxt AI?

The npm package robotstxt-ai-mcp was installed 33 times in the last week. Week over week that is +3%. We show installs rather than GitHub stars on purpose: a star is a bookmark, an install is someone actually running it.

Is Robotstxt AI open source?

Yes — it is published under the MIT licence, written in TypeScript and 1 stars on GitHub. The source link is on this page, so you can read exactly what it does with your data before you connect it.