Robots Txt AI Crawler Audit MCP Server

by jmshinhwa Your server? Claim it

local only

Robots Txt AI Crawler Audit runs on your own machine — the client starts it, so there is no endpoint to ping. 310 installs a week from npm.

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

The linked repository no longer exists on GitHub — it was deleted or made private.

Installs per day peak 193 · avg 10

a month agotoday

310

Installs / week

npm · @readystack/robots-txt-ai-crawler-audit

—

Stars

on GitHub

—

Last commit

0 releases in 90 days

—

License

language unknown

This one has been quiet for a while

Quiet is not dead — but it is worth knowing when it wakes up, or when someone else takes it over. We watch the repository and tell you either way.

Connect this server

This server runs on your own machine — install it with the package manager and the client starts it for you. Package name taken from the official registry entry.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-audit

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ],
      "command": "npx"
    }
  }
}

~/.codex/config.toml

[mcp_servers.robots-txt-ai-crawler-audit]
command = "npx"
args = ["-y", "@readystack/robots-txt-ai-crawler-audit"]

.cursor/mcp.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ],
      "command": "npx"
    }
  }
}

.vscode/mcp.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ],
      "command": "npx"
    }
  }
}

Alternatives to Robots Txt AI Crawler Audit

same job, measured the same way

llms.txt Checker - AI Readiness Audit

by apifymcpfactory-droid

Checks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.

answering

ActableSite AI Crawler Monitor

by actablesite

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

5 tools answering

Scrape-LE

by nolindnaidoo

Analyse robots.txt content and report whether a path may be crawled.

41 installs/wk local only

Brainiall Web

by brainiall

Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.

5 tools answering

Robotstxt AI

by sharozdawa

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

29 installs/wk local only

Geo Tool Check

by shufflethis

Is this page readable for AI search? Crawler access, structure and citability checks, runs locally.

47 installs/wk local only

Geolint

by iliasabk

ESLint for AI search: audits AI-crawler access, llms.txt, schema and citability.

1 147 installs/wk local only

AI Visibility Checker

by krisdiallo

AI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.

local only

Robots Txt AI Crawler Audit — questions

Answers built from our own checks of this server.

Why is there no uptime for Robots Txt AI Crawler Audit?

Robots Txt AI Crawler Audit runs on your own machine over stdio — there is no network address to reach, so uptime cannot be measured for it by anyone. What can be measured is adoption: the npm package @readystack/robots-txt-ai-crawler-audit was installed 310 times last week.

Is Robots Txt AI Crawler Audit still maintained?

The linked repository no longer exists on GitHub — it was deleted or made private. We show this because it changes what you can expect: an unmaintained server may keep answering for months and then stop without warning.

How do I connect Robots Txt AI Crawler Audit?

Copy the ready config from this page — we generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each with the file path that client actually reads. It runs locally, so the command pulls @readystack/robots-txt-ai-crawler-audit straight from npm; nothing to host, nothing to sign up for.

How many people use Robots Txt AI Crawler Audit?

The npm package @readystack/robots-txt-ai-crawler-audit was installed 310 times in the last week. We show installs rather than GitHub stars on purpose: a star is a bookmark, an install is someone actually running it.