Robots Txt AI Crawler Audit runs on your own machine — the client starts it, so there is no endpoint to ping. 59 installs a week from npm. Last commit 2 Nov 2026.
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
Nothing serious here today
Today is the operative word: we check Robots Txt AI Crawler Audit every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.
Connect this server
This server runs on your own machine — install it with the package manager and the client starts it for you. Package name taken from the official registry entry.
claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-audit{
"mcpServers": {
"robots-txt-ai-crawler-audit": {
"command": "npx",
"args": [
"-y",
"@readystack/robots-txt-ai-crawler-audit"
]
}
}
}[mcp_servers.robots-txt-ai-crawler-audit]
command = "npx"
args = ["-y", "@readystack/robots-txt-ai-crawler-audit"]{
"mcpServers": {
"robots-txt-ai-crawler-audit": {
"command": "npx",
"args": [
"-y",
"@readystack/robots-txt-ai-crawler-audit"
]
}
}
}{
"servers": {
"robots-txt-ai-crawler-audit": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"@readystack/robots-txt-ai-crawler-audit"
]
}
}
}Alternatives to Robots Txt AI Crawler Audit
same job, measured the same wayChecks a site build: sitemaps, structured data, robots.txt and what crawlers see.
Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
Checks a site for AI crawler access, robots.txt and llms.txt. Runs locally, no telemetry.
Checks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.
AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
What search and AI crawlers extract from any public URL. Five of nine tools need no token.
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
29 free read-only growth tools: CAC, ROAS, PPC, schema, meta tags, robots.txt and AI crawler checks.
Robots Txt AI Crawler Audit — questions
Answers built from our own checks of this server.