polite-crawler runs on your own machine — the client starts it, so there is no endpoint to ping. 27 installs a week from npm. Last commit 25 Oct 2026.
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
What the code does
We read the source, 52 min ago · rules 995398eba69f
Things with no honest explanation: a promise that contradicts the code, code that runs at install time while hiding what it does, data leaving the machine.
[v4("169.254.169.254"), 32, "metadata"],
What this server is able to do. For an MCP server this is often the job itself — a terminal server runs commands because that is what it is for. Listed so you know what you are plugging in, not as an accusation.
execSync(`pnpm add ${filter} ${presetPackages.dependencies.join(" ")}`, {
for (const [name, value] of Object.entries(process.env)) {
Is this your server and something here is wrong? Tell us — corrections are free and do not require a plan.
We found things in this code
Code changes quietly between releases, and nobody reads the diff of a dependency. We do, on every release — watch polite-crawler and you get told the day something new turns up.
Connect this server
This server runs on your own machine — install it with the package manager and the client starts it for you. Package name taken from the official registry entry.
claude mcp add polite-crawler -- npx -y polite-crawler-mcp{
"mcpServers": {
"polite-crawler": {
"command": "npx",
"args": [
"-y",
"polite-crawler-mcp"
]
}
}
}[mcp_servers.polite-crawler]
command = "npx"
args = ["-y", "polite-crawler-mcp"]{
"mcpServers": {
"polite-crawler": {
"command": "npx",
"args": [
"-y",
"polite-crawler-mcp"
]
}
}
}{
"servers": {
"polite-crawler": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"polite-crawler-mcp"
]
}
}
}Optional settings from the registry entry, left out of the block above:
POLITE_CRAWLER_DIR (The folder of the store. The default is ~/.polite-crawler), POLITE_CRAWLER_CONTACT (An email address or URL for the From header of each request), POLITE_CRAWLER_ALLOW_PRIVATE (Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them), POLITE_CRAWLER_ALLOW_USER_PROVIDED (Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm), POLITE_CRAWLER_SIGN_KEY (The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT), POLITE_CRAWLER_SIGNATURE_AGENT (Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set).
Alternatives to polite-crawler
same job, measured the same wayRobots-aware website audits, pre-crawl quotes, and structured evidence for people and agents.
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.
Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
Live web search, image search, topic filters and full-text fetch over our own crawled index.
Free MCP tools for AI-search visibility: crawler checks, page audits, and llms.txt generation.
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
polite-crawler — questions
Answers built from our own checks of this server.