TheCrawler MCP Server

by manchittlab Your server? Claim it

local only

TheCrawler runs on your own machine — the client starts it, so there is no endpoint to ping. 44 installs a week from npm. Last commit 2 Jul 2026.

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

Installs per day peak 14 · avg 6 · -20% w/w

a month agotoday

Installs / week

npm · thecrawler

Stars

1 open issues

2 Jul 2026

Last commit

0 releases in 90 days

AGPL-3.0

License

TypeScript

What the code does

We read the source, 20 h ago · rules 3dff92dd89df

Capabilities

What this server is able to do. For an MCP server this is often the job itself — a terminal server runs commands because that is what it is for. Listed so you know what you are plugging in, not as an accusation.

File ships in the package but is absent from the source [пакет] bin/thecrawler.js:1

этот файл ставится пользователю, но в репозитории его нет

Is this your server and something here is wrong? Tell us — corrections are free and do not require a plan.

This code can reach further than it looks

We found places where it runs commands, builds paths or queries from values it is given. None of that is a flaw by itself — it becomes one when the code changes, and code changes quietly between releases. We re-read it on every one.

Connect this server

This server runs on your own machine — install it with the package manager and the client starts it for you. Package name taken from the official registry entry.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add thecrawler -- npx -y thecrawler

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "thecrawler": {
      "args": [
        "-y",
        "thecrawler"
      ],
      "command": "npx"
    }
  }
}

~/.codex/config.toml

[mcp_servers.thecrawler]
command = "npx"
args = ["-y", "thecrawler"]

.cursor/mcp.json

{
  "mcpServers": {
    "thecrawler": {
      "args": [
        "-y",
        "thecrawler"
      ],
      "command": "npx"
    }
  }
}

.vscode/mcp.json

{
  "mcpServers": {
    "thecrawler": {
      "args": [
        "-y",
        "thecrawler"
      ],
      "command": "npx"
    }
  }
}

Alternatives to TheCrawler

same job, measured the same way

RAG Vault

by robthepcguy

Local RAG MCP server with hybrid search, PDF/DOCX support, and zero-config setup

69 installs/wk local only

doc.page PDF Extraction

by doc

Extract PDFs to Markdown, RAG chunks and cited tables; publish tracked Doc Links with read stats.

7 tools answering

SQL Server MCP (macOS)

by tharanabope

SQL Server MCP with RAG capabilities for macOS (Docker SQL Server support)

25 installs/wk local only

Clean Markdown

by damonwill78-a11y

Any URL to clean, LLM-ready Markdown for RAG. Strips ads, nav, and boilerplate.

36 installs/wk local only

PageIndex MCP

by vectifyai

Reasoning-based RAG system for chatting with long PDFs. Supports local and online files.

64 installs/wk local only

OmniClip RAG

by msjsc001

Read-only local-first MCP server for private Markdown, PDF, and Tika-backed search on Windows.

local only

OmniClip RAG

by ellismorrow

Read-only local-first MCP server for private Markdown, PDF, and Tika-backed search on Windows.

local only

RAG

by fieldcure

MCP RAG server with hybrid search, multi-KB support, and AI-powered chunk contextualization.

local only

TheCrawler — questions

Answers built from our own checks of this server.

Why is there no uptime for TheCrawler?

TheCrawler runs on your own machine over stdio — there is no network address to reach, so uptime cannot be measured for it by anyone. What can be measured is adoption: the npm package thecrawler was installed 44 times last week.

How do I connect TheCrawler?

Copy the ready config from this page — we generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each with the file path that client actually reads. It runs locally, so the command pulls thecrawler straight from npm; nothing to host, nothing to sign up for.

How many people use TheCrawler?

The npm package thecrawler was installed 44 times in the last week. Week over week that is -20%. We show installs rather than GitHub stars on purpose: a star is a bookmark, an install is someone actually running it.

Is TheCrawler open source?

Yes — it is published under the AGPL-3.0 licence, written in TypeScript, 2 stars on GitHub and 1 open issue. The source link is on this page, so you can read exactly what it does with your data before you connect it.