Evalsmith MCP runs on your own machine — the client starts it, so there is no endpoint to ping. 379 installs a week from npm. Last commit 28 Oct 2026.
Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.
Nothing serious here today
Today is the operative word: we check Evalsmith MCP every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.
Connect this server
This server runs on your own machine — install it with the package manager and the client starts it for you. Package name taken from the official registry entry.
claude mcp add evalsmith-mcp -- npx -y evalsmith-mcp{
"mcpServers": {
"evalsmith-mcp": {
"command": "npx",
"args": [
"-y",
"evalsmith-mcp"
]
}
}
}[mcp_servers.evalsmith-mcp]
command = "npx"
args = ["-y", "evalsmith-mcp"]{
"mcpServers": {
"evalsmith-mcp": {
"command": "npx",
"args": [
"-y",
"evalsmith-mcp"
]
}
}
}{
"servers": {
"evalsmith-mcp": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"evalsmith-mcp"
]
}
}
}Optional settings from the registry entry, left out of the block above:
EVALSMITH_JUDGE_BASE_URL (OpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost)), EVALSMITH_JUDGE_MODEL (Model used by the optional LLM judge), EVALSMITH_JUDGE_API_KEY (API key for the judge endpoint, if it needs one), LOG_LEVEL (pino log level; logs go to stderr).
Alternatives to Evalsmith MCP
same job, measured the same wayDeterministic multi-hop graph retrieval for RAG. Zero LLM calls in the query path.
RAG retrieval IR metrics: recall@k, hit@k, MRR, NDCG@k, evaluate_batch.
Local-first agentic RAG with citations - hybrid search, reranking, multimodal document retrieval
Trace, evaluate, and optimize your LLM, RAG, and agent apps with Noveum observability.
Apple developer docs and WWDC transcripts with RAG retrieval and Jev relevance ranking.
Local-first memory for AI agents: SQLite FTS5, deterministic recall, no vector DB, no cloud.
Local FAISS vector database for RAG with document ingestion, semantic search, and MCP prompts.
Agent-first resource directory for AI agents: protocols, security, RAG, memory, evals, and more.
Evalsmith MCP — questions
Answers built from our own checks of this server.