evalpair MCP Server

by srikarmk Your server? Claim it

local only

evalpair runs on your own machine — the client starts it, so there is no endpoint to ping. 752 installs a week from pypi. Last commit 5 Nov 2026.

Run LLM eval suites on local or hosted models; compare runs with intervals and a noise floor.

Installs per day peak 645 · avg 25

a month agotoday

752

Installs / week

pypi · evalpair

Stars

0 open issues

5 Nov 2026

Last commit

from GitHub

MIT

License

Python

Nothing serious here today

Today is the operative word: we check evalpair every 15 minutes and re-read its code on every release. Watch it and you find out the day that stops being true.

Connect this server

This server runs on your own machine — install it with the package manager and the client starts it for you. Package name taken from the official registry entry.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add evalpair -- uvx evalpair

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ]
    }
  }
}

~/.codex/config.toml

[mcp_servers.evalpair]
command = "uvx"
args = ["evalpair"]

.cursor/mcp.json

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ]
    }
  }
}

.vscode/mcp.json

{
  "servers": {
    "evalpair": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalpair"
      ]
    }
  }
}

Optional settings from the registry entry, left out of the block above: EVALPAIR_HOME (Folder for the call cache, saved runs and your own suites (default ~/.evalpair)), EVALPAIR_API_KEY (Bearer token for an OpenAI-compatible server, if it needs one (OPENAI_API_KEY also works)).

Alternatives to evalpair

same job, measured the same way

apimodels

by stimqq

Image, video, chat and speech models with one account — local (npx) or hosted with OAuth sign-in.

203 installs/wk answering

LLM Eval Harness

by razilzul

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

local only

RuralCheck

by ruralcheck

Location analysis for rural property in Europe: flood, noise, sun hours, land register and more.

answering

Loci

by askloci

Run local models on your Apple silicon Mac with Loci. Requires macOS 26.2 or later.

local only

Welt

by ergodic-welt

Run Welt's tabular, causal and forecasting models on your data. Hosted; OAuth sign-in.

answering

CodeZaiku

by wyrdsekai

A coding and development harness on local models or hosted APIs, for code, review, research and ops.

481 installs/wk local only

LLMSlim memory

by thanatos9404

Long-term memory and context fitting for agents, with no LLM in the loop. Local SQLite.

817 installs/wk local only

Casatoo

by casatoo

Search and compare current homes across Portuguese real-estate portals with Casatoo's hosted MCP.

6 tools answering

evalpair — questions

Answers built from our own checks of this server.

Why is there no uptime for evalpair?

evalpair runs on your own machine over stdio — there is no network address to reach, so uptime cannot be measured for it by anyone. What can be measured is adoption: the pypi package evalpair was installed 752 times last week.

How do I connect evalpair?

Copy the ready config from this page. We generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each in the form that client actually reads. It runs locally, so the command pulls evalpair straight from pypi; nothing to host, nothing to sign up for.

How many people use evalpair?

The pypi package evalpair was installed 752 times in the last week. We show installs rather than GitHub stars on purpose: a star is a bookmark, an install is someone actually running it.

Is evalpair open source?

Yes — it is published under the MIT licence, written in Python and 0 stars on GitHub. The source link is on this page, so you can read exactly what it does with your data before you connect it.