CompletionKit MCP Server

by completionkit Your server? Claim it

answering

CompletionKit is answering right now. Last checked 5 min ago. It exposes 54 tools. Last commit 16 Sep 2026.

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

Uptime history 48 days of history

48 days agonow

100.0%

Uptime 24h

92 of 92 checks

Tools

read from the server

502 ms

Response time

average over 24h

Stars

last commit 16 Sep 2026

What changed 3

Every tool that appeared, vanished or quietly changed what it asks for. Recorded since 12 August 2026. No other catalogue keeps this.

16 Sep a tool changed version

12 Aug a tool changed version2 times that day

and 1 more, back to 12 August 2026

What the code does

We read the source, 22 h ago · tools taken from the live server · rules 3dff92dd89df

Capabilities

What this server is able to do. For an MCP server this is often the job itself — a terminal server runs commands because that is what it is for. Listed so you know what you are plugging in, not as an accusation.

Builds a database query by concatenation app/services/completion_kit/response_query.rb:47

      kept = sort_by_score(scored.select { |id| in_range?(averages[id]) }, averages)
      filtering? ? kept : kept + unscored

Is this your server and something here is wrong? Tell us — corrections are free and do not require a plan.

This code can reach further than it looks

We found places where it runs commands, builds paths or queries from values it is given. None of that is a flaw by itself — it becomes one when the code changes, and code changes quietly between releases. We re-read it on every one.

Connect this server

Endpoint below is the one we actually reach during checks — not the one copied from a README. Last verified 5 min ago.

Claude Code Claude Desktop Codex Cursor VS Code

run in your terminal

claude mcp add evals --transport http https://completionkit.com/mcp

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "evals": {
      "url": "https://completionkit.com/mcp"
    }
  }
}

~/.codex/config.toml

[mcp_servers.evals]
url = "https://completionkit.com/mcp"

.cursor/mcp.json

{
  "mcpServers": {
    "evals": {
      "url": "https://completionkit.com/mcp"
    }
  }
}

.vscode/mcp.json

{
  "mcpServers": {
    "evals": {
      "url": "https://completionkit.com/mcp"
    }
  }
}

Available tools 54

Read directly from the server with tools/list, grouped by what they act on. If a tool disappears, we record the date.

runs

runs_create

Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.

runs_delete

Delete a run

runs_generate

Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.

runs_get

Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.

runs_list

List all runs

runs_regrade

Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.

runs_rerun

Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.

runs_retry_failures

Re-run only the failed responses of a run, optionally limited to specific response ids via "only".

runs_update

Update a run

metric

metric_groups_create

Create a metric group

metric_groups_delete

Delete a metric group

metric_groups_get

Get a metric group by ID

metric_groups_list

List all metric groups

metric_groups_update

Update a metric group

metric_versions_dismiss

Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as current instead.

metric_versions_list

List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.

metric_versions_publish

Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.

prompts

prompts_create

Create a prompt

prompts_delete

Delete a prompt

prompts_get

Get a prompt by ID

prompts_list

List all prompts

prompts_publish

Publish a prompt version, making it the current version

prompts_suggest_improvement

Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).

prompts_update

Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.

datasets

datasets_create

Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.

datasets_create_from_url

Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.

datasets_delete

Delete a dataset

datasets_get

Get a dataset by ID

datasets_list

List all datasets

datasets_update

Update a dataset

metrics

metrics_create

Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.

metrics_delete

Delete a metric

metrics_get

Get a metric by ID

metrics_list

List all metrics

metrics_suggest_variants

Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.

metrics_update

Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.

provider

provider_credentials_create

Create a provider credential

provider_credentials_delete

Delete a provider credential

provider_credentials_get

Get a provider credential by ID (API key is not exposed)

provider_credentials_list

List all provider credentials (API keys are not exposed)

provider_credentials_update

Update a provider credential

Endpoints

URL	Transport	State	Latency	Checked
https://completionkit.com/mcp	streamable-http	answering	387 ms	5 min ago

Alternatives to CompletionKit

same job, measured the same way

Ollama / vLLM Bridge

by setheerwagen

Call your local Ollama or vLLM model over MCP with schema-validated JSON output

1 236 installs/wk local only

Dali by Lulu

by lulu-the-narwhal

The prediction MCP — score your prompt before you generate, so you never waste a credit.

303 installs/wk 16 tools answering

Ingressward

by ingressward

Block prompt injection, evasive spacing, and obscene content before it reaches your LLM.

22 installs/wk local only

Spec Score MCP

by openpoem

Score your specs before feeding them to an LLM. MCP server with radar chart visualization.

local only

Everyrow MCP Server

by futuresearch

Give your AI a research team. Forecast, score, classify, or research every row of a dataset.

345 installs/wk local only

Pixlint

by amitsingh-24

Lint, curate & prepare computer-vision datasets from your AI assistant — MCP server, 67 tools

89 installs/wk local only

retriEVAL

by hcarrillo001

LLM evals as MCP tools: score outputs for faithfulness, relevancy, and hallucination.

local only

mcp-audit

by saagpatel

Scan, enumerate, and risk-score every MCP server configured on your machine.

190 installs/wk local only

CompletionKit — questions

Answers built from our own checks of this server.

What can CompletionKit do?

It exposes 54 tools, read directly from the server on our last check. Among them: agreements_create, agreements_list, datasets_create, datasets_create_from_url, datasets_delete, datasets_get and 48 more. The full list with descriptions is on this page — we take it from the server itself via tools/list, not from a README. How MCP servers expose tools in the first place →

What is CompletionKit mostly used for?

Its tools cluster around runs, metric and prompts. That is what this server is built to work with — the grouping comes from the actual tool names, not from a category we assigned.

Is CompletionKit working right now?

We send a real MCP handshake every 15 minutes. Over the last 24 hours 92 of 92 checks got a reply (100.0%), average response time 502 ms. The bar chart above shows every period we have measured.

How do I connect CompletionKit?

Copy the ready config from this page — we generate it for Claude Code, Claude Desktop, Codex, Cursor and VS Code, each with the file path that client actually reads. It is a remote server, so there is nothing to install — the client connects to the address.

Does CompletionKit need an API key?

No. CompletionKit completed a full MCP handshake with us as an anonymous client and listed its tools without asking for anything. All 54 of them are readable on this page. This is what we observed, not what the docs claim.

How fast is CompletionKit?

It answers our handshake in 502 ms on average, which is faster than 29% of all working MCP servers we measure. The comparison comes from our own checks across the whole registry, every 15 minutes.

Is CompletionKit open source?

We cannot say either way: written in Ruby, 3 stars on GitHub and 14 open issues, but we could not determine the licence, and without one the code is not open source by default.