Add Runner Eval Skill for Codex

Add a Runner Eval

Use this skill for the Runner Evals family: a real Runner/provider session

against a seeded mock control plane. Product browser/server/database/Daytona

coverage belongs in add-product-e2e-eval.

Locate the Paperclip checkout using PAPERCLIP_ROOT when supplied, or

git rev-parse --show-toplevel from a checkout. From outside Git, inspect the

workspace roots (for example ~/paperclipai/paperclip) and verify that the

selected root contains packages/paperclip-runner and tests/runner-e2e.

Locate paperclip-evals using PAPERCLIP_EVALS_ROOT or a discovered sibling;

a worktree's parent directory need not contain that repository. Read

doc/evals.md and packages/paperclip-runner/docs/runner-protocol-live-evals.md,

then inspect the nearest existing case, roster, schema, and report test before

editing. Definitions and authored cases belong in the sibling

paperclip-evals/evals/paperclip-runner; Runner integration, aggregation,

viewer, and publication behavior belongs in packages/paperclip-runner.

Keep the control-plane boundary explicit in names and documentation.

The sibling eval README is the concrete map: cases live under cases/,

company fixtures under fixtures/, runtime/model settings under configs/,

selections under rosters/, and maintained campaign membership under

campaigns/live-direct-full.json. Update inventory/coverage mappings when the

program requires them; a new file alone does not join the maintained campaign.

From the Evals repository root, adapt these provider-free checks to the case

and roster you changed. Run the reliability-plan validator only when that

separate plan changes:

python3 evals/paperclip-runner/tools/eval_program.py validate \
  --case evals/paperclip-runner/cases/get-task-context.json \
  --config evals/paperclip-runner/configs/live-codex-pinned.json
python3 evals/paperclip-runner/tools/run_live_roster.py validate \
  --roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
  --campaign evals/paperclip-runner/campaigns/live-direct-full.json
python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
  --plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json

Use nearby positive and negative grader cases/fixtures to calibrate the new

assertion, including malformed or missing evidence where the grader must fail

closed. Preserve the existing machine disposition and grade; product,

model/provider, grading, and infrastructure labels are analytical annotations,

not instructions to rewrite classifiers.

Author one bounded case with a deterministic semantic assertion and an

inspectable result. Declare its expected operation, state effect, provider

lane/profile, timeout and retry policy, and any required evidence. Do not grade

hidden reasoning, infer success from a provider terminal message, or invent

conversation/tool evidence. Public output follows the reviewed projection:

sanitized fixture conversation and allowlisted tool outcomes may be published;

raw trusted artifacts, credentials, secrets, private references, and hidden

reasoning may not.

Validate without provider calls first using the commands above and the relevant

report/render validation documented in the Runner docs. When a live run is

authorized, pin the Paperclip commit and exact 40-character

paperclip-evals commit, select the smallest useful roster, and retain the

complete provenance and cost record.

Update authoritative detailed docs when the contract or command changes, then

link from doc/evals.md rather than duplicating the Runner runbook. Keep public

reports immutable and use the reviewed projection; sanitized fixture

conversation and allowlisted tool outcomes may be public, while credentials,

secrets, private references, raw trusted payloads, and hidden reasoning must not

be exposed.