Model Evaluation Agent Skill

Model-Evaluation Skill

Purpose

This skill makes a medical-imaging model's held-out evaluation task-correct and honest: the right

metric for the task and the prevalence, with uncertainty, calibration, and subgroup performance. It

emits a per-case metric table that the publication statistics build on, and gates the metric choice

against Metrics Reloaded (Maier-Hein & Reinke et al., *Nat Methods* 2024) and CLAIM 2024.

It sits between /model-validation (which audits the split / design) and /analyze-stats (which owns

the comparative inference). It computes the imaging-specific per-case metrics (surface distances, FROC,

ECE of a softmax head); /analyze-stats owns DeLong / NRI / IDI / decision curves / MRMC. Like

/analyze-stats, it generates and executes code on your predictions — numbers are never hand-typed.

When to use

• You have held-out predictions + ground truth and need task-correct metrics with CIs, calibration, and

subgroup slices, plus a per-case table for the manuscript statistics.

When NOT to use

• Auditing the validation design / leakage → /model-validation.
• DeLong / NRI / IDI / decision curves / MRMC reader study → /analyze-stats.
• Building / training the model → /model-scaffold; LLM / MLLM → /mllm-eval.
• Figure rendering → /make-figures.

Workflow

Phase 1 — Fix the analysis unit and the task

State the task (segmentation / classification / detection / interactive / generative) and the analysis unit the metric must

respect (per-patient vs per-lesion vs per-image). A per-lesion metric must not be reported as

per-patient.

Phase 2 — Compute task-correct metrics

Generate evaluation code that computes, on the held-out predictions:

• segmentation: Dice/IoU and a boundary metric (HD95 / NSD), per structure not only a global

mean, with bootstrap 95% CIs.

• classification: AUROC and AUPRC with bootstrap CIs, sensitivity/specificity, and PPV/NPV **at

the deployment prevalence** (not a balanced set).

• detection: FROC / mAP with the IoU match criterion stated.
• interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive): the segmentation

metrics above plus the interaction axis — Dice-vs-interactions / number-of-clicks (NoC) to a

target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time

(see the metric guide; the study design is in /design-study + /model-validation).

• generative / synthesis (image generation or modification): full-reference similarity

(MSE/RMSE/PSNR/SSIM) or no-reference quality (SNR/CNR, standardized visual scores), **plus a

downstream-task evaluation** — image quality is not clinical utility (Park et al., *Radiol Med* 2024).

For multiclass classification, state the aggregation scheme (one-vs-rest / macro / micro /

pairwise / Obuchowski); time-to-event discrimination (Harrell's C, time-dependent ROC) is handed

to /analyze-stats.

Add calibration (reliability diagram / ECE) and subgroup slices (the Model Card Factors).

See ${CLAUDE_SKILL_DIR}/references/metric_guide.md. Emit a per-case CSV for /analyze-stats.

Phase 3 — Gate the metric choice (deterministic)

python3 ${CLAUDE_SKILL_DIR}/scripts/check_metric_reporting.py \
  --report results.md --task segmentation|classification|detection|interactive|generative --strict

PIXEL_ACCURACY_SEG / NO_BOUNDARY_METRIC / ACCURACY_ONLY / DETECTION_METRIC_MISSING must be zero.

Phase 4 — Hand off

The per-case table → /analyze-stats (DeLong / NRI / IDI / decision curves, publication tables);

figures → /make-figures; the numbers + subgroup performance → /model-card; Methods/Results →

/write-paper; compliance → /check-reporting.

Anti-Hallucination

• Never fabricate a metric value. Every number comes from executed code on the supplied predictions;

if predictions or ground truth are missing, say so and stop — do not invent a result.

• Never report pixel/voxel accuracy for segmentation or bare accuracy under imbalance — the gate

flags these; report Dice + a boundary metric, or AUROC + AUPRC with CIs.

• Never report a per-lesion metric as if it were per-patient — respect the analysis unit.
• If a metric definition or its CI method is uncertain, flag [VERIFY] and ask.

Deterministic gate

scripts/check_metric_reporting.py — flags a task-metric mismatch / missing uncertainty (stdlib,

network-free). Reproducible challenge: bash ${CLAUDE_SKILL_DIR}/scripts/metric_reporting_challenge/verify.sh.

Reference Files

Load on demand (keep SKILL.md short):

• ${CLAUDE_SKILL_DIR}/references/metric_guide.md — operational checklist: the task-correct metric

per task (segmentation Dice + HD95/NSD per structure; classification AUROC + AUPRC + sens/spec at

deployment prevalence; detection FROC/mAP with a stated IoU), plus calibration, subgroup slices,

run-variance, and the per-case CSV hand-off.

• ${CLAUDE_SKILL_DIR}/references/metric_selection_grounding.md — the standards grounding behind

those choices: the Metrics Reloaded task-fingerprint principle, why each metric pairing is

required, calibration vs discrimination, disaggregated reporting, and the CLAIM 2024

reporting-fit map (/check-reporting owns the item audit).

Boundaries

model-validation (design) -> model-evaluation (this skill: per-case task-correct metrics + CIs)
  -> analyze-stats (DeLong / NRI / IDI / decision curves, publication tables) -> make-figures
  -> model-card (numbers + subgroup) -> write-paper + check-reporting