Model Validation Agent Skill

Model-Validation Skill

Purpose

This skill pressure-tests the validation study for an engineer-built medical-imaging model — the

common case where a clinical team receives a trained segmentation / classification / detection model

from an engineering collaborator and must validate it and write it up. It is the imaging-model

specialization of /design-study: where design-study covers general validity and

design-ai-benchmarking covers AI-versus-human-expert reader studies, this skill owns the

partition, leakage, reproducibility, and metric-selection mechanics that decide whether a reported

Dice / AUROC / sensitivity is trustworthy.

It is advisory and deterministic-audit only. It writes decision notes and runs a stdlib gate on the

split table; it never builds, trains, or alters the model, and it never replaces MONAI / nnU-Net /

TorchIO — those produce the model, this validates and publishes it.

When to use

• A trained imaging model (in-house, vendor, or open-weights) needs a clinical-validation study designed

or audited before submission.

• You have, or can produce, the split-assignment table (which patient went to train / val / test).

When NOT to use

• Building or training the model → out of scope (integrate MONAI / nnU-Net).
• AI-versus-human-expert reader study → /design-ai-benchmarking.
• LLM / MLLM evaluation → /mllm-eval (when available).
• General study/validity review → /design-study.
• Statistical execution (DeLong, ICC, bootstrap CIs, calibration tables) → /analyze-stats.
• Item-by-item reporting-guideline audit of a finished manuscript → /check-reporting.
• Reviewing a finished manuscript → /self-review or /peer-review (which load the MD0–MD8

reviewer-side probe).

Workflow

The design/audit rationale behind Phases 2–7 — the full data-leakage taxonomy, the

internal-vs-genuine-external validation ladder, comparator design, single-run vs multi-seed

variance, test-set sizing, and the CLAIM 2024 / TRIPOD+AI / STARD-AI reporting map — is in

${CLAUDE_SKILL_DIR}/references/validation_design.md (load on demand). The patient-disjointness

verdict itself is proven by scripts/check_split_leakage.py (Phase 2), not from that prose.

Phase 1 — Reconstruct the task, the intended-use horizon, and the analysis unit

State the model's task (segmentation / classification / detection), its intended-use horizon

(screening, triage, pre-procedure, post-hoc), the single headline metric the conclusion leans on,

and the analysis unit the metric must respect (per-patient vs per-lesion vs per-image). Everything

downstream is read against this.

Phase 2 — Leakage audit (the deterministic gate, run first)

The most metric-inflating defect is a split that is not disjoint at the patient level. Produce the

emitted split-assignment table (patient_id,split) and run the gate:

python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \
  --splits <split_assignment.csv> --out qc/split_leakage.json --strict

PATIENT_OVERLAP (a patient in ≥ 2 partitions) and MISSING_SEED (an unreproducible split) are

proven by set arithmetic — not heuristics. Then walk the rest of the leakage taxonomy (Kapoor &

Narayanan, *Patterns* 2023) that the table cannot show: preprocessing-before-split (normalisation,

resampling, foundation-model embeddings, or ComBat harmonisation fit on the whole cohort before

partitioning), site / scanner / burned-in-label shortcuts, and temporal leakage (a random split

where future and past coexist). The decisive question: *could any value used in training have been

computed only with knowledge of a test case?*

Phase 3 — Validation tier (internal split vs genuine external)

Classify the evidence honestly: apparent → internal random split → cross-validation → temporal →

geographic / external (different site, scanner, vendor) → multi-site external. Cross-validation and

bootstrap are development-time optimism corrections, not external validation. Flag a generalisability

or deployment claim that outruns an internal-only design, and "developed with external validation" where

the single external set was used for tuning. Also confirm the test set was touched once — no

architecture search, hyperparameter sweep, early-stopping, or operating-point / threshold choice read

the test set.

Phase 4 — Comparator design

Decide what the model is compared against: clinical-only baseline, incremental value over an existing

score, or reader comparison. For a reader comparison, hand the rubric / inter-rater design to

/design-ai-benchmarking.

Phase 5 — Metric selection (task-correct, prevalence-aware)

Match the metric to the task and the prevalence (Metrics Reloaded — Maier-Hein & Reinke et al.,

*Nat Methods* 2024): segmentation pairs an overlap metric (Dice / IoU) with a boundary metric

(HD95 / Normalised Surface Distance), per-structure not only global; classification under imbalance

reports AUROC and AUPRC with CIs plus sensitivity / specificity and prevalence-dependent PPV / NPV,

never bare accuracy on a balanced set; detection reports FROC / mAP with the IoU match criterion

stated. Require the headline metric as mean ± SD across ≥ 3 seeds / runs, or a fixed reported seed

with the determinism caveat. The per-case metric computation + the deterministic metric-reporting gate

live in /model-evaluation (which emits the per-case table for /analyze-stats). For **interactive /

promptable segmentation** (SAM2 / MedSAM2 / nnInteractive) the metric set adds the interaction axis —

number of clicks / interactions-to-threshold (NoC), initial-vs-converged Dice, and per-case

interaction / inference time (/model-evaluation --task interactive). When the evaluation runs two arms

(simulated prompting + human-operator validation), record protocol fidelity across arms — identical

prompt types, stopping rule, target threshold, and seeds — as an explicit validity item: arm-to-arm

comparability is the precondition for reading the human-operator arm as validating the simulated one,

and the human-operator arm design is in /design-study.

Phase 6 — Test-set sizing

Check the events per class in the test set, not the cohort total — a metric on a sparse positive set

has a confidence interval spanning much of the usable range, and calibration needs roughly ≥ 100 events.

Hand the formal sizing (diagnostic-accuracy / AUC precision / agreement) to /calc-sample-size.

Phase 6.5 — Prospective evaluation & deployment-monitoring horizon

Retrospective external validation shows accuracy *transfers*; it does not show the model

is safe and useful *in the clinical workflow*. If the claim is clinical use — not just

discrimination — design the higher tier explicitly: silent / shadow deployment (live

cases, no care impact, prospective performance + calibration targets) → **prospective

comparative / impact study or RCT on a clinical endpoint → post-deployment monitoring**

for performance / dataset-shift / calibration drift with recalibration-or-withdrawal

triggers and ongoing subgroup-performance audit. See references/validation_design.md §2b.

Scope the claim to the tier reached — a retrospective external study must not claim

deployment readiness or clinical-outcome benefit.

Phase 7 — Reporting-guideline fit

Map the study to its reporting standard via /check-reporting: CLAIM 2024 (diagnostic imaging AI),

TRIPOD+AI (prediction model), STARD-AI (diagnostic accuracy), PROBAST+AI (risk of bias),

and — for a prospective/live evaluation (Phase 6.5) — DECIDE-AI (early clinical evaluation of

decision-support AI) or CONSORT-AI / SPIRIT-AI (full AI trials / protocols).

Phase 8 — Handoffs

Carry the audited design into /write-paper (Methods), /calc-sample-size (sizing), /check-reporting

(compliance), and — for the reviewer-side audit of the finished draft — /self-review, which loads the

model_development.md (MD0–MD8) probe.

Deterministic gate

scripts/check_split_leakage.py — proves patient-level split disjointness + seed presence on the emitted

split-assignment table (stdlib, network-free). Verdicts: PATIENT_OVERLAP (Major), MISSING_SEED

(Major), SINGLE_PARTITION (Minor). Reproducible challenge:

bash ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage_challenge/verify.sh.

Anti-Hallucination

• Never fabricate performance metrics, split assignments, event counts, or seeds. Every number comes

from the engineer's executed code, the supplied split table, or a re-run of the deterministic gate —

never invented. A reported Dice / AUROC / overlap count with no underlying record is the failure mode

this skill exists to prevent.

• Never report a split-audit "pass" without running check_split_leakage.py. The patient-disjointness

verdict is proven by the script, not asserted from prose.

• Never invent references, reporting-guideline items, or metric-selection rules. Verify citations via

/search-lit (confirmed DOI / PMID); mark unverified ones [UNVERIFIED - NEEDS MANUAL CHECK]. If a

CLAIM 2024 / TRIPOD+AI / Metrics-Reloaded item is uncertain, flag [VERIFY] and ask the user rather

than guessing.

• **Do not claim external validation, generalisability, or deployment readiness the design does not

support** — classify the validation tier honestly and let the evidence cap the claim.

Boundaries — which skill to use, in what order

design-study (general validity)
  └─ model-validation (this skill: leakage, split, comparator, metric, sizing handoff)
       ├─ check_split_leakage.py  (deterministic patient-disjointness gate)
       ├─ calc-sample-size        (test-set / event sizing)
       ├─ design-ai-benchmarking  (reader-comparison rubric / IRR)
       ├─ check-reporting         (CLAIM 2024 / TRIPOD+AI / STARD-AI)
       └─ write-paper -> self-review / peer-review (MD0–MD8 reviewer probe)

It does not build the model (integrate MONAI / nnU-Net), compute publication statistics (/analyze-stats

owns DeLong / ICC / calibration tables), or evaluate an LLM / MLLM (/mllm-eval).