Model Sourcing

Model-Sourcing Skill

Purpose

/architecture-zoo answers a literature question — which family of model suits this task. That

question has a stable answer. The next question does not: *which concrete artifact do I run?*

A repository, a revision, a checkpoint. That is a provenance question, and the two facts a

careful researcher usually checks are the two that cannot answer it.

The licence tells you whether you may use it. The citation count tells you whether others did.

Neither tells you whether the number you are about to report means what you will say it means.

The failure this skill exists for is the quietest one in the lane. A method developed and tuned

against a benchmark family gets evaluated by the next person *on that same family*, and the

resulting figure reads like validation while sitting much closer to a training-set score. Nothing

in the repository says so. The licence is clean, the paper is peer-reviewed and highly cited, the

task matches, the code runs on your GPU. The conflict lives in the relationship between two

facts that are documented in different places — what the model was developed on, and what you are

about to evaluate it on — and it becomes visible only when they are written down side by side.

Writing them down side by side is what the dossier is for.

When to use

• You have a concrete candidate (a GitHub repo, a Hugging Face checkpoint, a paper's released

weights) and are about to build a study on it.

• You are writing the Methods paragraph that says which model you used, and it has to survive a

reviewer asking what it was trained on.

• You inherited a pipeline whose model came from somewhere nobody recorded.

When NOT to use

• Choosing an architecture *family* → /architecture-zoo (archetypes and the task-to-architecture

logic; deliberately not a live leaderboard).

• Building the training repo → /model-scaffold. Designing the validation study →

/model-validation. Computing held-out metrics → /model-evaluation.

• Documenting a model you built → /model-card (Model Card + Datasheet).
• Auditing your own dataset before modelling → /profile-imaging.
• Evaluating an LLM/multimodal system on a clinical task → /mllm-eval (which owns

pretraining-contamination of public benchmarks for that setting).

Workflow

Step 1 — write the dossier

One JSON file recording what is *known*, with unknowns left unstated rather than guessed:

{
  "model": "OrganSeg-3D v2.5.1",
  "source": {"kind": "github", "url": "...", "version": "v2.5.1", "commit": "abc1234"},
  "licence": {"spdx": "Apache-2.0", "verified_from": "LICENSE at commit abc1234"},
  "intended_use": "research",
  "weights": {"pretrained": false},
  "task": {"model": "3d_ct_organ_segmentation", "study": "3d_ct_organ_segmentation"},
  "reported_validation": [{"dataset": "ExampleBench", "metric": "Dice", "source": "J Ex 2021"}],
  "developed_on": ["ExampleBench"],
  "evaluation_arms": [{"name": "external", "dataset": "OtherCohort-2026"}],
  "hardware": {"claimed": "any CUDA GPU", "verified_on": "GTX 1080 Ti", "verified": true}
}

Each field is read from the artifact, not from memory: the licence from the LICENSE file at the

pinned commit (a README badge is not the licence), developed_on from the paper's own account of

where the method was built and tuned, hardware.verified only after it has actually run.

developed_on is the field people skip, and it is the one the gate needs. A method that won a

challenge was tuned on that challenge.

Step 2 — gate it

python3 scripts/check_model_provenance.py --dossier model_dossier.json \
    --out qc/model_provenance.json --strict

Stdlib-only, network-free — no repository is fetched and no licence resolved online, so the audit

re-runs anywhere the JSON travels. Verdicts:

| Verdict | Severity | Fires when |

|---|---|---|

| BENCHMARK_PROVENANCE_CONFLICT | Major | an evaluation arm uses a dataset the model was developed or tuned on |

| EVAL_DATA_IN_TRAINING | Major | an evaluation arm's dataset is inside the pretraining corpus |

| LICENCE_UNSTATED | Major | no licence recorded — which is not the same as a permissive one |

| LICENCE_INCOMPATIBLE | Major | a non-commercial / research-only licence under commercial or deployment intent |

| WEIGHTS_PROVENANCE_UNKNOWN | Major | pretrained weights whose training corpus is not stated |

| TASK_MISMATCH | Minor | the model's task is not the study's task |

| NO_VERSION_PIN | Minor | no commit, tag or revision |

| VALIDATION_UNREPORTED | Minor | no reported validation (dataset + metric + source) |

| HARDWARE_UNVERIFIED | Minor | hardware support claimed but never executed |

| LICENCE_UNVERIFIED | Minor | a licence is named but the file it was read from is not |

The gate flags a relationship, not a reputation. A dossier that declares

developed_on: ExampleBench passes cleanly as long as no evaluation arm uses ExampleBench.

Being developed on a benchmark is not a defect; evaluating on it and calling that independent is.

The clean fixture exists to prove exactly that distinction.

Dataset names are matched as token sequences with a small family-alias table, so

MSD Task09 Spleen matches MSD and MS Cohort 2026 does not. Matching never falls back to

substring search.

Step 3 — turn a Major into a study decision

A BENCHMARK_PROVENANCE_CONFLICT is rarely a reason to abandon the model — it is usually the

best-engineered option precisely because it was tuned hard. It is a reason to change **what the

arm is claimed to establish**:

• Report that arm as a demonstration that the pipeline runs end to end, not as evidence the

method works.

• Put the evidential weight on an arm whose data post-dates the model, and say so with dates.
• State the conflict in Methods and Limitations rather than leaving a reviewer to find it.

An EVAL_DATA_IN_TRAINING is different in kind: that arm produces a training-set score and cannot

be reported as validation at all.

Carry the dossier forward — /model-validation (arm design), /model-evaluation (what each arm

may claim), /model-card (provenance section), /write-paper (Methods + Limitations).

Outputs

• model_dossier.json — the provenance record downstream skills and the Methods section read.
• qc/model_provenance.json — deterministic audit with verdicts.
• The arm-by-arm decision from Step 3, written into the study record.

Anti-Hallucination

• Never infer a fact the dossier does not state. An unstated licence is LICENCE_UNSTATED,

never "probably MIT"; an unstated pretraining corpus is a Major finding, never an assumption.

• Never record a licence from a badge, a model card summary, or memory — only from the licence

file at the pinned revision, and record which file that was.

• Never mark hardware verified without executing it. A support matrix and what the stack

actually runs is a different claim; a CUDA capability the compiler accepts may still be refused

by a compiler in the same stack.

• Never report an arm as independent validation when the gate flags a provenance conflict.
• If a provenance fact cannot be established from the artifact, leave it unstated and let the

gate say so.

Deterministic gate

scripts/check_model_provenance.py — 10 verdicts by set arithmetic over the dossier, stdlib-only

and network-free. Reproducible challenge:

bash ${CLAUDE_SKILL_DIR}/scripts/check_model_provenance_challenge/verify.sh.

Regression suite: bash ${CLAUDE_SKILL_DIR}/tests/test_model_provenance.sh.

Boundaries

architecture-zoo (which family?) -> model-sourcing (this skill: which artifact, and what may its
  numbers claim?) -> profile-imaging / preprocess-imaging -> model-scaffold -> model-validation
  -> model-evaluation -> model-card -> write-paper