Design Study

Design-Study Skill

Purpose

This skill pressure-tests whether a study is answerable, interpretable, and defensible before large amounts of drafting or analysis work accumulate.

Use it when:

• a study question is known but the analysis plan is still fluid
• the user wants a methods sanity check
• a manuscript feels vulnerable to reviewer criticism
• a peer review requires explicit methodological diagnosis

Communication Rules

• Communicate with the user in their preferred language.
• Use English for statistical, radiologic, and reporting-guideline terminology.
• Be direct about validity risks, but always propose the smallest feasible fix first.

Core Review Questions

Always inspect these dimensions:

• What is the exact research question?
• What is the analysis unit: patient, lesion, exam, study, phase, report?
• What is the index date or decision point?
• How are inclusion and exclusion criteria applied?
• Is there any information leakage?
• What is the reference standard or endpoint definition?
• What comparator is clinically meaningful?
• What validation strategy is used?
• What uncertainty reporting is required?

10. Which reporting guideline best fits?

11. Are exposure/outcome/covariate definitions literature-grounded, or invented ad-hoc from the data dictionary? If ad-hoc, defer to /define-variables before drafting Methods.

Standard Output

## Study Design Review
Question: ...
Study type: ...
Analysis unit: ...
Index date / prediction timepoint: ...

### Strengths
- ...

### Major validity risks
1. ...
2. ...

### Minimal fixes
- ...

### Reporting fit
- Recommended guideline: ...

### Decision
- Ready for analysis / Needs redesign / Drafting can proceed with limitations

Workflow

Phase 1: Reconstruct the study

Extract from protocol, draft, slides, tables, or notes:

• clinical problem
• intended use case
• population
• inputs
• outputs
• outcome definition
• timing of variable availability

Gate: Present the reconstructed study summary (question, analysis unit, intended use)

to the user. Confirm before proceeding — if the reconstruction is wrong, the entire

validity review will be misdirected.

Phase 2: Check structural validity

A. Analysis unit

Look for mismatches such as:

• patient-level claim from lesion-level analysis
• exam-level split with patient overlap
• phase-level samples treated as independent

B. Leakage

Look for:

• postoperative features used for preoperative prediction
• normalization or thresholding performed before data split
• repeated exams across train/test
• reader annotations derived from outcome information
• input-text contamination for NLP/LLM extraction tasks: if the model input includes report

sections such as clinical history, indication, impression, prior diagnosis, or referral text, confirm

that those fields do not literally name or strongly imply the target label. If the target is already

present in the supplied text, the task is information retrieval under label leakage, not phenotype

inference; redesign the input mask, report a sensitivity analysis excluding leaky fields, or reframe the

claim.

• construct dependence (a predictor that is a definitional component of the outcome). Two cases:

(i) *mathematical definition* — an input that computes the outcome (when the outcome is HOMA-IR =

f(fasting insulin, fasting glucose), those two inputs are not independent predictors); (ii)

*near-tautological composite* — a ratio or score built from the outcome's defining components, which

shows an inflated, near-circular association. Test: "could this predictor be derived, in whole or

part, from the outcome's definition or the same measurement?" If yes, exclude it, or retain it only

as a labeled calibration probe rather than a reported discovery.

F. Time origin & survivorship (incident / transition models)

For any time-to-event or incident/transition design, check before drafting:

• Time origin per model. Each incident model starts its at-risk clock at the correct origin. Watch for immortal-time bias (a span in which the event cannot occur, misattributed to one group) and left-truncation / delayed entry (subjects entering the risk set after the origin).
• Mediator-ascertainment-window survivorship. A "progressor" / transition label that is conditional on *surviving to* a later ascertainment (a second scan, a follow-up visit) is survivorship-biased; plan a landmark time or an explicit intermediate-state (multistate / illness-death) model.
• Primary-analysis-set selection. If the primary will not be the full cohort (e.g., complete-case while a large fraction is missing), pre-specify the selection justification and a MAR rationale; do not let the complete-case model become primary because it is the significant one (an outcome-dependent choice).
• A design that cannot yet answer these should say so honestly — but note that at review time a Methods/Limitations admission that the issue was *"not formally assessed"* is escalated to a MAJOR by the survival probe (S1), not waved through as a limitation.

C. Reference standard

Check:

• who established ground truth
• when it was established
• whether blinding was possible
• whether only a subset had gold standard verification
• Construct ↔ nominal-definition match. Does the exposure/finding *construct* stay inside its stated definition, or does it quietly exceed it? An "incidentaloma" defined as an *indeterminate* finding must not include frank malignancy reads; a label that overshoots its definition inflates the apparent cohort and breaks the κ. For each construct, restate the nominal definition and confirm every included case satisfies it.
• Per-flag reference-standard concordance. When the index finding is flagged against a reference standard, report the concordance *per flag category* (not just overall). A construct where a large fraction of flags do not match the reference standard (e.g., ~86% non-match) is measuring something other than the named construct.
• Manuscript definition ↔ variable_operationalization.md. The variable definitions written in Methods must match the operationalization table verbatim (dictionary-first). A blinded re-classification form must quote the analytic protocol's definition verbatim — paraphrase / "common-sense extension" in the form (but not the Methods) is the documented cause of a low κ that is a *definition mismatch*, not real disagreement. Cross-check with /define-variables output before drafting.

D. Validation

Classify:

• apparent only
• internal split
• cross-validation
• temporal validation
• external validation
• multi-center external validation

E. Reader / expert-elicitation studies (load on demand)

When the study elicits expert ratings — a reader study, an annotation panel, an AI-output

evaluation — the design decisions that matter are made before data collection, and the

acceptance ceiling of a perceptual / reader AI study is fixed at design time: no quality of

execution lifts a ceiling baked into the comparator, the estimand, or the reader cohort.

For an AI-system-versus-human-expert benchmark specifically, route to /design-ai-benchmarking,

which extends this subsection with arm definition, LLM-as-judge versus human-as-judge

adjudication, and a structured export schema.

| File | Read it when | Cost if read blindly |

|---|---|---|

| references/reader_elicitation_design.md | the design has a human-rater or expert-elicitation arm — rubric axes, calibration probes, operational rigor, human-as-operator, and the six ceiling decisions | ~2,400 tokens, none of which applies to a design with no reader arm |

| references/dag_adjustment.md | confounding control needs an explicit adjustment set | — |

| references/target_trial_emulation.md | the design emulates a target trial | — |

| references/venue_accept_recipe.md | it is a clinical DL / AI-validation study and the question is which venue tier the achievable design can be accepted at, and the one design move that reaches the tier above (the design→acceptance-tier ladder + the five acceptance levers, reverse-engineered from accepted OA papers; the bridge into /find-journal) | ~1,800 tokens; skip for a design with no publication-tier decision |

| references/combine_models_ablation_design.md | the model is built by combining / adapting / fine-tuning existing models (nnU-Net, TotalSegmentator, SAM/MedSAM, a pretrained backbone) — how to design the comparator as an ablation that proves the combination earns its complexity (beat the un-adapted base + the best single component + direct-train), reverse-engineered from accepted OA papers | ~1,600 tokens; skip if the model is trained de novo with no reused component |

| references/multi_model_comparison_design.md | the study's contribution is comparing several models / architectures head-to-head (CNN vs Transformer vs foundation backbone; N segmentation networks) — how to make the comparison fair: one frozen split + one preprocessing through every model, a strong self-configuring baseline (nnU-Net) not a hobbled one, matched training/HPO/compute budget (the #1 threat — "new≠better, just tuned harder") or disclosed, variance-over-single-run, a pre-specified primary metric + a paired delta test, honest ranking. Reverse-engineered from accepted OA papers | ~1,700 tokens; skip for a single-model study (use combine_models_ablation_design.md for an ablation of one model, /design-ai-benchmarking for AI-vs-human) |

| references/segmentation_failure_characterization_design.md | the claim is that a segmentation model is clinically usable, not that it scores well — a pre-specified failure taxonomy (boundary drift / missed / hallucinated / catastrophic), an acceptability endpoint with a stated scale, named judges and an adjudication rule, the tail (per-case distribution, worst percentile, catastrophic count) beside the mean, edit effort paired against manual-from-scratch and disaggregated per structure and site, and failures stratified by what predicts them. Reverse-engineered from accepted OA papers | ~1,700 tokens; skip when the endpoint is benchmark accuracy with no usability claim (metric choice → /model-evaluation; abstention / risk–coverage → /uncertainty-imaging) |

Phase 3: Clinical framing

Ask whether the comparator and endpoint support the stated claim:

• is the model better than current practice or just another model?
• is the endpoint clinically meaningful?
• does performance translate to action?
• incremental value: if the study frames the model/marker as adding value *beyond* / *on top of* / *incremental to* an existing tool (a clinical score, a routine test, a baseline model), the design must pre-specify the baseline comparator built from the in-routine-use predictors and an incremental-value metric — ΔC-index / ΔAUC (with a paired CI, e.g. DeLong), categorical or continuous NRI, IDI, or decision-curve net benefit. A standalone discrimination number ("our model's AUC was 0.84") does not support a "beyond X" claim; without the nested-model comparison the finding may be real but redundant. Plan this at design time — it cannot be added post hoc without the baseline model.
• fine-tuning contribution baseline: if an NLP/LLM study claims that fine-tuning, LoRA, prompt

engineering, or a multi-agent wrapper improves extraction/classification, pre-specify a same-backbone

zero-shot or few-shot comparator on the identical input, output schema, and test split. A comparison

only against a weaker or unrelated baseline cannot establish that the proposed adaptation adds value. For an imaging model built by combining / adapting / fine-tuning existing models (nnU-Net, a foundation model, a pretrained backbone), design the full ablation ladder — un-adapted base, best single component, direct-train vs transfer — per references/combine_models_ablation_design.md. When the contribution is instead a head-to-head comparison of several models (which architecture wins), the decisive design question is comparison *fairness* — one frozen split/preprocessing through every model, a strong fairly-tuned baseline, a matched (or disclosed) compute budget, and a paired delta test — per references/multi_model_comparison_design.md. When the claim is not that a segmentation model *scores* well but that it is clinically usable, the design must carry a pre-specified failure taxonomy, an acceptability endpoint with a named judge and adjudication rule, the tail beside the mean, and edit effort paired against manual-from-scratch — per references/segmentation_failure_characterization_design.md; a mean DSC cannot be converted into a usability claim after the fact.

• endpoint↔conclusion scope: decide up front what *kind* of conclusion the design can support, so the manuscript does not overreach. A cross-sectional / single-visit / prevalence design cannot support a prognostic or surveillance claim (rescreen interval, disease progression) — that needs longitudinal follow-up. A binary surrogate endpoint (present/absent, >0, dichotomized) is risk stratification, not a patient-care directive (defer/withhold/initiate therapy). At review time /self-review §D + check_scope_coherence.py flag CROSS_SECTIONAL_PROGNOSTIC / SURROGATE_CARE_DIRECTIVE against the conclusion.

Phase 4: Reporting fit

Recommend one primary guideline:

• TRIPOD-AI
• CLAIM
• STARD
• STROBE
• PRISMA
• CARE
• ARRIVE
• journal-specific additions if needed

Frequent Failure Modes

Diagnostic AI

• no clinically relevant comparator
• exam-level split instead of patient-level split
• unclear reference standard
• AUROC-only reporting without threshold metrics

Prognostic modeling

• unclear time zero
• immortal time bias
• feature timing mismatch
• no calibration

Retrospective cohort / screening database

• time zero misalignment: cohort entry ≠ follow-up start → immortal time bias
• interval-censored outcomes treated as exact → underestimation of event times
• healthy volunteer bias unacknowledged → inflated external validity claims
• surveillance bias from unequal follow-up frequency between groups
• 3 bias classification (Hernan/Robins): selection bias (who enters), information bias (how measured), confounding (what else differs) — explicitly map each threat
• comparative / causal question → emulate a target trial. For a treatment-vs-treatment, screening-vs-no-screening, or drug-A-vs-drug-B question on routinely-collected data, specify the seven target-trial components (eligibility, strategies, assignment, time zero, outcome, causal contrast, analysis plan) before extraction — this is what prevents the immortal-time / prevalent-user / confounding-by-indication trio above and turns an association into a defensible causal contrast. New-user + active-comparator design, grace-period clone-censor-weight, and negative controls are in references/target_trial_emulation.md.
• confounding completeness: pre-specify the adjustment set from a DAG (not a Table-1 p < 0.05 rule), and plan to report whether any measured covariate that turns out imbalanced by exposure but outside the adjustment set leaves the primary estimate robust (an extended-adjustment sensitivity model). Build the DAG and pre-screen the proposed covariates with scripts/adjustment_set_helper.py (flags mediator / collider / descendant adjustment and omitted confounders, and proposes a candidate backdoor set), then derive the minimal sufficient set with dagitty — see references/dag_adjustment.md. At review time /self-review Phase 2.5e + the O1–O12 probes in observational_confounding.md check this against Table 1 (including O7 over-adjustment, O10 overlapping-subset-gradient discipline, for complex-survey data O11 design-based weighting and O12 data-driven-threshold mining, O13 — a cross-sectional mediation claim cannot order X→M→Y, and O14 — a synergy/joint-effect claim needs the additive interaction scale (RERI/AP/S), not a multiplicative-only test).

Multimodal LLM / report generation

• no clear rubric for clinical correctness
• benchmark labels derived from noisy reports without adjudication
• unsupported claims about safety or workflow benefit
• input text contains the target label or diagnosis being predicted
• no same-backbone zero-shot/few-shot baseline for a fine-tuning or prompt-engineering claim

Imaging meta-analysis

• overlapping cohorts
• paired modalities analyzed as independent
• heterogeneity metrics missing
• zero-cell handling unspecified

Minimal-Fix Principle

Whenever possible, recommend the smallest feasible repair first:

• clarify the claim
• narrow the target population
• add a limitation statement
• add a clinically relevant baseline
• re-run one key sensitivity analysis
• redefine the endpoint more explicitly

Escalate to redesign only when the central claim is not defensible otherwise.

Handoff Rules

• route to analyze-stats when the design is basically sound but analysis details need refinement
• route to check-reporting after the design is locked
• route to self-review when the user wants a pre-submission quality check on their own manuscript
• route back to write-paper only after the main validity risks are documented

What This Skill Does NOT Do

• It does not compute statistics directly
• It does not draft full manuscript prose
• It does not resolve raw data engineering issues
• It does not replace a full peer review when journal-facing tone is required

Anti-Hallucination

• Never fabricate references. All citations must be verified via /search-lit with confirmed DOI or PMID. Mark unverified references as [UNVERIFIED - NEEDS MANUAL CHECK].
• Never invent clinical definitions, diagnostic criteria, or guideline recommendations. If uncertain, flag with [VERIFY] and ask the user.
• Never fabricate numerical results — compliance percentages, scores, effect sizes, or sample sizes must come from actual data or analysis output.
• If a reporting guideline item, journal policy, or clinical standard is uncertain, state the uncertainty rather than guessing.