Scenario Design Agent Skill

Turn a change spec into adversarial, real-life test scenarios — designed to

break the system, not confirm it works. A scenario is only as good as it is

*concrete and executable*. If the spec can't supply the concrete bits, that is a

spec defect, surfaced as a clarification — not a guess.

Input: A change/feature spec. --change <name> (OpenSpec), or infer from

context / point the skill at any spec doc.

Mode: --stage proposal|design|apply (default: infer — see Gate).

Output: a standalone test-plan.md catalog written to your change/spec's

test-plan location (OpenSpec: openspec/changes/<name>/test-plan.md).

Core mechanism — the Triple

Every scenario MUST resolve three concrete slots:

   ┌─────────────┬──────────────────────┬────────────────────────────┐
   │  INPUT      │  TRIGGER             │  EXPECTED OBSERVABLE OUTCOME │
   │  concrete   │  the condition /     │  a measurable, visible fact  │
   │  data /     │  action that fires   │  (status, value, latency,    │
   │  state      │  the behaviour       │   DOM, log line, exit code)  │
   └─────────────┴──────────────────────┴────────────────────────────┘

Rule (from spec-coding edge-case practice): **if any slot is a verb without a

noun, or an adjective instead of a number, the slot is unfillable → spec gap.**

"Handles errors gracefully" is not a Triple. "POST /api/restart while server

already restarting (input) → second caller (trigger) → receives 409 within

500ms, no second orchestrator spawned (observable)" is.

Stance: falsify, don't confirm. For each requirement, the job is to find the

input+trigger that makes the observable *wrong*. Happy path is table stakes;

the scenario value is in the boundaries and failures.

Phase 1 — Read the spec, classify requirements

• Read what exists (OpenSpec layout shown; in a non-OpenSpec project read

whatever spec/design/task docs the user points at — do not fail on a missing

openspec/ dir or CLI):

• openspec/changes/<name>/proposal.md (always)
• openspec/changes/<name>/design.md (if present — decisions, invariants)
• openspec/changes/<name>/specs/**/spec.md (requirement deltas)
• openspec/changes/<name>/tasks.md (if present — to align section numbers)

• Extract every testable requirement (each SHALL/MUST, each scenario block,

each acceptance criterion). For each, tag its shape — this picks the

technique:

| Requirement shape | Technique to apply | Scenario class |

|---|---|---|

| Input range / numeric / size / count | Equivalence Partitioning + Boundary Value Analysis | edge-case |

| Multiple boolean/enum flags combine | Decision Table | edge-case |

| Lifecycle / status transitions / reconnect / restart | State-Transition | frontend-quirk + error-handling |

| Async / WebSocket / polling / optimistic UI | State-convergence + invariant assertions (not UI-visibility) | frontend-quirk |

| Latency / throughput / memory / long-run | tail-latency (p95/p99) + soak + threshold | performance |

| Depends on network / disk / subprocess / other service | fault injection (delay + abort) | error-handling |

See references/technique-cheatsheet.md for how to apply each.

Phase 2 — Generate scenarios via the Triple (or a gap)

For each requirement, walk its technique and try to emit one or more Triples.

• EP+BVA: emit min, just-below-min (invalid), nominal, just-below-max,

max, just-above-max (invalid). Six Triples from one numeric requirement.

• Decision table: one Triple per reachable flag combination; mark

impossible combos.

• State-transition: one Triple per legal edge AND per *illegal* edge

(event fired in a state that shouldn't accept it).

• Async/convergence: assert the eventual invariant and the intermediate

states, never "element is visible after N ms".

• Performance: state the workload, the metric (p95/p99/RSS), the threshold,

and the measurement window. No threshold in spec → gap.

• Fault injection: for each dependency, a delay Triple and an abort Triple;

assert retry/timeout/degradation behaviour.

When a slot won't fill → STOP generating that scenario. Record a gap with

the unfillable slot named (see Gate). Do not invent the missing value.

Phase 3 — The clarification Gate (configurable)

Whether an unfillable Triple blocks or just annotates depends on stage:

   stage = proposal | design   →  HARD gate
   stage = apply               →  SOFT gate
   (no --stage)                →  infer: tasks.md absent ⇒ proposal/design (hard)
                                          tasks.md present ⇒ apply (soft)

• HARD gate: collect all gaps, then call ask_user with decision-forcing

questions and STOP. Do not write test-plan.md until answered. The spec is not

yet testable; clarify before locking scenarios.

• SOFT gate: write the scenario row with a `[NEEDS CLARIFICATION: <slot> —

<question>]` marker, continue, and list all markers in a banner at the top of

test-plan.md.

Decision-forcing question rules (from ambiguity-detection practice):

• Name the missing slot and *why* it blocks a scenario.
• Offer concrete candidate answers, never propose a solution/implementation.
• One question per genuine decision; do not pad.

Example: *"Restart quiesce window: tasks say bridges 'suppress auto-start for

the quiesce window'. To test the boundary I need the exact value — is it 5s

(restart) / 60s (shutdown) per AGENTS.md, or spec-defined elsewhere? Without a

number I cannot write the just-after-window re-spawn scenario."*

Phase 4 — Route each scenario to a test level

Every scenario carries a level tag fixing where it would be authored, and a

disposition (automated | manual-only). Map each scenario's *nature* to

one of your project's actual test levels — the routing *method* is fixed; the

level names and paths are yours to fill. Do not assume a level/harness the

project lacks.

| Scenario nature | Route to the project level that is… |

|---|---|

| pure logic / boundary / decision table / pure state | the fast in-process unit tier |

| process / install / spawn / multi-OS runtime | the process/CLI smoke tier (NO rendered-UI asserts) |

| rendered UI / WS-driven view / convergence / quirk | the browser/e2e tier |

| micro perf (fn-level) | the unit tier, timed |

| process/load perf, soak | the smoke tier (or a dedicated perf harness) |

| aesthetics / hardware / "feels right" / subjective | manual-only → no fold, no test task (disposition=manual-only, level —) |

Keep the rendered-UI-vs-smoke boundary sacred: a UI-visible assertion never

lives in a process/CLI smoke row.

> Example — pi-agent-dashboard levels (this repo's concrete routing; other

> projects substitute their own). Honour the AGENTS.md hard rule: rendered-UI

> assertions are Playwright only; qa/ stays CLI/process smoke.

>

> `text

> ┌────────────────────────────┬──────────────────────────────────────────┐

> │ Scenario nature │ Level → location │

> ├────────────────────────────┼──────────────────────────────────────────┤

> │ pure logic / boundary / │ L1 unit → packages/*/src/**/__tests__/ │

> │ decision table / state pure│ *.test.ts (vitest) │

> │ process / install / spawn │ L2 smoke → qa/tests/*.sh|*.ps1 │

> │ / multi-OS runtime │ (NO rendered-UI asserts) │

> │ rendered UI / WS-driven │ L3 e2e → tests/e2e/*.spec.ts │

> │ view / convergence / quirk│ (Playwright vs docker harness port †) │

> │ micro perf (fn-level) │ L1 unit (timed) │

> │ process/load perf, soak │ L2 smoke (or dedicated harness) │

> │ aesthetics / hardware / │ manual-only → no fold, no test task │

> │ "feels right" / subjective │ (disposition=manual-only, level —) │

> └────────────────────────────┴──────────────────────────────────────────┘

> `

>

> † The docker e2e harness port is NOT a fixed :18000 — docker/test-up.sh

> hash-derives a free port per worktree and records it in .pi-test-harness.json

> (dashboardPort). An L3 scenario's observable is read against that derived

> port; never hardcode :18000.

manual-only routing outcome (additive to L1/L2/L3): a scenario whose

expected observable is a human judgment with no automatable signal — visual

aesthetics, a hardware behaviour, "feels right / looks correct", subjective UX —

is NOT routed to a test level. Its manifest row records `disposition:

manual-only (level —`), and no test task is folded for it; it is deferred to

post-merge manual verification by ship-change. Every routable scenario keeps

its L1/L2/L3 level and disposition: automated — this outcome only diverts the

truly un-automatable rows; existing L1/L2/L3 logic is unchanged.

If a scenario implies a brand-new level/harness, flag it in the plan's "New

infra needed" section rather than silently assuming it exists.

Phase 5 — Write test-plan.md

Write the test-plan.md to your change/spec's test-plan location (OpenSpec:

openspec/changes/<name>/test-plan.md) using

references/test-plan-schema.md. It is a standalone catalog, separate from

tasks.md. Each scenario is a numbered row with: id, class, technique, level,

disposition (automated | manual-only), the full Triple, and (soft gate)

any clarification marker. The disposition column is mandatory on every row —

it is the manifest's source-of-truth signal that the fold step (in

plan-proposal) and the defer rule (in ship-change) both read.

End with a short offer (do not auto-act): *"Want me to fold these into the

## Tests / ## Validate sections of tasks.md as checklist items?"* — folding

is a separate, explicit step.

Guardrails

• Never invent a missing value to make a scenario "work" — that hides the

spec gap this skill exists to expose.

• Never write app/test code here — this skill drafts the *catalog*. Authoring

the actual *.test.ts / *.spec.ts is implementation (use implement /

openspec-apply-change).

• Don't downgrade scenarios to smoke to make them easy. A scenario that only

checks "it exists / exit 0" belongs in qa/ smoke already — this skill's output

is the layer *above* that.

• Honour the level boundary — no rendered-UI assertion in a qa/ smoke row.
• Offer, don't auto-fold into tasks.md.

References

• references/technique-cheatsheet.md — how to apply each ISTQB + resilience

technique, with project-specific examples.

• references/test-plan-schema.md — exact test-plan.md layout.