Designing Distributed System Tests

Designing Distributed-System Tests

The default for testing distributed and stateful systems — write a few

integration tests and call it done — finds a small fraction of the bugs

that actually break these systems in production. This skill enforces an

opinionated workflow: scope the change, generate failure-mode hypotheses

that cover the categories the literature says matter most, pick

techniques from a curated catalog, and emit a structured plan file that

the executing-distributed-system-tests skill (or a human) can run.

Plan modes

This skill produces two shapes of plan. Decide which one applies before

you start; the steps below branch on it.

• Change-scoped — the default. Use when the caller names a commit,

PR, branch-diff, or feature. The plan covers what *this change* could

regress, scoped by its blast radius.

• Project-wide — use when the caller asks for a "release-validation

plan", "stability plan for the whole system", "test plan to enough

coverage", "what should we be testing", or otherwise frames the

request without a specific change. The plan covers what *the system*

should be tested for, with an explicit inventory of existing tests

and a gap analysis driving the new-scenario list.

If the framing is ambiguous, ask once before starting — the modes

diverge enough that retrofitting one into the other wastes work.

Process

Follow these steps in order. Do not skip; the order matters because

later steps depend on artifacts the earlier steps produce.

1. Scope the system

Read the project's entry points: README, AGENTS.md or CLAUDE.md,

top-level docs/, any existing test-plan or runbook files. Note:

• Tenancy / isolation model
• Persistence model (what is durable, fsync contract)
• Replication / consensus protocol, quorum, leadership
• Ordering guarantee exposed to clients
• Network boundaries (which RPCs / streams)
• Retry / idempotency contract
• Observability (logs, metrics, traces) available to an oracle

Write this as a one-paragraph SUT model. If anything is ambiguous from

the repo, ask the user before proceeding — do not invent guarantees.

1b. Extract claims and guarantees

A good test plan exists to falsify what the product *claims*. Before

generating hypotheses, write down what the SUT promises its users.

This is the spine the rest of the plan hangs off — every hypothesis,

every scenario, every oracle should be traceable back to a claim it

either confirms or refutes.

Sources to mine:

• README "guarantees" / "what we offer" sections
• API docs / reference manuals
• ARCHITECTURE / DESIGN docs (claims about consistency, durability,

replication, fault tolerance)

• Public blog posts, talks, marketing material (if any)
• The code itself: function names, doc-comments on public APIs,

error types (IdempotencyConflict, StaleRead, etc.) imply

guarantees the system claims to enforce

• Existing test names (a test called linearizable_under_partition

implies a linearizability claim under partition)

Categorise each claim:

• Safety — "the system never returns a stale read", "no

acknowledged write is ever lost", "linearizable per key"

• Liveness — "every accepted operation eventually commits",

"leader election completes within N seconds of crash"

• Durability — "fsync'd writes survive crash", "replicated

writes survive single-AZ loss"

• Performance / SLO — "p99 append latency ≤ X ms at Y ops/s

per session"

• Operational — "rolling upgrade is non-disruptive",

"configuration changes are atomic"

• Idempotency / dedup — "same idempotency key never produces

two committed effects"

• Isolation — "tenant A's reads never observe tenant B's writes",

"no read returns data from a transaction that has not yet

committed"

• Ordering — "consumers always see messages in the order the

producer sent them", "every reader sees a prefix of the global

log order"

• Membership — "a node that fails its liveness probe is removed

from the cluster membership view within N seconds", "every joined

member appears in the membership table exactly once"

• Boundary — access-boundary semantics: "tenant A's data is never

reachable from tenant B on any surface", "a request scoped to

namespace X never routes to namespace Y". Subsumes tenancy / authz

/ namespace / routing / multi-protocol; do not file those as

separate categories. Triggers the §7.M.S surface-decomposition

discipline (see step 3 and references/boundary-and-isolation-testing.md).

• Fairness — per-group performance and noisy-neighbor isolation:

"no tenant can starve another for throughput", "one shard's load

does not blow another shard's p99". Group can be tenant, shard,

queue, partition, region, priority class, user, table, or workload

class. Also triggers §7.M.S.

If the project does NOT explicitly document a claim that appears

in the code, write it as an *inferred* claim and mark it as such —

inferred claims are still testable, and surfacing them often

catches places where the docs lie or are silent about real

guarantees the implementation depends on.

When done, you should have a numbered claims list (C1, C2, …). The

hypothesis-generation step (step 3) will reference these by number,

the coverage matrix (template §5) tracks claim × hypothesis, and

scenarios (template §7) state which claim(s) each is trying to

falsify. If a hypothesis cannot

be tied back to a claim, either name the missing claim explicitly

or drop the hypothesis — untethered hypotheses produce ceremonial

scenarios.

Missing claims are a first-class finding. During hypothesis

generation (step 3) you will encounter behaviors the implementation

relies on that no claim covers — Unicode normalisation policy,

specific timeout windows, edge-case error semantics. List these

in the plan's "Missing claims discovered" section (template §1c).

Surfacing them is one of the highest-value outputs of the whole

exercise: it tells the maintainer where docs and implementation

have drifted apart.

2. Scope the change OR the project

Change-scoped: Identify the commit, PR, or feature under test.

List every file touched and the surfaces (RPCs, on-disk formats,

replication messages, public APIs) affected. Build a one-paragraph

blast-radius statement.

Project-wide: No specific change. Instead, enumerate the system's

externally observable surfaces (public APIs, on-disk formats, wire

protocols, replication/consensus, background jobs, operational

controls) and the invariants each must preserve. Declare what is

in-scope and what is explicitly out-of-scope (adapters, ancillary

tools, demo apps) — a project-wide plan that tries to cover

everything covers nothing well.

2b. Inventory existing tests (project-wide only)

Walk the SUT's test surface: unit tests, integration tests, fault-

injection / stability harnesses, smoke scripts, CI workflows, and

any test-plan / runbook docs. For each notable test or harness,

capture: what subsystem, what invariant it pins, and what failure

modes it would catch. This becomes the left-hand column of the

coverage matrix in step 4b.

Do not re-test what is already covered well. The point of the gap

analysis is to surface what is NOT covered.

3. Generate failure-mode hypotheses

For each claim from step 1b, ask: under what conditions could the

SUT fail to honor this claim? Each hypothesis must be tied to one

or more claims by number ("could falsify C3 and C7"). Tests exist

to refute claims, not to "check that things work" — a passing test

should mean "this claim survived this fault", and a failing test

should name the claim it falsified.

Walk the pitfall catalog. Before generating hypotheses from

intuition, open references/common-distributed-systems-pitfalls.md.

It lists 16 failure modes that recur across the Jepsen analyses

corpus, each with a hypothesis template ready to paste-adapt. For

every pitfall, decide if it applies to this SUT: y / n / maybe.

Every y and most maybes become hypothesis rows. This shortcut

prevents the common failure mode of plans that only test what the

agent already thought of.

Generate hypotheses for each touched surface (change-scoped) or

in-scope surface (project-wide) across these categories:

correctness, durability, liveness, partial failure, idempotency /

replay, upgrade / rollback, configuration, performance / fairness.

If a category is genuinely not applicable, say so explicitly. The act

of writing "N/A because…" surfaces wrong assumptions more often than

it sounds like it would.

Boundary and fairness claims trigger §7.M.S. When you encounter a

claim about tenant isolation, authz, namespace, routing,

multi-protocol access, compatibility across API surfaces, or

per-group fairness (noisy-neighbor, queue-group, per-region), tag it

with the boundary or fairness category in §1b. Both categories

trigger the surface-decomposition discipline in §7.M.S of every

scenario that falsifies them — see

references/boundary-and-isolation-testing.md for the boundary

claim matrix template and surface catalogs.

In project-wide mode the list is typically larger (the system has

more surfaces than any single change). Group hypotheses by subsystem

so the gap-analysis table stays readable.

4. Select techniques

Open references/catalog-index.md and find the techniques that match

your hypotheses. For each technique you pick, open its reference file

and write down in the plan: which hypotheses it addresses, what it

would catch that other techniques would miss, the typical cost.

For scenarios that will be serious (any claim in `{safety, durability,

idempotency, isolation, ordering, membership}`), also open the

executing skill's references/oracle-patterns.md and use the

"Checker picker" table at the top to pick the checker(s) matching

your model and claim category. The checker choice is part of the

plan, not a runtime decision.

A change usually warrants 2–4 techniques in combination. One technique

is suspicious — re-check whether you've collapsed multiple distinct

hypotheses into one. A project-wide plan typically reaches further

across the catalog (5–7 techniques) because the surface is larger.

4b. Map coverage and identify gaps (project-wide only)

Build a table indexed by claim, not just by hypothesis. Each row:

the claim (C-number), the hypothesis that would falsify it, the

existing test(s) (from step 2b) that exercise it, the verdict

(covered / partial / not covered), and the gap kind (no test /

shallow test / oracle too weak / no fault-injection variant).

Sort by claim severity × gap so the highest-leverage gaps end up

at the top.

This table is the heart of the project-wide plan. It tells the

maintainer where the product's claims are unverified. Without it

the plan is just a wishlist.

**For very large systems (50+ claims, 100+ hypotheses), split the

matrix.** A per-claim summary table (one row per claim with rolled-

up verdict) gives the maintainer the at-a-glance view; a per-

hypothesis detail table keeps the granular gap-kind information.

Without the split, a single matrix where load-bearing claims appear

in many rows becomes unreadable.

4c. Declare environment requirements

For each technique you picked, list the runtime dependencies the

executing skill will need on the test box: container runtime

(docker / podman + compose), language toolchains (Rust, Go, Node,

Python at specific minima), database / object-store backends

(Postgres N+, MinIO / S3-compatible), fault-injection facilities

(iptables, tc/netem, libfaketime, dm-flakey, Toxiproxy), kernel

features (network namespaces for asymmetric partitions, cgroups

for IO throttling), observability tooling (Prometheus, OTLP

collector), and any project-specific binaries.

Put this in the plan's "Environment requirements" section as a

checklist with version floors where they matter. The executing

skill consults this list at its environment-capability probe step

and uses it to either guide the operator through install or mark

dependent scenarios INCONCLUSIVE.

5. Design scenarios

For each technique, write concrete scenarios. Each scenario must

specify: workload (what generator, rate, distribution, duration);

faults (schedule of what is injected when); oracle (the property

checked and how); observability required; and the three budget

tiers (Smoke / Hardening / Release — see the field list below).

Resist "logs look fine" as an oracle. The oracle must be a

machine-checkable property or a metric SLO with a defined threshold.

In project-wide mode, prioritise scenarios that fill the highest-

leverage gaps from step 4b, and tag each scenario with both the

hypothesis-rows it closes AND the claim(s) it tries to falsify.

Long-tail "nice to have" scenarios go into template §9 (open

questions / followups), not the actionable scenario list in

template §7 — the plan should be actionable, not aspirational.

Every scenario name should encode the claim it targets:

linearizable_per_session_under_partition,

durability_survives_fsync_loss, idempotent_replay_across_restart.

A test named after its claim is harder to weaken; a test named

after its setup ("3-node cluster with chaos") tells you nothing

about what it actually verifies.

Each scenario is an executable spec. Beyond the prose

fields (Workload, Faults, Oracle, Observability, and the three

budget tiers), emit two more:

• Target test file — the relative SUT path where this test

will live if/when it becomes a permanent regression. Follow

the SUT's test conventions: for Rust crates crates/<crate>/tests/auto/<S_id>_<slug>.rs,

for Go modules <module>/<pkg>_test.go, for Python pytest

tests/auto/test_<slug>.py. The auto/ subdirectory makes

generated tests easy to find and review separately from

hand-authored ones.

• Skeleton — a language-specific code block with imports,

the test function signature, and TODO regions for the

workload / faults / oracle bodies. The skeleton MUST include

an AUTO-GENERATED header comment with the plan path and

scenario id so a reviewer can trace any committed test back

to its spec.

The skeleton is what the executing skill (in author mode) writes

to the target path and then fills the TODOs from. The plan +

the generated test are traceable back to each other; if the

plan's prose changes, the test should be regenerated.

Fill §7.M for serious scenarios. If any claim in this scenario's

Falsifies if it FAILs row belongs to `{safety, durability,

idempotency, isolation, ordering, membership}`, the scenario is

*serious* and must fill the §7.M sub-block in the plan template.

A scenario that decomposes into §7.M.S arms (boundary / fairness) is

also serious: each arm is always serious and carries its own §7.M

block, regardless of claim category — the executing skill scores

every arm through the §7.M checker node of the verdict decision tree.

The §7.M sub-block:

• Model under test — pick from the picker in

references/history-discipline.md.

• Operation history — which of the default 11 fields the recorder

captures; the recording mechanism (in-process / external / server-

side / combined).

• Checker — name from the executing skill's

references/oracle-patterns.md "Checker picker" table at the top

of that file. Or, if no checker, write the justification.

• Nemesis + landing evidence — nemesis from the executing skill's

references/fault-injection-howto.md, plus the observable signal

that proves the fault landed.

• Ambiguous outcomes — how the recorder treats timeouts, unknowns,

retries, duplicates.

• Reduction plan — minimisation recipe + the SUT/harness/checker/

env classification step from the executing skill's

references/test-case-reduction.md.

For non-serious scenarios (perf-SLO, liveness, operational), write

§7.M: not applicable (no gated claim category falsified) and move

on. Do not invent a model just to fill the field.

The §7d confidence statement should lean on the chain ("checker X

consumed history Y under nemesis Z with landing evidence E") for

every serious scenario. A serious scenario whose §7.M is partially

filled cannot contribute to a hardening claim.

Fill §7.M.S for boundary and fairness scenarios. If any claim in

this scenario's Falsifies if it FAILs row belongs to

{boundary, fairness}, the scenario is *surface-decomposition

mandatory* and must fill the §7.M.S sub-block in the plan template:

• Surfaces — drawn from the catalog in

references/boundary-and-isolation-testing.md, or SUT-specific

surfaces the catalog does not cover. Minimum three per boundary

claim or written justification for fewer.

• Operations — per surface, which operations the scenario

exercises.

• Adversarial inputs — confusable identifiers from the catalog

in boundary-and-isolation-testing.md.

• Positive controls — what legitimate access must still succeed.
• Negative controls — what illegitimate access must be denied

AND not observable in metrics / logs / side channels.

• Delayed / async paths — background jobs, retries, GC,

compaction, CDC, exports.

• Observability paths — metrics, traces, audit logs that could

themselves leak across the boundary.

• Scenario arms — per-arm IDs (S<n>/api, S<n>/sdk, etc.).

Apply the split-into-arms rule: if the scenario spans more than 3

surfaces, more than 3 claim categories, or requires more than 1

independent oracle, split into arms with independent verdicts.

For non-boundary scenarios, write `§7.M.S: not applicable (no

boundary or fairness claim falsified)` and skip the fields. Do not

invent surfaces just to fill the field.

The §7.M.S sub-block is a *sibling* to §7.M (model / history /

checker), not a replacement. A scenario falsifying both a

consistency claim and a boundary claim fills both blocks.

Every scenario declares three budget tiers. Replace the legacy

single Exit criteria field with explicit Smoke / Hardening /

Release budgets per scenario:

• Smoke budget — minimum config + duration + faults + seeds

required for PASS-smoke.

• Hardening budget — strictly stronger than smoke on every

dimension; required for PASS-hardening.

• Release budget — long / repeated / statistical gate, OR an

explicit not provided — <reason>. Revisit when: <condition>.

declaration. Empty / "TBD" / "see §6b" are explicitly disallowed.

The execute skill uses the budget tier actually met as a verdict

precondition; the findings report surfaces every "not provided"

release budget in a dedicated Release-budget disclosures section.

5b. Argue coverage adequacy

A test plan that lists scenarios without arguing they are *enough*

is not a test plan — it's a wishlist. Before writing the plan file,

build the argument for adequacy. Cover three things:

1. Architectural summary. A one-page (≤ 30 lines) summary of the

system's actual architecture: the major components, how data flows

between them, where state is durable, where consensus runs, where

trust boundaries live. This is not the catalog (catalog is reference

material) — this is the system as it actually exists, written so a

reviewer who has never seen the codebase can follow the test plan.

The architectural summary makes it possible for the reviewer to

spot a missing test ("you have nothing exercising the

storage→index handoff") that a flat scenario list would hide.

2. Coverage adequacy argument. For each claim, demonstrate that

the chosen scenarios — *taken together* — would falsify the claim

if it were violated. The form is: "claim Cn could be violated under

threats T1, T2, …; scenarios Sa, Sb, Sc exercise those threats

under conditions X, Y, Z; therefore if Cn is wrong, at least one

of Sa/Sb/Sc would catch it." A reviewer should be able to read

this and either accept the argument or point at a specific gap

("scenario Sa doesn't actually inject T2 — it only injects T1").

3. Residual uncertainty. Honestly list what the plan does NOT

falsify and why that is acceptable. "Claim Cn is not exercised

under multi-AZ failure because the harness cannot inject AZ-level

faults today; we accept this risk because production deploys are

single-AZ for now." This section is what turns a plan from "tests"

into "an argument for shipping."

These three sections together are the "confidence" the reader

needs. Without them, the plan answers "what would we test" but

not "is testing this enough to ship."

For boundary or fairness claims (any claim whose §7.M.S decomposes

into multiple arms, or whose oracle covers multiple groups), the

§7d statement must include a per-aspect

confidence table after the leading paragraph. The paragraph alone

hides which arms are well-tested vs. which are deferred; the table

makes it visible. A single conservative paragraph that says

"confidence is moderate" without naming which aspects are moderate

and which are low is the specific failure mode the table prevents.

6. Write the plan file

Copy assets/plan-template.md to the plan destination and fill it in.

Default destination is docs/testing-plans/<short-slug>.md in the SUT

repo; the user may override (e.g. when they don't want the agent

writing into their repo, fall back to whatever path they specify, or

to ./testing-plans/<short-slug>.md in the current working directory

if no path was given).

If the parent directory does not exist, create it before writing. Many

repos won't have a docs/testing-plans/ directory the first time this

skill runs; mkdir -p it without ceremony.

The plan slug is the only handoff to the executing skill. Pick a

descriptive slug — durable-idempotent-append-replay, not

plan-1.

7. Self-check

Read the plan back. Every hypothesis has at least one scenario.

Every scenario has an oracle that is not "logs look fine". Every

chosen technique cites its reference file. If anything fails the

check, fix it in the plan; do not move on with known gaps.

The adequacy test. Imagine a reviewer who has never seen the

codebase reading the plan cover to cover. Then they're asked: "if

all of these scenarios pass, would you be comfortable shipping

this code?" If the plan does not contain enough material for them

to answer yes/no with confidence — specifically: the architectural

summary, the coverage-adequacy argument per claim, the residual

uncertainty list — the plan is not done. A list of scenarios is

not a confidence argument.

Anti-pattern checks (run before declaring the plan done).

• Surface decomposition implied by name but missing in arms. If

a scenario name contains "across all surfaces" or names multiple

surfaces but only one Target test file is declared — split into

arms.

• Boundary claim without negative controls. If a claim is in

{boundary} but §7.M.S Negative controls is empty — required

negative controls missing. (Tenancy / authz / namespace / routing

claims are subsumed under boundary; they do not appear as

separate categories.)

• Fairness claim without per-group formula. If a claim is in

{fairness} but the oracle is not a per-group formula from the

executing skill's references/oracle-patterns.md §14 — fairness

criterion missing.

• Vague oracle. A scenario whose Oracle field reads as "no

leaks," "no unauthorised access," or similar prose without a

model / state comparison or formula — sharpen with a concrete

checker pattern.

• Confidence statement missing untested-surface disclosure. If

§7d does not name the untested surfaces for any boundary claim

whose §7.M.S arms include NOT-RUN or PARTIAL-surface — the

§7d surface-coverage disclosure rule requires that naming.

• Release budget without disclosure. If any scenario's

Release budget field is empty, equals "TBD", "see §6b", or any

value not matching either a concrete budget specification OR the

not provided — <reason>. Revisit when: <condition>. template

— absence must be an explicit disclosure, not a silent gap.

• **Scenario name promises decomposition that §7.M.S does not

deliver.** If a scenario name contains "routing", "tenant isolation",

"blast radius", "multi-tenant", "cell", "region", "shard",

"namespace", "availability zone", "replica set", "placement

pool", "failure domain", or similar architectural-boundary

keyword, AND the scenario's §7.M.S Surfaces field is empty or

names only one surface — the plan is incomplete. Either fill

§7.M.S with the surface decomposition the boundary keyword

implies, or rename the scenario so its name does not promise a

decomposition the plan does not deliver. The expert framing:

tests tend to validate that the boundary mechanism *exists*,

not that it *actually contains failure*; this check forces the

plan author to confront which one their scenario tests.

Early exit

If the change genuinely does not warrant a distributed test plan —

for example, a docs-only change, a typo fix, a refactor with no

behavior change covered by existing unit tests — say so explicitly

and recommend the appropriate lighter-weight testing. Do not produce

a ceremonial plan for changes that don't need one.

What this skill does not do

• It does not execute the plan. That's the

executing-distributed-system-tests skill.

• It does not author Jepsen tests, TLA+ specs, or fuzz harnesses.

It tells the engineer which to reach for; building them stays the

engineer's job.

• It does not replace project-specific stability plans. It produces

change-scoped plans that complement them.

Reference files

• references/catalog-index.md — start here; selector page
• references/jepsen-and-elle.md
• references/deterministic-simulation.md
• references/chaos-and-fault-injection.md
• references/fuzzing.md
• references/formal-methods-tla.md
• references/property-and-metamorphic.md
• references/performance-and-benchmarking.md
• references/crash-recovery-and-upgrade.md
• references/common-distributed-systems-pitfalls.md — 16 pitfalls

with hypothesis templates (walk this during step 3)

• references/history-discipline.md — operation-history schema and

ambiguous-outcome handling (required reading when any scenario

will be serious)

• references/boundary-and-isolation-testing.md — surface catalogs,

boundary claim matrix, confusable-identifier catalog,

negative-control anti-patterns (required reading when any scenario

falsifies a boundary or fairness claim)

Each reference file follows the same shape: when to reach for it,

what it detects well, what it misses, concrete tools, papers,

cost / wall-clock signal, plan checklist. The discipline references

(common-distributed-systems-pitfalls.md, history-discipline.md)

instead follow an enumeration + anti-pattern shape — they are walked

exhaustively rather than picked from.

Asset

• assets/plan-template.md — the structure to fill in.