Gating

gating

A gate is a check that blocks. Its only job is to go red when it should.

The characteristic failure is not a wrong check — a wrong check gets noticed.

It is a check that cannot fail, which reports PASS forever and is

indistinguishable from a working one from the outside. That is what makes this

different from ordinary testing: the object under suspicion is the check.

The three obligations

Every gate owes these. A gate missing any of them is not yet a gate.

1. An anchor outside your own code. Something the check compares against

that your implementation did not produce: a published constant, a closed-form

answer, a conservation law, a degenerate case with a known result, an

independent implementation. A check that compares this run to the last run only

ever tells you the code still does what it did. See references/anchors.md.

2. A known-bad it demonstrably rejects. Break the subject the way it would

plausibly break, run the gate, confirm red. Until you have done this you have

not shown the gate works — you have shown it runs. This is the obligation

people skip, because a passing gate feels like evidence.

Two things about known-bads that are easy to get wrong:

• Validate it at the configuration it will run in. A case tuned on a small

or fast setting can stop being bad at full size. An "untrained" grid built

from one Lloyd iteration was genuinely zero-gain at m=2/K=16 and earned a

real +0.10 dB at m=8/K=65536, where one iteration relocates ~63,000 empty

cells toward the mode. It passed the fast gate and certified nothing about

the real one. This matters more than it sounds, because mutate.py needs a

fast gate variant and it is tempting to validate everything there.

• Measure its *reach*. One known-bad is the floor, not the goal. Name which

checks it exercises (known_bad(..., covers=(...))); the harness prints the

checks no known-bad reaches. An audited gate had a single known-bad covering

1 of 8 checks — and the check its whole result rested on *accepted* the same

bad case.

3. A written statement of what it cannot catch. Coverage holes are

invisible from inside a green run: the gate is silent about the thing it does

not look at, in exactly the same tone it uses for the thing it looked at and

approved. The author has to assert the hole; nothing else will.

scripts/gate.py enforces obligations 2 and 3 mechanically — it returns exit

code 2 (INCONCLUSIVE, not PASS) when a gate registers no known-bad or no

coverage limit.

Building a gate

Work in this order. The first step is the one that determines whether the rest

is worth anything.

Name the wrong conclusion, not the component. Not "check the quantizer is

correct" but "prevent shipping *scalar wins at high bit rates* when that would

really be an optimizer artifact." A gate aimed at a conclusion knows what

counts as a near-miss; a gate aimed at a component just exercises the code.

Find an anchor. references/anchors.md lists the kinds, in rough order of

strength, with the questions that find each one.

Prefer brackets to point checks. Assert a value lies strictly between two

things it cannot legitimately pass: better than a baseline, worse than a

theoretical bound. A one-sided check passes for a result that collapsed as

readily as for one that is right — which is how an implementation that

silently does nothing gets certified.

Derive the tolerance from measured noise. Run the thing several times, see

how much it moves, put the threshold outside that. A tolerance picked for

comfort tends to land wider than the defect you are trying to catch, and then

it swallows it.

Then check it is not too tight to mean anything. The opposite failure is

real and less obvious: a margin can be statistically impeccable and practically

empty. A *paired* estimator — scoring both arms on one shared sample so the

common fluctuation cancels — is the right way to measure a difference, and its

standard error *shrinks as the two arms converge*. So "beats the baseline by

3 se" degenerates: a codebook perturbed by N(0, 1e-3) gained +0.0001 dB against

a 3-se margin of 1.2e-06 and was accepted, while real ones gained 0.35–1.41 dB.

The check certified *the effect is real*, not *the effect is worth having*.

Those are different assertions and need different thresholds — and the second

one has to come from an anchor (there, a published lattice codebook), never

from the estimator, which knows nothing about what magnitude would matter.

Build the known-bad and confirm red. Then run scripts/mutate.py for the

failures you did not think of.

Wire it to a non-zero exit and run it before the thing it gates, not after.

A gate that runs after the results are written is a report.

Auditing an existing check suite

Given tests, a linter config, a CI job, or a gate someone already wrote, the

question is not "do these pass" but "can these fail". Full procedure in

references/auditing.md; the fast version:

• For each assertion, name a concrete input that makes it fail. If you cannot,

it is decoration — delete it or fix it.

• Check each oracle's range against the range you actually operate in. A

published table that stops short of your regime is a hole with a green light

on it.

• Run scripts/mutate.py against the code the suite covers. Every survivor is

a behaviour nothing checks.

• Look for assertions whose truth does not depend on the subject at all.
• Look for tolerances wider than the effect being measured.

Scripts

# harness: refuses to report PASS without a known-bad and a coverage limit
python3 scripts/gate.py           # importable; see the module docstring

# mutation pass: which single-token changes does the gate NOT notice?
python3 scripts/mutate.py --target src/codec.py -- python3 calibrate.py
python3 scripts/mutate.py --target grids.py --max 40 -- pytest -q

mutate.py requires the gate to pass on unmutated code first and refuses to

run otherwise, because survivor counts against an already-red gate mean

nothing. It restores the file even on interrupt, and uses tokenize so string

literals and comments are never corrupted. It is the zero-dependency pass that

works against *any* gate command; once it stops finding survivors, mutmut or

cosmic-ray go deeper on Python test suites specifically.

Anti-patterns

Each of these has shipped a wrong result somewhere. They are ordered by how

convincingly they impersonate a working gate.

| Anti-pattern | Why it survives review |

|---|---|

| Slack wider than the defect | A tolerance chosen for comfort. The gate passes the real thing *and* the broken thing, and reports PASS for both. Derive the threshold from noise, then confirm the known-bad falls outside it. |

| An oracle with a coverage hole | Published anchors end somewhere. If the defect is past the end of the table, the check is structurally incapable of catching it and looks fine. State the range the anchor covers. |

| An assertion whose truth doesn't depend on the subject | "The output has at least N distinct colours" is equally true of an unchanged frame. Prefer differential checks: the state must *change* when it should, and a toggle applied twice must return to the byte-identical original. |

| Confirming the check ran, not that it can fail | "Invoke it and confirm the step appears in the output" catches a check that was never wired up. It says nothing about a check that is wired up and toothless. |

| Comparing against your own previous output | Regenerated goldens ratify drift. If the golden came from the code under test, it is a changelog, not an oracle. |

| A cache keyed on the problem rather than the method | cache[(m, K)] cannot notice that the code producing the value changed. Version-stamp the artifact and delete on mismatch instead of trusting. |

| A self-matching predicate | until ! pgrep -f trainer never exits, because the watching shell's own argv contains trainer. Worse, a malformed variant exits immediately and reports the job finished while it runs. Wait on a PID. |

| A margin that is significant but not meaningful | A threshold derived purely from estimator noise certifies that an effect is *real*, not that it is *worth having* — and a paired estimator's noise shrinks as the arms converge, so the margin can approach zero. Pair every noise-derived floor with a magnitude an anchor says would matter. |

| A strict bracket at an attainable optimum | A theoretical bound is often reachable, and reaching it is the best possible outcome. A strict edge then goes red on a perfect result and blocks real work. Ask of each edge whether the subject can legitimately sit exactly there. |

| A gate written by whatever produced the artifact | Shared assumptions produce shared blind spots, and the convention both inherited is the one neither questions. Anchors are the defence, because an anchor is the one input the producer did not choose. |

Division of labour

This skill is for results and pipelines where the failure mode is a

plausible wrong number that would survive a careful read.

| Use | When the risk is |

|---|---|

| gating (this) | A number or empirical result is about to be published or acted on, and a wrong-but-reasonable value would pass unnoticed. Output: a gate that blocks. |

| challenging | An artifact would draw a specific objection from a skeptical reader — prose, analysis, a recommendation, a diff. LLM judgement against a persona. Output: findings and a SHIP/REVISE/RETHINK verdict. |

| verifying-claims | Documentation says something about code that is no longer true. Output: prose-vs-code disagreements. |

| A test suite / TDD | Code you wrote does not behave as specified. Output: red tests. |

challenging asks *would a careful reader object?* gating asks *can this

check go red?* They are complements and they miss different things: an

adversarial reviewer will not recompute your constants, and a gate will not

notice that your framing is wrong.

A gate checks correctness, and cannot check comparability. If two arms of a

comparison are each individually correct but not *comparably implemented*, no

anchor and no mutant will see it: nothing is broken, so nothing goes red. A

published ablation reported one transform 11–24× slower than another and

carried a caveat saying the number was implementation-bound; both arms passed

every correctness check, and the real finding — one arm was a tuned BLAS call

and the other an interpreted loop — was found by a human reviewer months of

gate-work later. Related: **performance claims have no anchor in this

framework at all.** There is no published constant for how fast your code

should be. Wall-clock belongs to benchmarking discipline (matched

implementation effort, min-of-trials, stated hardware), not to gating.

One caution about pairing them. A same-model reviewer is an independent

*context*, not an independent *reviewer* — it shares your priors, so the

convention you did not question is the one it will not question either. Where

that matters, an anchor beats a reviewer, because an anchor is not

negotiable.

References

• references/anchors.md — kinds of oracle, strongest first, and how to find

one when nothing published exists.

• references/auditing.md — the full "can this fail?" pass over an existing

suite, including how to read a mutation report.