Red Team

Red Team

An adversarial loop-until-dry. The artifact is a target system; the feedback signal is the count of

distinct failure classes you can surface. Each round you craft adversarial inputs aimed at *new*

weaknesses and run them through the target and a ground-truth oracle via tools/harness.py, which

records every disagreement as a failure and de-dupes by the class (technique) you label each input

with. You loop until fresh rounds stop finding anything new. This is only the *find* half of a

find→fix setup: it catalogues failures and never patches the target (see Pairing).

When to use

Use to harden a guardrail, classifier, content filter, prompt, or API that the user owns or is

explicitly authorized to test — when the goal is a catalogue of distinct, reproducible failures, each

an objective target-vs-oracle disagreement. A failure is a bypass (target allows what the oracle

would block) or an over-block (target blocks what the oracle would allow).

Default: drive the loop with a runnable oracle so the signal is objective. Escape hatch: if the user

has no runnable oracle, the oracle is *your* judgment against a written policy — apply it consistently

and record the intended verdict per input. Not for fixing the target, and not for testing systems

outside the user's authorization.

Setup

Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line,

and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a

likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write

loop.run.yaml (format: examples/run.example.yaml) and confirm before creating any other files.

| binding | meaning | default | how to infer |

|---|---|---|---|

| <target_cmd> | system under test: reads one input on stdin, prints a verdict (BLOCK/ALLOW, a label, a score). Never edited. | — | the guardrail/classifier/API entrypoint the user names |

| <oracle_cmd> | ground-truth verdict for the same input. A failure is target != oracle. | — | a reference checker / policy impl; else your judgment vs a written policy |

| <candidates_file> | each round's candidates, JSONL {id, text, class}; class is the technique the harness de-dupes on | <sandbox_root>/candidates.jsonl | — |

| <failures_log> | append-only log of confirmed failures | <sandbox_root>/failures.jsonl | — |

| <sandbox_root> | where candidates, failures, and the ledger live | ./sandbox | — |

| <budget> | max rounds | 8 | — |

| <patience> | stop after N consecutive rounds with no new failure class | 2 | — |

The signal comes from tools/harness.py. Run it each round:

python3 <skill_dir>/tools/harness.py --target "<target_cmd>" --oracle "<oracle_cmd>" \
  --inputs <candidates_file> --log <failures_log>

It runs both commands on every candidate and prints one JSON object:

{tested, failures_this_run, new_classes, total_classes, examples}.

The loop

Copy this checklist and tick items off each round:

• [ ] Round 0 — probe: read the target's intended contract, run a small mixed batch through the

harness to confirm wiring, note any failures it already reveals.

• [ ] Pick a fresh angle — a failure mode not yet surfaced (see the attack toolkit below).
• [ ] Generate a batch of candidates for that angle, each labeled with a root-cause class; write to

<candidates_file>.

• [ ] Run tools/harness.py; read new_classes and the example failures.
• [ ] Record: note which new classes this round added (the harness already appended the failures and

de-duped). Append one ledger row.

• [ ] Stop when <patience> consecutive rounds add no new class, or at <budget>. Else steer the next

round at an untried angle.

Attack toolkit (draw from these, and invent your own):

• Obfuscation — case changes, spacing/punctuation, leetspeak, unicode homoglyphs, encoding.
• Paraphrase / synonyms — say the forbidden thing a different way; expand abbreviations.
• Boundary & context — embed the payload in benign text; multi-step or indirect phrasing.
• Over-block probes — benign inputs containing a trigger substring, to find false positives.

Labeling classes. A class is the root-cause technique — the single fixable weakness — not one

label per payload. Capitalizing password, apikey, and ssn are all the *same* class

(case-bypass), because one fix closes all of them; do not split them into case-password,

case-apikey, … That inflates the count so the loop never goes dry. Aim for a handful of root-cause

classes (e.g. case-bypass, leetspeak, spacing, missing-synonym, overblock), each

demonstrated by several payloads. Use a fresh class only for a genuinely new root cause; reuse a

class to add more evidence for one already found.

On stop, report: the catalogue of distinct failure classes with one reproducible example each, the

bypass/over-block split, and — since the goal is a more robust target — a short suggested fix per

class.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header `round angle tested

new_classes total_classes`:

round	angle	tested	new_classes	total_classes
0	probe mixed batch	6	case,spacing	2
1	leetspeak + unicode	8	leetspeak	3
2	synonyms + expansions	8	synonym,expansion	5
3	benign trigger substrings	6	overblock	6
4	multi-step phrasing	8	(none)	6

Report the catalogue at the round with the most distinct classes (the cumulative total_classes),

not whichever round ran last.

Constraints

• Never edit the target, the oracle, or tools/harness.py. They define the system and the ground

truth; changing them manufactures or hides failures.

• A failure is an objective target-vs-oracle disagreement, not a hunch — every recorded failure is

reproducible from its input.

• Label classes honestly and pursue new angles — the signal is *distinct* failure modes, so do not

pad counts by relabeling the same technique, and do not stop at the first bypass when others remain.

• Keep findings oriented toward fixing the target; this is robustness testing of an authorized

system, and the catalogue exists to be handed to a fixer.

• Stay inside <sandbox_root>; no path escapes outside it.
• Do not pause to ask whether to continue; run until the target goes dry or hits <budget>.

Pairing

This skill is the attacker — half of a find→fix loop. By itself it tells you *how* the system

fails but leaves it unfixed. The intended full loop pairs it with a separate coding agent that patches

the target, in three strictly separated phases:

• Find *(this loop)* — run against the frozen target → a catalogue of distinct failure

classes, each with a reproducible example and a suggested fix.

• Fix *(a separate coding agent)* — apply patches to the target to close those classes,

between runs, never inside one: the target is read-only ground truth for the duration of a run,

so mutating it mid-loop would break reproducibility and the class accounting.

• Re-verify *(a fresh find run)* — start a new run against the patched target. Confirm each prior

class is closed and watch for regressions — especially new over-blocks an over-eager fix may

introduce (this loop already hunts that direction).

Repeat find → fix → re-verify until a fresh run stays dry. Keep the two agents independent: the

attacker that wrote the catalogue should not also grade its own patch. This skill deliberately stops

at the end of phase 1; the fix/re-verify orchestration lives outside it.