Weak Agent Test

Adversarial review — weak-agent harness for docx-cli

This harness answers one question: **can weak agents actually use docx-cli to get

real work done, and what should we fix first?** It runs the weak-agent-test

workflow (.claude/workflows/weak-agent-test.js), which fans out one weak exercise

agent per scenario (Haiku by default — swappable to Sonnet via args.model), renders

every output with Microsoft Word, grades each against ground-truth criteria with an

opus judge, and has opus synthesize a prioritized improvement report.

Exercise agents do NOT self-report tool counts — every tool-economy and token number

is measured after the run (agents under-count their own calls ~2×, so self-reports

were dropped): from the agent transcripts for the Claude arms, from each scenario's

exercise.json ledger for the local arm. Both roll up into the same Run-metrics table

(tokens, wall-clock, tool split, correctness) via exercise-metrics.ts.

The test corpus is bundled with this skill under scenarios/, one folder per

scenario, named after its key (scenarios/mnda/, scenarios/invoice/, …). Each

scenario folder is self-describing and holds everything that scenario needs:

• task.md — the AGENT-FACING request, written as a human delegating the work:

the goal, the data, the intent — and no tool vocabulary (no docx commands,

locators, or OOXML terms), because discovering which features deliver the outcome

is part of what's measured,

• criteria.md — the JUDGE-ONLY grading rubric (the precise, tool-specific checks).

The stage step withholds it from the agent's run workspace, and the judge reads

it from the pristine source — the agent never sees the answer key,

• the fixture .docx to work on (edit scenarios only; authoring scenarios create

their output fresh),

• assets/ — any additional inputs (data files, images; empty for most edit

scenarios).

The workflow's SCENARIOS manifest holds only the per-scenario routing metadata

(key, bucket label, edit/author kind, the doc filename); whether a baseline gets

rendered is DERIVED from the kind (every edit scenario has a pristine source, so it

gets one — see hasBaseline()), not a stored field. The actual

request/criteria/fixture/assets all live in the folder. The skill is

therefore self-contained and travels with its test corpus. To change what a scenario

tests, edit the files in its folder. (Heavy, ephemeral run outputs — edited docx,

renders, reviews, the report — are dumped to ./tmp/docx-weak-agent-test/<ts>/,

never into the repo.)

Staging is ONE code path for every backend: scripts/stage-scenario.ts copies a

scenario folder, strips the judge-only criteria.md, and verifies the inputs landed.

The workflow's Stage agent runs it per scenario; the local corpus runner imports it.

Each run produces, under the timestamped run dir, one result folder per scenario

(named after its key) plus the run-level report and metrics:

<RUN_DIR>/
  REPORT.md            ← synthesized report; the Metrics phase appends the measured
                          run-metrics section (local: in-run; Claude: your post-run pass)
  exercise-metrics.md  ← measured per-exercise-agent tokens/time/tool split
  exercise-metrics.json
  <key>/               ← one per scenario; the worked-on copy lives here
    task.md  assets/   ← (criteria.md is withheld from this copy — judge-only)
    <doc>.docx         ← the edited/authored document
    renders/output/    ← the OUTPUT: Word-rendered page PNGs + read.md (markdown read view)
    renders/baseline/  ← the pristine "before": page PNGs + read.md (every EDIT scenario;
                          absent only for the authored eliot-journal — no source to diff)
    review.md          ← the judge's saved review for this task (written in-run)
    verdict.json       ← the judge's structured verdict incl. taskSuccess (written in-run
                          by the judge — the correctness source the Metrics phase reads)
    metrics.json       ← this task's measured tokens/time/tool split + correctness
                          (local: in-run Metrics phase; Claude: your post-run pass)

The render step fires the moment each task finishes (for both arms) and produces,

for the OUTPUT and — whenever a pristine source exists (every edit scenario) — its

BASELINE "before", BOTH deliverables in each render dir: the page PNGs AND a read.md

(the markdown read view of that doc). The judge reads all four (output PNGs + read.md,

baseline PNGs + read.md) to compare before/after both visually and textually. The

workflow's render step is idempotent: for the local backend the corpus runner

already produced the SAME artifacts at the SAME paths as it went, so the render step

just reuses them (re-rendering only anything missing) — no double-render; for the

Claude backend nothing is pre-rendered, so it does the full Word render. Either

way the judge grades Word-rendered PNGs (the local harness runs on the mac, where the

corpus's default render engine IS Word).

Steps

Run these in order from the repo root. Do NOT skip the build — the global docx on

PATH is a stale binary; the harness must test the CURRENT working tree.

1. Preflight — ALWAYS rebuild (mandatory gate)

The whole harness is meaningless if it tests a stale binary, so the build is a hard

gate, not an optional step. **Always run bun run build:binary, even if dist/docx

already exists** — never reuse a prior build. Abort the whole run if any check below

fails.

REPO="$(git rev-parse --show-toplevel)"
cd "$REPO"
SCENARIOS_DIR="$REPO/.claude/skills/weak-agent-test/scenarios"   # this skill's bundled corpus (one folder per scenario)

# Word must be installed (this harness renders with Word, not LibreOffice).
test -d "/Applications/Microsoft Word.app" || echo "WARNING: Microsoft Word not found — render phase will fail."

# (1) Build the CURRENT working tree into a fresh standalone binary. Abort on failure.
bun run build:binary || { echo "BUILD FAILED — abort"; exit 1; }
BINARY="$REPO/dist/docx"

# (2) Hard gate: the fresh binary must match package.json's version AND have `render`.
# `--version` prints "docx X.Y.Z"; take the 2nd space-delimited field. NOTE: use `cut`,
# NOT an awk field reference — a literal dollar-N positional token gets clobbered by
# slash-command positional-arg substitution when this skill runs with arguments, mangling
# the gate. Keep this whole block free of dollar-N tokens for the same reason.
EXPECTED="$(bun -e 'console.log(require("./package.json").version)')"
GOT="$("$BINARY" --version | cut -d' ' -f2)"
echo "built docx $GOT (package.json: $EXPECTED)"
[ "$GOT" = "$EXPECTED" ] || { echo "VERSION MISMATCH ($GOT != $EXPECTED) — build is stale, abort"; exit 1; }
"$BINARY" render --help >/dev/null 2>&1 || { echo "render MISSING — build stale/broken, abort"; exit 1; }
echo "preflight OK: fresh $GOT binary with render"

If the version mismatches or render is missing, the build did not reflect the

working tree — stop and fix it before running. Do not proceed on a stale binary.

> First-run note: Word-for-Mac rendering triggers a one-time macOS Automation

> permission prompt for the controlling terminal. If the render phase fails on a

> fresh machine, grant it under System Settings → Privacy & Security → Automation and

> re-run.

2. Make an isolated run workspace (under ./tmp/)

Create an empty timestamped ./tmp/ run dir per workflow run. **Do NOT copy the

scenarios here — the workflow's Stage** phase runs scripts/stage-scenario.ts for

_only the active scenarios_, seeding one subfolder per scenario ($RUN_DIR/<key>/), so

originals stay untouched, the repo stays clean, and a single-scenario run doesn't drag

the whole corpus along:

TS="$(date +%Y.%m.%d-%H%M%S)"
RUN_DIR="./tmp/docx-weak-agent-test/$TS"
mkdir -p "$RUN_DIR"   # empty; the workflow's Stage phase seeds one subfolder per active scenario from $SCENARIOS_DIR
echo "RUN_DIR=$RUN_DIR"

3. Launch the workflow (up to 3 concurrently)

Invoke the Workflow tool with scriptPath pointing at the workflow file and pass

the absolute paths as args:

Workflow({
  scriptPath: "<REPO>/.claude/workflows/weak-agent-test.js",
  args: {
    runDir: "<RUN_DIR from step 2>",
    binary: "<BINARY from step 1>",
    scenariosDir: "<SCENARIOS_DIR from step 1>",
    model: "haiku",              // the exercise model: "haiku" (default) or "sonnet"
    only: <optional scenario filter — see below>
  }
})

> Exercise agent type. The exercise agents run as the repo's weak-exercise

> agent type (.claude/agents/weak-exercise.md): minimal tools and no Skill

> tool, so the session's skills catalog stays OUT of their context (it's a

> per-turn token tax and leaks docx-cli/harness names into the

> "capable-but-fresh agent" premise). The agent registry loads at SESSION

> start — in a session older than that file, the workflow aborts with

> "agent type 'weak-exercise' not found"; pass

> exerciseAgentType: "general-purpose" to override for that session (and note

> the run's base context is then ~4k tokens/turn heavier, so its token numbers

> aren't comparable to weak-exercise runs).

> Never resume a benchmark run whose exercise phase failed. If an exercise

> agent dies (API error → that scenario reports no exercise/verdict), re-run

> the WHOLE run in a FRESH run dir. resumeFromRunId replays the cached stage

> step without re-copying fixtures, so re-run exercise agents would edit

> already-edited documents — double redlines, double fills, unusable verdicts

> (this voided run r2 on 2026-07-15, twice). The failure is worse than it

> looks because the resume cache is PREFIX-based, not keyed: everything

> issued AFTER the first missing/changed result re-runs live, not just the

> dead agent. So a dead exercise for a MANIFEST-EARLY scenario (mnda is

> first) re-runs EVERY exercise against edited docs even if you restore that

> one scenario's staging state — while a dead LAST scenario (eliot-journal)

> happens to resume cleanly. Don't gamble on manifest position: exercise-phase

> failure → fresh run dir, no exceptions. Resume is only safe for failures at

> or after the render phase (dead judge/synth), where nothing mutates

> documents no matter how much of the suffix re-runs.

Running 3 at a time (the fast path to averaged numbers). The benchmark

methodology is 3 runs per arm/model, and runs can go concurrently: launch up to

three Workflow invocations in one message, each with its OWN RUN_DIR from step 2

(suffix the timestamp, e.g. $TS-r1, $TS-r2, $TS-r3). This is safe because the

only shared mutable resource is Microsoft Word, and the CLI itself serializes Word

access across processes with an advisory lock (src/core/render/engines/word-mac.ts)

— concurrent runs' renders queue instead of corrupting each other. Don't go beyond ~3:

renders start spending more time queueing than rendering. A haiku-vs-sonnet

comparison is just two batches: three runs with model: "haiku", three with

model: "sonnet" (never mix models within one run dir).

only restricts the run to a subset of scenarios (omit it to run all 6). To run a

single task, pass its key as a plain string — only: "mnda". It also accepts an

array (only: ["mnda", "invoice"]) or a comma/space-separated string; all forms are

normalized to the same list. The keys are the folder names under $SCENARIOS_DIR

(run ls "$SCENARIOS_DIR" if you need to confirm them); unknown keys abort the run

with a "No scenarios matched" error listing the valid ones.

> Use scriptPath, NOT name: "weak-agent-test". Launching by name resolves to a

> copy cached at session start, so any edit to the workflow made during the session is

> ignored; scriptPath always reads the current file from disk. (The workflow also

> tolerates args arriving as a JSON string — the runtime stringifies it — so passing

> a plain object is fine.)

When the tool returns, note each run's Transcript dir: path it prints — call it

TRANSCRIPT_DIR (it looks like …/subagents/workflows/wf_<id>). You need it in

step 4 to measure per-agent tokens and time. With concurrent runs, keep each

run's (RUN_DIR, TRANSCRIPT_DIR) pair matched.

Scenario keys (omit only to run all 6):

mnda, invoice, resume, contract-markup, contract-finalize, eliot-journal.

If the user passed scenario keys as arguments to this skill (e.g.

/weak-agent-test mnda invoice), parse them into the only array. Otherwise run

everything.

Each run is heavy (6 exercise agents, serialized Word rendering, 6 opus judges + an

opus synthesis pass); it can take many minutes. Watch live progress with /workflows.

4. Save the report + measure the exercise metrics

When a workflow completes, its return value is

{ arm, report, runDir, binary, exercises, verdicts }. The report contains the

scoreboard, per-task merits/demerits, and prioritized fixes — deliberately without

tool-call or token numbers (nothing self-reports them).

Most of this is now written in-run — don't re-do it. The workflow's synth agent

writes REPORT.md to disk itself, the judge writes each <key>/verdict.json, and —

for the local backend — the workflow's final Metrics phase already ran

exercise-metrics.ts --append-report, so REPORT.md already ends with the measured

Run metrics section and exercise-metrics.{md,json} + per-<key>/metrics.json

already exist. So:

• Do NOT overwrite $RUN_DIR/REPORT.md. It's authoritative on disk (synth wrote

it; the Metrics phase appended to it). Only write it from the returned report as

a *fallback* if the file is somehow missing — never over an existing one, or you'll

clobber the appended metrics.

• Metrics — the **measured per-exercise tokens (input AND output) + wall-clock +

docx/non-docx tool split + correctness**. The workflow can't measure tokens/time

itself (the runtime gives its JS no token API and bans clocks), so this is a script

pass — but only the Claude backend still needs you to run it:

• Local (exerciseBackend: "local") — **already done by the workflow's Metrics

phase** (reads each <key>/exercise.json _local block + verdict.json). Nothing

to run; just confirm REPORT.md ends with a "Run metrics" section.

• Claude (exerciseBackend: "claude") — run it now (the token pass reconstructs

from the transcripts, and TRANSCRIPT_DIR — the path you noted in step 3 — is only

known after launch, so the workflow can't do this itself). The 4th arg is the

exercise model (args.model, default haiku — pass sonnet if you ran sonnet,

or it matches no agents and emits an empty table). --append-report adds the

section to REPORT.md with no shell redirect:

     bun "$REPO/.claude/skills/weak-agent-test/scripts/exercise-metrics.ts" \
       "<TRANSCRIPT_DIR>" "$RUN_DIR" "$BINARY" "haiku" --append-report

Repeat per concurrent run (match each RUN_DIR with its own TRANSCRIPT_DIR).

Either way you end up with the Run metrics section on REPORT.md, run-level

$RUN_DIR/exercise-metrics.{md,json} (tagged with backend), and each scenario's

measured row in $RUN_DIR/<key>/metrics.json. Token cost is reported as **effective

input** (cache-weighted: fresh/non-cache input + cache write ×1.25 + cache read

×0.1) plus output, kept separate — NOT a single "total tokens", because cache

reads are ~10× cheaper than fresh input and lumping them in overstates cost. The raw

cache split is in the Totals table and exercise-metrics.json.

• Present in chat: the Executive summary, the per-task merits/demerits, and

the measured metrics — correctness (N/6 success), total docx vs other calls +

docx share, fresh/cache input + output tokens, total wall-clock, and the

per-scenario outliers. For a multi-run batch, also give the across-runs averages

(tasks solved of 6, effective input, output, wall-clock). Tell the user where the

artifacts live:

• <RUN_DIR>/REPORT.md — findings + scoreboard + per-task merits/demerits + measured metrics table
• <RUN_DIR>/exercise-metrics.json — the raw numbers
• <RUN_DIR>/<key>/ — one folder per scenario, each holding that task's worked-on

.docx, its read.md (markdown read view) and renders/ (the Word PNGs the

judge looked at), review.md + verdict.json (the judge's saved review + verdict),

and metrics.json (that task's measured tokens/time/tool split + correctness)

Backends & arms

The exercise slot is swappable; everything downstream (render → opus judge →

opus synthesis, all against the same rubrics) is identical for every backend and

arm — that's what makes the numbers comparable.

• Exercise model (args.model): "haiku" (default) or "sonnet" — same

workflow, same prompts, only the exercise agents' model changes.

• Local harness (args.exerciseBackend: "local"): the exercises run OUT OF BAND

on the local-first agent harness (model built in), then the workflow

renders/judges/synthesizes the results identically. Two steps:

• bun "$REPO/.claude/skills/weak-agent-test/scripts/run-local-corpus.ts" "$SCENARIOS_DIR" "$RUN_DIR" "$BINARY" <HARNESS_DIR> [--context N] [--timeout SEC] [key...]

— serial (single GPU); stages via the same stage-scenario.ts, runs the

harness per scenario, and parses each session ledger into

$RUN_DIR/<key>/exercise.json (it also writes a run-level $RUN_DIR/corpus.log

orchestration log itself — no stdout redirect needed). Every number is

ledger-MEASURED (the local model

is never asked to self-report), including a code-computed status

(completed = the harness process ran to its own stop, failed = the watchdog

killed it or it crashed on a signal — lifecycle only; the judge owns quality).

LOCAL_MODEL_PATH/LOCAL_MMPROJ_PATH env vars override the harness's built-in

model for control runs.

• Collect the results DETERMINISTICALLY and pass them to the workflow inline, so it

skips its LLM LOAD agent and the code-computed status/account reach the judge

straight from disk:

     EXERCISES="$(bun "$REPO/.claude/skills/weak-agent-test/scripts/collect-exercises.ts" "$RUN_DIR")"

then launch with `{ runDir, binary, scenariosDir, exerciseBackend: "local",

modelLabel: "<harness/model name>", exercises: <the collected array> }`. The

workflow runs the normal Render/Judge/Synthesize pipeline on them. (If you omit

exercises, the workflow falls back to an LLM LOAD agent that reads the

exercise.json files itself — the status is still code-computed on disk, but

prefer the deterministic collect so nothing re-reads it through a model.)

The point of this arm is marketing the local harness by its **competitiveness with

Haiku**: same tasks, same judge, same rubrics — only the exercise brain differs.

Its cost/effort is ledger-measured into each exercise.json under _local, and the

workflow's final Metrics phase rolls it up (via exercise-metrics.ts --local)

into the SAME Run-metrics table the Claude arms get — tokens, wall-clock, tool split,

correctness — appended to REPORT.md automatically, in-run (no post-run step for

this backend), so the local-vs-Haiku numbers are directly comparable.

• Competitor arm (args.arm: "anthropic-docx-skill"): the A/B bake-off against

Anthropic's bundled docx skill. First provision it with

bun "$REPO/.claude/skills/weak-agent-test/scripts/stage-competitor.ts" <SKILL_DEST> [RUN_DIR] (fetches the real skill and

installs/verifies its full toolset — fairness gate), then pass

arm: "anthropic-docx-skill", competitorDir: "<SKILL_DEST>". Only the exercise

agents' tool instructions differ; grading is identical.

Notes

• This harness is re-runnable: each invocation rebuilds the binary (mandatory),

stages a fresh ./tmp/ run dir, and never mutates the bundled scenarios/.

• The headline benchmark metrics are correctness (tasks solved of 6), the tool

economy (docx-cli calls vs other calls), and token cost as **effective input +

output** — all measured by exercise-metrics.ts (transcripts for Claude, the

_local ledger for local), never self-reported.

• The weak agents invoke the binary at an allowlisted absolute path

(dist/docx), so they should not hit permission prompts for the CLI itself. The

benign shell commands they and the render step use (mkdir, cp, ls, cat,

bun) are NOT yet allowlisted — if you get prompted, add them via the

update-config skill or run with edits allowed. See .claude/settings.local.json.

• To add a scenario, create a folder under this skill's scenarios/<key>/ holding

task.md (the agent-facing request, in human voice — NO tool vocabulary, so the

agent must discover the features), criteria.md (the judge-only grading rubric —

withheld from the agent's run workspace, read by the judge from the pristine source),

the fixture .docx (edit scenarios only), and an assets/ folder, then add a

routing entry to SCENARIOS in the workflow (.claude/workflows/weak-agent-test.js,

shape { key, bucket, kind, doc }) AND to the MANIFEST in

scripts/run-local-corpus.ts (shape { key, doc, kind }). To change what an existing

scenario tests, edit the files in its folder — the request/criteria/fixture/assets all

live there, not in the workflow.

Scripts

All Bun/TypeScript (this is a Bun-first repo — no shell scripts):

• scripts/stage-scenario.ts — stage ONE scenario (copy + strip criteria.md +

verify). The single staging path: the workflow's Stage agent runs it; the local

corpus runner imports it.

• scripts/exercise-metrics.ts — post-run run-metrics rollup (both backends):

measured tokens (fresh/cache input + output), wall-clock, docx/other tool split, and

correctness (from the judge verdicts), per scenario + totals + run-over-run

comparison. Claude reads the transcripts; --local <runDir> <label> reads each

exercise.json _local block.

• scripts/run-local-corpus.ts — run the exercise phase on the local agent harness

(serial, watchdogged), producing exercise.json per scenario (with a code-computed

status) for exerciseBackend: "local".

• scripts/collect-exercises.ts — deterministically read the run's exercise.json

files into the args.exercises array, so the workflow's local backend skips its

LLM LOAD agent (the status reaches the judge from disk, not via a model).

• scripts/parse-local-ledger.ts — parse one local-harness session ledger into the

exercise shape (ledger-measured tool calls, tokens, timings, and the process

status).

• scripts/local-exercise-prompt.md — the prompt template the local runner renders

per scenario (task inlined for the small model).

• scripts/stage-competitor.ts — provision the Anthropic docx skill + its full

toolset for the competitor arm (fairness gate).