Docs Corpus Audit Agent Skill

docs-corpus-audit -- whole-corpus regrounding pass

The docs corpus drifts silently between releases. docs-sync catches

drift introduced by individual PRs at PR-open time. This skill catches

the accumulated drift that slips past per-PR review -- stale flag

names, dead nav links from past IA reshuffles, deprecation banners

that outlived their version targets, factual claims whose source-side

truth has moved.

The pattern is **A1 PANEL + WAVE EXECUTION + S7 DETERMINISTIC TOOL

BRIDGE + A8 ALIGNMENT LOOP + A9 SUPERVISED EXECUTION**. The corpus is

split into disjoint page scopes; one verifier subagent owns each

scope; agents extract factual claims, S7-verify against source, apply

surgical fixes inline. The orchestrator then runs an alignment-loop

pass to re-verify that applied edits actually ground out true.

This skill is ADVISORY but ACTIONABLE: agents apply edits inline on a

working branch. The orchestrator is the sole writer to git -- stages,

commits, pushes. Maintainer reviews the resulting PR.

Sibling contract with docs-sync

These two skills share substrate. Be explicit:

| Shared resource | Owner | Both use |

|---|---|---|

| .apm/docs-index.yml (corpus map) | docs-sync | yes |

| doc-writer persona | shared | yes (per-page edits) |

| python-architect persona | shared | yes (S7 verification) |

| editorial-owner persona | shared | optional (voice pass at scale) |

| cdo persona | shared | yes (final synthesis) |

| assets/panelist-return-schema.json | docs-sync (mirrored) | yes |

Trigger boundary (avoid DISPATCH COLLISION):

• docs-sync triggers on a PR event ("PR opened/synchronized",

source-diff-driven).

• docs-corpus-audit triggers on a maintainer ask for a

WHOLE-CORPUS pass ("audit the corpus", "reground", "pre-release

sweep") -- no PR required, no diff required, the whole corpus

is the input.

If a maintainer asks "review this PR's doc impact", route to

docs-sync. If they ask "audit all our docs" or "the docs feel

stale everywhere", route here.

Architecture invariants

• Wave-batched, not flat. Pages are partitioned into 6-8 disjoint

scopes; each scope is one verifier subagent. Cost scales with

wave size, not corpus size. A wave of 6 agents on ~10 pages each

is the canonical shape.

• Disjoint page ownership. Each subagent has EDIT AUTHORITY on

its scope only. No two agents touch the same file -- guarantees

no merge conflicts during fan-in.

• S7 verification is mandatory. Every factual claim is verified

against deterministic source: uv run apm <verb> --help for CLI,

grep -n src/apm_cli/ for symbols, python -c "import ..." for

module shape, file-existence checks for nav links. Never assert

from LLM recall.

• Surgical edits only. 1-3 line patches per drift, preserving

voice. Restructuring is deferred to the orchestrator post-pass,

never auto-applied by per-scope agents.

• Single-writer interlock for git. Subagents NEVER run

git commit, git push, or gh pr <write>. Orchestrator

commits per wave; pushes once per session.

• Alignment loop (A8). After waves return, orchestrator

re-greps the corpus for the patterns the agents claimed to fix.

Any residue triggers a targeted re-dispatch (max 2 redrafts) or

is escalated to maintainer.

Roster (composition, not invention)

Reuse docs-sync's personas. Do NOT invent a one-off "grounding-

verifier" role; that's R3 EXTRACT in reverse.

| Role | Persona | Always active? |

|---|---|---|

| Per-scope verifier+editor | python-architect (S7) and doc-writer (edits), bundled into one subagent prompt per scope | Yes -- one per page scope, parallel fan-out |

| Cross-corpus post-pass | orchestrator (deterministic greps via scripts/scan-cross-corpus-drift.sh) | Yes -- once after waves return |

| Alignment-loop checker | orchestrator (deterministic re-grep + targeted re-dispatch) | Yes -- once after post-pass |

| Voice pass (optional) | editorial-owner | Only when >20 edits to keep tone coherent |

| Final synthesis | cdo | Once, for the PR summary comment |

The per-scope subagent prompt that composes python-architect +

doc-writer is in assets/subagent-prompt-template.md -- the

orchestrator substitutes scope + working dir + branch and dispatches

via the task tool.

Process

1. PROBE (A9 SUPERVISED EXECUTION)
   - Check working tree: docs/src/content/docs/ exists?
   - Check working tree: packages/apm-guide/.apm/skills/apm-usage/
     exists? (Rule-4 backfill target. If missing, the audit cannot
     close Rule 4; ask maintainer before continuing.)
   - Check `.apm/docs-index.yml` reachable.
   - Verify on a working branch (not main).

2. RISK-TRIAGE (orchestrator, ~1 LLM call)
   - Read .apm/docs-index.yml only (NOT the corpus body).
   - Bucket pages by drift risk: HIGH (CLI ref, schemas, consumer
     flows), MEDIUM (producer, enterprise policy), LOW (concepts,
     contributing, troubleshooting, integrations).
   - Decide wave order: HIGH first, MEDIUM next, LOW last.

3. WAVE-PLANNER (orchestrator, deterministic)
   - Partition pages into 6-8 disjoint scopes per wave.
   - Each agent gets ~9 pages, mixed surface types.

4. WAVE EXECUTION (parallel, one subagent per scope)
   - Orchestrator dispatches one task per scope using the prompt
     template in assets/subagent-prompt-template.md.
   - Subagents read pages, extract claims, S7-verify, apply
     surgical edits, return JSON per the docs-sync panelist
     schema (mirrored at assets/panelist-return-schema.json).
   - Validate every return against the schema; reject malformed
     JSON.

5. CROSS-CORPUS POST-PASS (orchestrator, deterministic)
   - Run scripts/scan-cross-corpus-drift.sh to grep for patterns
     a per-scope agent cannot see (IA-reshuffle dead links, stale
     deprecation version targets, phantom flag references).
   - Patch residue inline.

6. ALIGNMENT LOOP (orchestrator, deterministic)
   - Re-run scripts/scan-cross-corpus-drift.sh.
   - Re-grep for claims the agents marked DRIFTED-FIXED.
   - If residue: targeted re-dispatch to the owning agent
     (bounded: max 2 redrafts per wave).

7. COMMIT + PUSH (orchestrator, single writer)
   - One commit per wave; structured message naming closed items.
   - Push to working branch.

8. PR + SUMMARY COMMENT (orchestrator)
   - If no PR exists: open one with the [pr-description-skill]
     (../pr-description-skill/SKILL.md).
   - Post per-wave summary comment: pages audited, drift caught,
     fixes applied, items deferred, alignment-loop residue.

Bundled assets

• assets/subagent-prompt-template.md -- the per-scope prompt the

orchestrator substitutes and dispatches. Composes python-architect

(S7) + doc-writer (surgical edit). Loaded once per scope.

• assets/panelist-return-schema.json -- subagent return schema,

mirrored from docs-sync. Loaded once at wave start; validated

against every return.

• scripts/scan-cross-corpus-drift.sh -- deterministic grep sweep

for cross-corpus patterns (IA dead links, stale deprecation

targets, phantom flags). Non-interactive; emits structured

matches on stdout, diagnostics on stderr. Run --help for

pattern list. Update this script after each major IA reshuffle.

Cost model

| Wave size | Pages | Subagents | LLM dispatches | Wall time |

|---:|---:|---:|---:|---:|

| Small | ~30 | 4 | ~5 | ~3 min |

| Medium (default) | ~55 | 6 | ~7 | ~5 min |

| Large | ~110 (full corpus) | 12 (two medium waves) | ~14 | ~10 min |

Compared to docs-sync (15-call flat ceiling), this skill scales as

O(waves), not O(claims), because per-agent work fits in one context

window. S7 verification dominates wall-time, not LLM cost.

Boundary (what this skill does NOT do)

• Per-PR doc-impact review -- use docs-sync.
• Single-page typo or copy edit -- direct edit is faster.
• Writing docs for a brand-new feature -- use docs-impact-architect

and doc-writer directly.

• Auto-merging or pushing without maintainer review.
• Reviewing code quality, security, or test coverage (out of scope).

Evals

See evals/:

• evals/content-evals.json -- 3 corpus snapshots with seeded drift

(stale CLI flag, dead nav link, expired deprecation target);

expected behavior is that the skill catches all three and applies

surgical fixes that ground out true on re-verification.

• evals/trigger-evals.json -- 10 should-trigger + 10 should-NOT-

trigger queries, 60/40 train/val. The val split is the ship gate

(>=0.5 should-trigger AND <0.5 should-not-trigger).

• evals/README.md -- how to run.

Provenance

This skill was extracted from a real session that audited the

microsoft/apm corpus across 3 waves (PR #1511, 2026-05-27):

112/112 pages audited, 49 surgical fixes, ~25 LLM dispatches,

~30 min wall-time. The session design artifact (genesis hand-off

packet) lives in session state, not in this bundle (maintainer-

scope, not runtime-loaded).