Docs Grounding Verifier

docs-grounding-verifier

CLAIM-LEVEL grounding verification. Adapts the RAGAS faithfulness-eval

pattern (proven in RAG literature) to docs/code instead of generated-

answers/retrieved-context. Source code is the ground truth; docs

paragraphs are the candidate text under audit.

python-architect persona

doc-writer persona

Sibling contract

This skill is a SIBLING of docs-corpus-audit and docs-sync. The

boundary is load-bearing:

| Skill | Trigger | Scope | Granularity |

| ---------------------- | -------------------------------------- | ---------------- | --------------- |

| docs-sync | PR opened/synchronized | PR diff only | Page-level |

| docs-corpus-audit | Maintainer asks for whole-corpus pass | Entire corpus | Page-level |

| docs-grounding-verifier | Verify specific pages factually | 1..N pages | CLAIM-level |

docs-corpus-audit invokes this skill in its VERIFY phase on the

highest-risk pages of each wave. docs-sync can invoke it on the

specific pages in a PR diff. The skill is also runnable standalone.

When to activate

• Maintainer says "verify <page> against the code".
• An audit wave wants per-claim grounding scores for its highest-risk pages.
• A PR review wants to confirm that prose changes are not just plausible

but actually consistent with the implementation.

• A "fact-check" or "grounding" or "drift hunt" request.

When NOT to activate

• Whole-corpus sweep with no specific page list -> use docs-corpus-audit.
• PR review with mixed code+docs diff -> use docs-sync.
• Editorial / tone review -> use editorial-owner persona directly.

Architecture (PIPELINE-of-PANELS)

PARENT
  -> [Stage 1: EXTRACT claims, fan-out PANEL]
       per page -> LLM extracts atomic factual claims as JSON
       script: scripts/extract-claims.py
  -> [Stage 2: RETRIEVE evidence, deterministic S7]
       per claim -> grep over src/ via keywords + hints
       script: scripts/retrieve-evidence.sh   (NO LLM)
  -> [Stage 3: JUDGE grounding, adversarial A7]
       per (claim, evidence) -> LLM rules GROUNDED|PARTIAL|CONTRADICTED|UNSUPPORTED
       asset: assets/judge-prompt.md
  -> [Stage 4: SYNTHESIZE]
       aggregate ungrounded -> doc-writer for fix
       re-verify after fix (A8 ALIGNMENT LOOP)

Stage 2 is the load-bearing design choice: evidence retrieval is

DETERMINISTIC (grep + AST hints), not LLM. The judge in Stage 3 can

only rule on evidence it actually receives -- it cannot hallucinate

support that the retriever did not find. This is the structural

guard against the failure mode "the LLM convinces itself the docs

match the code."

Phase 1: SCOPE

Input: list of page paths to verify (1..N). If a risk_class is

attached (e.g. "high-stakes"), prefer it; otherwise treat all as equal.

Out-of-scope:

• Pages outside docs/src/content/docs/ or

packages/apm-guide/.apm/skills/apm-usage/.

• Pages with no factual claims (pure editorial / landing). Skip

rather than force-extract.

Phase 2: EXTRACT (parallel)

For each page, dispatch ONE claim-extractor agent:

• Prompt template: scripts/extract-claims.py <page> produces the

prompt and embeds the page content.

• Returns: JSON `{"page", "claims":[{"id","text","section","keywords",

"expected_source_areas"}]}` capped at 15 claims per page.

Parallel safe; no shared state between extractors.

Phase 3: RETRIEVE (deterministic, batched)

For each claim, pipe to scripts/retrieve-evidence.sh:

• Uses keywords + expected_source_areas to grep src/.
• Returns one-line JSON: `{"claim_id","claim_text","evidence":[...],

"evidence_count"}`.

Sequential is fine (grep is fast). No LLM. Diagnostics on stderr,

data on stdout.

Phase 4: JUDGE (parallel)

For each (claim, evidence) tuple, dispatch ONE grounding-judge agent:

• Load assets/judge-prompt.md.
• Send the prompt + the tuple.
• Returns: JSON verdict per the schema in judge-prompt.md.

Batching across claims-of-one-page into a single judge call is fine

(prompt with all tuples at once). Across pages, fan out.

Phase 5: SYNTHESIZE

Aggregate verdicts. Materialize the report:

{
  "summary": {
    "pages_verified": N,
    "claims_total": N,
    "grounded": N, "partial": N, "contradicted": N, "unsupported": N,
    "grounding_rate": N/total
  },
  "actionable": [
    {"page", "claim", "verdict", "evidence_cited", "fix_suggestion"}
  ]
}

CONTRADICTED and PARTIAL are doc-writer work items. UNSUPPORTED is

split: if retrieval_fix_suggestion is plausible, retry retrieval

with the suggested keywords; if still empty, treat as CONTRADICTED.

Phase 6: ALIGNMENT LOOP (A8)

Hand actionable items to doc-writer (one subagent per page). After

edits, RE-RUN the pipeline on the same pages. The grounding_rate

must MONOTONICALLY INCREASE between iterations or the loop has

diverged -- stop and escalate to the operator.

Ship gate

• grounding_rate >= 0.9 on each verified page after the alignment loop.
• Every CONTRADICTED claim cited a specific code file:line that

disproves it -- not vague "the code doesn't say that".

• The eval-runner (see evals/) passes on the trigger evals and

the content evals before the skill is treated as production-ready.

Bundled assets

• scripts/extract-claims.py -- Stage 1 prompt builder. --help, --schema.
• scripts/retrieve-evidence.sh -- Stage 2 retriever. Deterministic. --help.
• scripts/verify-page.sh -- end-to-end orchestrator. --help.
• assets/judge-prompt.md -- Stage 3 adversarial judge prompt.
• evals/trigger-evals.json -- 20 dispatch queries (10 should, 10 shouldn't).
• evals/content-evals.json -- seeded-drift recall scenarios.
• evals/run-evals.sh -- the eval-runner that turns JSON into metrics.

Failure modes guarded against

• Hallucinated grounding: Stage 2 is deterministic; judge sees only

real evidence.

• Adversarial weakness: Stage 3 prompt defaults to SKEPTICAL.
• Page-level averaging: claim-level granularity surfaces partials.
• Bundle leakage: design notes / one-time scripts stay in session

state, never in references/.

• Phantom dependency: SKILL.md links its persona deps via relative

paths; A9 PROBE before invoking docs-corpus-audit's substrate.

• Dispatch collision with sibling skills: trigger-eval validation

split is the ship gate (must distinguish from docs-sync /

docs-corpus-audit triggers).