wanshuiyin/citation-forensics
Citation-integrity forensics: is every reference real, correctly attributed, and used in a context the cited work actually supports? Catches hallucinated references (no paper at the claimed arXiv id/DOI/venue, fabricated authors/year), metadata drift (wrong year/venue/version), and wrong-context citations (a real paper cited for a claim it never makes — or argues against). A hot zone for machine-generated papers. Decidable at L0 (text + canonical sources). Span-anchored to the evidence ledger (claims.json); the executor gathers canonical facts (DBLP / arXiv / DOI), then one FRESH cross-model thread per cited key proposes findings; reviewer != adjudicator. Emits citation-forensics.findings.json; NEVER computes the verdict. Triggers: \"citation forensics\", \"check the references\", \"hallucinated citations\", \"wrong-context citation\", \"verify references\", \"引用核对\".
npx skills add https://github.com/wanshuiyin/Anti-Autoresearch --skill citation-forensicsAudit citation integrity for: $ARGUMENTS (requires claims.json from
/evidence-ledger; reasons over its type:"citation" claims). Emit span-anchored
citation-forensics.findings.json. This skill computes no verdict.
> 🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is
> verdict-bearing input — it proposes the citation findings the deterministic
> adjudicator turns into the report. Re-firing it on a wall-clock timer adds no
> signal: its output changes only when the paper / ledger / bibliography
> changes, not with the clock, and each run spends real cross-model + lookup budget
> per cited key. Schedule the *external wait that precedes it* — bibliography
> finalized → ledger rebuilt → audit once. (Mirrors ARIS's external-cadence
> doctrine.)
> Adapted from ARIS citation-audit, re-wired onto this repo's evidence ledger and
> the reviewer≠adjudicator contract, and reframed from "audit + rewrite the bib"
> to "emit ledger-anchored findings, never touch the paper." Three layers, ported
> verbatim: existence → metadata → context. Following the repo's
> baseline-comparison-audit pattern, the executor gathers the canonical facts
> (DBLP / arXiv / DOI) as neutral evidence; a fresh cross-model reviewer judges
> existence + metadata + context over those facts. The reviewer never grades — the
> deterministic adjudicator does.
An autoresearch pipeline (or a rushed human) generates a bibliography in a separate
pass from the prose and never reconciles the two. The failure modes are not
wildly fake entries — those are easy to spot. The dangerous ones are:
authors, title, or year are fabricated. (HP-CITE-HALLUC, critical)
preprint number used after the work appeared at CVPR/ICML/NeurIPS, or vice versa),
or a v1 title silently merged with a v3 retitle. (HP-CITE-HALLUC, major)
make, or argues *against* (e.g. citing a self-refinement paper to support
"self-feedback yields correlated errors" when the cited paper argues the
opposite). (HP-CITE-CONTEXT, major)
None of this needs the code, the data, or a re-run — only the citing sentence (from
the ledger) checked against the cited work's public record (DBLP / arXiv /
publisher). That is why this layer is L0-decidable (observability-wise — no repo
or result files needed) and independently defensible. (The citing-sentence *claims* it
anchors to still enter the ledger only via the LaTeX path; a pure PDF-text ledger
yields none — see Step 0.)
**Ledger-anchored, span-verified, canonical-fact-checked-or-handed-off,
reviewer≠adjudicator, detect-only.**
(claims.json, type:"citation"), never from re-reading the PDF.
.bib metadata) and gathers a reproducible resolution snapshot of the
canonical record (DBLP / arXiv / DOI). It gathers facts, never a judgment —
whether a mismatch is fabrication, a typo, or a preprint→venue migration is the
reviewer's call.
*proposes* findings, judging existence + metadata + context over those facts.
claim_id + a verbatim span of theciting sentence (references/integrity-forensics-contract.md rules 1–2). The
bib entry, the canonical record, and any URL live in the finding's description,
never as the anchor span.
tools/adjudicate_findings.py decides(references/reviewer-independence.md Layer 2). This skill emits findings only.
source (snapshot record + URL in description). Cannot settle → `verdict_local:
needs_external_check` — never a guessed "fabricated". A false hallucination
flag is a serious error.
> The anchor is the citing sentence, not the bib line. tools/build_claim_ledger.py
> puts type:"citation" claims in the ledger whose text_span is the *sentence that
> contains \cite{key}* and whose refs are the cite keys — it does not parse
> the .bib. So a citation finding always anchors to that citing sentence (quote a
> verbatim substring of it, e.g. \cite{smith2024bar} or the surrounding phrase).
> What the .bib claims and what DBLP/arXiv actually return go in description —
> there is no ledger claim for the .bib line to anchor to. A bib key that is in the
> .bib but never \cited has no citation claim, cannot be anchored, and is
> therefore out of scope (detect-only; not flagged).
| Auditor | Question it answers | Level |
|---------|---------------------|------|
| citation-forensics (this) | Do the cited papers exist, with correct metadata, and support the claim they are used for? | L0 |
| consistency-audit | Does the paper contradict ITSELF / described method = evaluated method? | L0 |
| baseline-comparison-audit | Are the right baselines present, tuned, and is "SOTA" earned? | L0 stated / L2 verified |
| experiment-forensics | Are the reported numbers what the code actually computes? (fake GT, self-norm, phantom) | L2 |
| presentation-signals | Surface "AI-flavor" hints (auxiliary, capped at minor) | L0 |
| adversarial-case-builder | Strongest evidence-bound rejection memo (no verdict weight) | any |
Do NOT raise here (hand off instead): numeric self-contradiction / method drift →
consistency-audit; "first / SOTA / beats prior work" as an *empirical* claim →
baseline-comparison-audit (emit needs_external_check); code/result-level fraud →
experiment-forensics (needs L2); surface / AI-flavor of the prose →
presentation-signals; the rejection memo → adversarial-case-builder. **Stay in
lane:** this skill judges only whether *the cited work* exists, is described
correctly, and supports the citing sentence — not whether the citing paper's own
claim is true.
REVIEWER_MODEL = gpt-5.6-sol # different family from executor (Claude)
REVIEWER_REASONING = xhigh # always; effort never lowers reviewer quality
REVIEWER_SANDBOX = read-only # detect-only; never mutate the paper or .bib
REVIEWER_CWD = <PAPER_DIR> # so it can re-open claims.json + the .bib to confirm a span
THREAD_POLICY = ONE fresh mcp__codex__codex per CITED KEY; NEVER mcp__codex__codex-reply across keys
TAXONOMY_VERSION = 0.5 # references/hack-pattern-taxonomy.md §E
PATTERNS = HP-CITE-HALLUC (existence/metadata) | HP-CITE-CONTEXT (wrong context) | HP-CITE-RETRACTED (retracted/withdrawn)
OBS_REQUIRED = 0 for all three patterns # decidable at L0 (text + canonical / retraction sources)
FACT_GATHERING = executor, Step 2: DBLP MCP + WebSearch/WebFetch -> resolution.json (FACTS, never a verdict)
DOSSIER = <PAPER_DIR>/.aris/citation-forensics/dossier.json # Step 1 (rehashable; NOT /tmp)
RESOLUTION = <PAPER_DIR>/.aris/citation-forensics/resolution.json # Step 2 (rehashable; NOT /tmp)
FINDINGS = <PAPER_DIR>/citation-forensics.findings.json # Step 4 output
TRACE_POLICY = forensic (never silently dropped)
TRACE_DIR = <PAPER_DIR>/.aris/traces/citation-forensics/<YYYY-MM-DD>_run<NN>/
from the ledger, assembles the per-key dossier, gathers the canonical resolution
facts, validates the reviewer's spans, and writes the findings file. It never
summarizes the paper, pre-judges "this reference is fake", or leaks an opinion into
the prompt — only structured inputs (the dossier + the neutral resolution snapshot
+ the checklist). (reviewer-independence.md Layer 1.)
the executor's facts and self-reports false_positive_risk. It is the
evidence-weigher, not the judge. Like the other auditors it runs
config: {"model_reasoning_effort": "xhigh"}, sandbox: read-only; the lookups
are the executor's job (Step 2), not the reviewer's.
codex-reply is intentionally absent fromallowed-tools; never carry one key's conclusion into another (the bias guard).
Edit in allowed-tools — this skill never rewrites the.bib or any .tex (that is ARIS citation-audit's job, not a forensics tool's).
The ledger is the only structure this skill reasons over. Resolve it and read the
run's observability level L, paper_id, the set of cited keys, and the
bibliography path(s) (each Bash block is self-contained — shell state does not
persist, so re-derive paths every time):
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# $ARGUMENTS is a paper-dir OR a claims.json path:
LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json"
# Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that
# resolves to a missing claims.json must NOT silently fall back to $(pwd) — that
# could audit the wrong paper; let the NO_LEDGER check below fire instead.
[ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json"
python3 - "$LEDGER" <<'PY'
import json, sys, os, glob
p = sys.argv[1]
if not os.path.isfile(p):
sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST "
"(it writes artifact_manifest.json + claims.json).")
d = json.load(open(p, encoding="utf-8"))
paper_dir = os.path.dirname(os.path.abspath(p)) or "."
cites = [c for c in d.get("claims", []) if c.get("type") == "citation"]
keys = sorted({k for c in cites for k in (c.get("refs") or [])})
bibs = glob.glob(os.path.join(paper_dir, "**", "*.bib"), recursive=True)
print("LEDGER =", os.path.abspath(p))
print("PAPER_DIR =", paper_dir)
print("PAPER_ID =", d.get("paper_id", "?"))
print("RUN_LEVEL_L =", d.get("observability_level", 0))
print("CITE_CLAIMS =", len(cites))
print("CITE_KEYS =", len(keys), keys)
print("BIB_FILES =", bibs or "NONE (metadata layer will be partial)")
PY
Then create the run directories (start NN at 01, bump if it exists):
PAPER_DIR="<PAPER_DIR printed above>"
DATE=$(date -u +%Y-%m-%d)
TBASE="$PAPER_DIR/.aris/traces/citation-forensics"
NN=01; while [ -d "$TBASE/${DATE}_run$NN" ]; do NN=$(printf '%02d' $((10#$NN + 1))); done
RUN="${DATE}_run$NN"
mkdir -p "$TBASE/$RUN" "$PAPER_DIR/.aris/citation-forensics"
echo "RUN = $RUN"
echo "TRACE_DIR = $TBASE/$RUN"
echo "DOSSIER = $PAPER_DIR/.aris/citation-forensics/dossier.json"
echo "RESOLUTION = $PAPER_DIR/.aris/citation-forensics/resolution.json"
echo "FINDINGS = $PAPER_DIR/citation-forensics.findings.json"
Failure / edge handling.
NO_LEDGER → stop; tell the user to run /evidence-ledger first. This skillnever re-reads the raw PDF and invents structure (contract rule 1).
CITE_CLAIMS = 0 → the ledger has no type:"citation" claims. This is expected onan L0 PDF-text-only run: extract_from_text in tools/build_claim_ledger.py
extracts numbers + scope language but not citations (only the LaTeX path
extract_from_latex emits citation claims). Skip to Step 4, write
citation-forensics.findings.json = [], and note: *"no citation claims in ledger
— re-run /evidence-ledger with the LaTeX source, or the paper has no \cite."*
Silent skip is forbidden — the file must exist.
BIB_FILES = NONE (or only a .bbl) → continue. The citing sentences still anchorthe audit, and existence + context are checkable from the cite key + the sentence +
canonical sources. The metadata layer is then partial (no claimed
authors/year/venue to compare) — say so in the dossier and tell the reviewer the
bib entry is unavailable for that key. That is honest, not a failure.
Carry the absolute LEDGER, PAPER_DIR, PAPER_ID, L, RUN, TRACE_DIR,
DOSSIER, and RESOLUTION paths into the steps below.
Group every type:"citation" claim by cite key (a \cite{a,b} claim contributes to
both a and b) and, best-effort, attach the claimed .bib metadata for each
key by balanced-brace extraction from any *.bib under the paper dir. Pure assembly —
no web, no opinion. Stage it under .aris/ so the verifier can rehash it (never
/tmp):
LEDGER="<abs LEDGER>"; PAPER_DIR="<abs PAPER_DIR>"
OUT="$PAPER_DIR/.aris/citation-forensics/dossier.json"
python3 - "$LEDGER" "$PAPER_DIR" "$OUT" <<'PY'
import json, os, re, sys, glob
ledger_path, paper_dir, out = sys.argv[1], sys.argv[2], sys.argv[3]
L = json.load(open(ledger_path, encoding="utf-8"))
cites = [c for c in L.get("claims", []) if c.get("type") == "citation"]
# 1) group citing spans by cite key
keys = {}
for c in cites:
for k in (c.get("refs") or []):
keys.setdefault(k, []).append({
"claim_id": c["claim_id"],
"span": c.get("text_span", ""),
"location": c.get("location", {}),
})
# 2) best-effort: parse claimed metadata per key from any .bib (balanced-brace)
bibtext, bibfiles = "", []
for p in glob.glob(os.path.join(paper_dir, "**", "*.bib"), recursive=True):
try: # best-effort: skip an unreadable .bib
txt = open(p, encoding="utf-8", errors="replace").read()
except OSError:
continue
bibfiles.append(p)
bibtext += txt + "\n"
def bib_entry(key): # @type{key, ... } with brace matching
m = re.search(r"@\w+\s*\{\s*" + re.escape(key) + r"\s*,", bibtext)
if not m:
return None
i, depth = m.end(), 1
while i < len(bibtext) and depth:
depth += (bibtext[i] == "{") - (bibtext[i] == "}")
i += 1
return bibtext[m.start():i][:1200]
entries = [{"key": k, "n_cites": len(keys[k]),
"bib_entry": bib_entry(k), "citing": keys[k]} for k in sorted(keys)]
os.makedirs(os.path.dirname(out), exist_ok=True)
json.dump({"n_keys": len(entries), "bib_files": bibfiles, "entries": entries},
open(out, "w", encoding="utf-8"), indent=2, ensure_ascii=False)
missing = [e["key"] for e in entries if not e["bib_entry"]]
print(f"dossier: {len(entries)} cited keys from {len(cites)} citation claims; "
f"bib_files={bibfiles or 'NONE'}; no-bib-entry keys={missing} -> {out}")
PY
Gate — sanity. The printed cited keys count should match what you expect for
the paper. 0 keys with CITE_CLAIMS > 0 (rare) means the refs were empty —
spot-read a citation claim in claims.json. Keys flagged no-bib-entry (no .bib,
or the key missing from it) are audited existence + context only — the metadata
layer is partial because there is no claimed authors/year/venue to compare. That is
honest; the reviewer infers the intended work from the cite key + the citing
sentences.
For each cited key, gather a reproducible snapshot of what the canonical record
actually is — so a finding is reproducible against staged facts even if DBLP changes
later, and so the reviewer has the evidence it needs to judge. **This step records
facts, never a verdict**: whether a mismatch is fabrication, a typo, or a
preprint→venue migration is the reviewer's call in Step 3. (Same executor-gathers /
reviewer-weighs division as baseline-comparison-audit Step 2.)
For each key, seed the queries from the dossier's bib title / authors / venue. If
bib_entry is null (no .bib, or only a .bbl), seed instead from the cite-key
tokens (author / year / keyword) plus distinctive words from that key's citing
sentence(s) in the dossier — existence + context stay checkable; the metadata layer is
then partial (nothing claimed to compare against), which is honest, not a failure:
mcp__mcp-dblp__fuzzy_title_search with title=<bib title>,similarity_threshold=0.7 (lower to ~0.5 only if no hit). Returns canonical
{title, authors, venue, year, doi, ee, url} for the top matches.
mcp__mcp-dblp__search withquery="<first-author surname> and <distinctive title word>" (the query supports
only and/or, no parentheses). Confirms author↔title↔year coherence;
venue_filter / year_from / year_to narrow it.
mcp__mcp-dblp__get_venue_info with venue_name=<bib venue> toconfirm the venue is real and of the claimed type (Conference or Workshop /
Journal / Repository).
WebFetch the arXiv abstract page built fromthe bib eprint/arxiv id (https://arxiv.org/abs/<id>) and capture the returned
title + authors and the abstract text (the abstract is what lets the reviewer
judge *context*); WebFetch https://doi.org/<doi> to confirm the DOI resolves to
this work. Use WebSearch as a fallback for very recent papers (< ~2 weeks) not yet
in DBLP.
Write one neutral record per key to the staged snapshot (use Write), and copy it
into the trace dir (Step 5):
// .aris/citation-forensics/resolution.json
{
"smith2024bar": {
"dblp_fuzzy_top": [{"title": "...", "authors": ["..."], "venue": "...",
"year": 2024, "doi": "...", "url": "https://dblp.org/..."}],
"venue_info": {"venue": "...", "type": "Conference or Workshop"},
"arxiv": {"id": "2401.01234", "resolves": true, "title_returned": "...",
"abstract": "... fetched abstract text, for the context layer ..."},
"doi": {"doi": "10.1145/...", "resolves": true},
"notes": "facts only — not a verdict"
}
}
Failure handling (non-blocking). If a DBLP MCP call errors, or web access is
unavailable, write {"<key>": {"status": "unavailable", "reason": "..."}} for that
key and continue — the Step 3 reviewer then falls back to
verdict_local: needs_external_check for that key rather than guessing. Note the skip
in the trace. Never hand-author a "fabricated" conclusion from a failed lookup.
Existence and metadata are mechanical fact-checks against the snapshot; *context*
needs judgment. Read dossier.json + resolution.json. For each entry, issue
one fresh mcp__codex__codex call (key order; tag 001, 002, …), `cwd =
PAPER_DIR so the reviewer can re-open claims.json / the .bib` to confirm a span.
Inject that one key's dossier record and resolution record in place of the two
bracketed blocks. Send EXACTLY this — it is the reviewer's complete instruction set;
add no commentary of your own about the paper:
mcp__codex__codex:
model: gpt-5.6-sol
config: {"model_reasoning_effort": "xhigh"}
sandbox: read-only
cwd: <absolute PAPER_DIR from Step 0>
prompt: |
You are an integrity-forensics reviewer auditing ONE bibliographic citation key
of a research paper. You judge three things and NOTHING ELSE: does the cited
paper EXIST, is its METADATA correct, and does it actually SUPPORT the claim each
citing sentence uses it for. You do NOT judge whether the citing paper's own
result is true, and you NEVER accuse anyone of misconduct. You audit integrity,
not authorship.
Judge existence + metadata from the RESOLUTION SNAPSHOT below — canonical FACTS
(DBLP / arXiv / DOI) gathered by the executor. It is evidence, NOT a verdict:
cross-check the .bib's self-report against it. If the snapshot is "unavailable",
or does not settle a key either way, say so and set verdict_local
"needs_external_check"; do NOT guess existence, and NEVER fabricate the cited
paper's contents. Judge CONTEXT only from the snapshot's abstract/title (the
fetched record — never from memory of the cited paper); if the snapshot lacks
enough of the cited paper's content to decide, set "needs_external_check" and tell
the human which section of the cited paper to read.
## THE ENTRY UNDER AUDIT (from the evidence ledger; claims.json is in your cwd —
## you MAY re-open it to confirm a span is real, but you may NOT introduce a
## claim_id that is not listed here):
[DOSSIER RECORD FOR THIS KEY — dossier.json["entries"][i]:
{key, n_cites, bib_entry (claimed metadata; null if no .bib), citing:[{claim_id, span, location}]}]
## CANONICAL RESOLUTION gathered by the executor (FACTS; may be empty/unavailable):
[RESOLUTION RECORD FOR THIS KEY — resolution.json["<key>"]]
## WHAT TO CHECK (run all three layers for THIS key; one finding per concrete discrepancy)
(A) EXISTENCE — does a paper exist at the claimed arXiv id / DOI / venue with the
claimed title (or, if bib_entry is null, the work implied by the key + the
citing sentences)? [HP-CITE-HALLUC]
* No record resolves anywhere; authors/title/year fabricated -> severity critical.
* The id is a TYPO but the paper plainly exists at the corrected id -> severity
minor (a FIX, not a fabrication), false_positive_risk medium.
* Very recent work (<~2 weeks) not yet indexed / snapshot unavailable -> set
needs_external_check, severity info; do NOT call it fabricated.
* IDENTIFIER-HIJACKING: the arXiv id / DOI RESOLVES, but the resolved record's
title and/or authors in the snapshot do NOT match the citation -> the existence
check alone is NOT enough; the load-bearing test is the title/author MATCH against
the resolved record. Mismatch -> severity critical (resolves to an unrelated work).
Keep wording neutral (state the metadata mismatch; do NOT use "deception"). FP: the
id resolves to a newer VERSION of the same work.
* PLACEHOLDER CITATION: the bib/citing span is a leftover stub ("[ref?]", "[CITATION]",
"\cite{XXX}", "TODO: cite", "?") never replaced -> severity major if load-bearing,
else minor; false_positive_risk medium (a clearly-marked draft).
(B) METADATA — real paper, but wrong year / wrong venue (arXiv number used though it
appeared at NeurIPS) / wrong-or-missing authors / v1<->v3 retitle. [HP-CITE-HALLUC]
* severity major. A preprint->published migration (arXiv 2023 -> CVPR 2024) is
a COMMON legitimate case: severity minor, false_positive_risk high.
(C) CONTEXT — for EACH citing sentence: does the cited paper actually establish what
the sentence uses it for? Flag a real paper cited to support a claim it does
NOT make, or argues AGAINST. [HP-CITE-CONTEXT]
* severity major. In `description`, state what the cited paper actually
establishes vs how the sentence uses it. false_positive_risk high if a
"see also / contrast with / unlike" reading is plausible, or the load-bearing
claim is the citing paper's OWN contribution.
* Also flag SEMANTIC-HALLUCINATION: a real reference attached to a finding the
cited paper does not contain, or an attribution of a specific claim/number the
cited work never makes (paper real; attributed content not). Judge ONLY from the
snapshot's abstract/title; if insufficient, set needs_external_check.
* AUXILIARY INTENT LABEL: optionally add `intent` ∈ {support|contrast|mention} with
a confidence in `description`. It only SHARPENS candidates (a contrast/mention
reading is the common FP; only a `support` cite whose work doesn't support the
claim is dangerous) — NEVER a verdict on its own; do not raise severity on it.
(D) RETRACTION — does the RESOLUTION SNAPSHOT report the cited work as RETRACTED or
withdrawn (Crossref / Retraction-Watch open metadata)? If so, and a citing sentence
RELIES on it to support a claim with no note of the retraction, flag it. [HP-CITE-RETRACTED]
* severity major when the retracted reference is load-bearing.
* If the sentence cites it EXPRESSLY to discuss the retraction, or the retraction
POST-DATES submission -> severity info/minor, false_positive_risk high (honest use).
* An "expression of concern" / erratum / correction is NOT a full retraction -> do
not flag as retracted. If the snapshot has no retraction record -> do NOT infer one.
* Retraction is a FACT about the cited work (source + date in `description`), never an
accusation against the citing authors.
## HARD RULES (a finding that breaks any of these is worthless)
1. ANCHOR. Every finding above severity "info" MUST carry >=1 evidence entry
{claim_id, span}, where claim_id is ONE OF the citing claim_ids above and span
is a VERBATIM substring of THAT citing sentence (no paraphrase — e.g. quote
"\cite{<key>}" or a phrase of the sentence). Put the .bib metadata, the
canonical record, and the URL in `description`, NOT in `span` — there is no
ledger claim for the .bib line or a DBLP URL to anchor to. If you cannot quote
a verbatim substring of a listed sentence, keep the finding at "info".
2. FACT OR HAND OFF — never guess from memory. Settle existence/metadata against
the snapshot (cite the record/URL in `description`). If the snapshot cannot
settle it, emit verdict_local "needs_external_check", requires_external_check
true, severity "info", and say what to look up. A false "this citation is
fabricated" is a serious error.
3. OBSERVABILITY. Set observability_level_required = 0 on every finding: citation
existence, metadata, and wrong-context are decidable from text + the public
record (you do not need the repo or result files).
4. DISCREPANCY, NOT ACCUSATION. `description` and `recommended_reviewer_action`
say what a human should CHECK or ASK. Never "reject", "fabricated by the
authors", "the authors faked X".
5. HONEST FP RISK — set it truthfully. COMMON false positives: typo'd-but-
resolvable id (a FIX); preprint->published migration; "see also / contrast"
framing; 6+-author "and others" truncation; the claim being the citing paper's
own contribution.
6. pattern_id is exactly one of: "HP-CITE-HALLUC", "HP-CITE-CONTEXT", "HP-CITE-RETRACTED".
## OUTPUT — a single JSON array, and NOTHING ELSE (no prose, no code fence). Each
element conforms to schemas/finding.schema.json:
{
"finding_id": "F001",
"skill": "citation-forensics",
"pattern_id": "HP-CITE-HALLUC | HP-CITE-CONTEXT | HP-CITE-RETRACTED",
"title": "short, neutral",
"description": "the discrepancy: what the .bib claims, what the canonical "
"source returns (+URL), and/or what the cited paper actually "
"establishes vs how it is used",
"severity": "critical|major|minor|info",
"observability_level_required": 0,
"evidence": [{"claim_id": "C0xx", "span": "verbatim substring of the citing sentence",
"location": {"file": "...", "section": "..."}}],
"verdict_local": "fail|warn|clean|needs_external_check",
"requires_external_check": false,
"false_positive_risk": "low|medium|high",
"alternative_explanation_checked": "<criticals ONLY — the benign readings you explicitly ruled out: rounding/display precision, unit or metric convention, statistical reporting convention, scope difference. Anchoring proves the text exists, NOT that your interpretation is right; a critical without this field demotes to major>",
"recommended_reviewer_action": "what to CHECK or ASK — never 'reject'"
}
If this key is clean on all three layers, emit []. An empty array is a valid,
honest result.
Immediately after each call returns, persist the trace (Step 5) before the
next key's call: write the FULL raw reply with Write to
<TRACE_DIR>/<NNN>-<key>.response.md, the exact prompt sent to
<NNN>-<key>.request.json, and <NNN>-<key>.meta.json
({model, reasoning, sandbox, thread_id}). The .response.md files are the immutable
input to Step 4.
Reference output — what good findings look like (the shape the validator keeps):
// HP-CITE-CONTEXT (major) — the dangerous case: real paper, wrong claim
{
"finding_id": "F001", "skill": "citation-forensics", "pattern_id": "HP-CITE-CONTEXT",
"title": "Self-refinement work cited for the opposite of what it shows",
"description": "The sentence cites \\cite{madaan2023selfrefine} to support that self-feedback yields correlated errors. The cited paper (Self-Refine, NeurIPS 2023; DBLP https://dblp.org/... confirms title/venue/authors; abstract in resolution.json) demonstrates that iterative self-feedback IMPROVES outputs — it does not establish correlated self-feedback errors. The citation supports a claim the cited work does not make.",
"severity": "major", "observability_level_required": 0,
"evidence": [{"claim_id": "C042", "span": "\\cite{madaan2023selfrefine}",
"location": {"file": "sections/2.overview.tex", "section": "method"}}],
"verdict_local": "fail", "false_positive_risk": "medium",
"recommended_reviewer_action": "Re-read Self-Refine §1; ask which result supports 'correlated errors', or re-attribute the claim to a paper that establishes it."
}
// HP-CITE-HALLUC (critical) — no canonical record resolves
{
"finding_id": "F002", "skill": "citation-forensics", "pattern_id": "HP-CITE-HALLUC",
"title": "No paper resolves at the claimed arXiv id / title",
"description": "\\cite{kim2024neuralcompress} claims arXiv:2407.99999 with authors {Kim, Park}. resolution.json: arXiv:2407.99999 does not resolve; DBLP fuzzy+boolean return no paper with this title and author set. The reference appears to have no canonical record.",
"severity": "critical", "observability_level_required": 0,
"evidence": [{"claim_id": "C051", "span": "\\cite{kim2024neuralcompress}",
"location": {"file": "sections/6.related.tex", "section": "related"}}],
"verdict_local": "fail", "false_positive_risk": "low",
"recommended_reviewer_action": "Ask the authors for a resolvable arXiv id / DOI; verify it exists before relying on the surrounding claim."
}
> Contrast (NOT critical): a typo'd-but-resolvable id (2407.9999 → real 2407.09999)
> is a FIX → severity: minor/info, false_positive_risk: low,
> recommended_reviewer_action: "correct the arXiv id". A preprint→venue migration
> (arXiv 2023 → CVPR 2024) is metadata drift at most (major only if the wrong record
> is load-bearing), never critical.
Budget / fan-out. Default: one fresh thread per cited key (the bias guard).
For a long bibliography you MAY group a handful of keys into one fresh
mcp__codex__codex call only if the prompt keeps each key in its own clearly
labeled block (with its own dossier + resolution records) and emits a flat array of
findings each anchored to its key's citing claim — but never use codex-reply to
carry one key's conclusion into another.
Failure handling.
a fresh mcp__codex__codex call (gpt-5.6-sol, xhigh) — never codex-reply.
outermost [...]; if there is none, re-ask that one key with "Output ONLY the JSON
array, nothing else." Never hand-author findings on the reviewer's behalf.
info /needs_external_check for existence/metadata. Do not upgrade them.
Coverage gate (fail closed). First confirm Step 3 actually ran: count
ls "$TRACE_DIR"/*.response.md | wc -l and require one .response.md per cited key
(fewer only if you deliberately batched keys into a shared response — then each
batched key must still appear in some .response.md). **Zero response files while
CITE_CLAIMS > 0 means Step 3 never ran — STOP and run it; do NOT emit a clean [].**
The *only* legitimate empty-findings path is the CITE_CLAIMS = 0 short-circuit from
Step 0.
Enforce the ANCHOR gate before keeping anything — it mirrors the verbatim-span
rule tools/adjudicate_findings.py re-applies (so nothing you keep is silently
rejected downstream), plus one citation-specific check: a citation finding must
rest on a type:"citation" (citing-sentence) claim, not a stray number/scope claim.
This reads every *.response.md in the trace dir, applies schema hygiene, merges, and
renumbers. The span must be a verbatim, whitespace-normalized substring of the
cited claim (span in claim, never claim in span — appending hallucinated text to a
real sentence must fail):
LEDGER="<abs LEDGER>"; TRACE_DIR="<abs TRACE_DIR>"
OUT="<abs PAPER_DIR>/citation-forensics.findings.json"
python3 - "$LEDGER" "$TRACE_DIR" "$OUT" <<'PY'
import json, re, sys, glob, os
ledger_path, trace_dir, out_path = sys.argv[1], sys.argv[2], sys.argv[3]
def nw(s): # mirror adjudicator _norm_ws (whitespace only)
return " ".join((s or "").split())
ALLOWED = {"HP-CITE-HALLUC", "HP-CITE-CONTEXT", "HP-CITE-RETRACTED"} # the ONLY patterns this skill emits
OBS = {"HP-CITE-HALLUC": 0, "HP-CITE-CONTEXT": 0, "HP-CITE-RETRACTED": 0} # all decidable at L0 (taxonomy 0.5 §E)
SEV = {"critical", "major", "minor", "info"}
VL = {"fail", "warn", "clean", "needs_external_check"}
FPR = {"low", "medium", "high"}
ABOVE_INFO = {"critical", "major", "minor"}
L = json.load(open(ledger_path, encoding="utf-8"))
claims = {c["claim_id"]: c for c in L.get("claims", []) if c.get("claim_id")}
kept, dropped, demoted, n = [], 0, 0, 0
files = sorted(glob.glob(os.path.join(trace_dir, "*.response.md")))
for fp in files:
raw = open(fp, encoding="utf-8", errors="replace").read()
m = re.search(r"\[.*\]", raw, re.S) # tolerate prose / code-fence wrapping
if not m:
print(f" note: no JSON array in {os.path.basename(fp)} (treated as []) -- if this persists after the one retry, record citation-forensics=review_unavailable in the run's coverage.json: a partial key sweep must not read as completed"); continue
try:
arr = json.loads(m.group(0))
except Exception as e:
print(f" WARN: unparseable JSON in {os.path.basename(fp)}: {e} (treated as [])"); continue
if isinstance(arr, dict): # tolerate {"findings": [...]}
arr = arr.get("findings", [])
for f in arr:
if not isinstance(f, dict):
dropped += 1; continue
pid = f.get("pattern_id")
if pid not in ALLOWED: # stray HP-* / surface signal -> not this skill's to emit
dropped += 1; continue
n += 1
f["finding_id"] = f"F{n:03d}" # FORCE renumber — per-key arrays each start at F001
f["skill"] = "citation-forensics" # force-correct the skill tag
# enum hygiene: any illegal value -> safe default
if f.get("severity") not in SEV: f["severity"] = "info"
if f.get("verdict_local") not in VL: f["verdict_local"] = "warn"
if f.get("false_positive_risk") not in FPR: f["false_positive_risk"] = "high"
if f["verdict_local"] == "needs_external_check":
f["requires_external_check"] = True
# ANCHOR gate: span is a verbatim ws-normalized SUBSTRING of its cited claim,
# AND that claim must be a type:"citation" (citing-sentence) claim.
anchored, has_cite_anchor = [], False
for ev in (f.get("evidence") or []):
cid, span = ev.get("claim_id"), nw(ev.get("span", ""))
c = claims.get(cid)
if c and span and span in nw(c.get("text_span", "")): # span IN claim, not claim IN span
ev.setdefault("location", c.get("location", {})) # enrich for human navigation
ev.setdefault("artifact_hash", c.get("evidence_anchor", ""))
anchored.append(ev)
if c.get("type") == "citation":
has_cite_anchor = True
f["evidence"] = anchored
if f["severity"] in ABOVE_INFO and not (anchored and has_cite_anchor):
f["severity"] = "info"; demoted += 1 # unanchored / non-citation anchor -> info
# observability fallback: a real int 0-3 (JSON bool is an int subclass -> reject)
olr = f.get("observability_level_required")
if isinstance(olr, bool) or not isinstance(olr, int) or not (0 <= olr <= 3):
f["observability_level_required"] = OBS.get(pid, 0)
# cross-model provenance (reviewer-independence: a proposal, not a verdict)
import os as _aris_os
RESOLVED_MODEL = _aris_os.environ["ARIS_RESOLVED_MODEL"] # exported by the executor from the call that ACTUALLY ran
RESOLVED_REASONING = _aris_os.environ["ARIS_RESOLVED_REASONING"] # (fail LOUD if unset — never stamp a target default)
f["reviewer"] = {"model": RESOLVED_MODEL, "reasoning": RESOLVED_REASONING, "deterministic": False}
kept.append(f)
json.dump(kept, open(out_path, "w", encoding="utf-8"), indent=2, ensure_ascii=False)
print(f"validated {len(kept)} citation findings from {len(files)} entry response(s) "
f"({demoted} demoted to info for unanchored/non-citation span, "
f"{dropped} dropped: non-citation-pattern/malformed) -> {out_path}")
PY
Scope of this gate: anchoring + schema hygiene — verbatim-span anchoring (the span
must be a substring of a type:"citation" claim), enum coercion, non-citation-pattern
rejection, observability fallback, and cross-model provenance — so every kept finding
is well-formed and honestly anchored. It does not compute the verdict, the FP-risk
cap, or the observability *downgrade* against the run level; those belong to
tools/adjudicate_findings.py, the single decider.
Always emit. Write citation-forensics.findings.json even when it is [] (no
citation claims, or every key clean) — silent skip is forbidden; the orchestrator
and the standalone adjudicate command both expect the file at a predictable path. If
the CITE_CLAIMS = 0 short-circuit from Step 0 fired, simply
echo '[]' > "<PAPER_DIR>/citation-forensics.findings.json" here. This skill has no
deterministic-tool pass — there is no check_citations.py; existence/metadata require
the live public record, which the executor fetches in Step 2.
Failure handling. A bad response file is reported and skipped (treated as []) so
one malformed entry never aborts the merge — re-run that key's Step 3 with the
strict-JSON reminder if you want its findings back. A finding that loses all evidence
is kept as info, never silently dropped (the forensic record stays).
Save every reviewer call under
<PAPER_DIR>/.aris/traces/citation-forensics/<YYYY-MM-DD>_run<NN>/. This repo ships
no save_trace.sh, so write the files directly, one set per cited key (mirror ARIS
review-tracing — fresh thread per key, full reply preserved; the .response.md is also
the forensic record of what the reviewer concluded over the gathered facts):
.aris/traces/citation-forensics/<date>_run<NN>/
run.meta.json # {skill, paper_id, run_level_L, n_keys, generated_at}
resolution.json # the executor's Step 2 canonical-facts snapshot (copy)
001-<key>.request.json # the EXACT prompt sent for this key (dossier+resolution+checklist; no paper digest)
001-<key>.response.md # the FULL raw reviewer reply (input to Step 4)
001-<key>.meta.json # {model:"gpt-5.6-sol", reasoning:"xhigh", sandbox:"read-only", thread_id}
002-<key>.request.json ...
TRACE_DIR="<abs TRACE_DIR>"
python3 - "$TRACE_DIR" "<PAPER_ID>" "<L>" "<N_KEYS>" <<'PY'
import json, sys, datetime
d, pid, L, n = sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4])
json.dump({"skill": "citation-forensics", "paper_id": pid, "run_level_L": L,
"n_keys": n,
"generated_at": datetime.datetime.now(datetime.timezone.utc)
.isoformat().replace("+00:00", "Z")},
open(d + "/run.meta.json", "w", encoding="utf-8"), indent=2)
print("wrote", d + "/run.meta.json")
PY
Each request.json is the independence audit trail — it must show the executor sent
only the dossier record + the neutral resolution facts + the checklist, never a
hunch like "this looks AI-generated".
Within /anti-autoresearch, stop here: the orchestrator globs every
*.findings.json, runs the adjudicator, and emits REPORT.md + report.json. When
running this skill alone, you may produce the report yourself — --ledger is
required (it re-verifies each finding quotes a real ledger span; without it every
above-info finding fails closed to info):
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
LEDGER="<abs LEDGER>"; D="$(dirname "$LEDGER")"
python3 "$ROOT/tools/adjudicate_findings.py" \
--findings "$D/citation-forensics.findings.json" \
--ledger "$LEDGER" \
--paper-id "<PAPER_ID>" --observability-level <L> --taxonomy-version 0.5 \
--out "$D/report.json" --md "$D/REPORT.md"
The adjudicator applies, in order: ANCHOR → OBSERVABILITY → FP-RISK → MEMO → SURFACE
gates, then computes overall_verdict ∈ {CLEAN_GIVEN_EVIDENCE, SOFT_FLAGS,
HARD_FLAGS}. A span-anchored, low-FP HP-CITE-HALLUC critical (a reference with no
canonical record) decidable at L0 → HARD_FLAGS; a high-FP context flag survives at
most as minor → SOFT_FLAGS. No model is in the final decision.
This skill always writes, into the ledger's directory:
citation-forensics.findings.json — Step 4, a JSON array conforming toschemas/finding.schema.json (or [] when there are no citation claims or all are
clean — written regardless; silent skip is forbidden). Each above-info finding
carries evidence[].claim_id + a verbatim span of the citing sentence,
pattern_id ∈ {HP-CITE-HALLUC, HP-CITE-CONTEXT, HP-CITE-RETRACTED}, and
observability_level_required: 0.
.aris/citation-forensics/dossier.json + resolution.json — Steps 1–2, therehashable staged inputs (per-key citing spans + claimed bib metadata + canonical
facts).
.aris/traces/citation-forensics/<date>_run<NN>/ — Step 5, the raw per-key reviewercalls (run.meta.json + resolution.json + per-key
request.json / response.md / meta.json).
It writes no verdict and no report of its own — report.json / REPORT.md come
only from tools/adjudicate_findings.py (Step 6 / the orchestrator).
substring of a type:"citation" claim's text_span; the .bib line, the
canonical record, and any URL go in description (the ledger has no bib claim to
anchor to). span in claim, whitespace-normalized — never claim in span.
the executor's Step 2 (resolution.json); the reviewer judges existence + metadata
+ context over those facts. Same division as baseline-comparison-audit.
canonical record (cited in description). Cannot settle → needs_external_check,
never a guessed "fabricated". A false hallucination flag is a serious error.
.bib entry's metadata is the *claim under audit*, notground truth — verify it against DBLP / arXiv / the publisher.
more dangerous than a typo'd author — and is the headline value of this skill.
observability_level_required: 0 for all three patterns;they are decidable from text + canonical sources, no repo or results. This skill
never asserts code/result-level fraud (that needs L2 → experiment-forensics).
(gpt-5.6-sol xhigh); each key is a new mcp__codex__codex thread; codex-reply is
never used across keys.
adjudicate_findings.pydecides the verdict. This skill emits findings only.
reject; the tool audits citation integrity, not authorship — preprint migrations,
"see also" framing, and and others truncation are honest FPs, not fraud.
to; detect-only, not flagged unless explicitly requested (no budget spent on them).
.bib or any .tex (no Edit in allowed-tools;reviewer sandbox read-only). Rewriting citations is ARIS citation-audit's job.
verdict.
claims.json yet → run /evidence-ledger first; this skill never inventsstructure from the raw PDF.
extractor does not pull citations) → there is nothing to anchor to. Re-run
/evidence-ledger with the LaTeX source so citation claims enter the ledger;
otherwise emit [] and say why.
/consistency-audit./baseline-comparison-audit (+ hand off via needs_external_check); this skill
judges only whether *the cited work* supports the sentence, not whether the citing
paper's own result is true.
→ /experiment-forensics at L2.
hints live in /presentation-signals (auxiliary, capped at minor); this repo is
not an AI-text classifier.
.bib/.tex auto-fixed → that is ARIS citation-audit(co-author mode). This skill is detect-only.
/loop / /schedule / CronCreate this skill; re-fire onlywhen the paper / ledger / bibliography changes (see the fence at the top).
Policy: forensic — never silently skip. The full per-key trace layout
(run.meta.json + the resolution.json snapshot + per-key
NNN-<key>.request.json / .response.md / .meta.json) is defined in Step 5;
write each .response.md during Step 3, immediately after its reviewer call. Each
request.json must hold only the dossier record + neutral resolution facts + the
checklist (the reviewer-independence audit trail) — never a hunch.
Take wanshuiyin/citation-forensics from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.