Research Proposal

Research Proposal Loop

The artifact is a research proposal; the feedback signal is ScholarEval (literature-grounded

Soundness + Contribution) turned into a 0-100 grade by a Judge against the fixed rubric.md.

Each iteration evaluates → grades → revises until the grade clears <pass_threshold> or the

budget runs out.

North star. Clearing the threshold is the stopping condition, not the goal. The goal is the

strongest, most novel, genuinely publishable version the proposal can *honestly* become — every

revision should ask "does this make the work more significant and more novel?", not just "does this

patch a flaw?". This is grounded ambition: the lift comes from better-justified methods, a

sharper-but-defensible novelty claim, and stronger baselines, all backed by real retrieved evidence.

Overclaiming *lowers* the grade (evidence gate + Contribution axis); it never raises it.

The cast (all in this folder):

• roles/ScholarEval.md — the two-module literature-grounded evaluator; emits scholareval.json.
• roles/Judge.md — grades the feedback → 0-100 + ranked fixes; emits verdict.json (decides pass).
• roles/Reviser.md — rewrites the proposal to address the fixes (guards the research intent).
• rubrics/rubric.md — the fixed grading rubric (the Judge never edits it).
• schemas/scholareval.schema.json, schemas/verdict.schema.json — the two validated outputs.

Spawn-or-degrade. On Claude Code, spawn ScholarEval / Reviser as real Agent subagents;

otherwise adopt each role inline. You are the orchestrator and the Judge.

When to use

Use when a written proposal exists and the user wants it pushed past a quality bar with

literature-grounded critique. Default: run the full evaluate→grade→revise loop below. Escape hatch: if

the user only wants the critique (no rewriting), run one ScholarEval + Judge pass and stop. Not for

writing a proposal from a blank page, and not for a standalone literature survey.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it and skip to

the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for

each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text

prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm every value,

<intent>, and the live/degraded literature tier before creating any other files.

| binding | meaning | default | how to infer |

|---|---|---|---|

| <proposal_path> | file with problem + methodology + planned experiments | — | scan for a likely .md/.txt/.pdf; if only prose is pasted, save it to <sandbox_root>/iter1/proposal.md |

| <intent> | core research question + headline contribution, 2-3 sentences — frozen; the Reviser may never change it | — | read the proposal, extract it, have the user confirm |

| <pass_threshold> | grade (0-100) the proposal must reach to pass | 75 | a solid, well-grounded proposal without demanding perfection |

| <budget> | max iterations | 6 | — |

| <patience> | stop after this many consecutive no-improvement iterations | 2 | — |

| <eval_scale> | how much literature ScholarEval pulls per iteration (low/medium/high, see below) | medium | — |

| grade_weights | soundness · contribution · evidence_quality (must sum to 1; frozen for the run) | 0.45 · 0.35 · 0.20 | rubric.md; recommend default |

| <sandbox_root> | where snapshots, ledger, and lit cache live | ./sandbox | — |

Evaluation depth dial (<eval_scale> caps per iteration — methods · dimensions examined ·

queries each · papers read full-text):

| preset | methods · dims | queries each | fulltext reads | effort |

|---|---|---|---|---|

| low | 2 · 2 | 1 | 0 (snippet/abstract only) | low |

| medium *(recommended)* | 4 · 3 | 2 | 2 | medium |

| high | 6 · 5 | 3 | 5 | high |

Literature toolchain. Paper search goes through the sibling literature-search skill —

resolve <lit_skill_dir> (it installs as a sibling, e.g. ~/.claude/skills/literature-search/),

<lit_py> = python3, and <lit> = <lit_skill_dir>/tools/lit_search.py; append

--cache-dir <sandbox_root>/literature/.cache after a subcommand to reuse the cache. Confirm

<lit> --help works at setup; if the skill is absent, tell the user and either install it

(copy the repo's loops/literature-search folder into ~/.claude/skills/) or degrade all retrieval to WebSearch/WebFetch

(no ranked snippets or citation-graph expansion). The keyless S2 + arXiv core

(search/snippet/cite/fulltext) needs no setup; a free S2_API_KEY makes snippet+cite

(ScholarEval's two defining moves) reliable.

API key (optional, never block). The literature-search skill owns the key convention: run

<lit> keys --init, then have the user fill the printed keys.env themselves and never paste secrets

into chat. Re-run <lit> keys to confirm presence and record the tier in loop.run.yaml

(literature_tiers, presence only). A missing key just degrades to the keyless pool → WebSearch.

Initialise the sandbox once bindings are confirmed:

<sandbox_root>/
├── loop.run.yaml        ← resolved bindings + <intent> + grade_weights + literature_tiers
├── ledger.tsv           ← header only (see Ledger)
├── literature/.cache/   ← lit_search on-disk cache
└── iter1/proposal.md    ← the input proposal (the baseline)

The loop

<N> starts at 1; iteration 1 evaluates the unmodified proposal (the baseline grade — no

revision before it). Re-evaluate fresh every iteration: the grade comes only from a *new* ScholarEval

pass on the *revised* proposal, never carried over.

Copy this checklist and tick items off:

• [ ] Evaluate — run roles/ScholarEval.md (spawn-or-degrade) on iter<N>/proposal.md with the <eval_scale> caps and <lit>; it writes iter<N>/scholareval.json (validates against schemas/scholareval.schema.json).
• [ ] Grade — as the Judge (roles/Judge.md) apply rubrics/rubric.md to scholareval.json: evidence gate → 0-5 per axis → weighted grade → hard gates → pass + ranked fixes; write iter<N>/verdict.json (validates against schemas/verdict.schema.json).
• [ ] Log — append one ledger.tsv row (see Ledger).
• [ ] Stop check — verdict.pass == true, or N == <budget>, or grade flat for <patience> iterations → stop (see Stops).
• [ ] Revise — run roles/Reviser.md (spawn-or-degrade) with verdict.json, scholareval.json, <intent>, <lit>, and a small revision search budget (≈<eval_scale> searches + a couple of reads); it applies one focused fix batch and writes iter<N+1>/proposal.md + iter<N+1>/revision_notes.md.
• [ ] N = N + 1 and repeat.

Every cited snippet must come from a real retrieval that iteration — never fabricated. When a lit tool

returns {"error","fallback"}, fall back to WebSearch/WebFetch; never invent a paper.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in free text:

iter	grade	pass	soundness	contribution	evidence_quality	top_fix	revision_summary
1	58.0	no	3	2	4	add MM-GBSA re-scoring	baseline (no revision)
2	71.0	no	4	3	4	add head-to-head vs [C2] pipeline	added re-scoring + scoped affinity claim
3	82.0	yes	4	4	4	-	reframed contribution around integration + new benchmark

The matching scholareval.json and verdict.json for each iteration live in iter<N>/. Report the

best-grade iteration when stopping on budget/plateau, not necessarily the last. Leave ledger.tsv,

iter*/, and literature/ untracked.

Constraints

• Never fabricate citations or snippets — every evidence entry comes from a real <lit>/WebFetch

result retrieved that iteration, verbatim; the evidence gate exists to catch fabrication.

• The rubric is fixed — the Judge never edits rubrics/rubric.md, so the passing threshold stays

meaningful across iterations.

• Protect <intent> — the Reviser may strengthen but never replace the core research question or

headline contribution, and never gut a central method just to lift the grade.

• Grade through evidence, not prose — eloquence earns nothing; grounded support and surviving

novelty earn the score.

• One focused revision batch per iteration, so each grade move is attributable.
• No installs — the sibling literature-search skill is stdlib-only; never print or commit API keys

(keys.env stays gitignored at the project root). The sandbox is self-contained — no ../ escapes.

Stops

The loop stops on the first of:

• Pass — verdict.pass == true. Report the final proposal (iter<N>/proposal.md), its grade, and

the grade trajectory.

• Budget — N == <budget>. Report the best-grade iteration as the deliverable.
• Plateau — grade hasn't improved for <patience> consecutive iterations. Report the best

iteration and the standing priority-1 fixes.

Always end with the deliverable proposal path, its grade and pass/fail, the grade trajectory from

ledger.tsv, and — if it did not pass — the standing blockers between this proposal and the bar.