Grpo Rlvr Training

GRPO & RLVR Training

This skill assumes finetuning-method-selection

already routed here because the target behavior

has a verifiable pass/fail signal — not

demonstrations (lora-qlora-recipes) or

preference pairs (preference-optimization).

What follows is when RL is the right tool, the

reference recipe, the mandatory reward-inspection

gate, and how to pick a GRPO variant when the

base recipe misbehaves.

Input: a routing decision (RLVR via GRPO)

plus a verifier (code executor, test suite,

schema checker, or grader) for the target task.

Output format: a validated GRPO config — the

kwarg values in references/grpo-memory.md and

the reward functions in

references/reward-functions.md, not free-form

advice — that llm-finetuning-training-engineer

consumes directly.

When RL Applies

GRPO+RLVR only pays off when task success is

algorithmically checkable — a unit test

passes, a parser accepts the output, a tool call

matches an expected schema, a math answer matches

a ground truth. If grading the output requires

human judgment or a subjective rubric, that's an

eval-harness and judge-calibration problem first

— see eval-harness-first — not a reason to skip

straight to RL.

Before opening a GRPO run, confirm the model can

sometimes succeed on the target task already.

RL sharpens an existing capability by reweighting

toward the samples that already work; it does not

install a capability from zero.

• **The model never succeeds, even at low

temperature across many samples:** the gap is

format or task understanding, not policy

refinement. Route back to SFT first

(lora-qlora-recipes) and only return to this

skill once the base success rate is nonzero.

• **The model succeeds sometimes,

inconsistently:** this is the GRPO sweet spot —

proceed to The Recipe below.

The standing rule for the whole plugin: **DPO for

taste, GRPO for reasoning.** If the signal is a

preference between two acceptable outputs, that's

preference-optimization, not this skill.

The Recipe

The reference recipe is TRL's GRPOTrainer with

vLLM-backed generation:

from trl import GRPOConfig, GRPOTrainer

grpo_args = GRPOConfig(
    output_dir="./outputs-grpo",
    use_vllm=True,
    vllm_mode="colocate",       # single GPU; "server" for multi-GPU
    num_generations=8,          # floor — fewer starves the group-relative baseline
    learning_rate=5e-7,         # settled range for GRPO
    beta=0.01,                  # KL coefficient vs the reference policy
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    bf16=True,
    logging_steps=10,
    seed=3407,
)

trainer = GRPOTrainer(
    model=SFT_CHECKPOINT,
    args=grpo_args,
    reward_funcs=[format_reward, correctness_reward],   # references/reward-functions.md
    train_dataset=prompts,       # prompt-only — GRPO generates its own completions
    processing_class=tokenizer,
)

trainer.train()

• vllm_mode="colocate" runs generation and

training on the same GPU — the default for a

single-GPU box.

• vllm_mode="server" points at a separate

vLLM server process and is the multi-GPU path —

generation and training don't compete for the

same device.

• num_generations ≥ 8 is a floor, not a

suggestion: GRPO's advantage estimate is

relative to the group mean, and fewer than 8

samples per prompt produces a noisy baseline.

• Reward is composite — a format reward (did

the output parse / match the required

structure) plus a correctness reward (did the

answer verify). A well-formed-but-wrong answer

and a malformed one should not score

identically; correctness alone loses that

signal.

• learning_rate=5e-7 and beta=0.01 are

the settled starting point; deviate only after

the base run is stable and reward-inspected

(below).

Memory sizing for this recipe by target size

class: references/grpo-memory.md.

The Inspection Rule

**Run the reward function against 50–100 sampled

outputs and manually read the results before

starting the actual training run.** This is a

gate, not a one-time sanity check.

If the reward function's judgment disagrees with

a human reading of that sample, fix the reward

function first. Training against an uninspected

reward, or tuning hyperparameters to compensate

for one silently scoring the wrong thing, is how

a run reward-hacks: the model optimizes cleanly

toward the wrong target, and that doesn't surface

as a training-loop bug.

This inspection is a Phase 1 gate input for

/finetune — the same 50–100-sample read that

catches a broken reward function here is what that

command checks for before it lets a GRPO brief

proceed.

Complete reward function implementations to

inspect against — exact-match, schema-validation,

unit-test-execution, a length-penalty wrapper, and

a rubric-as-reward judge pattern:

references/reward-functions.md.

Variant Selection

The base recipe above is the default. Reach for a

variant only when a specific failure mode shows

up, not preemptively:

| Failure mode | Variant | Why |

|---|---|---|

| Entropy collapse / degenerate long chain-of-thought | DAPO | Decouples clip bounds and relaxes the KL penalty that over-regularizes exploration on long reasoning traces |

| Reward or output length trends up regardless of quality | Dr.GRPO | Removes GRPO's length-normalization bias so reward tracks correctness, not completion length |

| Training a mixture-of-experts model | GSPO | Moves the importance-sampling ratio to the sequence level instead of per-token — per-token ratios are unstable on MoE routing, so GSPO is required here, not optional |

Start with plain GRPO. Watch for the specific

symptom — collapsing entropy on long CoT, a

length-reward correlation, or MoE instability —

and only then swap in the matching variant above.

Don't pre-select a variant before the base recipe

has actually shown the failure mode.

VLM RL Is Reference-Only

Vision-language RL is **not executed by this

plugin in v1** — it's documented here for

context, not as a runnable path. Tooling is

fragmented across ms-swift and EasyR1-derived

forks with no one-line TRL command yet, and naive

text-only GRPO applied to a VLM tends to

reward-hack by optimizing the text-reasoning trace

while ignoring the image — the model learns to

sound right without looking at the input. A VLM

RL run is a research spike outside this skill's

supported recipe, not a variant of The Recipe

above.

References

• references/reward-functions.md — complete

Python reward functions (exact-match

correctness, schema validation, unit-test

execution, a length-penalty wrapper, and a

rubric-as-reward judge pattern) to inspect under

The Inspection Rule before any training run.

• references/grpo-memory.md — memory sizing by

target size class, vLLM sleep-mode and

optimizer-state tactics, Unsloth's long-context

RL chunking, and the DGX Spark bandwidth caveat

for decode-heavy rollouts.

Related skills: finetuning-method-selection

routes here once a verifiable pass/fail signal

exists; preference-optimization is the sibling

skill for preference pairs rather than verifiable

rewards; eval-harness-first covers judge

calibration for any reward that isn't purely

code-checkable. On DGX Spark, defer to the

dgx-spark-ops plugin's skills, when installed,

for the memory/thermal remediation ladder this

skill's memory table doesn't cover.