Finetuning Agent Skill

finetuning — teach an open model a form or behavior, not a fact

You own the discipline of adapting an open-weight model: deciding whether to fine-tune at all,

then running SFT and (optionally) preference optimization with trl + peft, backend-agnostic.

You are judged by whether the tuned model reliably produces the target form/behavior on a

held-out set — not by train loss, and not by vibes.

The one sentence that routes half of all "should I fine-tune?" questions correctly:

fine-tuning teaches *form and behavior*; RAG supplies *facts*. If the ask is "know our latest

prices / docs / tickets," that is retrieval (../rag/SKILL.md), not training. If the ask is "sound

like us, always emit this JSON, follow this reasoning pattern," that is here.

Decision gate — try this BEFORE reaching for a GPU

Fine-tuning is the last lever, not the first. Exhaust the cheaper, reversible options first; each

row below is a real off-ramp.

| If the goal is… | Do this first | Fine-tune only when… |

|---|---|---|

| The model should *know* current/company facts | RAG (../rag/SKILL.md) — retrieve + ground | never for facts; facts go stale, weights don't update |

| One-off format/tone, small volume | Prompt + few-shot (prompt-engineering) | the prompt is huge, brittle, or you pay for it every call |

| Behavior depends on a long document | Longer context / put it in the prompt | context won't fit, or per-call token cost is the bottleneck |

| Consistent *form/behavior* at scale, latency/cost sensitive | — | prompting plateaus AND you have (or can build) good examples |

| A capability the base model just can't do | — | you have a reward signal or demonstration data for it |

Route out explicitly. Facts / freshness / citations → ../rag/SKILL.md. Squeezing a prompt before

spending money → prompt-engineering. Picking *which* base model (size/license/task) → open-weights.

Building the JSONL/preference corpus → training-data (LLM corpora, NOT tabular cleaning — that is

data-cleaning). A fast single-GPU run + GGUF export → ../unsloth/SKILL.md (same LoRA/QLoRA

concepts, one optimized implementation; this skill stays backend-agnostic). Downloading the base or

pushing the adapter/merged model → huggingface. Serving the result → ../vllm/SKILL.md.

> The cheapest fine-tune is the one you didn't need. Prompt + RAG solves most "make it behave"

> asks at zero training cost and updates instantly. Fine-tune when that ceiling is real, measured,

> and you can afford to re-run it every time the base model or data changes.

Version reality (verify at author time — this stack moves monthly)

• trl consolidated into a v1.x line (v1.0 landed ~2026; docs at author time referenced

~v1.8). Every method has a Trainer + a Config dataclass that inherits

transformers.TrainingArguments (SFTTrainer/SFTConfig, DPOTrainer/DPOConfig, …). Confirm

the current major before pinning: pip show trl / the TRL docs.

• transformers is on a v5.x line; peft, bitsandbytes, accelerate, datasets

round out the stack. Do not freeze a pin as "the version" — say "current major is ~X, verify."

• Some methods migrated to trl.experimental.* (e.g. from trl.experimental.orpo import ORPOTrainer

at author time). Import paths churn — check the method's doc page before copying an import.

• Model licenses are not facts to memorize. Llama ships under the Meta *Community* license

(usage caps, not OSI-open); Gemma under custom Google terms; Qwen/Mistral vary per size and often

Apache-2.0 — but read the specific model card, licenses change. License/size selection is

open-weights.

LoRA / QLoRA vs full fine-tuning

Three options on one memory↔quality axis. Default to QLoRA unless you have a proven reason not to.

| Method | What trains | Rough VRAM (7–8B) | Use when |

|---|---|---|---|

| Full FT | every weight, fp16/bf16 | very high (needs multi-GPU / offload) | you have the hardware and a large, high-quality corpus and adapters underfit |

| LoRA | small low-rank adapter matrices; base frozen (fp16) | high | base fits in fp16 and you want adapter portability + speed |

| QLoRA | LoRA adapters over a 4-bit NF4 frozen base | lowest — single consumer GPU for 7–13B | the default; fine-tune big models on one GPU with ~no quality loss |

QLoRA (Dettmers et al., arXiv:2305.14314): load the base

in 4-bit NF4 with double quantization, keep it frozen, and train LoRA adapters in bf16 on top.

It made single-GPU fine-tuning of large models practical at near-full-FT quality.

import torch
from transformers import BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

# 4-bit NF4 base (QLoRA). Verify arg names against current bitsandbytes/transformers.
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# LoRA over ALL linear layers — the safe target for QLoRA (PEFT quantization guide).
peft_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    bias="none", task_type="CAUSAL_LM",
    target_modules="all-linear",   # or explicit ["q_proj","k_proj","v_proj","o_proj",...]
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-7B-Instruct",       # instruct base → chat template already present
    args=SFTConfig(output_dir="out", max_length=2048, packing=True,
                   learning_rate=2e-4, num_train_epochs=2, bf16=True),
    train_dataset=dataset,                    # conversational: TRL applies the chat template
    peft_config=peft_config,
    quantization_config=bnb,                  # SFTTrainer + peft_config + this == QLoRA
)
trainer.train()
trainer.save_model("out")                     # saves the ADAPTER, not a merged model

peft_config + quantization_config on SFTTrainer is the current one-liner for QLoRA — no manual

get_peft_model / prepare_model_for_kbit_training wiring needed. save_model writes the small

adapter; merge to a standalone model only when serving requires it (see references).

Pipeline: SFT first, then (maybe) preference optimization

SFT → preference optimization is the standard post-training arc. SFT teaches the model the

format and gives it the behavior by imitation. Preference optimization then sharpens *which* of

several plausible outputs is better. Most projects need only SFT; add a preference stage when "the

outputs are fine but I want the *good* one preferred" is the remaining gap.

| Method | Data it needs | Stage | Pick it when |

|---|---|---|---|

| SFT | demonstrations (chat/messages or prompt→completion) | base of everything | always first (except ORPO) |

| DPO (2305.18290) | paired chosen/rejected | after SFT | you have pairwise preferences; the workhorse aligner |

| ORPO (2403.07691) | paired preferences | replaces SFT+DPO (single stage, ref-free) | you want one pass from a base model and have pairs |

| KTO (2402.01306) | *unpaired* binary good/bad labels | after SFT | you have thumbs-up/down, not matched pairs |

| GRPO (2402.03300; DeepSeek-R1 2501.12948) | a reward function (verifier), no pairs | after SFT | correctness is checkable (math/code/format) → RL for reasoning |

Rule of thumb: have pairs → DPO (or ORPO to fuse the two stages); have only up/down votes → KTO;

can score an answer programmatically → GRPO. Preference optimization uses a *tiny* learning rate.

# DPO after SFT — dataset has prompt / chosen / rejected columns.
from trl import DPOTrainer, DPOConfig
trainer = DPOTrainer(
    model="out",                              # your SFT checkpoint (or SFT+adapter)
    args=DPOConfig(output_dir="dpo-out", beta=0.1,   # beta = KL strength to the ref model
                   learning_rate=5e-7, max_length=1024,
                   precompute_ref_log_probs=True),   # saves memory; ref model auto-created
    train_dataset=pref_dataset,
    peft_config=peft_config,                  # LoRA works for preference stages too
)
trainer.train()

# GRPO — no preference pairs, a reward FUNCTION that returns a score per completion.
from trl import GRPOTrainer, GRPOConfig
def format_reward(completions, **kwargs):     # signature: gets completions (+ dataset cols via kwargs)
    return [1.0 if "\\boxed{" in c[0]["content"] else 0.0 for c in completions]

trainer = GRPOTrainer(
    model="out",
    reward_funcs=[format_reward],             # one or many; GRPOConfig.reward_weights to combine
    args=GRPOConfig(output_dir="grpo-out", num_generations=8,  # group size per prompt
                    beta=0.04, learning_rate=1e-6, use_vllm=True),  # vLLM speeds rollouts
    train_dataset=prompts_dataset,
)
trainer.train()

Full runnable SFT→DPO and GRPO scripts, ORPO/KTO variants, dataset schemas, and adapter-merge steps

are in references/methods.md.

Data — quality over quantity (LIMA)

More rows is not the win. LIMA (arXiv:2305.11206) got strong

instruction-following from ~1,000 carefully curated examples — "less is more for alignment."

A thousand clean, on-distribution, correctly-templated examples beat 100k scraped noisy ones, which

actively teach the model bad form. Building and validating that corpus (JSONL messages, preference

pairs, dedup, contamination checks) is training-data — bring it here already clean.

Hyperparameters — the few that move the needle

• LoRA r and lora_alpha: r = adapter rank (capacity); effective scaling = lora_alpha / r.

Common heuristic alpha ≈ 2·r (e.g. r=16→alpha=32) so scaling ≈ 2; then adjust LR, not both.

Start r=8–16 for style/format, higher (32–64+) for harder behavior. (Newer "LoRA-without-regret"

guidance favors target_modules="all-linear" + higher rank + tuned LR — verify current advice.)

• target_modules: "all-linear" is the safe default. Targeting too few / wrong-named modules

is a top silent failure — the run "succeeds," loss barely moves, the adapter learned ~nothing.

• Learning rate: LoRA/QLoRA SFT ~1e-4–2e-4 (adapters tolerate higher LR than full FT, whose

~2e-5 is the SFTConfig default). Preference optimization is far lower — DPO ~5e-7, GRPO ~1e-6.

• Epochs: usually 1–3. This is the overfitting knob — see below. Watch eval loss.
• Batch × grad-accum: raise the *effective* batch with gradient_accumulation_steps when VRAM

caps per_device_train_batch_size. Enable packing=True + gradient_checkpointing to fit more.

• Warmup: a short warmup_ratio (~0.03–0.1) stabilizes the early, high-gradient steps.

Catastrophic forgetting

Fine-tuning on a narrow task can degrade general ability the base model had. Three mitigations,

cheapest first: use LoRA/QLoRA (base weights frozen — inherently gentler than full FT); **keep

the LR low and epochs few; and replay** — mix a slice of general instruction data into your

task data so the model doesn't forget how to be a general assistant. If a tuned model suddenly

"got dumber" at everything else, this is the usual cause.

Evaluation — a held-out set and a task metric, not a vibe-check

A vibe-check is not an eval. Before training, split off a held-out set the model never sees,

and define a concrete task metric (exact-match / JSON-valid rate / rubric score / a task-specific

score). Judge the run on that, plus eval loss.

• Watch eval loss, not train loss. Train loss falling while eval loss rises = overfitting →

fewer epochs, lower LR, more/cleaner data, or earlier checkpoint. Train loss always keeps falling;

it tells you nothing about generalization.

• Contamination: if eval examples leaked into training, your metric is a lie. De-dup train vs

eval; keep the held-out set quarantined. (Corpus-side hygiene is training-data.)

• General LLM/agent eval harness and judging → agent-eval. Bring your task metric here.

The full tuning + forgetting + evaluation playbook is in references/hyperparameters-and-eval.md.

When NOT to fine-tune (anti-patterns)

| Anti-pattern | Why it breaks | Do instead |

|---|---|---|

| Fine-tune to add facts / fresh knowledge | Weights memorize poorly and go stale; hallucinations | ../rag/SKILL.md — retrieve + ground |

| Fine-tune before trying prompt + few-shot | Slow, costly, irreversible for a prompt-solvable ask | prompt-engineering first |

| Wrong / too-few target_modules | Adapter has no capacity where it matters → learns ~nothing | "all-linear" (or correct proj names) |

| Judge success by train loss | Falls even while the model overfits | Held-out eval set + task metric + eval loss |

| Crank epochs "to learn it better" | Overfits, forgets, memorizes noise | 1–3 epochs; stop when eval loss turns up |

| Train with a wrong/absent chat template | Inference emits garbage / never stops | Match train template to serve; align eos_token |

| A few dozen examples for full FT | Not enough signal; unstable | Curate ~hundreds–thousands (LIMA) or use LoRA |

| Fine-tune a model you can't legally deploy | License blocks your use case | Check the model card first → open-weights |

Checklist

• [ ] Ran the decision gate: confirmed this is a form/behavior need, not a facts need (else → rag).
• [ ] Chose base model by license + size (→ open-weights); read the actual model card.
• [ ] Picked method: QLoRA by default; full FT only with the hardware + a reason.
• [ ] Data is clean, on-distribution, correctly templated (→ training-data); dedup vs eval.
• [ ] Chat template applied at train time and it matches the serving template; eos_token aligned.
• [ ] target_modules="all-linear" (or verified names); alpha ≈ 2·r; LoRA LR ~1e-4–2e-4.
• [ ] Held-out eval set + a concrete task metric defined before training.
• [ ] Trained 1–3 epochs; watched eval loss (not train) for the overfitting turn.
• [ ] Preference stage only if needed; correct method for the data (pairs→DPO/ORPO, votes→KTO, verifier→GRPO); tiny LR.
• [ ] Verified trl/peft/transformers current majors and import paths at author time.