Eval Authoring Skill

Skill Eval Authoring

Author Vally evals that verify one Agent Skill's routing and behavior in isolation, under .github/skills/<skill>/evals/. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the repository-local eval authoring guide at .github/skills/eval-authoring/README.md. This skill exists to route you there with skill-eval context already loaded, not to duplicate it.

Triggers

USE FOR: write a skill eval, add trigger eval, test skill routing, add anti-trigger tests, create skill capability eval, harden skill graders

WHEN: "write a skill eval", "add trigger eval", "test skill routing", "add anti-trigger tests", "create skill capability eval", "harden skill graders"

DO NOT USE FOR: single MCP tool prompt-to-tool coverage (use eval-authoring-tool), multi-tool or multi-turn scenarios (use eval-authoring-workflow), authoring SKILL.md itself (use skill-authoring)

Steps

• Read the eval authoring guide (.github/skills/eval-authoring/README.md) Step 0 to find this repo's vallyRoot/evalGlobs for the skill tier, then the guide's "Skill" column throughout (naming, requirements, worked example).
• Read the target SKILL.md — its WHEN/DO NOT USE FOR boundaries, invoked tools — and its existing evals/.
• Write eval.yaml with routing stimuli (trigger + anti-trigger, skill-invocation grader) and capability stimuli together in the same file per the guide's naming convention and four-layer pattern. Split into additional <behavior>.eval.yaml files only once a skill's coverage genuinely grows large. For a boundary prompt, mount and require the intended competing skill while disallowing the skill under test — an anti-trigger with no competing skill mounted trivially "passes" (guide anti-pattern A7).
• Supply concrete identifiers (repo path, package name, anything needed to reach the call) in every stimulus prompt — a vague prompt fails tool-calls with zero recorded calls, which looks like a routing bug but is really a missing-context prompt bug.
• Validate locally per the guide's "Running evals locally" section. Do not finish or open a PR until the focused local eval passes; inspect eval-results.md and results.jsonl on failure.

Rules

• Cover both activation and non-activation; anti-triggers should target neighboring skills, not random unrelated prompts.
• Omit turn for conversation-wide assertions; pin a turn only when that turn owns the behavior unambiguously.
• Never copy secrets or production-write identifiers into fixtures.
• Update existing specs additively unless the requested contract changed.

References

• Eval authoring guide: .github/skills/eval-authoring/README.md — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with eval-authoring-tool/eval-authoring-workflow — update it there, not per-skill.
• Repository-local eval README/configuration discovered in Step 0 (if present)