Waza Runner Agent Skill
| Run evaluations on Agent Skills to measure their effectiveness. "check skill triggers", "skill compliance check", "measure skill performance", "run evals on [skill-name]", "grade skill execution". (use sensei), or general testing unrelated to skills.
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/microsoft/waza --skill waza-runnerWhat comes with it
3 171 bytes besides the instructionThe instruction itself
13 sections, as written by the authorSkill Eval Runner
> Evaluate Agent Skills like you evaluate AI Agents
This skill runs evaluations on other skills to measure their effectiveness using the same patterns that power AI agent evaluations.
When to Use
- Running quality evaluations on a skill
- Testing if a skill triggers on correct prompts
- Measuring skill behavior quality
- Generating eval reports for CI/CD
Commands
Run Evals
Run evals on <skill-name>
Initialize Eval Suite
Create evals for <skill-name>
Generate Report
Generate eval report for <skill-name>
Workflow
- Check for Eval Suite: Look for
eval.yamlin the skill directory - Load Tasks: Parse task definitions from
tasks/*.yaml - Execute: Run each task through the configured graders
- Report: Output results in JSON or Markdown format
Metrics Measured
| Metric | Description | Default Threshold |
|--------|-------------|-------------------|
| Task Completion | Did the skill accomplish the goal? | 80% |
| Trigger Accuracy | Was skill invoked on correct prompts? | 90% |
| Behavior Quality | Tool calls, efficiency, reasoning | 70% |
Grader Types
- Code Graders: Deterministic assertions, regex matching
- LLM Graders: Model-as-judge with configurable rubrics
- Human Graders: Manual review workflow
Example Usage
Running Evals
# From CLI
waza run ./my-skill/eval.yaml
# Output to file
waza run ./my-skill/eval.yaml -o results.json
Interpreting Results
{
"summary": {
"pass_rate": 0.85,
"composite_score": 0.82
},
"metrics": {
"task_completion": { "score": 0.9, "passed": true },
"trigger_accuracy": { "score": 0.95, "passed": true }
}
}
References
- Eval Specification - Full eval.yaml schema
- Writing Tasks - Task definition guide
- Grader Reference - Available graders
Other skills for the same job
different authors, same section of the catalogue
Multi-agent autonomous startup system for Claude Code. Triggers on "Loki Mode". Orchestrates 100+ specialized agents across engineering, QA, DevOps, security, data/ML, business operations, marketing, HR, and customer success. Takes PRD to fully deployed, revenue-generating product with zero human intervention. Features Task tool for subagent dispatch, parallel code review with 3 specialized reviewers, severity-based issue triage, distributed task queue with dead letter handling, automatic deployment to cloud providers, A/B testing, customer feedback loops, incident response, circuit breakers, and self-healing. Handles rate limits via distributed state checkpoints and auto-resume with exponential backoff. Requires --dangerously-skip-permissions flag.
Use when working with error debugging multi agent review
Build evaluation frameworks for agent systems. Use when testing agent performance systematically, validating context engineering choices, or measuring improvements over time.
Diagnoses and debugs A2A agent communication issues including agent status, message routing, transport connectivity, and log analysis. Use when agents aren't responding, messages aren't being delivered, routing is incorrect, or when debugging orchestrator, coder-agent, tester-agent communication problems.
Use when working with error debugging multi agent review
Rapidly creates atomic, focused skills optimized with evidence-based prompting, specialist agents, and systematic testing. Each micro-skill does one thing exceptionally well using self-consistency, program-of-thought, and plan-and-solve patterns. Enhanced with agent-creator principles and functionality-audit validation. Perfect for building composable workflow components.
Ultimate multi-agent framework for Google Antigravity. Orchestrates specialized domain agents (PM, Frontend, Backend, Mobile, QA, Debug) via Serena Memory.
This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", or mentions LLM-as-judge, multi-dimensional evaluation, agent testing, or quality gates for agent pipelines.
How to use it
Take microsoft/waza-runner from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.