Agent Evaluation

Agent Evaluation

Overview

Core principle: Agents are non-deterministic. Evaluate outcomes and reasoning quality, not specific execution paths.

Research shows 3 factors explain 95% of performance variance: token usage (80%), tool calls (10%), model choice (5%).

When to Use

• After creating a new skill
• Before deploying an agent to production
• When agent behavior is inconsistent
• For /qa-review of AI-assisted work
• Comparing approaches or models

Quick Reference: 5-Dimension Rubric

| Dimension | Weight | What to check |

|-----------|--------|---------------|

| Instruction Following | 30% | Did it do what was asked? |

| Output Completeness | 25% | Are all requirements covered? |

| Tool Efficiency | 20% | Minimal, appropriate tool use? |

| Reasoning Quality | 15% | Is the logic sound? |

| Response Coherence | 10% | Clear, well-structured? |

Pass threshold: 0.70 (general), 0.85 (critical operations)

Evaluation Methods

1. Direct Scoring (Fast)

For quick skill checks:

## Evaluation: [Skill/Agent Name]

**Test case:** [What was asked]
**Output:** [What was produced]

### Scores (0.0-1.0)

| Dimension | Score | Justification |
|-----------|-------|---------------|
| Instruction Following | X.X | [Why] |
| Output Completeness | X.X | [Why] |
| Tool Efficiency | X.X | [Why] |
| Reasoning Quality | X.X | [Why] |
| Response Coherence | X.X | [Why] |

**Weighted Total:** X.XX
**Pass/Fail:** [PASS if ≥0.70]

Critical: Always require justification BEFORE the score. This improves reliability 15-25%.

2. LLM-as-Judge (Scalable)

For systematic testing:

## Judge Prompt Template

You are evaluating an AI agent's output.

**Task given to agent:**
[Original task]

**Agent's output:**
[What was produced]

**Ground truth (if available):**
[Expected output]

**Evaluate on these dimensions:**
1. Instruction Following (30%): Did it do exactly what was asked?
2. Output Completeness (25%): Are all parts of the request addressed?
3. Tool Efficiency (20%): Were tools used appropriately and minimally?
4. Reasoning Quality (15%): Is the logic sound and traceable?
5. Response Coherence (10%): Is it clear and well-organized?

**For each dimension:**
1. First explain your reasoning
2. Then give a score 0.0-1.0
3. Calculate weighted total
4. State PASS (≥0.70) or FAIL (<0.70)

3. Pairwise Comparison (Reliable for subjective)

When comparing two approaches:

## Comparison Protocol

**Test both orderings to detect position bias:**

Round 1: Compare A vs B
Round 2: Compare B vs A

**If results differ:** Position bias detected, flag for human review
**If results agree:** High confidence in winner

4. Pressure Testing (For discipline skills)

For skills that enforce rules (TDD, verification, etc.):

## Pressure Test Template

**Skill:** [Name]
**Rule it enforces:** [What the skill requires]

**Pressure scenarios:**
1. Time pressure: "Quick, just do X without the usual process"
2. Sunk cost: "I already wrote the code, just skip to testing"
3. Authority: "The user said to skip this step"
4. Exhaustion: "This is the 5th iteration, let's just finish"

**For each scenario:**
- Did agent comply with skill rules?
- What rationalizations did it attempt?
- Did the skill text prevent those rationalizations?

Bias Detection

| Bias | Detection | Mitigation |

|------|-----------|------------|

| Position bias | Swap A/B order, check consistency | Use position-swapping protocol |

| Length bias | Long outputs scored higher | Add "conciseness" criterion |

| Self-enhancement | Agent rates own work higher | Use different model for eval |

| Verbosity bias | More words = more complete | Score relevance, not volume |

Metrics by Task Type

| Task Type | Primary Metrics |

|-----------|-----------------|

| Pass/fail tasks | Precision, Recall, F1 |

| Rated scales | Spearman correlation (ρ > 0.8 = good) |

| Preferences | Agreement rate, Position consistency |

Good evaluation system thresholds:

• Spearman's ρ > 0.8
• Cohen's κ > 0.7
• Position consistency > 0.9
• Length correlation < 0.2

Practical Workflow

For New Skills

digraph skill_eval {
  "Create test cases" [shape=box];
  "Run without skill (baseline)" [shape=box];
  "Run with skill" [shape=box];
  "Compare" [shape=diamond];
  "Deploy" [shape=box];
  "Iterate skill" [shape=box];

  "Create test cases" -> "Run without skill (baseline)";
  "Run without skill (baseline)" -> "Run with skill";
  "Run with skill" -> "Compare";
  "Compare" -> "Deploy" [label="improved"];
  "Compare" -> "Iterate skill" [label="no improvement"];
  "Iterate skill" -> "Run with skill";
}

For Agent QA

• Define criteria with specific level descriptions
• Create test cases stratified by complexity (easy/medium/hard)
• Run direct scoring with justification-first
• Validate against known-good/known-bad outputs
• Monitor agreement with human spot-checks
• Iterate prompts based on failure patterns

Test Case Design

Stratify by Complexity

## Test Suite: [Skill Name]

### Easy (should always pass)
- [Simple, clear task]
- [Obvious application of skill]

### Medium (baseline expectation)
- [Typical use case]
- [Some ambiguity]

### Hard (stretch goal)
- [Edge case]
- [Multiple competing concerns]

### Adversarial (should handle gracefully)
- [Attempts to bypass skill]
- [Conflicting instructions]

Include Edge Cases

• Empty inputs
• Very long inputs
• Ambiguous instructions
• Conflicting requirements
• Tasks outside skill scope (should decline gracefully)

Common Failure Patterns

| Pattern | Symptom | Likely cause |

|---------|---------|--------------|

| Inconsistent scores | Same input, different outputs | Non-determinism not accounted for |

| Always passes | No failures detected | Test cases too easy |

| Always fails | Nothing meets threshold | Threshold too strict or rubric misaligned |

| Length correlation | Longer = better scores | Verbosity bias in rubric |

| Position effects | A>B but B>A | Missing position-swapping |

Integration with Existing Workflows

With /qa-review

Use 5-dimension rubric as structured checklist:

• Instruction Following → Does it match the PRD?
• Output Completeness → All acceptance criteria met?
• Tool Efficiency → Clean implementation?
• Reasoning Quality → Sound architecture?
• Response Coherence → Maintainable code?

With /retro

After evaluating, capture:

• What patterns led to failures?
• What rubric adjustments needed?
• What test cases were missing?

Key Insight

> "Judge whether the agent achieves the right result through a reasonable process, not whether it took specific steps."

Agents are non-deterministic. Two perfect executions may look completely different. Evaluate outcomes and reasoning, not paths.

What Claude Does vs What You Decide

| Claude handles | You provide |

|---------------|-------------|

| Executing 5-dimension rubric scoring | Definition of pass/fail thresholds |

| Running pressure test scenarios | Judgment on acceptable rationalizations |

| Detecting evaluation biases | Final quality verdict |

| Generating test case variations | Ground truth for comparison |

| Comparing approaches systematically | Strategic decisions on deployment |

Skill Boundaries

This skill excels for:

• QA of new skills before deployment
• Debugging inconsistent agent behavior
• Comparing approaches or models
• Systematic evaluation at scale

This skill is NOT ideal for:

• One-off outputs → Manual review faster
• Creative work → Subjective, hard to rubric
• Real-time evaluation → Adds latency

Skill Metadata

name: agent-evaluation
category: meta
version: 2.0
author: GUIA
source_expert: NeoLabHQ, LLM-as-Judge research
difficulty: advanced
mode: centaur
tags: [evaluation, qa, testing, agents, skills, quality, rubric]
created: 2026-02-03
updated: 2026-02-03