Behavioral Evals

Behavioral Evals

Overview

Behavioral evaluations (evals) are tests that validate the agent's decision-making (e.g., tool choice) rather than pure functionality. They are critical for verifying prompt changes, debugging steerability, and preventing regressions.

> [!NOTE]

> Single Source of Truth: For core concepts, policies, running tests, and general best practices, always refer to evals/README.md.

🔄 Workflow Decision Tree

• Does a prompt/tool change need validation?
• *No* -> Normal integration tests.
• *Yes* -> Continue below.
• Is it UI/Interaction heavy?
• *Yes* -> Use appEvalTest (AppRig). See creating.md.
• *No* -> Use evalTest (TestRig). See creating.md.
• Is it a new test?
• *Yes* -> Set policy to USUALLY_PASSES.
• *No* -> ALWAYS_PASSES (locks in regression).
• Are you fixing a failure or promoting a test?
• *Fixing* -> See fixing.md.
• *Promoting* -> See promoting.md.

📋 Quick Checklist

1. Setup Workspace

Seed the workspace with necessary files using the files object to simulate a realistic scenario (e.g., NodeJS project with package.json).

• *Details in creating.md*

2. Write Assertions

Audit agent decisions using rig.setBreakpoint() (AppRig only) or index verification on rig.readToolLogs().

• *Details in creating.md*

3. Verify

Run single tests locally with Vitest. Confirm stability locally before relying on CI workflows.

• *See evals/README.md for running commands.*

📦 Bundled Resources

Detailed procedural guides:

• creating.md: Assertion strategies, Rig selection, Mock MCPs.
• fixing.md: Step-by-step automated investigation, architecture diagnosis guidelines.
• promoting.md: Candidate identification criteria and threshold guidelines.