mcpbeat Sign in

Evals Context Agent Skill

by foryourhealth111-pixel

Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).

2k tokens
context cost
the whole folder, loaded on every use
1
files
instructions only
0
copies elsewhere
how many repositories repackaged it
2583
stars on the repo
on the repository, not the skill itself

Install

one command, takes just this skill from the repository 
npx skills add https://github.com/foryourhealth111-pixel/Vibe-Skills --skill evals-context

The instruction itself

18 sections, as written by the author

Evals Codebase Context

When to Use This Skill

Use this skill when the task involves:

  • Modifying or debugging the evals execution infrastructure
  • Adding new eval exercises or languages
  • Working with the evals web interface (apps/web-evals)
  • Modifying the public evals display page on roocode.com
  • Understanding where evals code lives in this monorepo

When NOT to Use This Skill

Do NOT use this skill when:

  • Working on unrelated parts of the codebase (extension, webview-ui, etc.)
  • The task is purely about the VS Code extension's core functionality
  • Working on the main website pages that don't involve evals

Key Disambiguation: Two "Evals" Locations

This monorepo has two distinct evals-related locations that can cause confusion:

| Component | Path | Purpose |

| --------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- |

| Evals Execution System | packages/evals/ | Core eval infrastructure: CLI, DB schema, Docker configs |

| Evals Management UI | apps/web-evals/ | Next.js app for creating/monitoring eval runs (localhost:3446) |

| Website Evals Page | apps/web-roo-code/src/app/evals/ | Public roocode.com page displaying eval results |

| External Exercises Repo | Roo-Code-Evals | Actual coding exercises (NOT in this monorepo) |

Directory Structure Reference

packages/evals/ - Core Evals Package

packages/evals/
├── ARCHITECTURE.md          # Detailed architecture documentation
├── ADDING-EVALS.md          # Guide for adding new exercises/languages
├── README.md                # Setup and running instructions
├── docker-compose.yml       # Container orchestration
├── Dockerfile.runner        # Runner container definition
├── Dockerfile.web           # Web app container
├── drizzle.config.ts        # Database ORM config
├── src/
│   ├── index.ts             # Package exports
│   ├── cli/                 # CLI commands for running evals
│   │   ├── runEvals.ts      # Orchestrates complete eval runs
│   │   ├── runTask.ts       # Executes individual tasks in containers
│   │   ├── runUnitTest.ts   # Validates task completion via tests
│   │   └── redis.ts         # Redis pub/sub integration
│   ├── db/
│   │   ├── schema.ts        # Database schema (runs, tasks)
│   │   ├── queries/         # Database query functions
│   │   └── migrations/      # SQL migrations
│   └── exercises/
│       └── index.ts         # Exercise loading utilities
└── scripts/
    └── setup.sh             # Local macOS setup script

apps/web-evals/ - Evals Management Web App

apps/web-evals/
├── src/
│   ├── app/
│   │   ├── page.tsx         # Home page (runs list)
│   │   ├── runs/
│   │   │   ├── new/         # Create new eval run
│   │   │   └── [id]/        # View specific run status
│   │   └── api/runs/        # SSE streaming endpoint
│   ├── actions/             # Server actions
│   │   ├── runs.ts          # Run CRUD operations
│   │   ├── tasks.ts         # Task queries
│   │   ├── exercises.ts     # Exercise listing
│   │   └── heartbeat.ts     # Controller health checks
│   ├── hooks/               # React hooks (SSE, models, etc.)
│   └── lib/                 # Utilities and schemas

apps/web-roo-code/src/app/evals/ - Public Website Evals Page

apps/web-roo-code/src/app/evals/
├── page.tsx      # Fetches and displays public eval results
├── evals.tsx     # Main evals display component
├── plot.tsx      # Visualization component
└── types.ts      # EvalRun type (extends packages/evals types)

This page displays eval results on the public roocode.com website. It imports types from @roo-code/evals but does NOT run evals.

Architecture Overview

The evals system is a distributed evaluation platform that runs AI coding tasks in isolated VS Code environments:

┌─────────────────────────────────────────────────────────────┐
│  Web App (apps/web-evals)  ──────────────────────────────── │
│        │                                                    │
│        ▼                                                    │
│  PostgreSQL ◄────► Controller Container                     │
│        │               │                                    │
│        ▼               ▼                                    │
│     Redis ◄───► Runner Containers (1-25 parallel)           │
└─────────────────────────────────────────────────────────────┘

Key components:

  • Controller: Orchestrates eval runs, spawns runners, manages task queue (p-queue)
  • Runner: Isolated Docker container with VS Code + Roo Code extension + language runtimes
  • Redis: Pub/sub for real-time events (NOT task queuing)
  • PostgreSQL: Stores runs, tasks, metrics

Common Tasks Quick Reference

Adding a New Eval Exercise

  • Add exercise to Roo-Code-Evals repo (external)
  • See packages/evals/ADDING-EVALS.md for structure

Modifying Eval CLI Behavior

Edit files in packages/evals/src/cli/:

  • runEvals.ts - Run orchestration
  • runTask.ts - Task execution
  • runUnitTest.ts - Test validation

Modifying the Evals Web Interface

Edit files in apps/web-evals/src/:

  • app/runs/new/new-run.tsx - New run form
  • actions/runs.ts - Run server actions

Modifying the Public Evals Display Page

Edit files in apps/web-roo-code/src/app/evals/:

  • evals.tsx - Display component
  • plot.tsx - Charts

Database Schema Changes

  • Edit packages/evals/src/db/schema.ts
  • Generate migration: cd packages/evals && pnpm drizzle-kit generate
  • Apply migration: pnpm drizzle-kit migrate

Running Evals Locally

# From repo root
pnpm evals

# Opens web UI at http://localhost:3446

Ports (defaults):

  • PostgreSQL: 5433
  • Redis: 6380
  • Web: 3446

Testing

# packages/evals tests
cd packages/evals && npx vitest run

# apps/web-evals tests
cd apps/web-evals && npx vitest run

Key Types/Exports from @roo-code/evals

The package exports are defined in packages/evals/src/index.ts:

  • Database queries: getRuns, getTasks, getTaskMetrics, etc.
  • Schema types: Run, Task, TaskMetrics
  • Used by both apps/web-evals and apps/web-roo-code

Other skills for the same job

different authors, same section of the catalogue
Azure Kubernetes Automatic Readiness
by microsoft
vendor ×3

Assess Kubernetes workloads and cluster configuration for AKS Automatic compatibility. Identifies incompatibilities, generates fixes, and guides migration from AKS Standard to AKS Automatic. WHEN: migrate to AKS Automatic, check AKS Automatic readiness, validate manifests for Automatic, assess cluster for Automatic compatibility, fix deployment for Automatic compatibility, identify AKS Automatic migration blockers, is my cluster ready for AKS Automatic.

13k tokens
Capacity
by microsoft
vendor ×3

Discovers available Azure OpenAI model capacity across regions and projects. Analyzes quota limits, compares availability, and recommends optimal deployment locations based on capacity requirements. USE FOR: find capacity, check quota, where can I deploy, capacity discovery, best region for capacity, multi-project capacity search, quota analysis, model availability, region comparison, check TPM availability. DO NOT USE FOR: actual deployment (hand off to preset or customize after discovery), quota increase requests (direct user to Azure Portal), listing existing deployments.

6k tokens scripts
Customize
by microsoft
vendor ×3

Interactive guided deployment flow for Azure OpenAI models with full customization control. Step-by-step selection of model version, SKU (GlobalStandard/Standard/ProvisionedManaged), capacity, RAI policy (content filter), and advanced options (dynamic quota, priority processing, spillover). USE FOR: custom deployment, customize model deployment, choose version, select SKU, set capacity, configure content filter, RAI policy, deployment options, detailed deployment, advanced deployment, PTU deployment, provisioned throughput. DO NOT USE FOR: quick deployment to optimal region (use preset).

8k tokens
Deploy Model
by microsoft
vendor ×3

Unified Azure OpenAI model deployment skill with intelligent intent-based routing. Handles quick preset deployments, fully customized deployments (version/SKU/capacity/RAI policy), and capacity discovery across regions and projects. USE FOR: deploy model, deploy gpt, create deployment, model deployment, deploy openai model, set up model, provision model, find capacity, check model availability, where can I deploy, best region for model, capacity analysis. DO NOT USE FOR: listing existing deployments (use foundry_models_deployments_list MCP tool), deleting deployments, agent creation (use agent/create), project creation (use project/create).

26k tokens scripts
Preset
by microsoft
vendor ×3

Intelligently deploys Azure OpenAI models to optimal regions by analyzing capacity across all available regions. Automatically checks current region first and shows alternatives if needed. USE FOR: quick deployment, optimal region, best region, automatic region selection, fast setup, multi-region capacity check, high availability deployment, deploy to best location. DO NOT USE FOR: custom SKU selection (use customize), specific version selection (use customize), custom capacity configuration (use customize), PTU deployments (use customize).

9k tokens
Lamindb
by christophacham
×3

This skill should be used when working with LaminDB, an open-source data framework for biology that makes data queryable, traceable, reproducible, and FAIR. Use when managing biological datasets (scRNA-seq, spatial, flow cytometry, etc.), tracking computational workflows, curating and validating data with biological ontologies, building data lakehouses, or ensuring data lineage and reproducibility in biological research. Covers data management, annotation, ontologies (genes, cell types, diseases, tissues), schema validation, integrations with workflow managers (Nextflow, Snakemake) and MLOps platforms (W&B, MLflow), and deployment strategies.

22k tokens
Latchbio Integration
by christophacham
×3

Latch platform for bioinformatics workflows. Build pipelines with Latch SDK, @workflow/@task decorators, deploy serverless workflows, LatchFile/LatchDir, Nextflow/Snakemake integration.

12k tokens
Modal
by christophacham
×3

Run Python code in the cloud with serverless containers, GPUs, and autoscaling. Use when deploying ML models, running batch processing jobs, scheduling compute-intensive tasks, or serving APIs that require GPU acceleration or dynamic scaling.

17k tokens

How to use it

Copy the folder

Take foryourhealth111-pixel/evals-context from the repository into ~/.claude/skills for personal use, or into .claude/skills inside a project.

Check the name does not clash

The agent identifies a skill by the name field in its header. Two skills with the same name cannot sit side by side — one of them will be ignored.

Install what it needs

The instructions reference npx. Without those the skill loads but fails at the first command.