Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).
npxskills add roocodeinc/roo-code--skill evals-contextLoading…
Use this skill when the task involves:
Do NOT use this skill when:
This monorepo has two distinct evals-related locations that can cause confusion:
| Component | Path | Purpose |
|---|---|---|
| Evals Execution System | packages/evals/ | Core eval infrastructure: CLI, DB schema, Docker configs |
| Evals Management UI | apps/web-evals/ | Next.js app for creating/monitoring eval runs (localhost:3446) |
| Website Evals Page | apps/web-roo-code/src/app/evals/ | Public roocode.com page displaying eval results |
| External Exercises Repo | Roo-Code-Evals | Actual coding exercises (NOT in this monorepo) |
packages/evals/ - Core Evals Packagepackages/evals/
├── ARCHITECTURE.md # Detailed architecture documentation
├── ADDING-EVALS.md # Guide for adding new exercises/languages
├── README.md # Setup and running instructions
├── docker-compose.yml # Container orchestration
├── Dockerfile.runner # Runner container definition
├── Dockerfile.web # Web app container
├── drizzle.config.ts # Database ORM config
├── src/
│ ├── index.ts # Package exports
│ ├── cli/ # CLI commands for running evals
│ │ ├── runEvals.ts # Orchestrates complete eval runs
│ │ ├── runTask.ts # Executes individual tasks in containers
│ │ ├── runUnitTest.ts # Validates task completion via tests
│ │ └── redis.ts # Redis pub/sub integration
│ ├── db/
│ │ ├── schema.ts # Database schema (runs, tasks)
│ │ ├── queries/ # Database query functions
│ │ └── migrations/ # SQL migrations
│ └── exercises/
│ └── index.ts # Exercise loading utilities
└── scripts/
└── setup.sh # Local macOS setup script
apps/web-evals/ - Evals Management Web Appapps/web-evals/
├── src/
│ ├── app/
│ │ ├── page.tsx # Home page (runs list)
│ │ ├── runs/
│ │ │ ├── new/ # Create new eval run
│ │ │ └── [id]/ # View specific run status
│ │ └── api/runs/ # SSE streaming endpoint
│ ├── actions/ # Server actions
│ │ ├── runs.ts # Run CRUD operations
│ │ ├── tasks.ts # Task queries
│ │ ├── exercises.ts # Exercise listing
│ │ └── heartbeat.ts # Controller health checks
│ ├── hooks/ # React hooks (SSE, models, etc.)
│ └── lib/ # Utilities and schemas
apps/web-roo-code/src/app/evals/ - Public Website Evals Pageapps/web-roo-code/src/app/evals/
├── page.tsx # Fetches and displays public eval results
├── evals.tsx # Main evals display component
├── plot.tsx # Visualization component
└── types.ts # EvalRun type (extends packages/evals types)
This page displays eval results on the public roocode.com website. It imports types from @roo-code/evals but does NOT run evals.
The evals system is a distributed evaluation platform that runs AI coding tasks in isolated VS Code environments:
┌─────────────────────────────────────────────────────────────┐
│ Web App (apps/web-evals) ──────────────────────────────── │
│ │ │
│ ▼ │
│ PostgreSQL ◄────► Controller Container │
│ │ │ │
│ ▼ ▼ │
│ Redis ◄───► Runner Containers (1-25 parallel) │
└─────────────────────────────────────────────────────────────┘
Key components:
packages/evals/ADDING-EVALS.md for structureEdit files in packages/evals/src/cli/:
runEvals.ts - Run orchestrationrunTask.ts - Task executionrunUnitTest.ts - Test validationEdit files in apps/web-evals/src/:
app/runs/new/new-run.tsx - New run formactions/runs.ts - Run server actionsEdit files in apps/web-roo-code/src/app/evals/:
packages/evals/src/db/schema.tscd packages/evals && pnpm drizzle-kit generatepnpm drizzle-kit migrate# From repo root
pnpm evals
# Opens web UI at http://localhost:3446
Ports (defaults):
# packages/evals tests
cd packages/evals && npx vitest run
# apps/web-evals tests
cd apps/web-evals && npx vitest run
@roo-code/evalsThe package exports are defined in packages/evals/src/index.ts:
getRuns, getTasks, getTaskMetrics, etc.Run, Task, TaskMetricsapps/web-evals and apps/web-roo-code
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).