Chapter 6
Evaluate before expanding
Updated
Make failures visible and reproducible.
Read
- Demystifying evals for AI agents Task, trial, grader, trace, and outcome: a vocabulary for evidence-driven delivery.Focus: Read tasks, trials, graders, transcripts, outcomes, and repeated-run metrics. Apply: Build a release report with outcome checks, multiple trials where appropriate, and unresolved failure slices.
- Building self-improving tax agents with Codex How practitioner corrections and production traces become evaluation cases.
- AI Evals: Everything You Need to Know Error discovery, representative sampling, and validated automated judges.Focus: Start with error analysis, sampling traces, stale datasets, and validating an LLM judge against expert labels. Apply: Create a labeled failure taxonomy and report judge false accepts and false rejects on held-out examples.
Do
- Collect representative cases, edge cases, and unacceptable failures.
- Separate a development set from a held-out set and define graders.
- Compare the new workflow with a baseline on quality, latency, and cost.
You are done when: A versioned evaluation report with failure categories and release criteria.