Chapter 10
Close the feedback loop
Updated
Convert corrections into durable improvements.
Read
- Building self-improving tax agents with Codex How practitioner corrections and production traces become evaluation cases.
- Demystifying evals for AI agents Task, trial, grader, trace, and outcome: a vocabulary for evidence-driven delivery.Focus: Read tasks, trials, graders, transcripts, outcomes, and repeated-run metrics. Apply: Build a release report with outcome checks, multiple trials where appropriate, and unresolved failure slices.
- Effective context engineering for AI agents Select and refresh useful context as a workflow grows.Focus: Read just-in-time retrieval and techniques for long-running tasks, including compaction and structured notes. Apply: Record which evidence a task needs, when it is loaded, and what must survive compaction.
Do
- Classify reported failures as data, model, integration, policy, or UX issues.
- Turn three real failure reports into repeatable evaluation cases.
- Ship one improvement and check for regressions on held-out cases.
You are done when: A traceable chain from user feedback to a tested change.