Chapter 16
Find the first broken step in retrieval
Updated
A convincing answer cites the wrong policy despite an apparently strong language model.
Inspect
Check source coverage, access eligibility, freshness, chunk meaning, candidate retrieval, reranking, and final answer support in order. Test exact identifiers separately from broad semantic queries.
Decide
Measure evidence retrieval independently of generation. Compare a simple lexical or hybrid baseline before adding a larger model or a more autonomous search loop.
Bring to a review
A failure ledger that points each bad answer to the earliest failed stage and the correct source passage.
Common trap: Improving answer fluency can conceal that the evidence was never available to the model.
Read
- Contextual Retrieval Chunk context, lexical and semantic retrieval, and reranking.Focus: Read the ambiguous-chunk example, hybrid lexical/semantic retrieval, and reranking evaluation. Apply: Compare recall on exact identifiers versus semantic queries, holding the evaluation set fixed.
- Effective context engineering for AI agents Select and refresh useful context as a workflow grows.Focus: Read just-in-time retrieval and techniques for long-running tasks, including compaction and structured notes. Apply: Record which evidence a task needs, when it is loaded, and what must survive compaction.