Chapter 17
Lower token cost can raise cost per successful task
Updated
A cheaper model needs more retries and longer human review.
Inspect
Count inference, retrieval, infrastructure, review, rework, support, and amortized implementation. Use accepted task outcomes as the denominator; include failed attempts in the numerator.
Decide
Compare alternatives under the same quality and workload requirements. Separate cash savings, capacity released, and time that users actually redeploy to valuable work.
Bring to a review
A sensitivity table for task volume, success rate, review minutes, and support hours; use ranges rather than an unsupported ROI point estimate.
Common trap: Multiplying every minute saved by a salary rate does not establish realized financial savings.
Read
- AI Engineering BookA broad reference for model selection, adaptation, evaluation, and serving.Focus: Chapters 3–4 for evaluation; 6 for RAG/agents; 7 for fine-tuning; 8–10 for data, serving, and feedback. Apply: Build a decision table: prompt change, retrieval change, fine-tuning, or no model; include the evidence required.
- Eliminating Toil · The Site Reliability Workbook BookMeasure operational work and choose worthwhile automation.Focus: Read identifying, quantifying, and prioritizing toil; inspect the case examples. Apply: Record support time per successful task and compare automation cost with recurring manual effort.
- Online Controlled Experiments at Large Scale PaperExperiment trustworthiness, surprising results, and organizational decision making.Focus: Read organizational adoption, experiment interactions, and skepticism toward surprising results. Apply: Specify the comparison, unit of assignment, guardrails, and what would invalidate a claimed lift.