Opening
The problem starts after good retrieval
Every other chapter in this book is about getting the right information into the context window. This one starts where that story breaks.
Air Canada’s support chatbot had the bereavement-fare policy in its context. The correct text was retrieved. The model read it — and fabricated a retroactive fare-adjustment program that did not exist. A customer relied on the fabrication. A tribunal held the airline liable.
The document was right there. The model answered from somewhere else.
Now consider a different failure. A support bot’s instructions say: “Use only the customer’s current plan details. Do not mention other plans.” The bot pulls the right CRM record and formats a clean answer — then adds “Customers on our Premium tier enjoy priority routing for $49/month.” Every fact in that sentence may be true. The error is not fabrication. The model violated an explicit scope constraint.
These look similar from the outside — “the model ignored the context.” They are actually different problems. The Air Canada case is a groundedness failure: the model went beyond its evidence. The Premium-plan case is an instruction-following failure: the model broke a rule it was given. They have different causes, they need different metrics, and they require different fixes. If you don’t separate them, you end up in a whack-a-mole loop where fixing one makes the other worse.
One decomposition runs through the rest of this chapter. The assembled context carries requirements (rules the output must satisfy) and evidence (facts the output is allowed to use). Instruction following measures whether the output satisfied the requirements. Groundedness measures whether the output stayed inside the evidence. The diagnosis loop and control stack that follow both depend on knowing which obligation broke first.
The next section makes this decomposition precise.
Mental model
Two obligations, one answer
Every response owes two distinct kinds of faithfulness to its assembled context. Conflating them guarantees that your fixes create new bugs.
- Instruction following. Did the answer obey the rules—format constraints, scope limits, tool-use policies, ranking preferences, refusal conditions? This is a compliance check against explicit directives.
- Groundedness. Did the answer stay inside the local evidence—retrieved passages, tool outputs, database rows, verified state? This is a support check against provided facts.
A model can ace one obligation while failing the other. It can produce a perfectly grounded answer that violates the required format. It can satisfy every visible instruction and still slip in a single unsupported claim. If you treat both failures with the same fix — rewrite the prompt, usually — you oscillate between them indefinitely.
Understanding why these failures happen changes how you fix them. Language models are trained to be helpful — RLHF rewards completeness, fluency, and informativeness. When the evidence runs out, the model faces a choice: stop (which feels unhelpful) or fill the gap from parametric memory (which sounds better). The training signal overwhelmingly favors the second option. Groundedness failures are, in large part, helpfulness succeeding where it should not. Instruction-following failures have a related cause: parametric memory and contextual instructions compete for influence. When they conflict — the context says “do not mention other plans” but training says mentioning related products is helpful — the model may follow whichever signal is stronger.
Both failures get worse as context grows. Research on lost-in-the-middle effects (Liu et al., 2024) shows that models attend strongly to the beginning and end of context but poorly to the middle — a dead zone where instructions get dropped and evidence gets ignored. The needle-in-a-haystack test — can the model find a specific fact buried in a long document? — is a groundedness challenge in disguise. If your system is failing, one of the first diagnostic checks is whether the critical information sits in the attention dead zone.
Diagnosis loop
Four questions before you change anything
Knowing the two obligations exist is the mental model. These four questions turn it into a diagnosis.
- Which contract broke? Dropped rule, unsupported claim, or both.
- Was the needed context absent, present but buried, or present and ignored? Each points to a different component.
- Where did the failure originate? In the generator, or upstream in retrieval, reranking, or tool execution?
- What is the smallest measurable unit? A single requirement, a single claim, a citation link, or an abstention decision.
Observed failure. The answer cites the correct refund policy but adds “refunds usually land within five business days.” The source document never states a timeline.
- Contract broken: groundedness. The model fabricated a detail.
- Context status: present but incomplete — the policy was retrieved, but it contained no turnaround time. The model filled the gap from pre-training knowledge. (Also check: was the policy buried mid-context where attention is weakest?)
- Origin: generator. Retrieval did its job; generation extrapolated beyond the evidence.
- Measurable unit: one claim (“five business days”) with zero supporting sentences in the evidence pack.
- First intervention: tighter evidence representation plus a claim-level verification step—not a new formatting prompt.
Minimum viable evals
The smallest suite that tells the truth
The diagnosis loop tells you what failed on a single output. Evals tell you whether it is getting better or worse across your system.
Three outputs constitute the minimum viable eval suite for context adherence.
- Requirement satisfaction. Convert the response contract into natural-language unit tests. Score each requirement independently. A single “overall quality” label hides exactly the information you need. Measure as: satisfied requirements / total requirements.
- Groundedness. Decompose the answer into atomic claims. For each claim, check whether at least one passage in the evidence pack supports it. Measure as: supported claims / total claims.
- Failure map. Tally the dominant failure categories across your sample: dropped constraints, unsupported claims, broken citations, incorrect abstentions, upstream ranking errors. The distribution tells you which layer of the stack to fix first.
Nothing fancier is needed to start. The critical discipline is keeping these as separate numbers. A single blended quality score averages away the signal you are trying to find.
| Metric | Unit of measurement | Points you toward |
|---|---|---|
| Requirement satisfaction | Prompt criterion | Prompt design, output schema, deterministic enforcement |
| Groundedness | Atomic claim | Retrieval quality, evidence packing, verification pipeline |
| Abstention accuracy | Answer-or-refuse decision | Policy design, refusal prompting, tuned default behavior |
Model output. “Our refund policy allows returns within 30 days. Refunds are processed to the original payment method and typically arrive within 5–7 business days.”
Step 1: split into atomic claims. Each claim should be independently verifiable — one fact per unit.
- “The refund policy allows returns within 30 days.”
- “Refunds are processed to the original payment method.”
- “Refunds typically arrive within 5–7 business days.”
Step 2: check each claim against the evidence pack.
| Claim | Evidence | Verdict |
|---|---|---|
| Returns within 30 days | Policy doc §2: “Customers may return items within 30 calendar days.” | Supported |
| Processed to original payment method | Policy doc §3: “Refunds will be credited to the original form of payment.” | Supported |
| Arrive within 5–7 business days | No timeline stated anywhere in the evidence pack. | Unsupported |
Groundedness = 2/3 = 0.67. The output is mostly correct, but the timeline claim came from parametric memory, not the provided evidence. A holistic “looks good” review would likely miss this. Claim-level decomposition catches it mechanically.
The pattern behind effective adherence evaluation is always the same: decompose before scoring. Treat each requirement like a unit test and score pass/fail independently. Decompose answers into atomic claims and check each against the evidence. A single blended quality score will always hide exactly the signal you need.
Control stack
What to try today, this sprint, and this quarter
Now you know what failed and how to measure it. The control stack tells you what to fix, in what order.
| Layer | Typical owner | Latency / cost | Reach for it when |
|---|---|---|---|
| Model choice | Platform / ML lead | Low migration cost, zero extra latency | The base model fights the task on most inputs |
| Context representation | Application engineer | Low-to-medium engineering cost | Evidence is noisy, stale, or badly ordered |
| Prompt & output design | Application engineer | Low cost, usually zero added latency | Rules are local and salience-dependent |
| Verification / orchestration | ML engineer + platform | Medium cost, adds inference latency | Single-pass generation is too error-prone for the stakes |
| SFT / adapters | ML team | Training cost, minimal runtime overhead | Same failure persists despite prompt and system fixes |
| RL with verifiable rewards | Research / post-training | Highest data and compute cost | The default tradeoff itself is wrong (e.g., completeness over support) |
Applying the framework
Practical starting moves
Three concrete actions that pay off immediately — before you touch the model or redesign the pipeline.
1. Build your first eval set from real failures. Pull 20–50 actual failures from production or staging. For each one, name which contract broke and write down the specific requirement that was dropped or the specific claim that lacked support. Those labeled traces become your eval set, your regression suite, and eventually your training data. Synthetic examples are a distant second choice because they miss the failure patterns unique to your system.
2. Fix position effects before rewriting prompts. Move critical rules to the beginning or end of the prompt — away from the attention dead zone. Strip stale or duplicated evidence. Add a structured output schema for format requirements. Add one deterministic guard (a regex, a schema check, a keyword filter) for your most expensive failure class. These changes cost almost nothing and often resolve a surprising fraction of failures without any model changes.
3. Instrument every prompted stage. Log inputs and outputs at retrieval, reranking, generation, and verification. Without this, you cannot tell whether a failure originated upstream or in the generator — and fixing the wrong component is worse than fixing nothing.
Knowing when to climb
The signal that tells you prompting is not enough
The hardest judgment call in the control stack is not which layer to try. It is when to stop trying the current one.
You have already cleaned the evidence pack, made the response contract explicit with structured prompts and enumerated requirements, added a verification pass for the most expensive failure class — and the same unsupported completion or rule break still recurs across held-out traces and alternative phrasings.
The diagnostic question: is this failure prompt-sensitive (different phrasing fixes it) or default-behavior-sensitive (the model makes the same mistake regardless)? If you have tried three materially different prompt strategies and the failure persists, the model’s priors are the problem.
Escalate to SFT when the default behavior is wrong for your task — the model keeps reaching for parametric knowledge instead of stopping at the evidence boundary. SFT changes what “normal” looks like.
Escalate to RL when the tradeoff itself is miscalibrated — the model still prefers sounding complete over staying supported, or still answers when the correct move is to abstain. These are reward-shaping problems, not instruction problems.
Summary
What you should now be able to do
One sentence to carry away: separate dropped rules from unsupported claims, measure them with different instruments, and climb the control stack in cost order.
Full playbook: diagnose before rewriting prompts. Track requirement satisfaction and groundedness as independent numbers. Check for position effects before blaming the model. Fix the cheapest layer that plausibly owns the failure. Escalate to tuning only when recurring failures reveal that the model’s defaults — not the current prompt — are the actual problem.
You should be able to
Look at a bad model output and name whether it is an instruction-following break, a groundedness break, or an upstream failure. Stand up a minimum eval suite with two separate metrics. Explain why rewriting the prompt is not always the right first move.
The key tension to carry forward
Completeness and correctness pull in opposite directions. Pushing for thoroughness invites fabrication; pushing for precision invites excessive refusal. The control stack exists so you can place the system deliberately on that boundary, starting with the cheapest intervention that moves the right metric.
If you cannot name the unit that failed and the component that owns it, you do not have a diagnosis. You have a complaint. The gap between the two is exactly where debugging time disappears.