Opening
The problem starts after good retrieval
Every other chapter in this book is about getting the right information into the context window. This one starts where that story breaks.
Air Canada’s support chatbot had the bereavement-fare policy in its context. The correct text was retrieved. The model read it — and fabricated a retroactive fare-adjustment program that did not exist. A customer relied on the fabrication. A tribunal held the airline liable.
The document was right there. The model answered from somewhere else.
Now consider a different failure. A support bot’s instructions say: “Use only the customer’s current plan details. Do not mention other plans.” The bot pulls the right CRM record and formats a clean answer — then adds “Customers on our Premium tier enjoy priority routing for $49/month.” Every fact in that sentence may be true. The error is not fabrication. The model violated an explicit scope constraint.
These look similar from the outside — “the model ignored the context.” They are actually different problems. The Air Canada case is a groundedness failure: the model went beyond its evidence. The Premium-plan case is an instruction-following failure: the model broke a rule it was given. They have different causes, they need different metrics, and they require different fixes. If you don’t separate them, you end up in a whack-a-mole loop where fixing one makes the other worse.
One decomposition runs through the rest of this chapter. The assembled context carries requirements (rules the output must satisfy) and evidence (facts the output is allowed to use). Instruction following measures whether the output satisfied the requirements. Groundedness measures whether the output stayed inside the evidence. The diagnosis loop and control stack that follow both depend on knowing which obligation broke first.
Mental model
Two contracts, one context window
The assembled context is not one undifferentiated blob. It imposes two distinct obligations on the output.
By the time the model generates, it is looking at system instructions, a user query, retrieved passages, tool outputs, conversation history, memory, intermediate plans — sometimes verifier outputs too. All of that folds into one token sequence. But it does two jobs.
- Requirement-bearing context tells the model what the answer must look like: format constraints, scope limits, tool-use policies, ordering rules, refusal conditions.
- Evidence-bearing context tells the model what the answer is allowed to know: retrieved documents, database rows, tool return values, verified sub-results.
Break the first contract and you get an instruction-following failure. Break the second and you get a groundedness failure. A model can ace one obligation while failing the other. It can produce a perfectly grounded answer that violates the required format. It can satisfy every visible instruction and still slip in a single unsupported claim. These failures can originate at any stage of the pipeline: a reranker that ignores “prefer recent internal policy” can cause the generator to hallucinate even when generation itself is blameless.
Different products sit at different operating points. A policy bot may want very strict groundedness and moderate instruction following. A coding assistant may allow broader world knowledge but demand near-perfect instruction following. A creative tool may tolerate weak groundedness if the output is imaginative. The framework is the same; the acceptable failure profile is not.
Why this happens
RLHF is the direct cause of most groundedness failures in deployed systems. The reward model learned during RLHF training scores complete, fluent answers higher than bounded ones — so the model learned to fill gaps rather than flag them. When the evidence runs out, the model faces a choice: stop (which feels unhelpful and gets penalized during training) or fill the gap from parametric memory (which sounds better and gets rewarded). The training signal overwhelmingly favors the second option. This is not a reasoning error; it is the correct optimization of the wrong objective for your task. Groundedness failures are, in large part, helpfulness succeeding where it should not.
Instruction-following failures have a related but distinct cause. Parametric memory and contextual instructions compete for influence over generation. When they conflict — the context says “do not mention other plans” but the model’s training has seen thousands of examples where mentioning related products is helpful — the model may follow whichever signal is stronger. For well-rehearsed behaviors, parametric memory often wins. The model is not ignoring your instructions; it is being outweighed by thousands of training examples that say the opposite.
Both failures get worse as context grows. The lost-in-the-middle phenomenon (Liu et al., 2024) shows that models attend most to the beginning and end of their context window, creating a dead zone in the middle. This affects both contracts: instructions placed mid-context get dropped, and evidence placed mid-context gets ignored. The needle-in-a-haystack problem — whether a model can locate and use a specific fact buried in a long document — is a groundedness challenge in disguise. If your system is failing, one of the first things to check is whether the critical information sits in the attention dead zone.
Let x = (q, e) where q is the requirement bundle and e is the evidence bundle. The model returns response y. Two capabilities:
Diagnosis loop
Four questions before you change anything
The instinct is to rewrite the prompt. Resist it. The two-contract model only helps if you use it to diagnose before you intervene.
- Which contract broke? Dropped rule, unsupported claim, or both.
- Was the needed context absent, present but buried, or present and ignored? Each points to a different component.
- Where did the failure originate? In the generator, or upstream in retrieval, reranking, or tool execution?
- What is the smallest measurable unit? A single requirement, a single claim, a citation link, or an abstention decision.
Prompt. “Answer in JSON. Use exactly three bullets in steps. Cite the supporting policy line for each bullet. Do not mention plans other than the customer’s current plan.”
Output. Valid JSON, three bullets, citations present — but one bullet reads: “Customers on Premium pay $49/month.”
| Check | Pass? | Verdict |
|---|---|---|
| Requirement: valid JSON | Yes | Format contract satisfied. |
| Requirement: exactly three bullets | Yes | Structure contract satisfied. |
| Requirement: no other plans | No | Instruction-following failure. This is the bug. |
| Claim: Premium costs $49/month | Maybe true | Irrelevant — the requirement contract failed first. |
Decomposition tells you which axis owns the failure. A groundedness fix would not help here. A scope-enforcement fix would.
Observed failure. The answer cites the correct refund policy but adds “refunds usually land within five business days.” The source document never states a timeline.
- Contract broken: groundedness. The model fabricated a detail.
- Context status: present but incomplete — the policy was retrieved, but it contained no turnaround time. The model filled the gap from pre-training knowledge. (Also check: was the policy buried mid-context where attention is weakest?)
- Origin: generator. Retrieval did its job; generation extrapolated beyond the evidence.
- Measurable unit: one claim (“five business days”) with zero supporting sentences in the evidence pack.
- First intervention: tighter evidence representation plus a claim-level verification step — not a new formatting prompt.
Measuring adherence
The smallest eval suite that tells the truth
The diagnosis loop tells you what failed on a single output. Evals tell you whether it is getting better or worse across your system.
Three outputs constitute the minimum viable eval suite for context adherence.
- Requirement satisfaction. Convert the response contract into natural-language unit tests. Score each requirement independently. A single “overall quality” label hides exactly the information you need. Measure as: satisfied requirements / total requirements.
- Groundedness. Decompose the answer into atomic claims. For each claim, check whether at least one passage in the evidence pack supports it. Measure as: supported claims / total claims.
- Abstention correctness. Did the model answer when it could and refuse when it should? Tally correct abstain-or-answer decisions across your sample.
Nothing fancier is needed to start. The critical discipline is keeping these as separate numbers. A single blended quality score averages away the signal you are trying to find.
| Metric | Unit of measurement | Points you toward |
|---|---|---|
| Requirement satisfaction | Prompt criterion | Prompt design, output schema, deterministic enforcement |
| Groundedness | Atomic claim | Retrieval quality, evidence packing, verification pipeline |
| Abstention correctness | Answer-or-refuse decision | Policy design, refusal prompting, tuned default behavior |
Model output. “Our refund policy allows returns within 30 days. Refunds are processed to the original payment method and typically arrive within 5–7 business days.”
Step 1: split into atomic claims. Each claim should be independently verifiable — one fact per unit.
- “The refund policy allows returns within 30 days.”
- “Refunds are processed to the original payment method.”
- “Refunds typically arrive within 5–7 business days.”
Step 2: check each claim against the evidence pack.
| Claim | Evidence | Verdict |
|---|---|---|
| Returns within 30 days | Policy doc §2: “Customers may return items within 30 calendar days.” | Supported |
| Processed to original payment method | Policy doc §3: “Refunds will be credited to the original form of payment.” | Supported |
| Arrive within 5–7 business days | No timeline stated anywhere in the evidence pack. | Unsupported |
Groundedness = 2/3 = 0.67. The output is mostly correct, but the timeline claim came from parametric memory, not the provided evidence. A holistic “looks good” review would likely miss this. Claim-level decomposition catches it mechanically.
The pattern behind effective adherence evaluation is always the same: decompose before scoring. Treat each requirement like a unit test and score pass/fail independently (the approach behind LMUnit). Decompose answers into atomic claims and verify each one against the evidence (the approach behind FActScore and FACTS Grounding). The methodology matters more than the specific benchmark: decompose, then score per unit.
Control stack
Six layers, from context cleanup to weight changes
Now you know what failed and how to measure it. The control stack tells you what to fix, in what order.
Lower layers reshape the local state so the model’s existing behavior is more likely to succeed. Upper layers change what the model treats as normal behavior. That distinction matters: prompting asks the model to override its priors at inference time, every time. SFT and RL change which behavior feels locally easiest. The model stops fighting you because the default shifted.
| Layer | What it changes | Owner | Best for |
|---|---|---|---|
| 0. Model choice | Baseline behavior and failure profile | Platform / ML lead | The base model fights the product objective |
| 1. Context representation | Chunking, metadata, source ordering, state visibility | Application engineer | Evidence present but buried or conflicting |
| 2. Prompt and output design | Instruction salience, structure, abstention language | Application engineer | Dropped rules that respond to wording changes |
| 3. Orchestration and verification | Reranking, extract-then-answer, second-pass checks | ML engineer + platform | Recurring claim-level errors and upstream drift |
| 4. SFT / adapters | Default output style and evidence-use habits | ML team | Persistent failure modes that prompting cannot fix |
| 5. RL with verifiable rewards | Default policy under tradeoffs and abstention pressure | Research / post-training | Prompting still fights the weights; reward is checkable |
Most teams skip straight to prompt engineering because it feels productive, then skip straight to fine-tuning because prompt engineering did not work. They never fix the context representation — the ordering, the chunking, the stale passages — which is where most of the actual bugs live. The single highest-ROI intervention in production systems is almost always cleaning up the evidence pack, not rewriting the system prompt. Three patterns from recent benchmarking work reinforce this: instruction-following performance degrades non-linearly as constraints multiply (Jaroslawicz et al., 2025), holistic evaluation is too noisy to explain why (Saad-Falcon et al., 2024), and groundedness fails one claim at a time, not all at once (Min et al., 2023; Jacovi et al., 2025).
Grounded data and SFT
Grounded data is not more documents
The upper layers of the control stack — SFT and RL — need training data. Not just any data. The distinction matters.
“We need grounded data” usually means “more examples with citations.” That is too weak. Useful adherence data has structure: which requirements were active, which claims were licensed, what evidence supported them, and what the correct behavior was when the evidence ran out.
Start from production traces, not a generic instruction corpus. Real traces give you the failures that matter: near-duplicate queries with different retrievals, subtle rule conflicts, stale passages outranking fresh ones, fluent answers hiding one unsupported sentence. A model trained on “helpful assistant” data will never see those cases.
Input. Query: “Can I get a refund on my annual subscription?” Requirements: answer from evidence only; cite the policy section; do not mention other plan tiers.
Evidence pack. Refund policy doc (3 paragraphs): §1 covers eligibility, §2 covers the 30-day window, §3 covers payment method. No paragraph mentions turnaround time.
Artifact 1 — SFT gold answer. “Yes. Per §2 of the refund policy, annual subscriptions may be refunded within 30 calendar days of purchase. The refund will be credited to your original payment method (§3). The policy does not specify a processing timeline.” Every claim maps to a specific section. The gap is acknowledged, not filled.
Artifact 2 — correct abstention. Same query, but the evidence pack contains only §1 (eligibility for monthly plans). Gold answer: “The available policy excerpt covers monthly plans only. I cannot confirm annual subscription refund terms from this evidence.” The model learns to stop where evidence stops.
Artifact 3 — hard negative (evidence perturbation). Same query and requirements, but §2 is replaced with a version from six months ago that says “14 calendar days.” Gold answer uses the 14-day figure. Teaches the model to follow the evidence it actually has, not the answer it remembers.
Artifact 4 — hard negative (rule perturbation). Same query and evidence, but the “do not mention other plan tiers” requirement is moved from position 1 to position 5 in a list of 8 rules. Gold answer still respects it. Teaches the model to follow rules regardless of position.
SFT teaches the model where knowledge is allowed to come from
The core move: build demonstrations where the model answers from the provided evidence, cites what supports each claim, and abstains cleanly when the evidence falls short. Then train the model to imitate those trajectories.
The loss function is standard cross-entropy — nothing new there. What matters is the dataset, not the objective. A model fine-tuned on 10,000 examples of fluent-but-ungrounded answers will become better at fluent-but-ungrounded answers. A model fine-tuned on examples where the response systematically stops at the evidence boundary — citing what is supported, acknowledging what is not — will make that behavior the path of least resistance. For adherence, curation carries more weight than architecture.
Four kinds of demonstrations earn their place in an adherence SFT dataset:
- Positive grounded answers. Fully supported response, correct citations, no overclaiming.
- Correct abstentions. The model narrows its answer or refuses outright because the evidence does not settle the question.
- Counterfactual near-misses. Differs from the gold answer by exactly one dropped rule or one unsupported claim. Teaches the model where the boundary is.
- Stage-specific examples. Retrieval and reranking prompts get their own adherence demonstrations — do not assume the generator can compensate for upstream failures.
Before. “The policy does not specify a refund timeline, but customers usually receive refunds within 5–7 business days.”
After. “The policy excerpt provided does not specify a refund timeline. I cannot state one from this context.”
The model is not less capable. It has learned that “use only the local evidence” is the default for this class of task. The 5–7 day figure came from parametric memory. SFT taught the model to stop reaching for it.
Delete one critical line from the evidence bundle. Move one rule from the top of the prompt to the middle. Attach a citation that is related but insufficient. Swap a fresh internal policy for an older version with the same keywords. These small perturbations mirror how production actually fails. Giant synthetic corruptions do not.
Verifiable RL
RL changes which tradeoff wins by default
SFT teaches what good behavior looks like. RL teaches which tradeoff to make when good behavior in one dimension costs good behavior in the other.
A single scalar reward — “good answer” — is too blurry. It cannot tell the policy whether it lost points for dropping a rule, overclaiming beyond evidence, citing the wrong span, or refusing when it should have answered. If the chapter’s two-contract decomposition means anything, the reward must mirror it.
S_req rewards satisfied requirements. G rewards supported claims. Cite rewards sufficient (not decorative) citation. A rewards the answer/abstain boundary. The penalty term makes unsupported completion expensive.
Setup. Query: “What is the cancellation fee?” Requirements: (1) answer from evidence only, (2) cite the policy section, (3) use bullet format. Evidence: policy doc with §4 stating “$25 cancellation fee for monthly plans.”
Model output. “• The cancellation fee is $25 for monthly plans (§4). • Annual plans are subject to a prorated fee.” (No citation for the second bullet.)
| Component | Score | Reasoning |
|---|---|---|
| S_req (requirement satisfaction) | 2/3 = 0.67 | Bullet format ✓, evidence-only ✗ (second bullet unsupported), citation present ✓ |
| G (groundedness) | 1/2 = 0.50 | Claim 1 supported by §4 ✓, Claim 2 not in evidence ✗ |
| Cite (citation sufficiency) | 0.50 | One of two claims has a valid citation |
| A (abstention) | 0.0 | Should have abstained on annual plan info; did not |
| Unsupported penalty | 1.0 | One fabricated claim present |
With lambdas (0.3, 0.3, 0.1, 0.15, 0.4): r = 0.3(0.67) + 0.3(0.50) + 0.1(0.50) + 0.15(0.0) − 0.4(1.0) = 0.001. Near-zero reward: the fabrication penalty almost cancels the correct parts. A fully grounded one-bullet answer would score higher.
The optimizer choice is secondary. Scalar reward checks and sampled responses point toward PPO- or GRPO-style updates. Pairwise chosen/rejected data from verifiable checks fits DPO. The engineering principle stays constant: the reward must come from things you can check locally and repeat reliably.
To be direct: most teams reading this chapter should not be doing RL for adherence. If you have not built a grounded eval suite, have not cleaned your evidence packs, and have not tried SFT with a few hundred well-curated demonstrations, RL will optimize the wrong thing expensively. RL is for the team that has done all of the above and can still point to a specific, measured failure that persists. That team is rarer than the RL hype suggests.
| Signal | What it means | Response |
|---|---|---|
| Prompt edits help briefly, then regress | Wording is not the bottleneck | Shift default behavior via SFT or RL |
| Same unsupported claims recur in same domains | Evidence precedence is not the learned default | Add claim-verifiable and abstention rewards |
| Good eval exists but generation ignores it | You know what “correct” looks like; the policy does not | Use evaluator output as training signal |
| Hard constraints fail under long instruction bundles | The model needs stronger internal prioritization | Pair requirement-level rewards with adapters |
Instruction following and groundedness pull in opposite directions. Reward helpful completeness too aggressively and the model fills gaps with parametric guesses. Punish unsupported claims too harshly and the model becomes uselessly cautious. RL is not the first move because it is expensive. It is the upper-stack move because it changes what the model prefers to do when helpfulness, completeness, and local faithfulness conflict. That is categorically different from making the right behavior easier to notice at inference time. Verifiable reward design lets you place the system on that boundary deliberately — using checks you can audit, not intuitions you cannot.
Practical moves
What to do on Monday morning
The preceding sections give you the theory. Here is the operational sequence, ordered by speed and cost.
- Collect 20 real failures or near-misses from production or staging.
- Label each with the first failed contract: rules, evidence, both, or upstream stage drift.
- Extract requirement checks and claim checks from those traces — these become your first eval set.
- Move the highest-cost rule to the beginning of the prompt and strip stale evidence from the context pack.
- Add one deterministic check (regex, schema, keyword filter) for your single most expensive failure class.
- Stand up requirement satisfaction, groundedness, and abstention correctness as separate, automated metrics. Run them on every deploy.
- Instrument every prompted stage that can corrupt the final answer: retrieval, reranking, generation, verification.
- Mine near-miss examples — traces where only one rule breaks or one claim drifts — because these become your best hard cases for both eval and eventual training data.
- Decide whether the dominant failure is prompt-sensitive (different phrasing fixes it) or default-behavior-sensitive (the model makes the same mistake regardless of how you ask). This is the escalation decision.
You have already cleaned the evidence pack, made the response contract explicit with structured prompts and enumerated requirements, added a verification pass for the most expensive failure class — and the same unsupported completion or rule break still recurs across held-out traces and alternative phrasings.
The diagnostic question: is this failure prompt-sensitive or default-behavior-sensitive? If you have tried three materially different prompt strategies and the failure persists, the model’s priors are the problem.
Escalate to SFT when the default behavior is wrong for your task — the model keeps reaching for parametric knowledge instead of stopping at the evidence boundary. SFT changes what “normal” looks like.
Escalate to RL when the tradeoff itself is miscalibrated — the model still prefers sounding complete over staying supported, or still answers when the correct move is to abstain. These are reward-shaping problems, not instruction problems.
Summary
What you should now be able to do
The sentence to carry forward: when the model has the right context and still fails, separate dropped rules from unsupported claims, measure them with different instruments, and climb the control stack in cost order.
The mental model is compact. The assembled context imposes two contracts: a requirement contract (obey the rules) and an evidence contract (stay inside the facts). Those contracts fail differently, demand different metrics, and need different repairs. “Better prompting” is not a diagnosis.
The diagnosis loop follows directly. Classify the failure. Localize it — and check for position effects before assuming the model is ignoring your context. Measure it with the right unit. Only then climb the control stack. If the failure responds to prompt changes, stay low. If the same failure persists because the model’s default behavior is wrong, move upward — into grounded data, SFT, and verifiable-reward RL.
You should be able to
Look at a bad model output and name whether it is an instruction-following break, a groundedness break, or an upstream failure. Measure each dimension separately. Explain why the distinction changes the fix. Describe what grounded SFT teaches and why RL shifts the default tradeoff.
The key tension to carry forward
Completeness and correctness pull in opposite directions. Pushing for thoroughness invites fabrication; pushing for precision invites excessive refusal. Every intervention you apply moves the system along that boundary. The control stack exists so you can place the system deliberately — starting cheap, escalating only when the failure outlasts the lighter fix.
If you cannot name the unit that failed and the component that owns it, you do not have a diagnosis. You have a complaint. The gap between the two is exactly where debugging time disappears.