When agents hand off their KV-cache instead of re-sending text, a small budget of tokens must be recomputed to recover the cross-passage attention the splice dropped. Every prior selector scores the sender's cache error; CADENCE scores the receiver's decision instead — recompute the tokens the answer depends on, not the ones whose cache is merely most wrong.
This page reports a validated single-hop result honestly, but the study is not finished. The timing verdict is now final on both models: there is no wall-clock speedup at any budget (CADENCE-Grad runs at about 0.87x on Llama and 0.88x on Qwen, and decode is roughly 93% of end-to-end time, so the prefill saving cannot dominate). The accuracy numbers below are stable across two datasets and two models today; the following are still running and may revise the picture:
Multi-agent language-model systems and multi-passage retrieval pipelines have become a dominant deployment pattern through 2025 and 2026: several upstream agents each read a document or a tool result, and a downstream agent must produce an answer that draws on all of them. The conventional interface passes that evidence as text, which forces the receiver to re-read every passage from scratch and to pay a full prefill in order to rebuild the transformer's key/value tensors.
Prefill is the quadratic, compute-heavy stage of inference, and in long-context multi-agent settings it dominates the cost. A recent line of work therefore replaces the text hand-off with a KV hand-off: each upstream agent exports the key/value cache it has already computed for its passage, and the receiver splices those caches directly into its context and skips the prefill entirely. The intuition is that handing over a set of annotated notes is cheaper than asking the next reader to re-read the book.
This substitution is what makes the cache-reuse question pressing now rather than in 2022. Systems such as CacheBlend (EuroSys'25), KVCOMM (NeurIPS'25) and RelayCaching (2026) report several-fold reductions in time-to-first-token from exactly this move; KVCOMM measures up to a 7.82x lower TTFT (430 ms to 55 ms) in a five-agent configuration. As agent graphs grow to tens of coordinated calls over shared documents, the redundant prefill that KV hand-off removes grows in proportion, and the payoff of getting reuse right rises with it.
One mechanical detail matters throughout. Because a passage was cached at one range of positions but must live at another in the receiver's context, the splice includes a rotary re-alignment: a delta-RoPE rotation that slides each passage's keys to their correct global positions. This step is numerically exact, quantified in the next section, and the receiver then prefills only the short question suffix.
Handing over the cache verbatim is not without consequence, because a spliced cache is not the cache the receiver would have computed. When passages are encoded jointly, as a genuine re-prefill does, passage P3 attends to P1 and P2; when each passage is cached in isolation and later spliced, that cross-passage attention was never run. The spliced cache is therefore correct at layer 0 but drifts from the true dense cache at every layer above, and the gap widens for later passages, which miss more of their preceding context.
A useful subtlety follows from causality. In a causal decoder a token's cache cannot depend on future tokens, so the question suffix never corrupts the passage cache; the entire error lives within the reused passages, and its only missing ingredient is attention among them. That single observation is what makes the problem tractable: the receiver does not need to rebuild everything, only to restore the missing cross-passage interactions.
The effect is not subtle. Consider a two-passage question in the HotpotQA style: “Which Formula One champion had Richie Ginther as a teammate?”, where one passage introduces the champion and another lists Ginther's teammates. A Llama-3.2-3B receiver that splices the two passage caches and recomputes nothing answers “Richie Ginther” — it returns the teammate named in the prompt, because it never attended across the two passages to connect that teammate to the champion. A full re-prefill of the identical context answers “Graham Hill”, the gold answer; recomputing every reused token reproduces “Graham Hill” token-for-token. The failure is a direct, legible consequence of the missing cross-passage attention, not a garbled output.
Aggregated over the benchmark, discarding cross-passage attention removes between a third and two thirds of the accuracy the receiver would otherwise reach. On HotpotQA a full re-prefill reaches 64.2% exact-match; splicing every passage and recomputing nothing drops to 42.5% on Llama-3.1-8B and to 17.5% on Qwen3-14B. Before any recovery method is applied, the status quo for training-free reuse therefore leaves between 22 and 47 points of accuracy on the table. The bars are scaled to that 64.2% full-reprefill ceiling.
HotpotQA, zero recompute. Splicing every token and recomputing nothing drops accuracy well below the ceiling, and further on Qwen3-14B than on Llama-3.1-8B.
The recovery mechanism is due to CacheBlend, and its insight is that reuse need not be all-or-nothing. Rather than choosing between reusing nothing (a full prefill) and reusing everything (a sharp accuracy drop), the receiver keeps most of the reused KV but re-encodes a small budget of tokens through a fuse pass that lets exactly those tokens attend across the whole context again. At budget b=1 the procedure is a plain full prefill; at b=0 it is naive reuse; in between it trades accuracy against compute.
This exposes an accuracy-versus-budget dial and sharpens one question. Given a fixed budget of recomputable tokens, which tokens should the receiver re-encode? Every selector is nothing more than a scoring rule that assigns each token a score and recomputes the highest-scoring set; the whole method therefore reduces to the choice of that score.
The receiver recomputes the set of tokens with the highest scores under a budget of the ceiling of b times n tokens. Everything downstream is fixed; only the scoring rule s_t distinguishes one selector from another.
Every prior selector answers the “which tokens” question with a sender-side, intrinsic signal — a property of the cache itself, computed without reference to what the receiver is trying to decide. This section argues that the intrinsic signal is the wrong target, states the objective it should be replaced with, and positions the resulting method against the immediate prior work.
The obvious fix is to recompute the tokens whose cache is most wrong. CacheBlend's HKVD ranks tokens by their KV deviation, the norm of the difference between the reused and the dense cache, and recomputes the largest-deviation tokens; RelayCaching weights that deviation by accumulated attention influence, but the score is still computed on the sender side. Both minimize a reconstruction distortion — they make the spliced cache numerically close to the dense one — on the expectation that repairing the most-wrong tokens repairs the answer.
That expectation does not hold. At the b=0.2 operating point on HotpotQA-Llama, CacheBlend's deviation ranking recovers only 43.3% accuracy, which is indistinguishable from selecting tokens at random under the same budget. The reason is structural: the tokens carrying the largest cache error are frequently not the tokens the answer depends on, so a budget spent on them buys reconstruction fidelity that the decision never cashes in.
The pipeline is not paid to reconstruct the cache; it is paid to get the decision right, and the two objectives are not the same. A token can carry a large deviation yet exert no influence on the answer, and a token with a modest deviation can sit squarely on the reasoning path. The quantity worth minimizing is therefore the decision distortion — the probability that the receiver's answer flips relative to a full re-prefill — not the reconstruction error that deviation selectors minimize.
On the left, keep the answer produced by recomputing only the set S equal to the full-reprefill answer y-star — the dense model's own answer, not the gold label; accuracy is still scored against gold separately. On the right is what deviation selectors optimize. The two coincide only if deviation magnitude and decision-impact move together, and empirically they do not. The obstacle is that decision-impact is not visible from the cache alone: it is a property of the receiver's forward computation, which no sender-side signal observes.
Placed among the works a knowledgeable reader expects, the gap is a single shared assumption. The table records the scoring signal each method uses, whose information it is computed from, and what it leaves on the table; the common thread is that none conditions selection on the receiver's decision. Three of these four are the sibling pages in this survey's KV-reuse section.
| Method | Scoring signal | Side | What it leaves on the table |
|---|---|---|---|
| CacheBlend | KV deviation ‖ΔV_t‖ | sender | Query-blind; deviation is not decision-impact. |
| RelayCaching | deviation × attention influence | sender | Attention influence is a proxy for effect, not the effect on the answer. |
| KVCOMM | reuse-all + anchor offsets | sender | No per-token selection at all; an anchor miss forces a full prefill. |
| Ada-KV | per-head eviction budget | single stream | Compresses one model's own cache; no distinct receiver, no cross-agent splice. |
| CADENCE (this work) | decision-saliency gradient | receiver | Conditions selection on the receiver's own decision; no wall-clock speedup yet (Section 08). |
CADENCE keeps the entire selective-recompute machinery unchanged and alters only the score. It ranks each reused token by decision-saliency — how much the receiver's own answer would move if that token's cache were repaired — estimated with a single backward pass of the receiver's decision loss. The contribution is a scientific one about which signal to select on, and it is stated here before any efficiency claim so that the honest limits below are read alongside it.
The first selector to condition token selection on the receiver's decision rather than the sender's cache error, by differentiating the receiver's own answer loss with respect to each reused token's cache.
CADENCE-Grad needs one forward and one backward pass, with no dense reference, no partial prefill, no auxiliary model, and no training. It drops into any selective-recompute stack in place of the scoring rule.
At a fixed budget it matches or exceeds an exact-deviation selector handed the true KV-deviation, demonstrating empirically that high cache error and high decision-impact are distinct quantities.
The headline numbers are all measured at the pre-specified b=0.2 operating point on HotpotQA with Llama-3.1-8B, against the 64.2% full-reprefill ceiling.
The ideal score would test each token by actually recomputing it and re-running the model, which is far too expensive to do token by token. The construction below replaces that test with a first-order estimate that costs a single backward pass, and then states the exact quantity CADENCE differentiates.
Repairing token t means nudging its reused cache from the wrong value toward its true value, a nudge of size delta-t. A first-order Taylor expansion of the receiver's decision loss around the reused cache says the resulting change in the loss is simply the gradient dotted with the nudge.
The product has a clean reading. The gradient is decision-saliency — how much the answer cares about this token's cache — and delta is deviation, how wrong the cache is. A token matters to the decision only when it is both salient and wrong.
The decision loss is the negative log-likelihood of the receiver's own answer, produced from the reused cache and teacher-forced back through the model. No gold label is ever consulted; the signal is entirely self-generated, which is what makes it deployable at inference time.
Attributing that scalar to each token's cache to first order (attribution patching), summing keys and values across all layers, and taking the magnitude gives the gated CADENCE score. Both the gradient and the deviation appear, so the gate keeps only tokens that are salient and wrong at once.
Because the gate needs a dense reference it cannot afford, the variant that actually deploys discards the deviation term and keeps only the magnitude of the decision-saliency gradient. This variant needs no reference at all: one forward pass, which the receiver runs anyway to produce its answer, plus one backward pass of the decision scalar.
Empirically this reference-free score is statistically indistinguishable from the gated form throughout, so the gradient — which tokens the decision depends on — is doing all of the work, and the deviation gate earns nothing on the answers tested. The rest of this page reports CADENCE-Grad as the method of record and treats the gated form as a diagnostic.
The scorer runs three distinct passes, and keeping them separate is what makes the method both correct and self-contained. The first pass produces an answer used only to define the loss; the second scores every token; the third recomputes the chosen tokens and produces the answer that is actually graded. The pseudocode below shows the deployable path.
# 1. Forward from the spliced cache to obtain the receiver's own answer y_hat.
# This answer only defines the loss we differentiate; it is never itself graded.
y_hat = model.generate(reused_cache, question)
# 2. One backward pass of the decision loss scores every reused token.
L_dec = nll(model(reused_cache, question, y_hat), y_hat)
g = grad(L_dec, reused_cache.K, reused_cache.V) # dL_dec / dKV
score = (g.K.pow(2) + g.V.pow(2)).sum(dim=layers) # CADENCE-Grad, per token
S = topk(score, ceil(b * n)) # the salient set
# 3. Recompute only S with full attention, then answer from the repaired cache.
repaired = fuse_recompute(reused_cache, S)
answer = model.generate(repaired, question) # the final, graded answerTwo subtleties are worth stating. The answer y_hat in pass one is teacher-forced only to build a differentiable scalar, so a wrong first answer still yields a valid saliency map; the method does not assume the receiver is already correct. And because the gradient magnitude in pass two ranks tokens by how strongly the decision loss responds to each token's cache, the selection is invariant to the overall scale of the loss, which keeps the ranking stable across questions of very different difficulty.
The four stages below are the full receiver-side pipeline. Competitors share the first, second and fourth stages but score forward-only on the sender side; CADENCE adds the third stage, the backward pass, which is the only structural addition.
This is where overclaiming would be easy, so the accounting is deliberate. Scoring with CADENCE-Grad reuses the forward the receiver already runs and adds one backward pass of the decision scalar, whose cost is roughly 4Pn floating-point operations, about twice a forward pass. Deviation baselines use only a forward-side signal and pay no backward pass, so the comparison is honest only if the backward is counted in full.
The claim is therefore narrower, and it is the right claim for a selector. At a fixed recompute budget, CADENCE picks the tokens whose recompute most restores the decision, and it therefore reaches a target accuracy with fewer recomputed tokens. Accuracy per recomputed token is a hardware-independent quantity: it says the budget is better spent, not that the wall clock is shorter, and today those are different things. The method is training-free, single-model, and drops into any selective-recompute stack in place of the scoring rule.
Every method uses the same budget and the same recompute path; only the score differs. On HotpotQA-Llama at the pre-specified operating point b=0.2, the decision-conditioned signals lead, and the exact-deviation selector DenseDev — which pays for a full dense reference — still trails them. The bars are scaled to the 64.2% full-reprefill target.
Accuracy at b=0.2, HotpotQA with Llama-3.1-8B. CacheBlend dips to the level of random selection at this budget, while CADENCE recovers 94% of the 64.2% TEXT ceiling.
The table records what signal each selector uses, whose information it is computed from, its reference cost, and the headline number. Only CADENCE conditions on the receiver's decision, and only CADENCE-Grad achieves it without any reference.
| Selector | Scoring signal | Side | Reference cost | HotpotQA·Llama @ b=0.2 |
|---|---|---|---|---|
| TEXT | recompute everything | — | full prefill | 64.2 (target) |
| reuse-all | keep everything | — | none | 42.5 |
| CacheBlend / HKVD | KV deviation ‖ΔV_t‖ | sender | partial prefill ~10–18% | 43.3 |
| RelayCaching | deviation × attention influence | sender | partial prefill | 52.5 |
| DenseDev | exact deviation ‖Δ_t‖ | cache-only | full dense forward | 54.2 |
| CADENCE (gated) | grad · deviation ⟨∂L/∂KV, Δ⟩ | receiver | 1 backward + dense ref | 60.0 |
| CADENCE-Grad | grad magnitude ‖∂L/∂KV‖ | receiver | 1 backward (no ref) | 60.0 |
The clearest way to read the gap is in recomputed tokens at matched accuracy. Reaching 60% exact-match costs CADENCE-Grad roughly 297 recomputed tokens; CacheBlend does not reach 60% at any budget, and its best result of 57.5% needs about 705 recomputed tokens. CADENCE-Grad therefore exceeds CacheBlend's best-ever accuracy while recomputing about 2.4x fewer tokens. This is a statement about selection quality — accuracy per recomputed token — and not about wall-clock time or throughput; Section 08 explains why there is no latency win.
The single-hop result is robust but not uniform. Across the full grid of two datasets and two models — HotpotQA and 2WikiMultihopQA crossed with Llama-3.1-8B and Qwen3-14B, at 120 to 150 tasks per cell — the decision-saliency family is best or tied in 17 of 20 dataset-by-model-by-budget cells. The pre-registered confirmatory comparison, CADENCE above CacheBlend at b=0.2, is significant under a paired McNemar test with Holm correction in 3 of 4 cells, by up to 16.7 points. At the same budget CADENCE also matches or exceeds DenseDev, the exact-deviation selector, in all four cells, though the per-cell gaps are not individually significant after correction. Together these confirm the thesis that high KV-deviation is not the same as high decision-impact.
A method is only as trustworthy as its stated limits, and CADENCE has several. They are listed plainly here so that the single-hop accuracy result is not read as more than it is.