A premise running through a decade of GPU systems work — kernel-fusion compilers, cache-pinning runtimes, locality-scheduling proposals — holds that the leftover data sitting in the GPU's on-chip cache between computation steps quietly saves several percent of every step's time. This study measures whether that belief is true on real datacenter GPUs. A first, careless measurement agreed — and then turned out to be wrong by a factor of seven to twenty-three. This page tells the whole story from the ground up, for a reader who has never opened a GPU.
When a language model generates text, the chip performs the same long chain of small computations tens of thousands of times. Each computation writes its result somewhere, and the next one reads it back. The hardware has a small, fast on-chip memory — a cache — that briefly holds those results. The question this project asks is narrow but consequential: when the next computation finds its input still sitting in that cache, instead of having to fetch it from the slow main memory, how much wall-clock time does that actually save? A line of GPU systems work (named with citations in Section 2 and the References), and the first quick experiment we ran, both implied"a few percent." The careful answer is far smaller, and the interesting science is in why the quick experiment lied.
The leftover data really does survive in the cache between steps when nothing disturbs it — 86–99.7% of a 25 MB block is still there when the next computation arrives. We also found a hardware control, never documented for this use, that can deliberately protect that data even under heavy interference.
Bytes surviving in cache is not the same as time saved. We measured how each computation actually responds to memory speed, and converted surviving bytes into saved milliseconds. The most this reuse could ever recover, on the GPUs we tested, is under two-thirds of one percent of a step — and under realistic load, essentially zero.
Three standard ways of measuring this effect each report a big number, and each is measuring the wrong thing. The errors all push in the same direction — toward over-valuing the reuse. Diagnosing exactly how each instrument fools its user is the paper's most reusable contribution.
Before running the decisive experiment we wrote down, in advance, four pass/fail conditions our own measurement had to meet to be trusted. It failed them. We report a bounded, honest result instead of a confident wrong one — and the self-imposed referee that caught our own flawed setup is itself a tool others can reuse.
This section assumes no background. To understand the question the project asks, four ideas are enough: what a GPU is, what a "kernel" is, why memory comes in fast-small and slow-large flavors, and what an "activation handoff" is. Everything afterward builds on these four.
A large language model is, mathematically, a very long sequence of multiplications of big grids of numbers (matrices). A graphics processing unit (GPU) is a chip built to do enormous numbers of such multiplications at once: think of a factory floor with thousands of small workers, each doing one tiny piece of arithmetic in parallel. This is why GPUs, not ordinary processors, run modern AI. The specific chips in this study — NVIDIA's A100, H100, and H200 — are the datacenter GPUs that power most commercial AI services.
The factory does not do everything at once. It runs one operation at a time, and each operation is called a kernel — a single program that the whole floor of workers executes together before moving to the next one. Producing one layer of the model means running a fixed recipe of kernels in order: combine the input, compare every word with every other word (this step is called attention), reshape, normalize, and so on. The model used here, Llama-3.1-8B, has 32 such layers, and one pass through all of them produces the next word.
A worker who needs data has two places to get it. The first is main memory (called HBM, high-bandwidth memory): huge — tens of gigabytes, enough to hold the entire model — but comparatively slow and far away. Think of it as a warehouse across the yard. The second is a small on-chip cache (called L2): only tens of megabytes, roughly a thousand times smaller, but right next to the workers and several times faster. ("Faster" here means two things at once: data starts arriving sooner — lower latency, or waiting time — and more of it arrives each second — higher bandwidth, or flow rate.) Think of it as the countertop in the workshop. Fetching from the warehouse costs real time; finding the needed data already on the countertop costs almost none.
The memory ladder every byte must climb. The L2 countertop is small but fast; the HBM warehouse is large but slow. The question is how often the next kernel finds its input still on the countertop.
The intermediate results that kernels pass to each other are called activations. When kernel A finishes, it writes its activation out toward the warehouse — but on the way, a copy passes through and lingers on the countertop (the L2 cache). Two facts of how this cache works create the entire opportunity. First, writing puts data on the countertop and leaves it there until space runs out. Second, nothing wipes the countertop between kernels. So if kernel B launches soon enough, it may find A's freshly written activation still on the countertop and read it for almost free, skipping the slow warehouse trip entirely. That lucky, unplanned reuse is what this project calls the incidental handoff — and the question is simply: how much time does it save?
The incidental handoff. The whole study is a careful accounting of how often B reads from the fast countertop instead of the slow warehouse, and what that is worth in time.
Running large models is one of the most expensive activities in computing today; a fraction of a percent of every inference step, multiplied across billions of steps a day in a datacenter, is real money and real energy. So a belief that an effect is "worth a few percent" is not idle — it directs where engineers spend their effort. This section explains what the belief was, why it mattered, and how the most obvious way of checking it leads straight into a trap.
This is not a vague intuition; it is the stated or implied premise of several active lines of systems research. Fusion and megakernel compilers — Welder (OSDI '23), Souffle (ASPLOS '24), FlashAttention-2 (ICLR '24), and whole-model "megakernel" systems such as MPK, Ada-MK, and FlashFormer (2025–2026) — justify themselves partly by the boundary memory traffic they remove, which is only worth removing if that traffic is expensive. A second line keeps frequently-reused data resident on-chip ahead of its readers: KV-cache prefetching into L2 (Alibaba, 2025), weight and KV-cache prefetch (PRESERVE), learned L2 residency (AutoScratch, MLSys '23), persistent-kernel reuse in registers and shared memory (PERKS, ICS '23), and embedding-table L2 pinning for recommendation models (MICRO '24). A third treats inter-kernel L2 hit rates as a workload property worth characterizing (Wang et al., 2025), and a 2025 simulator study reports a 15.9% gain from priority-aware control of exactly this inter-kernel L2 traffic (Liu et al.). The assumption is vendor-endorsed: NVIDIA's own Ampere architecture guide states that data "repeatedly accessed" in the enlarged L2 is served "at much higher speed than reading from and writing to HBM," and that for DL inference "ping-pong buffers can be persistently cached in the L2 ... avoiding writebacks to DRAM." Underneath all of them sits one assumption none of them measures directly on shipping silicon: that the incidental handoff is worth a meaningful fraction of inference time, and is therefore worth protecting, scheduling around, or engineering away. Every work named here is listed with a full citation in the References section.
The natural way to measure the handoff's value seems easy. Between kernel A and kernel B, insert a small extra kernel that deliberately wipes A's data off the countertop, forcing B to fetch from the slow warehouse. Then time the difference. If the handoff was worth a lot, destroying it should make the step noticeably slower. We ran exactly this experiment on a live system: destroying the handoff cost +4.7% of step time on the H100 GPU and +5.2% on the H200. Read at face value, the folklore looked confirmed.
Even before measuring anything, a back-of-envelope argument caps how much the handoff could ever be worth. The countertop is only a few times faster than the warehouse — not a thousand times. So replacing a warehouse read with a countertop read can save, per byte, only the difference between those two speeds: roughly a factor of one-and-a-half to two-and-a-half. And the kernels that dominate the step's time are limited by arithmetic, not by memory speed at all, so for them the saving is near zero regardless. This is why the project was never framed as a big-speedup paper. It was framed, from the start, as an honest accounting: when does this resource exist, when does it turn into real time, and when can nothing recover it?
Five lines of prior research surround this question without answering it directly. The starred rows below (1, 2, and 4) are the ones whose case rests, in whole or part, on inter-kernel L2 reuse being worth recovering — they are precisely the "reuse is valuable" position this study puts a number on. Each representative work links to its full entry in the References.
| Camp | Representative work | What it claims / relies on | What it leaves unmeasured |
|---|---|---|---|
| 1. Fusion & megakernels ★ | Welder, Souffle, FlashAttention-2, MPK, Ada-MK, FlashFormer | Remove the kernel boundary outright; the value of removing it implicitly assumes the round-trip is costly. | Never measured what the handoff was worth before removing it. |
| 2. Read-side L2 staging ★ | KV-cache prefetch, PRESERVE, AutoScratch, PERKS, DLRM pinning | Deliberately pin or prefetch read-mostly data (weights, KV-cache, tables) into L2 ahead of its readers. | Targets read-mostly data; the one line that pins activations (AutoScratch) keeps them L2-resident for a whole step, and the closest written-then-read reuse (PERKS) uses registers and shared memory — neither isolates the transient dirty handoff window. |
| 3. Producer–consumer overlap | NVIDIA dependent launch, CUTLASS Ex.63 | Overlap a consumer's prologue with its producer's epilogue, citing inter-kernel L2 reuse as a side benefit. | Documents the mechanism, not its wall-clock value. |
| 4. CTA scheduling & simulated locality ★ | IKRA, CPC, Liu et al. (+15.9%) | Reorder CTA or page scheduling and credit cross-kernel data reuse for the speedup (IKRA, CPC), or report large gains from controlling inter-kernel L2 traffic (Liu et al.). | Mostly simulator-based; and the reuse they credit sits at L1 (IKRA, needing non-stock coherence) or the memory-page level (CPC), not stock-hardware L2. Our raster test shows the apparent L2-recency gain is actually an intra-kernel effect. |
| 5. Characterization & analytical ledgers | Wang et al., Lu et al. (I/O analysis) | Report L2 hit rates as a workload property, or derive byte ledgers from a cold-cache model. | The cold model over-counts the very traffic the handoff absorbs — up to 246× for attention (our correction). |
★ Camps 1, 2, and 4 are the ones whose case rests, in whole or part, on inter-kernel reuse being worth recovering — the "reuse is valuable" position. This study supplies the missing measured baseline.
As of 2026, no prior work simultaneously satisfies all five of: a real chip, an actual LLM inference workload, the official hardware cache controls, the transient dirty-activation case, and a causal attribution of the result to wall-clock time. That five-way intersection is the gap this project fills.
A measurement study is only as trustworthy as its instruments, so it is worth understanding each one in plain terms before looking at results. Four families of instrument carry the whole study. The first three each measure the resource a different way; the fourth is a referee that decides whether a measurement is allowed to be believed. None of these is exotic equipment — they are standard GPU profiling tools used with unusual care.
The first instrument answers the existence question: when kernel B reads its input, how many bytes did it pull from the slow warehouse, and how many did it find already on the countertop? Modern GPUs contain hardware counters — like odometers built into the chip — that can report exactly how many bytes a single kernel read from main memory. The technique is to run a three-kernel sequence: a producer writes a known block, an optional "polluter" floods the cache with junk to simulate interference, and a consumer reads the block back while the counter watches. If the consumer's main-memory reads are near zero, the data survived on the countertop; if they equal the full block size, it had been evicted to the warehouse. We call the directly counted version byte-true, because it is measured from hardware rather than inferred.
Knowing the data survived is not the same as knowing it saved time. A kernel that is limited by arithmetic does not care how fast its memory is, so for it, surviving bytes save nothing. The second instrument measures exactly how much each kind of kernel slows down when memory gets slower. The trick is to break the normal one-at-a-time rule on purpose and run a second kernel at the same time — a co-running"stealer" kernel that does nothing but consume warehouse bandwidth at an adjustable rate, deliberately starving the kernel being measured — like opening a tap to drain pressure from a shared pipe. By measuring how much each kernel slows down per unit of bandwidth stolen, the study learns each kernel's sensitivity to memory speed: a number, in milliseconds saved per gigabyte not fetched, that converts surviving bytes into saved time.
The first two instruments work on isolated kernels in a controlled rig. The third asks the production question directly: inside a real serving engine handling real traffic, does destroying the handoff cost measurable time? This is the +4.7% experiment from Section 2 — insert a scrubbing kernel into the live model, wipe the handoff, and time the result. The study also attributes time kernel-by-kernel, so the scrubbing kernel's own runtime can be excluded from the measurement. This is the instrument most prone to the artifacts described earlier, which is exactly why it does not get to report a result on its own — it has to pass the referee described next.
The fourth instrument is the most unusual, and the one the authors are proudest of. Before running the decisive live experiment, the team wrote down — in advance, so the conditions could not be quietly relaxed afterward to fit the answer they wanted — four pass/fail checks the measurement had to satisfy to be trustworthy. This is a research technique borrowed from clinical trials, called pre-registration: you commit to what counts as a valid result before you see the data, so you cannot fool yourself.
Two independent timing methods must report the same delta. If they disagree, neither can be trusted.
The scrubbing kernel's own time must grow predictably with its size, so it can be subtracted.
The more data destroyed, the bigger the slowdown should be. If they do not correlate, the signal is noise.
The extra warehouse traffic forced by the scrub must stay under a fixed budget, or the measurement is contaminated.
The three active instruments poke the system. A final, passive one simply watches a real serving engine and records, for every handoff, how much other traffic flows across the countertop between the producer and consumer. This "reuse-distance census" answers a question the controlled rig cannot: in actual production, does the data even have a chance to be evicted before the consumer reads it? As the results show, the answer reframes the whole problem.
Good empirical work states, before measuring, exactly what it intends to claim — so that the evidence can be checked against the claim rather than the claim quietly bent to fit whatever the evidence turned out to be. The project organizes its entire campaign around four claims, deliberately phrased in the cautious form the eventual evidence could actually support. Each claim maps to a different instrument and a different stage of the argument: does the resource exist, is it worth anything, does it exist in production, and where does the folklore number come from?
| Claim | Hypothesis in plain words | Instrument used |
|---|---|---|
| C-A Real |
The handoff data really does survive on the countertop between kernels, and a hardware control can protect it. | Byte counter (3.1) |
| C-B Bounded |
Even when it survives, the time it can save is capped at a tiny fraction of a step. | Bandwidth stealer (3.2) |
| C-C Already harvested |
In real production traffic, the handoff already happens naturally — there is little left to recover. | Census + live scrub (3.3, 3.5) |
| C-D Misattributed |
The big folklore numbers come from three instruments each measuring the wrong thing. | All three + decomposition (3.1–3.3) |
The findings line up with the four claims. The resource exists and can be protected; it is worth very little; in production there is little left to recover; and the folklore numbers are artifacts. The results below give each in turn, with the actual measured numbers from the three GPUs.
With no interference, a 25 MB activation block — the size produced by one real step of the model, and small enough that the countertop holds only about two of them — crosses the kernel boundary almost perfectly intact: 98.8% of it is still on the countertop on the H100, 99.7% on the H200, and 86.4% on the older A100. The resource genuinely exists. But its survival under interference does not fade gently; it falls off a cliff. As other data moving through the chip — traffic — floods the countertop, survival on the H100 drops from 85% to 52% to 3% to nearly zero as interference rises through one cache's worth of junk. Half the block is gone after only a quarter to a half of the countertop is overwritten by other work.
Survival of a 25 MB block on the H100 as interference rises (timing-implied estimate). The shape is a cliff, not a slope: a handoff either lands almost whole, or is gone entirely once roughly one cache of other traffic intervenes.
The study also found that a hardware control called the Access Policy Window — officially documented only for protecting read-only data like lookup tables — can be repurposed to protect the freshly written, dirty handoff. With this control switched on, the block stays protected even when the countertop is flooded with four times its capacity in junk: the consumer's warehouse reads stay at zero on two of the three GPUs. To the authors' knowledge this is the first systematic measurement of what this control does to dirty data.
With the protection control on, 72–76% of the block stays on the countertop under heavy interference (and 100% by direct byte count on two GPUs), versus near zero without it. A real, controllable resource — which makes the next result, that it is barely worth anything, all the more striking.
This is the heart of the result. The study combines, for every handoff in a model layer, three measured quantities: how much of the data survives, how many bytes it is, and how sensitive the consuming kernel is to memory speed. Multiplying these and summing over all the handoffs in all 32 layers gives the most time the entire incidental handoff could possibly save in one step. The formula is just an honest bookkeeping of "surviving bytes times how much each byte is worth":
Here $s(b,p)$ is the surviving fraction at a given block size $b$ and interference level $p$, and $\kappa$ is the consuming kernel's measured sensitivity (milliseconds per gigabyte). The answer, on all three GPUs under realistic running conditions, is small: a best case of about 0.12 to 0.31 milliseconds per step — under 0.65% of the step. Under any realistic interference it drops below 0.1%. Even if every uncertain assumption is pushed in the direction most favorable to the handoff, the bound stays at or below 1.6%. The 5% folklore had no room to be true.
The gap between belief and measurement. Bars scaled to the ~5% folklore figure. The real ceiling is roughly one-eighth of the folklore value at best, and a fiftieth of it under realistic load.
The census — passively watching a real serving engine — adds a second, independent reason the handoff is not a lever. It found that in production, dependent kernels run essentially back-to-back: the typical amount of other traffic between a producer and its consumer is between near-zero and 8 megabytes, far below the tens of megabytes it would take to evict the data. In other words, the handoff already succeeds naturally almost every time; there is nothing for a clever protection scheme to rescue. This pattern held identically across nine different serving configurations and concurrency levels, because the engine runs its kernels in a fixed serial line — changing the workload changes what is computed, not the order things sit in the cache. And real requests are small to begin with: a typical request produces handoff blocks of well under 5 megabytes, comfortably cache-resident.
The last result is the one with the broadest reach beyond this specific question. Three standard measurement techniques each report a large number for the handoff's value, and the study shows each is measuring the wrong thing — and all three err in the same direction, toward overvaluing the handoff. This is why careful engineers, using trusted tools, arrived at a wrong folklore figure.
The wiping kernel's own runtime, plus the warehouse save-out it forces, get charged to the handoff. Overstates the truth by 7–23×. Tell-tale sign: the apparent effect is largest when the least data is destroyed — backwards from a real effect.
Reordering a kernel to read recently written data first gives a real +4.6% — but the gain survives even when that data is fully evicted first. So it is not cross-kernel reuse at all; it is an internal effect of how one kernel walks its own data, misread as handoff reuse.
The standard profiler measures each kernel with a wiped cache, converting cache hits into warehouse reads that never happen in real running. It over-charges the attention kernel's memory traffic by 73–246×. Traffic models built from these counts inherit the error exactly where the handoff operates.
The third trap is worth a number to feel its size. When the standard profiler measures the attention kernel — which reads data the previous kernel just wrote — it reports that kernel reading up to 246 times more data from the warehouse than it really does in normal running, because the profiler wiped the countertop before measuring. The plain kernels that read the model's permanent weights, by contrast, measure honestly at roughly 1×. The lesson is that adding up per-kernel profiler numbers double-counts exactly the traffic the handoff absorbs — so the very models people use to argue the handoff is valuable are biased toward that conclusion.
How much the standard cold-cache profiler over-charges each kernel's warehouse traffic (log-like scale). Kernels reading freshly written handoff data are inflated up to 246×; kernels reading permanent weights stay honest. The bias lands exactly where the handoff lives.
We did not only cite these systems (Section 2) — we cloned and ran four of them on the same PACE GPUs (H100 / A100-80GB) and measured, in each case, whether the reproduced speedup (or, for DLRM, the L2-layer ablation) is attributable to the passive inter-kernel activation handoff this study bounds. The four span different mechanisms — register/SRAM staging, launch-overhead elimination, read-side embedding-table residency, and active KV prefetch — and all four corroborate the bound from different directions: each delivers a real benefit, and in none of them does that benefit come from the passive activation handoff.
| System | We reproduced (measured) | Is the win inter-kernel L2 activation reuse? | Full analysis |
|---|---|---|---|
| PERKS (ICS'23) | 1.86–2.13× iterative-stencil speedup (A100) | No — L2 hit-rate stays flat (~72%→79%), DRAM falls; the reuse is staged in registers / shared memory, routed around L2. | PERKS analysis |
| MPK / Mirage (OSDI'26) | 2.733× decode megakernel (H100) | No — it lands at 1.21× the weight-bandwidth roofline; the win is launch-overhead elimination, not activation/L2 reuse. | MPK analysis |
| DLRM L2-pinning (MICRO'24) | Embedding-row L2 pinning ablation (A100) | No — this is explicit read-side embedding-table residency, not the activation handoff. It pays only for hot rows (~92% hit vs ~30% random), and explicit pinning is about equal to register-prefetch at the L2 layer in the no-polluter run. | DLRM analysis |
| KV-cache prefetch (AAAI'26) | 1.80× e2e decode at batch 1 (H100) | A real active-prefetch speedup — but not a large L2-residency delta. The baseline is already ~85% L2-resident; the prefetch hides latency (+0.2–0.4 pp hit, DRAM flat). It is an engineered, active async prefetch of high-reuse KV-cache, categorically separable from the passive written-once activation handoff this study bounds — it sharpens, not contradicts, the <0.65%. | KV-prefetch analysis |
The common thread: none of the four gets its benefit from the passive, transient inter-kernel activation reuse this study bounds. The systems that do touch L2 either route around it (PERKS), reach a different bound (MPK), depend on data hotness (DLRM), or actively prefetch read-mostly data (KV-prefetch). That is consistent with, and independently corroborates, the <0.65% ceiling on the passive activation handoff — across four heterogeneous systems.
It is tempting to read "the effect is under 0.65%" as a disappointing result. It is the opposite: it is a decision that several groups of engineers can now make with confidence instead of folklore. The contributions below are what a reader from outside the field should take away.
Teams spend real effort fusing kernels and building cache-protection machinery, justified partly by the handoff's cost. This study shows that, for this workload, the incidental handoff already absorbs almost all of that cost on its own — so that part of the justification needs to be re-priced. The effort may still be worth it for other reasons, but not this one.
Anyone who builds memory-traffic models from a standard profiler is, without knowing it, over-counting the traffic that the handoff absorbs — by up to 246× for one kernel class. The paper publishes a per-kernel correction factor and a capture recipe that others can apply directly to avoid inheriting the bias.
The first systematic measurement of how a hardware cache-protection control behaves on freshly written, dirty data — a corner of the chip that the official documentation does not cover. Even though the protected bytes turn out to be worth little here, the characterization is new knowledge about the machine.
The pre-registered validity gate — committing in advance to what makes a measurement believable — caught the team's own flawed setup and prevented a confident wrong claim. It also exposed a genuine limitation of such gates: a test built to confirm an effect cannot certify its absence. Both the tool and that lesson are offered for others to reuse.
An honest measurement study is precise about its own boundaries. The numbers on this page hold for the cells that were actually measured, and the paper says so explicitly rather than over-generalizing. A professor evaluating this work should know exactly how far the claims reach.
Every claim this page makes about prior work traces to one of the entries below; each was resolved against DBLP, CrossRef, or the arXiv API. Groups A, B, and D contain the works whose case rests, in whole or part, on inter-kernel L2 reuse being worth recovering — the "reuse is valuable" position this study measures against.
cudaAccessPolicyWindow, persisting L2 set-aside). docs.nvidia.com/cuda/cuda-c-programming-guide--cache-control semantics. docs.nvidia.com/nsight-compute