The keystone: a direct measurement
In the host-bound regime the residual is large and does not scale with GPU work. We test this directly — no fit.
The host-exposed residual is large and order-of-magnitude stable — measured by differencing two nsys traces
vLLM V1 runs the model in a separate EngineCore subprocess, so in-process profilers capture nothing. Instead we let nsys passively capture the subprocess's kernels and run the identical decode twice — at 1 and at 512 generated tokens — then difference the cuda_gpu_kern_sum totals. The difference cancels model load, warmup, and prefill, leaving exactly the 511 pure decode steps. In eager mode kernels run serially on one stream, so the kernel-sum is faithful GPU-active time and the wall−GPU gap is host-exposed (launch + Python scheduling) time.
On an H100 (eager, N=1, ctx 2048): the Llama-3.1-8B step is 12.19 ms wall = 6.41 ms GPU + 5.78 ms host-exposed (47%); the Llama-3.2-1B step is 7.01 ms wall = 1.45 ms GPU + 5.56 ms host (79%). The host floor is 5.56 vs 5.78 ms — under 4% apart — even though GPU compute grows ~4.4×. GPU work scales steeply with the model; the residual is only loosely size-dependent. Replicate sweeps bound this as order-of-magnitude stability (~6–10 ms), not a tight invariant — a controlled same-family Llama sweep found a reproducible 3B excursion to 9.83 ms (see the status page).
Direct per-decode-step host/GPU split (H100, eager, N=1, ctx 2048). GPU-busy time (lower) grows ~4.4× from 1B to 8B; on this single anchor the host-exposed residual (upper, ~5.6–5.8 ms) is nearly flat (replicate sweeps bound this as order-of-magnitude stability, not a tight invariant). Source: nsys cuda_gpu_kern_sum differencing, no fit.
One invariant, five transforms
The same floor — measured against by each paper — is what these five system techniques each act on. A GPU-centric view explains none of them cleanly; the host term explains all five.
| Axis | Effect on the host floor | Headline measurement |
|---|---|---|
| Tensor parallelism | TP-collapses it — floor 5.79→0.48 ms; a separate additive term is k-invariant | 70B decode shrinks only 1.11× TP-2→TP-4 (not 2×); 8B floor flat 11.6→11.7 ms |
| Chunked prefill | Withdrawn — axis dropped after re-checking its traces | GPU kernel work DROPS 1.97× as K:1→16 (S fixed); wall is U-shaped (host paid K×) |
| Speculative decoding | Spec-amortized — spread over accepted tokens | 6.15× speedup at α≈1.0; per-cycle residual 0.77 ms = only 13% of the floor |
| CUDA-graph capture | Graph-removable — replays launch/dispatch as one launch | up to 2.24× (55.3% reduction); grid-mean 31.9%; collapses as GPU work grows |
| FP8 quantization | FP8-untouched — cuts the weight stream, not the floor | FP8 decode SLOWER in 30/30 cells (0.79×); FP8 prefill faster (1.45×) |
The operator takeaway is a single decision variable: measure the host fraction of the step first, then choose the axis. CUDA graphs and speculation pay in the host-bound regime (small batch, small model); chunk count and FP8 should be set against, not for, the host term; TP buys capacity but not decode latency.
The five papers
Each is an honest, well-scoped measurement study with its own figures and an explicit limitations section.
The Additive Term Does Not Shrink: A TP-Invariant Cost Predicts 70B Decode Latency (and Where It Fails)
…
Chunked Prefill Does Not Re-Read the KV Cache: An Exact K-Invariance Correction
…
Speculative Decoding as Host-Term Amortization: A Per-Output-Token Latency Model
…
When Does CUDA-Graph Capture Pay Off in vLLM Decode? An Empirical Regime Map
…
Does FP8's 2× GEMM Efficiency Transfer to Eager vLLM Decode? Prefill Yes, Decode No
…