The Whole Story in One Minute
Tensor parallelism (TP) is how you serve a model too big for one GPU: shard the weights across k GPUs and they each do roughly 1/k of the work. Every back-of-envelope model encodes the same hope — k GPUs make each decode step about k times faster, minus a small communication penalty. For the compute-bound prefill that mostly holds. For autoregressive decode, which dominates the latency a user feels, it does not. We measure why, and turn the why into a closed-form predictor.
1 · The step barely shrinks
Doubling the GPU count from TP-2 to TP-4 shrinks the 70B decode step by only 1.11× in-regime (1.18× including a disclosed outlier), nowhere near the ideal 2×. Even the paper's own composed baseline — sharded compute plus the real all-reduce — predicts only 1.36–1.53×, still well above what we measure. The gap is not an artifact of comparing to a fantasy ideal.
2 · A TP-invariant additive term recovers it
The missing quantity is an additive per-step term — anchored in the host/scheduler work the rank-0 driver pays once per decode step — that does not divide by k. Fixing it from an independent single-GPU 8B framework-overhead microbenchmark (zero TP-4 fitting; one global 70B compute scale fit on TP-2 only) predicts the 12 held-out H100 TP-4 cells to 4.4% mean MAPE (7.5% max), versus 16.9% mean (18.6% max) for the natural composed baseline on the same cells.
3 · The term is provably k-invariant
Against a suite of host-model forms all calibrated on TP-2, every k-invariant form (a free constant, a batch-linear intercept, or the fixed 8B microbench value) generalizes to held-out TP-4 within 4.4% — but forcing the term to scale with TP fails badly: H·k by 70.3%, H/k by 37.7%. The data strongly reject any k-scaling of the additive term. That is the load-bearing finding.
4 · A host floor exists — but the term is ~3× larger
Two distinct quantities. (i) A direct nsys split shows a model-size-invariant host-exposed floor exists: 5.78 ms at 8B, 5.56 ms at 1B (<4% apart) even as GPU compute grows ~4.4× — the signature of host/scheduler cost. (ii) The additive term the model actually uses is the fitted host line (~17–18 ms, intercept 16.9 ms), about 3× that directly-measured floor. So we claim only TP-invariance of the additive term plus the existence of a model-size-invariant host floor — not that the whole additive term is host. The excess (mis-calibrated compute, continuous-batching work) is disclosed, not yet fully attributed.
Measured 70B TP-2→TP-4 decode step-shrink (in-regime 1.11×) versus the ideal 2×. The step barely shrinks because the dominant component is an additive, TP-invariant term that does not divide by k.
The keystone direct measurement (8B, H100, eager, N=1, ctx 2048), from differencing two nsys runs. The 12.19 ms decode step splits into 6.41 ms GPU-kernel + 5.78 ms host-exposed (47%). The host-exposed floor is model-size-invariant: GPU time falls 4.4× from 8B to 1B (6.41→1.45 ms) while the host term holds within 4% (5.78→5.56 ms). This proves a model-size-invariant host floor exists; the fitted additive term used in the predictor is ~3× larger.
Why Adding GPUs Should — But Doesn't — Halve Decode
Three ideas are enough to follow the rest of the page: what tensor parallelism does, why decode is memory-bound while prefill is compute-bound, and what the host/scheduler actually does on every step. Each is developed in plain language before any vLLM-specific name appears.
Tensor parallelism splits the kernels, not the schedule
Under Megatron-style TP, each of the k GPUs holds 1/k of the projection weights, and two all-reduces per layer recombine the partial activations; the KV cache is sharded over the k devices too. So the matrix multiplications get k times narrower and the all-reduce is the price of stitching the shards back together. Crucially, TP shards the kernels — the GPU work — but it does not shard the schedule: there is still exactly one scheduler iteration per decode step, run by a single rank-0 driver, no matter how many GPUs participate.
Decode is memory-bound; prefill is compute-bound
In prefill the model processes a whole prompt at once, so the matrix multiplications are large and the GPU is genuinely compute-bound — there sharding pays off, near 2× for 2× the GPUs. In decode each request emits just one token per step, so the projection GEMMs are tall-skinny and the memory roof binds: the step is spent streaming weights and KV through HBM, not doing arithmetic. Halving the compute by adding GPUs therefore helps far less, because the compute was never the bottleneck — and worse, something else on the step does not shrink with k at all.
The host pays a per-step cost once, regardless of k
Modern engines like vLLM and Orca run an iteration-level scheduler: on every decode step the rank-0 driver dequeues the running batch, updates sequence states, maps the paged KV block tables, builds the kernel arguments, and issues the launches. In eager execution this host/scheduler work sits on the critical path. The decisive fact for TP is that this work is paid once per step, by one driver, and is structurally identical at k=2 and k=4. It is the only cost in the step that does not divide by k.
Host (rank-0 driver)
one scheduler iteration per step — same work at k=2 and k=4, does NOT divide by kGPU compute (sharded)
the projection GEMMs and KV read get k times narrower — this is the part that shrinksAll-reduce
2 per layer to recombine shards; only ~15.6% of the step on average at decode message sizes (H100)Because only the green box shrinks with k while the blue box stays fixed, the step as a whole shrinks far less than 2× — and the more GPUs you add, the larger a fraction of the step the fixed host cost becomes.
The failing composition
The natural back-of-envelope model composes the single-GPU mechanism cost with communication:
Since the compute term goes as 1/k and the all-reduce is small at decode message sizes, this predicts a near-ideal 2× shrink. The measured shrink is 1.11×, and the leftover after subtracting sharded compute and the real all-reduce is a positive term that grows as 1/k compute shrinks — exactly the signature of a fixed floor the /k scaling wrongly shrinks.
A Host Term That Does Not Divide by k
We add one term to the failing composition: a non-overlapped host cost H(N) that is independent of the TP degree. Every piece is determined separately, and we are explicit about what is measured versus model-assumed — the host term and the NCCL constants are measured on independent microbenchmarks, and the single fitted serving parameter is one global compute scale.
Compute Tcmp
A reused single-GPU roofline: each projection GEMM is the max of its FLOP roof and its HBM-streaming roof, summed over layers, with widths and KV heads divided by k. One global scale g (fit on TP-2 only) corrects the 8B-calibrated efficiencies; g = 0.30 in-regime, a substantial ~3× downward recalibration the host claims do not lean on.
All-reduce Tar
A latency-bandwidth fit on a standalone NCCL all-reduce run on the same allocation (λ = 21.2 μs, β = 307 GB/s on H100). A decode step issues 2L all-reduces. It is small — 15.6% of the step on average — and calibrated on-allocation, so it contributes little error.
Additive term H(N) — the contribution
H(N) = s0 + s1(N−1) is the linear fit to an 8B single-GPU framework-overhead microbenchmark (fitted intercept s0 = 16.9 ms, ~17–18 ms over N=1…8). We are precise: this fitted term is ~3× the directly-measured host-exposed floor (5.78 ms by nsys), so it is not a pure host floor — it also folds in compute the 8B-calibrated roofline over-predicts. Its defining property is no k dependence: the rank-0 driver runs one scheduler iteration per step regardless of sharding. It carries no free parameter.
The held-out protocol — the discipline that makes the claim credible
The whole point is to predict a configuration the model never saw. We fit the single compute scale g on TP-2 serving cells only; the host term is fixed from the 8B microbenchmark with no fit to 70B or to TP-4 at all; we then predict the 12 H100 TP-4 cells cold. Because measured decode time is roughly flat (~26–30 ms), many additive decompositions fit equally well on TP-2 — the discriminating test is held-out TP-4, which is what separates a k-invariant term from a k-scaling one.
Setup
Fresh runs on PACE Phoenix: NVIDIA H100-80GB (NVLink) and H200 nodes, vLLM 0.19.1.dev, PyTorch 2.10+cu126, eager mode; each job records git SHA, conda environment, and GPU UUID/driver at submit. Workload: Llama-3.1-70B-Instruct, TP-2 and TP-4 on H100 (the validated regime), plus TP-2 and TP-4 on a second SKU (H200, a fresh node) held out entirely — on which the additive model is falsified at TP-4. For N prompts of length ctx we generate 33 tokens with ignore_eos and greedy sampling, and report the steady-state per-step time (wall(T)−wall(1))/(T−1), which removes prefill (sampling and detokenize are included, and are part of the host floor). Grid: N ∈ {1,2,4,8}, ctx ∈ {512,2048,4096}, 12 timed reps/cell after 2 warmups, medians with bootstrap 95% CIs, 48 cells total.
The TP-Invariant Term Recovers H100 TP-4
Per-configuration accuracy on the held-out H100 TP-4 cells
The composed baseline (sharded compute + all-reduce) and the joint model (which adds the TP-invariant additive term) are compared per configuration. On the 12 held-out H100 TP-4 cells, the baseline mispredicts by 16.9% mean (18.6% max) while the model — with zero TP-4 fitting — lands at 4.4% mean (7.5% max). The H200 TP-2 SKU is also predicted within bar, but H200 TP-4 breaks the additive composition (see the honest negative below).
| SKU | TP | #cells | baseline MAPE | model MAPE |
|---|---|---|---|---|
| H100 | 4 (held-out) | 12 | 16.9% | 4.4% |
| H200 | 4 (held-out) | — | additive model falsified (6 in-regime cells below H + T_ar) | |
Baseline = T_cmp/k + T_ar (Eq. 1); Model adds the TP-invariant additive term (Eq. 2), with g fit on in-regime TP-2 only and H fixed from the 8B microbench (zero TP-4 fitting). The validated claim is scoped to H100; H200 TP-4 is reported as a falsifying limit.
The step does not shrink, and the leftover is a constant
The in-regime TP-2→TP-4 shrink is 1.11× (1.18× including the disclosed outlier). Subtracting unscaled sharded compute and the all-reduce term from the measured step leaves a residual that is small at TP-2 but a tight positive constant (≈8–9 ms) at TP-4: as 1/k compute shrinks, more of the step is unexplained by compute, exactly as a TP-invariant additive term predicts. In the decode-step composition below, the additive term (red) is the largest component and is identical at TP-2 and TP-4; only compute (green) shrinks. The additive term is 66.2% of the step; communication is just 15.6%.
Decode-step composition (H100, N=1, ctx 2048). The TP-invariant additive term (red) is the largest single component and is identical at TP-2 and TP-4; only compute (green) shrinks. All-reduce (amber) is a minor slice. (The additive term is the fitted ~17 ms line; its directly-measured host-exposed part is ~5.8 ms.)
Held-out H100 TP-4: 4.4% MAPE with zero TP-4 fitting
With g from in-regime TP-2 and H(N) from the 8B microbenchmark, the model predicts the 12 held-out H100 TP-4 cells to 4.4% mean, 7.0% p95, 7.5% max MAPE — versus 16.9% mean (18.6% max) for the composed baseline on the same cells. The error bars below are the 12-rep measurement 95% CIs. (A freely-fit or batch-linear intercept does marginally better, 1.6–1.7%; we make no best-accuracy claim — the contribution is the mechanism.)
Held-out TP-4 prediction (12 H100 cells), with measurement 95% CIs. Compute scale fit on TP-2 only; host term fixed from the 8B microbenchmark; zero TP-4 fitting.
The additive term is TP-invariant: k-scaling fails by 38–70%
This is the load-bearing finding. We calibrate five host-model forms on the in-regime TP-2 cells (the disclosed outlier excluded from training) and evaluate them held-out on TP-4. On TP-2 alone they are indistinguishable; only held-out TP-4 separates them. Every k-invariant form predicts TP-4 well — a free constant 1.7%, a batch-linear intercept 1.6%, the parameter-free 8B-microbench value 4.4% — whereas forcing the term to scale with TP fails: H·k gives 70.3% and H/k gives 37.7%. The data thus strongly reject any k-scaling of the additive term.
| host-model form | TP-4 mean MAPE | TP-4 max MAPE | verdict |
|---|---|---|---|
| H = free constant | 1.7% | 4.1% | k-invariant ✓ |
| H = C0 + C1·N (batch-linear) | 1.6% | 3.7% | k-invariant ✓ |
| H = 8B microbench S(N), fixed (0 params) | 4.4% | 7.5% | k-invariant ✓ |
| H = C·k (grows with TP) | 70.3% | 73.1% | k-scaling ✗ |
| H = C/k (shrinks with TP) | 37.7% | 38.9% | k-scaling ✗ |
Held-out TP-4 MAPE for host-model forms calibrated on in-regime TP-2. Every k-invariant form (green) generalizes; forcing the term to scale with TP (red) fails by 38–70%.
Direct evidence: the 8B floor is flat TP-2→TP-4
To measure the host floor directly, independent of the 70B model, we run an 8B model (which fits on one GPU, so all TP degrees are reachable) at TP-1/2/4 in the host-dominated small-N/small-ctx regime. The decode-step floor is 9.5 ms at TP-1, rising to 11.6 ms at TP-2 — the one-time cost of standing up the multi-process distributed runtime — then flat at 11.7 ms at TP-4. The TP-2→TP-4 invariance is the central claim, and here it is measured directly: the host floor does not shrink with TP.
Direct evidence: 8B decode-step floor (N=1). A one-time multi-process penalty at TP-1→TP-2 (9.5→11.6 ms), then flat TP-2→TP-4 (11.6→11.7 ms) — the floor does not shrink with TP. (This 8B floor corroborates the flatness, not the ~17–18 ms magnitude of the 70B continuous-batching term.)
The keystone: a model-size-invariant host floor exists (direct nsys split)
The TP-floor measurement shows the per-step floor is flat across TP, but not whether it is host or GPU work. A direct nsys decomposition settles it. Because vLLM V1 runs the model in a separate EngineCore subprocess (defeating in-process profilers), we trace the whole process and difference an 8B single-GPU decode at 1 versus 512 tokens, which cancels load, warmup, and prefill; in eager mode the kernel-time sum is GPU-active time and the wall-minus-kernel gap is host-exposed. The 8B step splits into 6.41 ms GPU-kernel + 5.78 ms host-exposed (47%). On Llama-3.2-1B the GPU-kernel time falls 4.4× (6.41→1.45 ms) while the host-exposed term holds within 4% (5.78→5.56 ms): the floor is model-size-invariant — the defining signature of host/scheduler cost rather than GPU work. This directly proves a model-size-invariant host-exposed floor of ~5.8 ms exists; it is not the ~17–18 ms magnitude the model attributes to the additive term (which is ~3× larger), and a rank-0 trace of the 70B TP-2/4 runs is still required to attribute the larger 70B residual — which we leave to future work.
Direct host-vs-GPU split of the 8B decode step (H100, eager, N=1, ctx 2048) from differencing two nsys runs. Host-exposed time is ~5.8 ms and model-size-invariant (GPU falls 4.4× 8B→1B; host holds within 4%). This is the single-GPU (TP-1) floor; the TP-2→TP-4 invariance is established separately above.
All 48 cells and the one disclosed outlier
We report all 48 cells under the joint model. One configuration — TP-2, N=8, ctx 4096, the memory-tightest (70B on two GPUs at the largest batch×context) — mispredicts by 43.3%; its measured step is about 2× neighboring cells, consistent with KV/activation memory pressure leaving the clean decode regime. The in-regime criterion is declared a priori from memory, not from model error. It is excluded from all training and starred throughout. Excluding it moves the fitted scale from g=0.32 to g=0.30 and the H100 shrink from 1.18× to 1.11×; we report both and never tune to it. The 12 held-out H100 TP-4 cells are all in-regime (four GPUs give more headroom), which is why the held-out prediction is clean.
All 48 cells under the joint model (Eq. 2), predicted versus measured. The single point well above the diagonal is the disclosed TP-2 memory-pressure cell (N=8, ctx 4096), excluded from training.
Honest negative: the additive model is falsified on H200 TP-4
Where Adding GPUs Stops Paying Off
The value of the result divides cleanly by audience. For systems researchers it is a measured correction to a common attribution; for operators it is a closed-form planning tool that says, before provisioning, the marginal latency benefit of adding GPUs.
For systems researchers
On H100, the remaining TP-decode inefficiency is not communication. All-reduce is only 15.6% of the step on average, so communication-overlap kernels — the standard fix — target the wrong slice on H100. The inefficiency is an additive per-step term that does not divide by k, anchored in host/scheduler work the rank-0 driver pays once and structurally identical at k=2 and k=4. A latency model that lets the additive term scale with TP has a measured counterexample here: held-out, k-scaling forms fail by 38–70%. (The additive term itself is ~3× the directly-measured 5.78 ms host-exposed floor, so we claim TP-invariance, not a full host attribution.)
For operators
On H100, the TP-invariant additive term behaves as a practical decode floor that more parallelism does not reduce — though it is SKU-specific, not a hard physical bound (it is falsified on H200 TP-4). The crossover where the additive term overtakes the sharded compute — where adding GPUs stops making decode faster — is, for 70B on H100 at the batch sizes we measure, already reached by k=2. The model predicts the H100 decode step at each TP degree in closed form, so you can compute the marginal benefit of more GPUs before buying them, and know that the only way below the floor is to attack the host path.
This is one of a family of measurement deepdives built on the same closed-form decode mechanism model. A companion page stress-tests the same host term in a different direction — asking what CUDA-graph capture actually removes from the decode step. Read the companion: Does the Latency Mechanism Survive CUDA-Graph Capture?
Exactly What Was Measured
The claims are deliberately bounded. Each item names a specific way the result could be over-read and the precise scope that remains valid.
- The compute term carries one global ~3× scale g (8B-calibrated efficiencies over-predict realized 70B decode compute; g=0.30 corrects it downward). We disclose it, and the TP-invariance claim is independent of it — the additive term carries no free parameter and survives both the held-out baseline suite and the 8B floor experiment, neither of which uses g for the additive term.
- The additive term is ~3× the measured host floor — we claim TP-invariance + floor-existence, not a full host decomposition. The directly-measured host-exposed part is only 5.78 ms (8B single-GPU, nsys), while the additive term the model uses is ~17–18 ms. The excess is distributed-runtime / launch coordination, continuous-batching scheduler work, and compute the g<1 correction over-predicts — disclosed and not yet attributed. The supported claim is the weaker one: a TP-invariant additive term (anchored by, but larger than, the measured host floor) recovers TP decode, and a k-shrinking term does not.
- The additive model is FALSIFIED on a held-out H200 TP-4 SKU. 6 in-regime H200 TP-4 cells measure below the additive lower bound H + T_ar, so it is not a hard physical bound. The validated claim is therefore scoped to H100; H200 is reported as a falsifying limit, not cross-SKU generalization. Only the k-scaling rejection survives pooling with H200.
- Eager mode only. Host work is on the critical path in eager execution; CUDA-graph capture overlaps part of it but the cost persists (independent measurements confirm multi-GPU slowdowns are often CPU-bound and survive graph capture).
- The model is a point predictor, not a statistical match. The held-out H100 percent errors are small (4.4% mean) but only a minority of the 12 TP-4 predictions fall within the per-cell 12-rep 95% CIs (the CIs are tight, < 1 ms). We present it as an accurate provisioning predictor, not as statistically indistinguishable from the measurement at every cell.
- Coverage: 70B, vLLM, H100 (validated) / H200 (held-out limit), TP-2/4, batch ≤ 8, ctx ≤ 4096. TP-1 for 70B (does not fit one GPU), TP-8, pipeline/expert parallelism, graph mode, larger batch, and other engines are out of scope. The 'adding GPUs stops helping' statement is scoped to the H100 regime.
- One disclosed outlier (TP-2, N=8, ctx 4096) mispredicts by 43.3% under memory pressure. It is flagged out-of-regime by an a-priori memory criterion, excluded from all training and starred; we report both the with- and without-outlier shrink (1.18× / 1.11×) and never tune to it.
The validated conclusions hold for Llama-3.1-70B in BF16 under vLLM eager on PACE H100, at TP-2 and TP-4, batch ≤ 8 and context ≤ 4096, with the 8B TP-1/2/4 floor and the direct nsys host/GPU split measured separately as direct corroboration; H200 TP-4 is reported as a falsifying limit. The central finding — that recovering H100 TP decode latency requires an additive term that does not shrink with the TP degree, and that a model-size-invariant host-exposed floor demonstrably exists — is the part expected to generalize beyond this grid.