A complete mechanistic model of one served LLM engine step — and an honest account of the Vidur comparison: a memorising random forest beats us on absolute accuracy on already-profiled hardware, yet cannot make the capacity-planning decision it exists for.
Llama-3.1-8B shapes · bf16 · A100-80 / H100 / H200 / L40S · NVML + Nsight-Compute + cuBLAS dispatch-log telemetry
This deep-dive documents everything from re-examining the per-step predictor ('m2'). It answers one question: how long does one served engine step take, given only the shapes of the work? The classic FLOPs/peak ÷ bytes/peak approach needs two numbers the silicon never sustains, patched with one unexplained fitted constant. We replace that constant with a small product of factors, each a concrete post-launch GPU behaviour — and pull the SM clock out as the most important.
One engine step's total time is the boxed equation below (src/mechanism/predictor.py). Every symbol is defined before use.
$\Tblocks$ is the GPU compute body. $\Nkern\,\Tlaunch$ is the host cost of launching every kernel; the $\max$ models CPU launch overlapping the GPU body. $\alpha(N,T_{\mathrm{new}})\,\Sn$ is host overhead scaling with a sampled-token function $\Sn$. $\Tsamp$ is the fixed sampling tail. The host terms are why calibrated mode fits one host-floor scalar per trace; everything in $\Tblocks$ is mechanism.
$\Tblocks$ expands over the $N_L=32$ layers as:
$t_{\mathrm{proj}}$ is one layer's four projection GEMMs — qkv, o, gate_up, down — and $t_{\mathrm{attn}}$ is that layer's attention; both per-layer. $t_{\mathrm{light}}$ (layernorms, RoPE, residual adds, activation, KV writes + LM head) carries its own layer count. Each operator term is itself a roofline max of a compute and a memory branch:
For a GEMM of token count $M$, output width $N$, contraction $K$: the compute branch divides $2MNK$ FLOPs by the achieved compute rate; the memory branch divides bytes (inputs + weight + output, bf16) by achieved bandwidth. $\Pboost$ is datasheet boost TFLOP/s, $\mathrm{BW}$ peak bandwidth; $\etac$ and $\etamem$ are the two efficiency factors the rest of this page is about. The larger branch is the bottleneck. This per-operator roofline-max is all of $\Tblocks$.
The compute-branch efficiency $\etac$ is the single number every roofline model hides in a fitted ceiling $\etabar$. We replace it with a product of factors, each a concrete post-launch GPU behaviour:
Sustained clock ÷ boost clock. A measured input per operating point. The term every boost model sets to 1 — and Section 3 shows it droops 20–44%.
How completely the grid fills the SMs. A partial last wave idles SMs. This is the factor Section 6's η_wave fix makes per-operator.
Cycles the tensor cores issue MMA vs stall on operand movement. Plateaus near 99% at large M under a pinned clock.
Steady-state duty of the compute pipe across the kernel, including ramp and tail. ncu-measured per cell.
A single ceiling works on its own card but conflates a transferable quantity (geometric efficiency) with a non-transferable one (the involuntary clock). We tested transfer with leave-one-out on 39 compute-bound GEMM cells across four SKUs. The deployable form — one per-architecture ceiling (same budget as the constant) times a derived $\etacluster$, no Nsight counter at deploy — beats the constant out of sample:
| GEMM efficiency model | In-sample MAPE | Leave-one-out MAPE |
|---|---|---|
| Per-architecture constant ceiling (baseline) | 16.87% | 18.75% |
| Deployed min(1, AI/ridge) | 18.10% | — |
| FIX: η_wave × per-arch ceiling (ours) | 9.86% | 11.36% |
| ncu-measured tc_util (oracle upper bound) | 7.06% | — |
Source: results/gemm_deployable_eta.json. Leave-one-out: 18.75% constant vs 11.36% ours — a ~7-pt gap that holds out of sample, so the win is transfer, not fitting.
Of Equation (2)'s four factors, the clock $\fclk$ is the one prior work most buries. Under sustained projection-GEMM load the SM clock droops with token count by 20–44% across data-centre GPUs, so a datasheet-boost roofline mis-prices throughput by a mean 29%. Exposing the clock recovers most of that error (to 9.6%).
Every data-centre card shows a piecewise-decreasing curve, not one clean step. Small $M$: low power, clock at boost. As $M$ grows, board power rises to the sustained limit; the clock falls to stay within the cap; at large $M$ it settles drooped. Throughput = $\Pboost\cdot\fclk\cdot\etanoclk$, where $\etanoclk=\etacluster\cdot\etaactive\cdot\etaduty$.
A clock depending on token count, power, and temperature is what a single ordered sweep can fake. Each GPU's full curve was measured five times in randomised order with cooldown, aggregated into a median with 95% CI. Half-widths are tight (8–33 MHz); on the H200 boost cells reproduce with zero variance.
| GPU | Boost (MHz) | Sustained @ M=16384 | 95% CI ± | qkv droop | Large-M throughput |
|---|---|---|---|---|---|
| A100-80GB | 1410 | 1125 | ±20.4 | 20.2% | rises (not tight-capped) |
| H200-SXM | 1980 | 1455 | ±10.2 | 26.5% | ≈ flat (−0.6%) |
| H100-SXM | 1980 | 1380 | ±8.3 | 30.3% | rises |
| L40S | 2520 | 1425 | ±33.3 | 43.5% | declines −15% |
Source: results/freq_study/clocktel_ci_summary.json + per-SKU files (n=5 randomised). Droop referenced to fixed datasheet boost.
The droop is universal but its throughput consequence is not. On well-provisioned cards rising $\etanoclk$ outpaces the falling clock (throughput climbs, droop masked); on the tight-cap L40S the clock overwhelms it (−15% at large M). A controlled cap sweep on a local RTX 5090 (575→400 W → 2730→2452 MHz) shows the clock tracks the cap monotonically. Honest null: that card never declined (not tight-capped enough at 575 W), so the full cap→clock→decline chain remains a bounded claim. Source: results/freq_study/cross_cap_causal_5090.json.
The right way to judge a predictor is the full scenario matrix. The wins are decisive and cheap; the one absolute-accuracy loss — against Vidur on a GPU it has already profiled — is real, and Section 5 shows it is a loss on the wrong metric.
| Scenario | Ours | Baseline | Note |
|---|---|---|---|
| DistServe OPT-13B decode | 1.78% | poly 2.27% | real engine trace · WIN |
| QLM estimator MAPE | 0.9–4.4% | native 111–123% | real steps; huge margin · WIN |
| GEMM deployable factor (LOO) | 11.4% | constant 18.8% | transfer, not fitting · WIN |
| freq explicit-clock | 9.6% | datasheet 29% | clock pulled out of η̄ · WIN |
| Vidur SKU-ranking decision (§5) | top-1 100% | RF: cannot rank | 0 profiling rows · WIN |
| Vidur absolute MAPE, A100 home (§5) | 8.4% all | RF 5.3% | RF memorises its table · LOSS |
Sources as listed. QLM live SLO (E3) tie; E4 neutral. The single absolute-MAPE loss is dissected in Section 5.
Vidur's predictor exists to make capacity-planning decisions — which SKU, engine, or parallelism is best — not absolute per-operator latency on already-profiled hardware. Judged on that purpose, the comparison inverts. Three honest parts: who is intrusive, where we genuinely lose, and the decision the memoriser cannot make.
m2 predicts from datasheet specs + mechanism, ZERO profiling traces. Vidur's RF is intrusive: it interpolates its own profiled per-operator table — the model must first run on each SKU (~44k rows/GPU) before it can predict there.
On the A100 home SKU, full per-step MAPE — ours 21.1% prefill / 7.6% decode / 8.4% all, vs RF 0.88% / 5.6% / 5.3%. We lose on absolute accuracy because the RF memorises the table that IS the ground truth, and we use none of it. Stated plainly.
Rank {A100-80, H100, H200, L40S} by GEMM step time per workload. m2, zero profiling, gets top-1 hit-rate 100% and mean Spearman +1.00 over 13 workloads. The RF can score only its one profiled home SKU — it cannot rank hardware it has never run, so it cannot make the decision at all.
The ranking is not trivial. The correct order changes across the token range — at small $M$ the bandwidth-rich H200 leads (projections are memory-bound), and the order shifts among H100/H200/A100 as the work becomes compute-bound. m2 tracks every ordering from datasheet specs alone, including the bandwidth-vs-compute crossover — exactly the judgement a capacity planner needs and a single-SKU profiler cannot supply.
Sources: results/vidur_decision_quality.json and part2_integrations/vidur/accuracy_summary.json.
Ranking hardware from specs only works if the per-step prediction is accurate enough that true orderings are not flipped by model error. We found and fixed three real GEMM-level errors eating into that margin. They are presented as the reason the decision result holds — not a standalone accuracy chase — and the last is a cautionary tale about which metric to optimise.
The ladder is read from the logs — qkv: 64×64 from M=64, 128×128 at M=256, 256×128 at M=512, 128×256 at M=4096. Held-out: 25/36 exact tile hits, η_wave MAPE 5.77%, so it generalises. η_wave more than halves the aggregate served-projection error (8.12% → 3.64%); the principal GEMM fix. Source: results/gemm_etawave_served_fix.json.
A second, independent bias lived in the large-M compute branch. A single $\etabar=0.75$ over-predicts the narrow projections — measured per-operator as qkv 0.675, o 0.703, gate_up 0.744, down 0.748. The single 0.75 made qkv/o under-predict large-M time by 8.5%/3.9%; per-operator ceilings cut the bias toward −0.4%. Caveat: $\etabar$ and η_wave are not freely separable (the η_wave tail constant was calibrated against 0.75), so they are applied together with the down projection gated out. This is the recommended deployable stack: it improves the served-projection aggregate from 3.64% to 3.28%. Source: results/gemm_perop_etabar.json, results/gemm_full_fix_vidur.json.
The third error was a per-operator memory-branch spread a single $\etamem=0.75$ cannot match: at M=1 o-proj is −10.5% while gate_up is +10.3%. The cause is real — achieved HBM bandwidth ramps with weight-read size, so a 33 MB o-proj weight saturates HBM far less than a 235 MB gate_up. The fix η_mem(W) = 0.891 − 1.171/√(W_MB) tightens the per-operator memory-branch MAPE (M≤16) from 5.3% to 2.8%, held-out 2.9%.
A fourth refinement — re-fitting the attention prefill constant (c_attn 2×10⁵ → 3×10⁵) — cuts prefill-attention MAPE 13.9% → 11.0% (held-out 10.9%); results/attn_prefill_refit.json. Step-level: results/gemm_full_fix_vidur.json.
Section 5's ranking is computed on GEMM step time. The cross-GPU transfer test (E2) is the full-step confirmation: evaluate the complete per-step predictor on a GPU the forest was never profiled on. There it must extrapolate, while our predictor only needs that architecture's datasheet constants and a cheap clock reading. E2 profiles on A100, evaluates on unseen H100/H200, comparing a 2×2 of (RF vs ours) × (home vs unseen).
What this is: a complete, mechanistic, non-intrusive per-step predictor with explainable trace-free constants, that wins the capacity-planning decision (rank unprofiled hardware: top-1 100%, Spearman +1.00, zero profiling rows) a memorising baseline cannot make. What it is not: a speedup, not the most accurate predictor on already-profiled hardware, and not yet a clock predictor. Honest boundaries throughout: the decline chain is proven only to cap → clock; the L40S ceiling rests on four cells; the η_mem law is correct per operator but kept out of the step model; the full-step cross-GPU confirmation is in flight. Every number traces to results/ or part2_integrations/.
Non-intrusive SKU ranking (top-1 100%, Spearman +1.00, 0 rows); five-factor LOO transfer; clock droop (n=5, four SKUs); cap→clock causal link; η_wave (8.1%→3.6%), config D (3.3%).
E2 cross-GPU transfer on PACE (H100/H200) — the full-step confirmation; folding config D + attention re-fit into predictor.py.
Forecast f_clk from the cap; a tight-cap controllable card to close cap→clock→decline; a context-length term for attention; the mid-M decode roofline knee.