Project Core Review

0. The honest framing — two halves, not a speedup paper

The two-half reframing is correct, but with one mismatch the data forces us to confront: the second half, as measured, does NOT demonstrate reduced inference time — it is a signal-fidelity and integration study with disclosed negative results.

Partial match. The paper calls it 'a per-step predictor plus a utility study'. 'Pure-input' overstates Half 1 (it needs microbenchmarks + ncu counters per architecture, plus one host-floor scalar per trace in calibrated mode). Half 2 is a utility study where two of four pre-registered claims FAIL; the one real improvement is keep-warm routing (−30 ms TTFT). A separate tier swaps the predictor into four tools, claiming accuracy-at-lower-cost, not speedup.

1. Part one — what the predictor actually computes

Half one predicts one served engine step. The classic approach decides the bottleneck per kernel from FLOPs vs bytes and divides by peak rates — needing two numbers a GPU rarely sustains, patched with an unexplained efficiency factor. We replace that factor with a small product of factors, each a concrete thing the GPU does after launch.

1.1 The step-time equation — and why it is not three corrections

No — three is an undercount; the honest structure is ~7 additive terms and 9+ factors. The step time is:

Here $T_{\mathrm{blocks}}$ is the GPU compute body of the step. It expands over the $N_L=32$ layers as:

where $t_{\mathrm{proj}}$ (the four projection GEMMs) and $t_{\mathrm{attn}}$ are per-layer, while $t_{\mathrm{light}}$ carries its own $N_L$. Each operator term in (1a) is itself a roofline max (eq 1c below); its compute branch efficiency is the five-factor model, clock as an explicit first-class factor:

Here $f_{\mathrm{clk}} = f_{\mathrm{sustained}}/f_{\mathrm{boost}}$ is the explicit clock factor, $\eta_{\mathrm{cluster}}$ is SM-fill, $\eta_{\mathrm{active}}$ the time-domain SM-occupancy, $\eta_{\mathrm{duty}}$ the tensor-pipe duty; delivery $\equiv 1$. The clock is its own factor — not absorbed into a fitted ceiling. Beyond compute the step adds a memory branch $\eta_{\mathrm{mem}}$, the attention term (four factors), the launch floor, the overlap leak, and sampling — ~7 terms / 9+ factors, not three.

Each operator term is a roofline max — compute branch vs memory branch

The five-factor $\eta_c$ of eq (1b) is only the COMPUTE branch. Each per-layer operator term in eq (1a) — the projection GEMMs of $t_{\mathrm{proj}}$ and the attention $t_{\mathrm{attn}}$ — is itself a roofline max. Take $t_{\mathrm{proj}}$ ($t_{\mathrm{attn}}$ has the same structure, §1.9): it sums the four projection GEMMs, each the larger of compute and memory time. For one GEMM: $M$ = new-token rows, $N_{op}\times K_{op}$ = weight-matrix shape, $\mathrm{BP}=2$ bytes, $P_{\mathrm{boost}}$ = datasheet boost FLOP/s, $\mathrm{BW}_{\mathrm{peak}}$ = datasheet HBM bandwidth. $\eta_c$ sets the compute ceiling, $\eta_{\mathrm{mem}}$ the memory ceiling:

At small $M$ (decode) the memory branch wins (full weight read → DRAM-bound); at large $M$ (prefill) compute wins — exactly the AI-vs-ridge regime column in Table A.

How $\eta_{\mathrm{mem}}$ is obtained — a measured pct-of-peak, not a derived stack

The memory branch's $\eta_{\mathrm{mem}}$ is a single number — the fraction of peak HBM bandwidth a memory-bound streaming kernel actually sustains, which is what turns the byte count $B$ into a time:

So it is neither derived from geometry (unlike $\eta_{\mathrm{cluster}}$) nor fitted (unlike $\eta_{\mathrm{duty}}$): it is read DIRECTLY off one hardware counter — Nsight Compute's gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed — at the $M{=}64$ DRAM-bound point. That percentage IS $\eta_{\mathrm{mem}}$, the memory-side analogue of $f_{\mathrm{clk}}$. One constant suffices because a streaming kernel saturates DRAM at a size-independent fraction of peak; only bytes $B$ and datasheet $\mathrm{BW}_{\mathrm{peak}}$ vary. Per-GPU ($M{=}64$ bench): H100 0.755, H200 0.669, L40S 0.884. A100-80 exception: bench 0.509 but engine $M{\le}32$ sustains 0.75 (kernel-dispatch gap) — 0.75 used.

How the four factors' values are determined (the exact counters)

Each factor maps to one specific measurement; only $f_{\mathrm{clk}}$ needs a per-deployment reading.

1.2 The frequency claim — valid, but sharper than the slogan

Partially valid. The clock droop is real and large — but from the real-DVFS loop, NOT the pinned ncu pass (which fixes the clock to isolate geometry, so its ratio is an artifact). Natural $f_{\mathrm{clk}}$ ≈ 0.68 (H100, 1350/1980, 32% droop), 0.74 (H200), 0.82 (A100-80), 0.56 (L40S). Under real DVFS H100 qkv tops near 681 TFLOP/s = 0.69 of the 989 spec, and the clock droop is essentially the whole gap (η_noclk near its ceiling at large M).

But 'lower clock makes GEMM slower' (a throughput decline) is FALSE on well-provisioned parts: H100/A100 decline 0.0%, H200 −0.6%, because rising geometric efficiency ($\eta_{\mathrm{noclk}}$ 0.18 to 0.95) masks the clock. Throughput only declines on tight-cap parts (L40S −15.4%, ±0.8 over 3 repeats; Blackwell ~3%, projected). A locked-clock control collapses the L40S decline to 0.0%.

See the companion deep dive for the full treatment. The Clock Is Not a Constant

Figure 1 (all four SKUs, qkv) — the masking mechanism, from measured data. Each panel is one GPU; H200 is the worked example. x: token count $M$. Red $f_{\mathrm{clk}}$ (real DVFS) steps down at the power cap; purple $\eta_{\mathrm{cluster}}$, orange $\eta_{\mathrm{active}}$ and blue $\eta_{\mathrm{duty}}$ (measured) rise; their product $\eta_{\mathrm{noclk}}$ rises 0.25 to 0.95; black achieved/peak rises then PLATEAUS — the clock droop is masked by rising geometry.

Every quantity behind the figure, traced to launch geometry: tile from the cuBLAS kernel name; cluster size / max-active from ncu (Hopper only); grid from launch__grid_size; output tiles $=\lceil M/M_t\rceil\lceil N/N_t\rceil$; tiles-per-CTA = output/grid; waves $=\lceil\text{grid}/N_{\mathrm{SM}}\rceil$.

Why $\eta_{\mathrm{duty}}$ rises with $M$ on Hopper — tiles-per-CTA, not more CTAs

The intuition "wave full ⇒ constant duty" holds for one-tile-per-CTA kernels, not Hopper: the grid saturates at the cluster cap (~112–120) beyond the knee and does not grow with $M$ — instead each persistent CTA walks tiles-per-CTA = output/grid tiles (1.0→3.4→12.8), amortising its one-time pipeline prologue. Measured duty is well described by $\eta_{\mathrm{duty}} = 1/(1 + 0.31/\text{tiles-per-CTA})$ to 3 decimals. $M$ adds tiles-per-CTA, not CTAs.

The four prediction formulas and their provenance — how far each factor is actually predicted

Given the tile + cluster size from the cuBLAS kernel name, what each factor of $\eta_c = f_{\mathrm{clk}}\,\eta_{\mathrm{cluster}}\,\eta_{\mathrm{active}}\,\eta_{\mathrm{duty}}$ is predicted by — four different provenance levels.

1.3 Why the SMs are not all filled — cluster co-residence cap, not a sync stall

Real, but the mechanism conflates two effects, and the Hopper-dominant one is not a sync stall. $\eta_{\mathrm{smused}}=\min(\text{grid},N_{\mathrm{SM}})/N_{\mathrm{SM}}$; at $M{=}2048$ the grid is 112 of 132 (0.848). cuBLAS uses 16-CTA (4×4) clusters; cudaOccupancyMaxActiveClusters caps co-residence at 7 (ncu launch__cluster_max_active-confirmed), so the persistent grid is 7×16=112 and 20 SMs are dark — GPC packing, not divisibility; a launch-time limit, not a runtime barrier.

On Ampere/Ada there are no clusters; the grid far exceeds SM count, $\eta_{\mathrm{smused}}=1.0$, and the residual is a wave tail (A100 11.1%, L40S 9.9% last-wave idle). At tiny sizes even Hopper has a true last-wave tail (o at $M{=}64$: 72 CTAs, $\eta_{\mathrm{wave}}=0.545$). The model captures both.

1.4 Why two factors, not one — occupancy time vs pipe duty

First, the premise mislabels them: the cluster-finishing effect is in the SM-fill factor (1.3), not here. The two factors here, defined in equation (3), are orthogonal: $\eta_{\mathrm{active}}$ is a TIME-domain occupancy (load-imbalance / last-wave idle time); $\eta_{\mathrm{duty}}$ is a PIPE-domain duty (the fraction of active cycles the tensor pipe is issuing).

How obtained: both from pinned-clock (--clock-control base) Nsight Compute counters — $\eta_{\mathrm{active}}$ = (sm__cycles_active.sum ÷ sm__cycles_elapsed.avg), $\eta_{\mathrm{duty}}$ = sm__pipe_tensor_cycles_active.avg.pct.../100 (H200 qkv M=2048: 0.95, 0.955). To estimate: $\eta_{\mathrm{cluster}}$ is pure-parameter from the grid; $\eta_{\mathrm{duty}}$ saturates to a per-generation constant at large $M$; only $f_{\mathrm{clk}}$ needs a per-deployment reading.

They multiply to the single proposed quantity, but staying split is correct — different physics and shape-dependence. At $M{=}2048$: H100 0.946/0.955, A100 0.883/0.918, L40S 0.898/0.962. $\eta_{\mathrm{active}}$ falls from many-wave tails (A100 runs 4 waves); $\eta_{\mathrm{duty}}$ falls from pipe stalls. For transfer the model folds $\eta_{\mathrm{active}}$ into the residual but keeps $\eta_{\mathrm{duty}}$ separate. Arithmetically one suffices; diagnostically and for transfer, two are needed.

1.5 The memory side and L2 reuse — a bytes correction, not a bandwidth α

It is a bytes-moved correction, the dual of the compute α: compute α derates the peak rate; L2 reuse derates the effective bytes charged. Cold profiling charges DRAM traffic warm execution never moves — cold/warm DRAM-read ratio for attention at 512 tokens is 73.5x (H100), 147x (H200), 246x (A100-80); collapses to ~1–2x at 2048 tokens. Modeled as a numerator (effective vs ideal bytes), never a sub-1 bandwidth factor.

The byte saving converts to almost no time: recoverable ceiling at $M{=}2048$ is 0.31/0.12/0.20 ms (under 0.65% of the ~50 ms pass; under 0.1% with pollution). This is a standalone paper ('Worth the Bytes?') — resource real, value bounded — whose causal gate failed, so it reports a descriptive bound.

1.6 The host overlap term — a bounded exposure, eager-mode α ≈ 1

A bounded overlap fraction from a two-resource argument; in eager mode it resolves to a measured, exposed host floor. The realised time lies between max (pure overlap) and sum (full serialisation). The host span splits into a shadowable dispatch part and an irreducible serial residual $S(N)$. With $T_{\mathrm{shadow}}=\max(0,T_{\mathrm{gpu}}-N_{\mathrm{kern}}T_{\mathrm{launch}})$, the exposed fraction is

Neither a fitted scalar nor max-plus-residual: a curve, fully exposed for short GPU steps, fully hidden when the GPU supplies shadow. $S(N)$ is calibrated once per SKU on a disjoint synthetic grid (floor 2.4–4.4 ms, slope ≤0.02 ms/req). Eager-mode exposure is $\alpha\approx1$: Hopper $N{=}1$ decode measures 15.6–16.6 ms while GPU work is 5.0–6.7 ms, leaving a 9.5–11.2 ms exposed host residual.

A SEPARATE host term, outside the max in (1), not on the GEMM path. The bounded form is load-bearing: a fitted scalar α implied per-cell exposures from −12.7 to +9.9 (52/108 cells outside [0,1]), proving a scalar was absorbing GPU-model error; the bounded form forces those errors back into the GPU terms.

1.7 Is it online learning? No.

An overstatement — no online/incremental/gradient learning anywhere. Everything is computed or measured once, offline; the per-architecture clock-independent geometric constants ($\eta_{\mathrm{cluster}}$, $\eta_{\mathrm{active}}$, $\eta_{\mathrm{duty}}$, $\eta_{\mathrm{mem}}$) come from standalone microbenchmarks, not eval traces, and the clock factor $f_{\mathrm{clk}}$ from one per-deployment reading. The only learned quantity is the host floor $S(N)=a+b(N-1)$, a one-shot closed-form least-squares per SKU (no iteration, so no convergence-in-steps). An optional mode recovers one scalar per trace. The only `.fit()` models in the repo are baseline competitors.

The honest phrasing: a mechanism model with microbenchmark-calibrated per-architecture constants, one closed-form floor curve per SKU, optionally one scalar per deployment. The sellable property is 'needs no per-workload training' — not 'online learning'.

1.8 How accurate are we, and where do we fail?

Ground truth = measured median step wall of a real vLLM engine. Two modes differ ~10x. FROZEN (no per-trace fit): arrival-driven decode 2.6–7.1%, but saturated all-at-once decode 18.6–30.9%. CALIBRATED (one scalar/trace): decode 0.8–3.7%, mixed 5.2–9.3%. Controlled grid: mean 4.34%, p95 9.33%, max 15.87% (200 cells).

1.9 The complete per-step model — every term, formula, and provenance

This is the whole model written out: every additive term, with its exact formula from the source of truth (src/mechanism/predictor.py) and a one-line provenance note — DERIVED (exact from geometry, no fit), MEASURED (a per-GPU constant from a standalone microbench/datasheet), or FITTED (a free parameter least-squares-fit to a microbench). Llama-3.1-8B constants: HID=4096, INTER=14336, HQ=32, HKV=8, DH=128, NL=32, VOCAB=128256, BP=2, QKV_OUT=6144, N_kern=12·NL+10=394. M = T_new = new tokens in the step; N = requests. The top equation is eq (1) above; below expands its terms.

Term 1 — $t_{\mathrm{proj}}$: the projection GEMMs (per layer; roofline max per operator)

Each of the four weight matrices (qkv, o, gate_up, down) is the larger of a compute time and a memory-streaming time; the four-factor $\eta_c$ and this roofline are derived in §1.1–§1.2, so we only restate the one line here.

Provenance. FLOPs and bytes are DERIVED-exact from the shapes; $\eta_c$ is the four-factor model (§1.2 Table C); $\eta_{\mathrm{mem}}$ is a MEASURED per-GPU constant (M=64 gpu__dram_throughput: H100 0.755, H200 0.669, A100-80 0.75, L40S 0.884).

Term 2 — $t_{\mathrm{attn}}$: attention (roofline max of a causal-work compute branch and a KV-read memory branch)

Attention is the larger of a prefill-shaped compute branch (only requests with $s_{\mathrm{new}}>1$ contribute the causal work $\sigma$) and a KV-cache-read memory branch (all requests). Its efficiency is three factors: a measured ceiling, a softmax-amortisation saturation, and a causal-shape penalty.

Provenance. $\sigma$, the FLOPs, and kv_bytes are DERIVED-exact; $\eta_{\mathrm{attn,ceil}}$ is MEASURED (per-GPU attention microbench); $c_{\mathrm{attn}}$ and $p_{\mathrm{attn}}$ are FITTED (softmax-amortisation + causal-shape, from the attention microbench); $\eta_{\mathrm{mem}}$ is MEASURED (same constant as Term 1).

Term 3 — $t_{\mathrm{light}}$: light kernels + LM head

Light kernels are the memory-bound elementwise passes per token per layer (two rmsnorms, two residual adds, rope on q and k, the KV-cache write, act_and_mul); the LM head is the final logit projection, weight-read-bound.

Provenance. per_tok_layer bytes are DERIVED-exact and trace-validated (L40S T=16384 predicted 134 ms vs 134 ms measured); $\eta_{\mathrm{mem}}$ is MEASURED; the LM head is weight-read-bound, so rows move it <0.15%.

Term 4 — $T_{\mathrm{dispatch}}=N_{\mathrm{kern}}\,T_{\mathrm{launch}}$: the kernel-launch / dispatch floor

A step cannot finish faster than the GPU can launch its kernels back to back; this floor enters as an explicit max against $T_{\mathrm{blocks}}$ (eq 1).

Provenance. $N_{\mathrm{kern}}$ is DERIVED-structural (kernels/layer × layers, +10); $T_{\mathrm{launch}}$ is MEASURED per-GPU (~7–8 µs).

Term 5 — $\alpha(N,T_{\mathrm{new}})\,S(N)$: the host-overlap leak

$S(N)$ is the scheduler / host step floor, growing with the batch; $\alpha$ is the exposed fraction. In eager mode (this paper's scope) $\alpha=1$ — the host runs serially on the critical path, fully exposed; an async / CUDA-graph engine would have $\alpha<1$, bounded by the GPU shadow.

Provenance. $S(N)$ is MEASURED per-GPU ($s_{\mathrm{floor}}$: H100 2.37, H200 3.73, A100-80 2.81, L40S 21.0 ms [generate-proto, recal pending]; $s_{\mathrm{slope}}\le 0.02$ ms/req); $\alpha=1$ is MEASURED for eager (the real-trace host residual is exposed).

Term 6 — $T_{\mathrm{samp}}$: sampling time

The token-sampling cost, a per-GPU constant (~1.5 ms).

Provenance. MEASURED constant.

Master provenance table

2. Part two — the utility story, told honestly

The reframing wants, per paper, the predictor it uses and where ours plugs in, ideally as an abstract 'tested on N systems, improved metric by X%'. Here is the per-system reality from the raw ledger — weaker than 'we improve everything'; the honest claim is a coverage-and-cost story.

2.1 The per-system integration table

Read the table left-to-right as the five questions: which system, the predictor it ships with, the code hook where ours is swapped in, a one-glance verdict, the head-to-head numbers (always ours vs theirs), and whether the evidence is a real trace or a simulator. The five marks:

2.2 Is Random Forest really worse? No — we are.

The RF is not worse — we are, on Vidur's home turf: decode 7.6% vs 5.6%, prefill 21% vs 0.9%, agg 8.4% vs 5.3% MAPE. The RF trained on exactly this data; we read none. Our only advantage is cost and coverage: ~0.001 s + 0 rows vs 158 s + 44k rows (~10⁵× cheaper), plus cells the RF cannot produce at all.

2.3 What we changed in QLM

We replaced `RWTEstimator.get_waiting_time` — a flat scalar (requests × 100 tokens / 700 tok/s, no sequence or hardware axis) — with our per-SKU, sequence-aware `predict_step`. The estimate's per-step MAPE drops from 111–123% to 0.9–4.4% across four GPUs, and cross-SKU ranking goes from degenerate to Spearman 1.0.

2.4 The honest abstract claim

The honest claim is coverage-and-cost, not improvement: 'We drop a fit-free predictor into the cost function of five published serving systems on their own code; with zero target-workload profiling it matches the profiled fit on its decode home turf (DistServe 1.8% vs 2.3%), replaces a hardware-blind scalar with a hardware-aware estimate (QLM 0.9–4.4% vs 111–123%), and supplies model×GPU cells two tools cannot model — at ~5 orders of magnitude lower calibration cost.'

2.5 Where precision actually pays — the value-of-precision map (E4)

This is the question the whole project exists to answer: four mechanisms, each tested by a controlled ε-sweep against MEASURED steps and a STRONG baseline. The honest finding is a map — precision pays only where the step cost is decision-sensitive AND the loss is asymmetric; elsewhere a margin absorbs the error for free.

Method: controlled ε-sweep + cross-predictor Spearman; ground truth = measured vLLM steps + one live H100 QLM run, never the predictor itself.

3. Literature anchors for the framing

The framing rests on two literature claims flagged for survey: prior work decides the bottleneck per kernel from FLOPs vs bytes, then multiplies peaks by an unexplained efficiency factor. A fresh survey is folded in below.

3.1 Per-kernel roofline / op-by-op bottleneck

Foundational: the Roofline model (Williams, Waterman, Patterson, CACM 2009) — attainable rate $=\min(\text{peak compute}, \text{peak bandwidth}\times\text{intensity})$, no discount. The cleanest verified op-by-op LLM instance is LLM-Viewer (Yuan et al., arXiv:2402.16363, 2024), raw peak with no efficiency factor; Pope et al. (MLSys 2023, arXiv:2211.05102) reason the same way; LIFE (arXiv:2508.00904) spreads efficiency per-operator. Class framing: bottleneck from per-kernel FLOPs vs bytes; efficiencies are constants, lookups, or learned residuals.

3.2 The unexplained efficiency factor

The clearest fitted-but-unexplained factor is GenZ (Bambhaniya et al., arXiv:2406.01698): per-op time $=\max(\text{compute}/(\text{FLOPS}\cdot\text{Eff}_C), \text{memory}/(\text{BW}\cdot\text{Eff}_{\mathrm{mem}}))$ with profiled values (A100 0.40, H100 0.55–0.75, MI300X 0.25), empirically profiled not derived; GenZ follows Vidur's profiling. The concept's origin is MFU from PaLM (Chowdhery et al., arXiv:2204.02311; PaLM-540B 46.2%, no closed form) and Megatron-LM (~52% of peak). FlashAttention (Dao et al., 2022) headlines an achieved-fraction; FlashAttention-3's reported 0.55 actually ranges 0.066–0.610 on H200. The sharpest foil is NeuSight (ASPLOS 2025, arXiv:2407.13853), which predicts utilisation with per-op MLPs instead of explaining it.

4. What the data cannot yet support — the missing experiments

These were the four missing experiments; E1, E3, E4 have now RUN (E4 = the §2.5 value-of-precision map), E2 still needs PACE staging. Each card records its outcome.

5. Mental model versus the data — a scorecard

A one-screen summary: what the reframing assumed versus what the raw files show. Green = supported, amber = partly / needs rephrasing, red = needs a new experiment.