Week 8 — Chunked Prefill

Learning Objectives

Key Concepts

Setup & Configuration

# Server 1: WITHOUT chunked prefill (HOL blocking baseline)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --no-enable-chunked-prefill \
  --port 8000 --disable-log-requests

# Server 2: WITH chunked prefill (default chunk size)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-chunked-prefill \
  --port 8002 --disable-log-requests

# Server 3: Aggressive chunking (512 tokens per chunk)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-chunked-prefill \
  --max-num-batched-tokens 512 \
  --port 8003 --disable-log-requests

Experiments

python benchmarks/benchmark_serving.py \
  --backend vllm --port 8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 --request-rate 4 \
  2>&1 | tee results_no_chunk.txt

# Key metric to look for: ITL p99 vs p50 ratio
# High ratio = high variance = HOL blocking present
grep "ITL" results_no_chunk.txt

python benchmarks/benchmark_serving.py \
  --backend vllm --port 8002 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 --request-rate 4 \
  2>&1 | tee results_chunk_default.txt

for tokens in 256 512 1024 2048 4096; do
  echo "=== max_batched_tokens=${tokens} ==="

  # Start server
  vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-chunked-prefill \
    --max-num-batched-tokens $tokens \
    --port 8010 --disable-log-requests &
  SERVER_PID=$!

  # Wait for server to be ready
  until curl -s http://localhost:8010/health >/dev/null; do
    echo "Waiting for server..."; sleep 5
  done

  # Run benchmark
  python benchmarks/benchmark_serving.py \
    --backend vllm --port 8010 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 100 --request-rate 4 \
    2>&1 | tee results_chunk_${tokens}.txt

  # Graceful shutdown (NOT kill -9)
  kill -2 $SERVER_PID
  wait $SERVER_PID
done

import re, json, matplotlib.pyplot as plt

def parse_itl_timeseries(filepath):
    itls = []
    with open(filepath) as f:
        for line in f:
            m = re.search(r"itls=\[(.*?)\]", line)
            if m:
                itls.extend([float(x) for x in m.group(1).split(",") if x.strip()])
    return itls

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 4))

itl_no_chunk = parse_itl_timeseries("results_no_chunk.txt")
itl_chunked  = parse_itl_timeseries("results_chunk_default.txt")

ax1.plot(itl_no_chunk, alpha=0.7, color="#e74c3c")
ax1.set_title("No Chunked Prefill — ITL Spikes")
ax1.set_ylabel("ITL (ms)")

ax2.plot(itl_chunked, alpha=0.7, color="#2ecc71")
ax2.set_title("Chunked Prefill — Smooth ITL")
ax2.set_ylabel("ITL (ms)")

plt.tight_layout()
plt.savefig("itl_comparison.png", dpi=150)
print("Saved itl_comparison.png")

import json, random

requests = []
# 20% long prefills (3000 tokens)
for _ in range(20):
    requests.append({
        "prompt": "Document: " + "Lorem ipsum dolor sit amet, " * 200,
        "max_tokens": 64
    })
# 80% short requests (50 tokens)
for _ in range(80):
    requests.append({
        "prompt": "What is 2+2?",
        "max_tokens": 128
    })

random.shuffle(requests)
json.dump(requests, open("mixed_workload.json", "w"))
print(f"Mixed workload: 20 long + 80 short = {len(requests)} requests")

import re, matplotlib.pyplot as plt

chunk_sizes = [256, 512, 1024, 2048, 4096]
ttft_p50 = []
itl_p99  = []
itl_ratio = []

for c in chunk_sizes:
    with open(f"results_chunk_{c}.txt") as f:
        text = f.read()
    t = float(re.search(r"Median TTFT.*?:\s+([\d.]+)", text).group(1))
    i = float(re.search(r"P99 TPOT.*?:\s+([\d.]+)", text).group(1))
    m = float(re.search(r"Median TPOT.*?:\s+([\d.]+)", text).group(1))
    ttft_p50.append(t); itl_p99.append(i); itl_ratio.append(i/m if m else None)

print("chunk\tTTFT_p50\tITL_p99\tITL_ratio")
for c, t, i, r in zip(chunk_sizes, ttft_p50, itl_p99, itl_ratio):
    print(f"{c}\t{t:.1f}ms\t{i:.1f}ms\t{r:.2f}x")

Experiment Results

Hardware

All experiments run on the PACE Phoenix cluster, using Llama-3.1-8B-Instruct in BF16. We tested four GPU classes — NVIDIA H200 (143 GB), H100 80GB HBM3, A100 80GB PCIe, and L40S 48GB — to see how chunked prefill behaves on different memory/compute profiles. Workload: ShareGPT-style prompts (median ~120 input tokens, 150 prompts at request rate 6 req/s).

Chunked vs No-Chunked Across GPUs

ShareGPT median input is ~120 tokens — too short to trigger HOL blocking, so the steady-state ITL p99/p50 ratio stays near 1.0× across all configs. The HOL-stress test below isolates the effect on long prompts.

max-num-batched-tokens Sweep (H200)

The throughput peak at chunk=512 on H200 is the steady-state interleaving sweet spot for short ShareGPT prompts: smaller chunks let the scheduler batch more decodes per step, smoothing the pipeline. Larger chunks (4096) give the smoothest p99/p50 ratio because each prefill burst dominates one step instead of leaking into the next.

HOL-Stress Test: One Long Prompt + Three Short (H200)

To isolate the chunked-prefill benefit, we send one 4096-token prompt followed immediately by three 256-token prompts. Without chunked prefill, the long prompt monopolizes one forward pass and short prompts must wait. With chunked prefill, the long prompt's compute is interleaved across multiple steps.

The long prompt gets 47% faster with chunked prefill on H200 because its prefill compute now overlaps with decode work on the short requests. Short prompt latency is unchanged — proving chunking does not regress the fast path.

Figure 1: max-num-batched-tokens vs throughput on H200 (left) and HOL-stress speedup for the long prompt across configs (right).

HOL Test — Shorts under Long Prefill (2026-04-11 rerun, cross-GPU)

The HOL test in the audit script originally had a timing bug: time.sleep(0.5) delayed the short requests so long that the long prompt finished before they arrived, producing zero overlap and making chunked-vs-no-chunked look identical. The 2026-04-11 rerun fixed this to a 30 ms stagger and replaced the long prompt with a 3500-token input that stays under max_model_len=4096. Three short requests then arrive while the long prefill is still executing, so any HOL blocking must manifest in their latency.

HOL blocking is most visible on slower GPUs. On the Turing RTX 6000, a 3500-token prefill takes multiple seconds, so every short request waiting behind it stalls for the full prefill duration — chunking slashes that wait by 2.3×. On H200, one 3500-token prefill is about 374 ms, still long enough for a three-short-request batch to be measurably slowed (1.58× speedup from chunking). On H100, prefill is so fast (~318 ms) that by the time the shorts arrive 30 ms after dispatch, the long is almost done, and chunked vs no-chunked are statistically indistinguishable.

Figure: Shorts mean latency (log-scale ms) with chunked prefill vs without, across four GPU classes. RTX 6000 Turing shows the most dramatic 2.30× improvement; H100 is too fast to show any HOL effect at this prompt size.

Expected vs Actual

Metrics to Collect

Source Code Reading

Core Source Code Walkthrough

Below are real excerpts from vllm-continuum that implement the concepts you measured. Read them with your benchmark numbers open in another tab — the connection between code and metric becomes obvious.

vllm/v1/core/sched/scheduler.py:625 — chunked prefill check

# chunked prefill has to be enabled explicitly to allow
# pooling requests to be chunked
if not self.scheduler_config.chunked_prefill_enabled and \
    num_new_tokens > token_budget:
    self.waiting.pop_request()
    skipped_waiting_requests.prepend_request(request)
    continue

num_new_tokens = min(num_new_tokens, token_budget)

Without chunked prefill, a long prompt that exceeds max_num_batched_tokens is skipped this step, causing head-of-line blocking. WITH chunked prefill, num_new_tokens is clamped to whatever budget remains, the request runs partially this step and continues next step. This is the line that determines whether your serving system hangs on long prompts.

Written Analysis — Reference Answers

The answers below are grounded in the H100/H200/L40S measurements above, especially the HOL-stress test on H200 where one 4096-token prompt is queued behind three 256-token prompts.

Q1: Why is chunked prefill important for decode latency?

Q2: How to choose max-num-batched-tokens?

Q3: When does chunked prefill hurt?