Week 7 — Continuous vs Static Batching

Learning Objectives

Key Concepts

Setup & Configuration

# Install dependencies (run in compute node via salloc — not on login node)
pip install vllm transformers accelerate

# Download ShareGPT for variable-length workload
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Verify vLLM version and locate benchmark scripts
python -c "import vllm; print(vllm.__version__)"
ls $(python -c "import vllm,os; print(os.path.dirname(vllm.__file__))")/../benchmarks/

# Generate sonnet (fixed-length) workload — all outputs ~128 tokens
python - <<'EOF'
import json
sonnets = [
    {"conversations": [{"from": "human", "value": f"Write a 14-line Shakespearean sonnet about topic number {i}. Follow strict iambic pentameter."}]}
    for i in range(300)
]
with open("sonnet_fixed.json", "w") as f:
    json.dump(sonnets, f)
print("Generated 300 fixed-length sonnet prompts")
EOF

# Check GPU availability
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader

Experiments

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, time

model_id = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id, padding_side="left")
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto")

prompts = ["Explain how large language models work."] * 16

def bench_static(max_new_tokens):
    enc = tok(prompts, return_tensors="pt", padding=True, truncation=True).to("cuda")
    t0 = time.time()
    out = model.generate(**enc, max_new_tokens=max_new_tokens, do_sample=False)
    elapsed = time.time() - t0
    n_out = (out.shape[1] - enc["input_ids"].shape[1]) * len(prompts)
    print(f"max_new_tokens={max_new_tokens}: {n_out/elapsed:.1f} tok/s, elapsed={elapsed:.1f}s")

bench_static(128)   # uniform-length baseline
bench_static(512)   # all padded to 512, simulates HOL blocking

# Start vLLM server
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 --disable-log-requests &
sleep 30

# Request-rate sweep: rr = 1, 4, 10, inf
for rr in 1 4 10 inf; do
  echo "=== request-rate: $rr ==="
  python benchmarks/benchmark_serving.py \
    --backend vllm \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 300 \
    --request-rate $rr \
    --port 8000 \
    2>&1 | tee results_vllm_rr${rr}.txt
done

kill %1

# Variance comparison at rr=10
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 &
sleep 30

for dataset in ShareGPT_V3_unfiltered_cleaned_split sonnet_fixed; do
  python benchmarks/benchmark_serving.py \
    --backend vllm \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name sharegpt \
    --dataset-path ${dataset}.json \
    --num-prompts 300 --request-rate 10 \
    2>&1 | tee results_variance_${dataset}.txt
done

kill %1

# vLLM offline mode (no server needed — maximum throughput)
python benchmarks/benchmark_throughput.py \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 300 \
  2>&1 | tee results_offline_sharegpt.txt

# vLLM offline on sonnet (fixed-length control)
python benchmarks/benchmark_throughput.py \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path sonnet_fixed.json \
  --num-prompts 300 \
  2>&1 | tee results_offline_sonnet.txt

# Run in a separate terminal alongside each experiment
nvidia-smi dmon -s u -d 1 -f gpu_util_hf_static.txt &
# ... run HF static experiment, then kill dmon ...
kill %1

nvidia-smi dmon -s u -d 1 -f gpu_util_vllm_cont.txt &
# ... run vLLM rr=10 experiment, then kill dmon ...
kill %1

# Quick matplotlib comparison (optional)
python - <<'EOF'
import matplotlib.pyplot as plt, re

def parse_dmon(p):
    return [int(m.group(1)) for l in open(p) if (m:=re.match(r'\s*\d+\s+(\d+)',l))]

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(13,4))
ax1.plot(parse_dmon("gpu_util_hf_static.txt")); ax1.set_ylim(0,105)
ax1.set_title("HF Static Batching — SM Util %")
ax2.plot(parse_dmon("gpu_util_vllm_cont.txt"), color="orange"); ax2.set_ylim(0,105)
ax2.set_title("vLLM Continuous Batching — SM Util %")
plt.tight_layout(); plt.savefig("gpu_util_comparison.png", dpi=150)
print("Saved gpu_util_comparison.png")
EOF

Experiment Results

Hardware

Experiments run on NVIDIA H200 SXM5 141GB, H100 SXM5 80GB HBM3, and L40S 48GB (PACE Phoenix cluster), using Llama-3.1-8B-Instruct in BF16, vLLM continuous batching, ShareGPT workload.

vLLM Continuous Batching — TTFT vs Request Rate

Variable vs Fixed Output Length — Throughput (rr=inf)

Figure 1: TTFT median vs request rate (rr=1,4,10,inf) and peak throughput — H200, H100, A100 PCIe, L40S continuous batching

Peak Throughput @ rr=inf (2026-04-11 rerun)

Cross-GPU comparison of sustained output throughput under infinite arrival rate (all 150 prompts dispatched immediately). This is the most important single number for batch/offline serving decisions: it tells you how much a GPU can push when you stop caring about tail latency and just want aggregate tokens-per-second. The ratios should match the HBM bandwidth hierarchy because decode is memory-bandwidth-bound for 8B models.

The ratios track HBM bandwidth almost perfectly for H200/H100/L40S. H200 is 1.43× H100 in bandwidth and 1.18× in throughput. L40S is bandwidth-limited: with 1/4 the bandwidth of H200, it gets roughly 1/4 the throughput. RTX 6000 (Turing) lags disproportionately because Turing lacks native BF16 support and vLLM auto-falls back to FP16, pushing decode into compute-bound territory where the weak Turing tensor cores become the bottleneck.

Figure: Peak output throughput (tok/s, log scale) at rr=inf across four GPU classes, with HBM bandwidth labelled for each. The bars follow the bandwidth hierarchy for H200/H100/L40S.

Expected vs Actual

Metrics to Collect

Source Code Reading

Core Source Code Walkthrough

Below are real excerpts from vllm-continuum that implement the concepts you measured. Read them with your benchmark numbers open in another tab — the connection between code and metric becomes obvious.

vllm/v1/core/sched/scheduler.py:280 — schedule()

def schedule(self) -> SchedulerOutput:
    # NOTE on the scheduling algorithm:
    # There's no "decoding phase" nor "prefill phase" in the scheduler.
    # Each request just has the num_computed_tokens and
    # num_tokens_with_spec.
    # At each step, the scheduler tries to assign tokens to the requests
    # so that each request's num_computed_tokens can catch up its
    # num_tokens_with_spec. This is general enough to cover
    # chunked prefills, prefix caching, speculative decoding,
    # and the "jump decoding" optimization in the future.

The single most important comment in vLLM. The unified abstraction 'no prefill phase, no decode phase' is what makes continuous batching possible. Each call returns a SchedulerOutput describing exactly which tokens to compute for which requests this step. New requests, in-progress decodes, and chunked prefill chunks all coexist in one batch.

Written Analysis — Reference Answers

Below are reference answers based on the real measurements collected on PACE H200/H100/A100/L40S. Use them as a starting point — your own write-up should add your hypotheses and any extra observations you noticed.

Q1: How does continuous batching avoid head-of-line blocking?

Q2: Comparing vLLM continuous vs HF static batching

Q3: Why does ITL only grow ~20% at high concurrency?