Week 11 — Speculative Decoding

Learning Objectives

Key Concepts

Three Proposer Strategies

Strategy Comparison Matrix

Setup & Configuration

# Verify vLLM installation and check version supports spec decode
pip show vllm

# Pre-download TinyLlama draft model weights
python -c "from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('TinyLlama/TinyLlama-1.1B-Chat-v1.0')"

# Baseline server: standard autoregressive decoding, no spec decode
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 --disable-log-requests &

# Draft-model speculative decoding server (TinyLlama, k=5)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
  --num-speculative-tokens 5 \
  --port 8001 --disable-log-requests &

# N-gram speculative decoding server (no extra model, k=5, ngram window=4)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-model [ngram] \
  --num-speculative-tokens 5 \
  --ngram-prompt-lookup-max 4 \
  --port 8002 --disable-log-requests &

Experiments

# Baseline: standard decoding, batch_size=1
python benchmarks/benchmark_latency.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --batch-size 1 --input-len 512 --output-len 128 \
  2>&1 | tee results_baseline_bs1.txt

# Record baseline tokens/s for speedup calculations
grep -E "Throughput|tokens/s" results_baseline_bs1.txt

# Draft-model spec decode, batch_size=1, k=5
python benchmarks/benchmark_latency.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --speculative-model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
  --num-speculative-tokens 5 \
  --batch-size 1 --input-len 512 --output-len 128 \
  2>&1 | tee results_draft_k5_bs1.txt

# Key metrics to extract:
#   acceptance_rate  — fraction of draft tokens the target accepted
#   mean_accepted_tokens — avg tokens produced per speculative step
#   Throughput — end-to-end tokens/s vs. baseline
grep -E "acceptance|accepted|Throughput" results_draft_k5_bs1.txt

# N-gram spec decode, k=5, ngram window=4
python benchmarks/benchmark_latency.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --speculative-model [ngram] \
  --num-speculative-tokens 5 \
  --ngram-prompt-lookup-max 4 \
  --batch-size 1 --input-len 512 --output-len 128 \
  2>&1 | tee results_ngram_k5_bs1.txt

# Expected: lower acceptance on random chat vs. draft-model
# N-gram excels when prompt and output share repeated n-grams
grep -E "acceptance|Throughput" results_ngram_k5_bs1.txt

for k in 1 3 5 7; do
  echo "=== k=$k ==="
  python benchmarks/benchmark_latency.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --speculative-model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --num-speculative-tokens $k \
    --batch-size 1 --input-len 512 --output-len 128 \
    2>&1 | tee results_draft_k${k}.txt
done

# Summarize tokens/s across k values
for k in 1 3 5 7; do
  printf "k=%-2s  " $k
  grep "Throughput" results_draft_k${k}.txt | tail -1
done

for mode in baseline spec; do
  for rate in 1 inf; do
    if [ $mode = "baseline" ]; then
      vllm serve meta-llama/Llama-3.1-8B-Instruct \
        --port 8000 &
    else
      vllm serve meta-llama/Llama-3.1-8B-Instruct \
        --speculative-model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
        --num-speculative-tokens 5 --port 8000 &
    fi
    sleep 60
    python benchmarks/benchmark_serving.py \
      --backend vllm --port 8000 \
      --model meta-llama/Llama-3.1-8B-Instruct \
      --dataset-name sharegpt \
      --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
      --num-prompts 200 --request-rate $rate \
      2>&1 | tee results_serving_${mode}_rate${rate}.txt
    kill %1; sleep 10
  done
done

Experiment Results

Hardware & Setup

Experiments run on vLLM 0.19.0 (vllm-pace build). Hardware: NVIDIA H200 SXM5 141GB HBM3e and H100 SXM5 80GB HBM3 (PACE Phoenix cluster). Target model: Llama-3.1-8B-Instruct BF16. All methods use k=5 speculative tokens, ShareGPT workload, request rate rr=2. Draft model: Llama-3.2-1B-Instruct. EAGLE model: yuhuili/EAGLE-LLaMA3.1-Instruct-8B.

H200 Results — All 4 Methods (rr=2, ShareGPT, k=5)

All spec decode methods are slower than baseline on median ITL and TTFT. N-gram is the only method with a TTFT improvement (21.48 → 18.52 ms, −14%), but ITL still increases (5.26 → 6.74 ms, +28%). EAGLE acceptance rate of 21.56% is far below the expected 85–95%. Draft model acceptance 52.49% is respectable but ITL balloons 5× (5.26 → 26.19 ms). Throughput (~427 tok/s) is nearly identical across all methods due to rr=2 rate limiting — the bottleneck is request arrival, not decode speed.

H100 Results — All 4 Methods (rr=2, ShareGPT, k=5)

H100 results are consistent with H200: N-gram is the only method with median TTFT improvement, all methods show ITL regression, and acceptance rates are hardware-independent (N-gram 41.87% vs 48.62% on H200 is within typical run-to-run variance on ShareGPT). EAGLE acceptance 20.72% confirms the low acceptance is a method/model-version issue, not hardware noise.

Greedy (temperature=0) Control Experiment — H200

To check whether sampling temperature affects acceptance rates, we reran baseline, N-gram, and EAGLE at temperature=0 (greedy decoding) on H200.

Temperature has minimal effect on the pattern: EAGLE acceptance rate increases slightly (21.56% → 24.05%) at temperature=0, confirming the fundamental issue is not sampling randomness. The low acceptance is structural — likely an incompatibility between the EAGLE model weights and vLLM v0.19.0's implementation.

Figure 1: 8B Median TTFT — Baseline vs N-gram, EAGLE, Draft Model (k=5) across H200, H100, A100. N-gram is the only method that improves TTFT; EAGLE and Draft Model both regress significantly.

Figure 2: 8B Median ITL — All spec decode methods increase ITL vs baseline. Draft Model (orange) is catastrophic (+162–398%). The gap between baseline and spec decode widens on faster GPUs (H200 5.3→26.2 ms, A100 12.1→31.7 ms).

Figure 3: 8B Acceptance Rate — EAGLE at ~21% is far below the expected 85%+ (red dashed line), consistent across all GPUs. Draft Model at ~53% and N-gram at ~47% are reasonable for ShareGPT but insufficient to overcome verification overhead.

70B Llama-3.1 — H100×4 TP=4 Serving Results

70B results follow the same pattern as 8B: N-gram improves TTFT (-19%) but worsens ITL (+18%). Draft model acceptance is higher (62.93% vs 52.49% on 8B) because Llama-3.2-1B is better aligned with Llama-3.1-70B, yet ITL still regresses +50%. Liu et al. report 1.96× speedup for 70B draft model at batch=1 — the discrepancy is likely due to their use of offline batch=1 mode vs our serving (rr=2).

Figure 4: 70B Llama-3.1 on H100×4 TP=4 — TTFT and ITL (serving, rr=2). N-gram improves TTFT but all methods worsen ITL.

8B Offline Throughput — H200 vs H100 (batch=auto)

Even in offline mode, draft model throughput is 0.52× (half of baseline). N-gram is neutral (0.99×). This confirms the issue is not serving overhead — the draft model's low acceptance rate (28.3%) combined with its forward pass cost makes it a net negative.

70B Offline Throughput — H200×2 vs H100×4 (batch=auto)

Positive speedups on 70B: Draft model achieves 1.12× on H200×2 — the combination of H200's 4.8 TB/s HBM bandwidth (minimizing draft overhead) and 70B's proportionally smaller proposing cost finally yields net acceleration. N-gram achieves 1.10× on H100×4 where baseline decode is slower. Key insight: spec decode benefits are GPU-dependent — H200 is fast enough that draft model overhead becomes negligible for 70B, while H100 is slow enough that n-gram's zero-cost proposals create measurable gains.

Expected vs Actual

Metrics to Collect

Source Code Reading

Core Source Code Walkthrough

Below are real excerpts from vllm-continuum that implement the concepts you measured. Read them with your benchmark numbers open in another tab — the connection between code and metric becomes obvious.

vllm/v1/spec_decode/ngram_proposer.py:11 — NgramProposer

class NgramProposer:
    def __init__(self, vllm_config: VllmConfig):
        self.min_n = vllm_config.speculative_config.prompt_lookup_min
        self.max_n = vllm_config.speculative_config.prompt_lookup_max
        self.k = vllm_config.speculative_config.num_speculative_tokens
        # Trigger Numba JIT compilation for N-gram proposer.
        self.propose(np.zeros(1024, dtype=np.int32))

    def propose(self, context_token_ids: np.ndarray) -> Optional[np.ndarray]:
        """Proposes the next sequence of tokens based on n-gram pattern
        matching in the context."""

The simplest speculative decoder — no draft model, just look back at the recent context for n-gram matches. If the model just generated [A, B, C, D] and we see [A, B, C] earlier in the prompt followed by [D, E, F], predict [E, F] as the next k tokens. The target model verifies all k+1 in one forward pass. Numba-JIT-compiled for speed. Only useful for repetitive text (code, structured output).

Discussion — Comparison with Liu et al. (2025)

Written Analysis — Reference Answers

Reference answers are grounded in our vLLM 0.19.0 measurements on H200 and H100, cross-referenced with Liu et al. (2025) findings.

Q1: Why does speculative decoding work in theory, and why did it not work here?

Q2: Why does speculative decoding hurt even at low QPS (rr=2)?

Q3: What acceptance rates did we observe, and what do they imply?