Week 5 — Prefix Caching (APC)

Learning Objectives

Key Concepts

Setup & Configuration

# Launch vLLM WITH APC enabled (port 8000)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --port 8000 \
  --disable-log-requests

# Launch vLLM WITHOUT APC (baseline, port 8002)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --no-enable-prefix-caching \
  --port 8002 \
  --disable-log-requests

# Launch vLLM with APC + FP8 KV cache (port 8003)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --port 8003 \
  --disable-log-requests

Experiments

# Cold run — APC enabled (first request computes & caches the prefix)
python benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 --port 8000 \
  2>&1 | tee results_apc_on_cold.txt

# Warm run — APC enabled (prefix already in cache)
python benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 --port 8000 \
  2>&1 | tee results_apc_on_warm.txt

# Baseline — APC disabled (no caching, always cold)
python benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 --port 8002 \
  2>&1 | tee results_apc_off.txt

for prefix_len in 0 256 512 1024 2048; do
  # First request warms the cache
  python benchmarks/benchmark_prefix_caching.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --shared-prefix-len $prefix_len \
    --num-prompts 5 --port 8000 >/dev/null 2>&1

  # Warm benchmark — cache already populated
  python benchmarks/benchmark_prefix_caching.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --shared-prefix-len $prefix_len \
    --num-prompts 200 --port 8000 \
    2>&1 | tee results_apc_prefix${prefix_len}.txt
done

# Check GPU blocks allocated — FP16 baseline (port 8000)
# Look for: "# GPU blocks: XXXX, # CPU blocks: YYYY"
grep "GPU blocks" vllm_startup_fp16.log

# Benchmark FP8 KV cache (port 8003)
python benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 --port 8003 \
  2>&1 | tee results_apc_fp8.txt

# Quick quality check: compare generation on a fixed prompt
for port in 8000 8003; do
  curl http://localhost:${port}/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"meta-llama/Llama-3.1-8B-Instruct",
         "prompt":"Explain the attention mechanism in detail.",
         "max_tokens":200}' | python -m json.tool
done

# Restart APC server with stats logging
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --port 8000 2>&1 | tee vllm_apc_stats.log &

# Run benchmark
python benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --shared-prefix-len 1024 \
  --num-prompts 200 --port 8000

# Extract cache hit rates from log
grep "prefix_cache_hit_rate" vllm_apc_stats.log

import json, random, string

# Build a synthetic RAG workload
shared_doc = "The following document describes AI inference optimization. " * 30  # ~512 tokens
data = []
for _ in range(200):
    unique_q = "Question: " + " ".join(random.choices(string.ascii_lowercase, k=40)) + "?"
    data.append({"prompt": shared_doc + unique_q, "output_len": 64})

json.dump(data, open("rag_workload.json", "w"))
print(f"Shared prefix ~{len(shared_doc.split())} words")

import re

def parse_benchmark(filepath):
    results = {}
    with open(filepath) as f:
        text = f.read()
    for key, pattern in [
        ("ttft_p50", r"Median TTFT.*?:\s+([\d.]+)"),
        ("ttft_p99", r"P99 TTFT.*?:\s+([\d.]+)"),
        ("throughput", r"Output throughput.*?:\s+([\d.]+)"),
    ]:
        m = re.search(pattern, text)
        results[key] = float(m.group(1)) if m else None
    return results

files = ["results_apc_off.txt", "results_apc_on_cold.txt",
         "results_apc_on_warm.txt", "results_apc_fp8.txt"]
for f in files:
    print(f, parse_benchmark(f))

Experiment Results

Hardware

Experiments run on NVIDIA H200 SXM5 141GB, H100 SXM5 80GB HBM3, and L40S 48GB (PACE Phoenix cluster), using Llama-3.1-8B-Instruct in BF16, ShareGPT workload.

APC On vs Off — Multi-GPU Comparison

Figure 1: APC on vs off — TTFT median and output throughput across H200, H100, A100 PCIe, L40S (ShareGPT workload)

Figure 2: FP16 vs FP8 KV cache — throughput impact on H200, H100, A100 PCIe, L40S

Cold vs Warm — properly isolated (2026-04-11 rerun)

The honest teaching point is that APC at 480 tokens barely moves the needle on fast modern GPUs. A 480-token prefill takes roughly 5–10 ms on H100/H200; the total request latency is dominated by the 64-token decode (50–100 ms), tokenizer, HTTP, and server overhead. Saving 5–10 ms out of ~470 ms is a 1–2% speedup, which is exactly what we measure. For APC to pay off visibly you need either (a) a much longer shared prefix (2000+ tokens) or (b) an older/slower GPU where prefill is a larger fraction of total latency.

Figure: Cold vs warm mean request latency for the 480-token shared-prefix workload across four GPU classes. Every bar pair shows ~1% speedup — the honest measurement when cold and warm are properly isolated.

Expected vs Actual

Metrics to Collect

Source Code Reading

Core Source Code Walkthrough

Below are real excerpts from vllm-continuum that implement the concepts you measured. Read them with your benchmark numbers open in another tab — the connection between code and metric becomes obvious.

vllm/v1/core/kv_cache_utils.py:539 — hash_block_tokens

def hash_block_tokens(
        hash_function: Callable[[Any], bytes],
        parent_block_hash: Optional[BlockHash],
        curr_block_token_ids: Sequence[int],
        extra_keys: Optional[tuple[Any, ...]] = None) -> BlockHash:
    """Computes a hash value corresponding to the contents of a block and
    the contents of the preceding block(s)."""
    if not parent_block_hash:
        parent_block_hash = NONE_HASH
    curr_block_token_ids_tuple = tuple(curr_block_token_ids)
    return BlockHash(
        hash_function(
            (parent_block_hash, curr_block_token_ids_tuple, extra_keys)))

This is how APC decides whether two requests share a prefix. Each KV block (16 tokens by default) gets hashed with its parent's hash chained in, like a Merkle tree. Two requests that start with identical token IDs produce identical block hashes, so vLLM can point both at the same physical KV pages. Our experiment showed this never fired on ShareGPT — because no two ShareGPT prompts share the first 16 tokens.

Written Analysis — Reference Answers

Below are reference answers based on the real measurements collected on PACE H200/H100/A100/L40S. Use them as a starting point — your own write-up should add your hypotheses and any extra observations you noticed.

Q1: Why didn't APC help on the ShareGPT workload?

Q2: When does APC provide large gains?

Q3: FP8 KV cache trade-offs