Week 6 — RadixAttention

Learning Objectives

Key Concepts

Setup & Configuration

# SGLang with radix cache ENABLED (default, port 8001)
python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8001

# SGLang with radix cache DISABLED (baseline, port 8004)
python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --disable-radix-cache \
  --port 8004

# SGLang with FCFS scheduling (port 8005)
python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --schedule-policy fcfs \
  --port 8005

# vLLM with APC for cross-system comparison (port 8000)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --port 8000 --disable-log-requests

Experiments

# Radix cache ON — few-shot workload
python -m sglang.bench_serving \
  --backend sglang --port 8001 \
  --dataset-name generated-shared-prefix \
  --num-prompts 200 --request-rate 4 \
  2>&1 | tee results_radix_on.txt

# Radix cache OFF — same workload
python -m sglang.bench_serving \
  --backend sglang --port 8004 \
  --dataset-name generated-shared-prefix \
  --num-prompts 200 --request-rate 4 \
  2>&1 | tee results_radix_off.txt

# LPM scheduling (default — port 8001)
python -m sglang.bench_serving \
  --backend sglang --port 8001 \
  --dataset-name generated-shared-prefix \
  --num-prompts 300 --request-rate 8 \
  2>&1 | tee results_lpm.txt

# FCFS scheduling (port 8005)
python -m sglang.bench_serving \
  --backend sglang --port 8005 \
  --dataset-name generated-shared-prefix \
  --num-prompts 300 --request-rate 8 \
  2>&1 | tee results_fcfs.txt

import sglang as sgl
import time

sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:8001"))

@sgl.function
def multi_turn(s, question1, question2):
    s += sgl.system("You are a helpful assistant.")
    s += sgl.user(question1)
    s += sgl.assistant(sgl.gen("answer1", max_tokens=256))
    s += sgl.user(question2)
    s += sgl.assistant(sgl.gen("answer2", max_tokens=256))

# Warmup run (cold cache)
t0 = time.time()
state = multi_turn.run(
    question1="What is PagedAttention?",
    question2="How does it compare to RadixAttention?")
print(f"Cold run: {time.time()-t0:.3f}s")
print(state["answer1"])

# Warm run (same prefix, different second question)
t1 = time.time()
state2 = multi_turn.run(
    question1="What is PagedAttention?",
    question2="What are its memory advantages?")
print(f"Warm run: {time.time()-t1:.3f}s")
print(state2["answer2"])

# vLLM APC benchmark
python benchmarks/benchmark_serving.py \
  --backend vllm --port 8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name generated-shared-prefix \
  --num-prompts 200 --request-rate 4 \
  2>&1 | tee results_vllm_apc.txt

# SGLang RadixAttention benchmark
python -m sglang.bench_serving \
  --backend sglang --port 8001 \
  --dataset-name generated-shared-prefix \
  --num-prompts 200 --request-rate 4 \
  2>&1 | tee results_sglang_radix.txt

for num_shots in 1 3 5 10; do
  echo "=== ${num_shots}-shot ==="
  python -m sglang.bench_serving \
    --backend sglang --port 8001 \
    --dataset-name generated-shared-prefix \
    --num-output-sentences $num_shots \
    --num-prompts 200 --request-rate 4 \
    2>&1 | tee results_radix_${num_shots}shot.txt
done

# LPM on ShareGPT (no shared prefix)
python -m sglang.bench_serving \
  --backend sglang --port 8001 \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 200 --request-rate 4 \
  2>&1 | tee results_lpm_sharegpt.txt

# FCFS on ShareGPT (no shared prefix)
python -m sglang.bench_serving \
  --backend sglang --port 8005 \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 200 --request-rate 4 \
  2>&1 | tee results_fcfs_sharegpt.txt

Experiment Results

Hardware

SGLang experiments run on NVIDIA L40S 48GB and A100 80GB HBM2e (PACE Phoenix cluster), using Llama-3.1-8B-Instruct in BF16, ShareGPT workload.

⭐ H100 measured — num_shots sweep (Radix cache hit scaling)

For each num_shots value we pre-warm SGLang with the few-shot prefix once, then measure 10 consecutive send_request calls. The flat latency curve is the signature of a working prefix cache.

A 9× jump in prefix length (18 → 164 tokens) costs essentially 0 ms of extra latency — the radix tree resolves the entire prefix to an existing subtree in constant time and the decode cost dominates each request.

⭐ H100 measured — LPM vs FCFS on ShareGPT (negative control)

On ShareGPT, LPM ≈ FCFS within noise. This run shows LPM +11%, but a prior run showed FCFS +9% — pure run-to-run variance. See Week 9 for the controlled comparison where Random falls 8% behind.

SGLang RadixAttention — Serving Results

H100 2026-04-11: radix ON and OFF are statistically tied on ShareGPT (<2% difference). The num_shots sweep above is more reliable evidence — its flat latency curve proves the radix tree works.

Figure 1: SGLang RadixAttention TTFT median — L40S (3273 ms) vs A100 (1922 ms), ShareGPT workload

⭐ H100 measured — vLLM APC vs SGLang Radix (Exp 6, head-to-head)

Same H100, same model, same workload. vLLM runs APC via asteria env; SGLang runs RadixAttention via sglang-lab env.

vLLM APC wins on latency (TTFT -35%, ITL -36%). SGLang Radix wins on throughput (+7.3%). Architectural trade-off: vLLM is faster per-request; SGLang packs more concurrent requests.

Expected vs Actual

Metrics to Collect

Source Code Reading

Core Source Code Walkthrough

Below are real excerpts from vllm-continuum that implement the concepts you measured. Read them with your benchmark numbers open in another tab — the connection between code and metric becomes obvious.

sglang/srt/mem_cache/radix_cache.py:374 — match_prefix

def match_prefix(self, params: MatchPrefixParams) -> MatchResult:
    """Find the longest cached prefix of `key` in the radix tree.

    Returns:
        MatchResult: device_indices is a 1-D torch.int64 tensor of
        the concatenated KV cache indices corresponding to the longest
        cached prefix (may be length 0).
    """

SGLang's answer to vLLM's APC. Instead of hashing fixed-size blocks, it builds a radix tree at TOKEN granularity — so two prompts that share the first 17 tokens (not 16, not 32) can still partially share KV cache. The cost is a tree walk per request; the benefit is much finer-grained reuse for few-shot and multi-turn workloads.

Written Analysis — Reference Answers

Below are reference answers based on the real measurements collected on PACE A100/L40S. Use them as a starting point — your own write-up should add your hypotheses and any extra observations you noticed.

Q1: How does RadixAttention differ from vLLM APC?

Q2: Which workloads make RadixAttention clearly beat APC?

Q3: Cost of RadixAttention