Hands-On Labs: AI Inference Infrastructure

Manual curl Test

Send individual requests to both servers and observe the response structure:

# vLLM completions API
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","prompt":"Explain transformers in 3 sentences.","max_tokens":128}'

# SGLang completions API
curl http://localhost:8001/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","prompt":"Explain transformers in 3 sentences.","max_tokens":128}'

vLLM Benchmark — Varying Request Rates

Run benchmark_serving.py at request rates 1, 4, 10, and inf (closed-loop):

for rate in 1 4 10 inf; do
  python -m vllm.entrypoints.openai.api_server &  # if not already running
  python benchmarks/benchmark_serving.py \
    --backend vllm \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 200 \
    --request-rate $rate \
    --port 8000 \
    2>&1 | tee results_vllm_rr${rate}.txt
done

SGLang Benchmark — Same Workload

for rate in 1 4 10 inf; do
  python -m sglang.bench_serving \
    --backend sglang \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 200 \
    --request-rate $rate \
    --port 8001 \
    2>&1 | tee results_sglang_rr${rate}.txt
done

Collect and Compare Results

Extract TTFT, ITL, throughput from each result file. Plot request rate vs. median TTFT and median ITL for both systems.

Offline Throughput — vLLM

python benchmarks/benchmark_throughput.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 500 \
  --output-json throughput_vllm.json

Offline Throughput — SGLang

python -m sglang.bench_one_batch \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --batch-size 32 64 128 \
  --input-len 512 \
  --output-len 256

Single-Request Latency Profiling

python benchmarks/benchmark_latency.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --batch-size 1 \
  --num-iters 10

PyTorch Profiler + Perfetto Trace

Enable the PyTorch profiler and export a trace for visualization in Perfetto UI:

# Run vLLM with profiling enabled
VLLM_TORCH_PROFILER_DIR=./profiles vllm serve \
  meta-llama/Llama-3.1-8B-Instruct --port 8000

# Send a few requests, then open the trace:
# Go to https://ui.perfetto.dev and load the .json trace file

GPU Monitoring Analysis

Analyze the gpu_monitor_w2.csv to observe GPU utilization patterns during offline vs. online workloads.

Nsys Profile — Full Serving Trace

nsys profile -o llm_trace --trace=cuda,nvtx \
  python benchmarks/benchmark_latency.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --batch-size 1 --input-len 512 --output-len 64 --num-iters 3

NCU Roofline — Per-Kernel Analysis

ncu --set roofline --target-processes all \
  -o roofline_report \
  python benchmarks/benchmark_latency.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --batch-size 1 --input-len 128 --output-len 16 --num-iters 1

Theoretical Decode Latency Calculation

For Llama-3.1-8B in BF16: model size ≈ 16 GB. A100 HBM bandwidth = 2 TB/s. Theoretical minimum decode step = 16 GB / 2 TB/s = 8 ms. Compare against your measured per-token latency from the nsys trace.

Prefill vs Decode Roofline

Open the NCU roofline report. Identify where attention kernels and GEMM kernels fall. Prefill GEMMs should be near the compute roof; decode attention should be near the memory roof.

HuggingFace Baseline

python hf_baseline.py

vLLM — max-num-seqs Sweep

for seqs in 1 4 16 64 256; do
  python benchmarks/benchmark_throughput.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --max-num-seqs $seqs \
    --dataset ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 300 \
    2>&1 | tee results_paged_seqs${seqs}.txt
done

Block Size Sweep

for bs in 8 16 32; do
  python benchmarks/benchmark_throughput.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --block-size $bs \
    --dataset ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 300 \
    2>&1 | tee results_paged_bs${bs}.txt
done

Memory Utilization Monitoring

During each run, log GPU memory usage. Note the KV cache block allocation reported in vLLM logs (e.g., '# GPU blocks: 1234').

APC vs No-APC Comparison

# With APC (port 8000)
python benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 --port 8000 \
  2>&1 | tee results_apc_on.txt

# Without APC (port 8002)
python benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 --port 8002 \
  2>&1 | tee results_apc_off.txt

Shared Prefix Length Sweep

for prefix_len in 0 256 512 1024 2048; do
  python benchmarks/benchmark_prefix_caching.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --shared-prefix-len $prefix_len \
    --num-prompts 200 --port 8000 \
    2>&1 | tee results_apc_prefix${prefix_len}.txt
done

FP8 KV Cache

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --port 8003

# Repeat prefix caching benchmark on port 8003
python benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 --port 8003 \
  2>&1 | tee results_apc_fp8.txt

Radix Cache On vs Off

# Few-shot workload benchmark
python -m sglang.bench_serving \
  --backend sglang --port 8001 \
  --dataset-name generated-shared-prefix \
  --num-prompts 200 --request-rate 4 \
  2>&1 | tee results_radix_on.txt

python -m sglang.bench_serving \
  --backend sglang --port 8004 \
  --dataset-name generated-shared-prefix \
  --num-prompts 200 --request-rate 4 \
  2>&1 | tee results_radix_off.txt

LPM vs FCFS Scheduling

SGLang uses Longest Prefix Match scheduling by default. Compare against FCFS by toggling the scheduling policy:

# LPM scheduling (default)
python -m sglang.bench_serving \
  --backend sglang --port 8001 \
  --dataset-name generated-shared-prefix \
  --num-prompts 300 --request-rate 8 \
  2>&1 | tee results_lpm.txt

# FCFS scheduling
python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --schedule-policy fcfs --port 8005
python -m sglang.bench_serving \
  --backend sglang --port 8005 \
  --dataset-name generated-shared-prefix \
  --num-prompts 300 --request-rate 8 \
  2>&1 | tee results_fcfs.txt

sgl.function DSL Example

import sglang as sgl

@sgl.function
def multi_turn(s, question1, question2):
    s += sgl.system("You are a helpful assistant.")
    s += sgl.user(question1)
    s += sgl.assistant(sgl.gen("answer1", max_tokens=256))
    s += sgl.user(question2)
    s += sgl.assistant(sgl.gen("answer2", max_tokens=256))

state = multi_turn.run(
    question1="What is PagedAttention?",
    question2="How does it compare to RadixAttention?")
print(state["answer1"])
print(state["answer2"])

HuggingFace Static Batching Baseline

Use the HuggingFace generate() script from Week 4 with varying output lengths. Observe that all sequences must wait for the longest one to finish.

vLLM Continuous Batching — Uniform Outputs

python benchmarks/benchmark_throughput.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 256 --output-len 128 \
  --num-prompts 300 \
  2>&1 | tee results_cb_uniform.txt

vLLM Continuous Batching — Variable Outputs

python benchmarks/benchmark_throughput.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 300 \
  2>&1 | tee results_cb_variable.txt

Batch Occupancy Analysis

Enable vLLM's --log-stats and observe how the running batch size fluctuates over time with continuous batching. With static batching, the batch size stays constant (wasting GPU cycles on padding).

HOL Blocking Demonstration

Create a workload mixing long prefills (2048+ tokens) with short streaming requests. Without chunked prefill, observe ITL spikes in the short requests.

python benchmarks/benchmark_serving.py \
  --backend vllm --port 8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 --request-rate 4 \
  2>&1 | tee results_no_chunk.txt

Chunked Prefill — Default

python benchmarks/benchmark_serving.py \
  --backend vllm --port 8002 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 --request-rate 4 \
  2>&1 | tee results_chunk_default.txt

max-num-batched-tokens Sweep

for tokens in 256 512 1024 2048 4096; do
  vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-chunked-prefill \
    --max-num-batched-tokens $tokens \
    --port 8010 &
  sleep 30  # wait for server startup
  python benchmarks/benchmark_serving.py \
    --backend vllm --port 8010 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 100 --request-rate 4 \
    2>&1 | tee results_chunk_${tokens}.txt
  kill %1
done

ITL Timeseries Analysis

Plot the per-token ITL over time for each configuration. Without chunked prefill, you should see periodic spikes. With chunked prefill, ITL should be smoother.

FCFS vs LPM vs DFS-weight

for policy in fcfs lpm dfs-weight; do
  python -m sglang.launch_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --schedule-policy $policy \
    --port 8001 &
  sleep 30
  python -m sglang.bench_serving \
    --backend sglang --port 8001 \
    --dataset-name generated-shared-prefix \
    --num-prompts 300 --request-rate 8 \
    2>&1 | tee results_sched_${policy}.txt
  kill %1; sleep 5
done

Cache-Aware DP Routing

If you have 2 GPUs, launch SGLang with data parallelism and the cache-aware router:

python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dp 2 --port 8001

# The sglang_router automatically routes requests to maximize cache hits
python -m sglang.bench_serving \
  --backend sglang --port 8001 \
  --dataset-name generated-shared-prefix \
  --num-prompts 500 --request-rate 10 \
  2>&1 | tee results_dp_router.txt

Eager Mode vs CUDA Graphs

# Eager mode (no CUDA graphs)
python benchmarks/benchmark_latency.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enforce-eager \
  --batch-size 1 --input-len 512 --output-len 128 \
  2>&1 | tee results_eager.txt

# CUDA graphs (default)
python benchmarks/benchmark_latency.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --batch-size 1 --input-len 512 --output-len 128 \
  2>&1 | tee results_cudagraph.txt

Startup Time Comparison

Time the server startup with and without CUDA graphs. The graph capture phase (warmup) adds significant startup time but reduces per-request latency.

# Time server startup
time vllm serve meta-llama/Llama-3.1-8B-Instruct --enforce-eager --port 8000 &
# vs
time vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8002 &

Memory Overhead Analysis

Compare GPU memory usage between eager and CUDA graph modes. CUDA graphs pre-allocate memory for captured operations, reducing available KV cache space.

Draft Model Speculation

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
  --num-speculative-tokens 5 \
  --port 8000

python benchmarks/benchmark_serving.py \
  --backend vllm --port 8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 --request-rate 2 \
  2>&1 | tee results_spec_draft.txt

N-gram Speculation

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-model [ngram] \
  --num-speculative-tokens 5 \
  --ngram-prompt-lookup-max 4 \
  --port 8002

python benchmarks/benchmark_serving.py \
  --backend vllm --port 8002 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 --request-rate 2 \
  2>&1 | tee results_spec_ngram.txt

num_speculative_tokens Sweep

for k in 1 3 5 7; do
  vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --speculative-model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --num-speculative-tokens $k \
    --port 8010 &
  sleep 30
  python benchmarks/benchmark_serving.py \
    --backend vllm --port 8010 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 100 --request-rate 2 \
    2>&1 | tee results_spec_k${k}.txt
  kill %1; sleep 5
done

Low QPS vs High QPS

Run the same speculative decoding config at request rates 1, 4, and 10. At low QPS, speculation reduces per-request latency. At high QPS, the overhead of draft model + verification can hurt throughput.

Throughput Comparison Across Precisions

# FP16 (baseline)
python benchmarks/benchmark_throughput.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 2>&1 | tee results_fp16.txt

# FP8
python benchmarks/benchmark_throughput.py \
  --model neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 \
  --num-prompts 200 2>&1 | tee results_fp8.txt

# AWQ-INT4
python benchmarks/benchmark_throughput.py \
  --model hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
  --quantization awq \
  --num-prompts 200 2>&1 | tee results_awq.txt

Accuracy Evaluation — GSM8K

# FP16
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
  --tasks gsm8k --batch_size auto \
  2>&1 | tee eval_fp16.txt

# FP8
lm_eval --model vllm \
  --model_args pretrained=neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 \
  --tasks gsm8k --batch_size auto \
  2>&1 | tee eval_fp8.txt

# AWQ-INT4
lm_eval --model vllm \
  --model_args pretrained=hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4,quantization=awq \
  --tasks gsm8k --batch_size auto \
  2>&1 | tee eval_awq.txt

Memory Usage Comparison

Record GPU memory used by each model variant (from vLLM startup logs or nvidia-smi). FP16 ≈ 16 GB, FP8 ≈ 8 GB, INT4 ≈ 4 GB for model weights.

TP Scaling — Latency

for tp in 1 2 4; do
  python benchmarks/benchmark_latency.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --tensor-parallel-size $tp \
    --batch-size 1 --input-len 512 --output-len 128 \
    2>&1 | tee results_tp${tp}_latency.txt
done

TP Scaling — Throughput

for tp in 1 2 4; do
  python benchmarks/benchmark_throughput.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --tensor-parallel-size $tp \
    --dataset ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 300 \
    2>&1 | tee results_tp${tp}_throughput.txt
done

NCCL All-Reduce Profiling

nsys profile -o tp2_trace --trace=cuda,nvtx,nccl \
  python benchmarks/benchmark_latency.py \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --tensor-parallel-size 2 \
    --batch-size 1 --input-len 256 --output-len 32 --num-iters 3

Open in nsys GUI and identify NCCL all-reduce calls. Measure their duration and frequency per decode step.

Serve Mixtral-8x7B

vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --tensor-parallel-size 2 \
  --port 8000

python benchmarks/benchmark_serving.py \
  --backend vllm --port 8000 \
  --model mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 100 --request-rate 4 \
  2>&1 | tee results_moe_tp2.txt

Expert Parallelism

# Expert parallelism distributes experts across GPUs
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --tensor-parallel-size 2 \
  --port 8002

# In SGLang with explicit EP
python -m sglang.launch_server \
  --model mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --tp 2 --port 8003

Memory vs Compute Analysis

Compare memory usage and throughput: Mixtral-8x7B has 46.7B total parameters but only activates ~12.9B per token (top-2 of 8 experts). Compare against Llama-3.1-8B which has fewer total parameters but activates all of them.

LMCache Integration with vLLM

# Launch vLLM with LMCache
LMCACHE_CONFIG_FILE=lmcache_config.yaml vllm serve \
  meta-llama/Llama-3.1-8B-Instruct \
  --kv-transfer-config lmcache_config.yaml \
  --port 8000

Cross-Restart Persistence Test

Send requests with a shared system prompt, shut down the server, restart, and send the same requests. With LMCache, the KV cache is restored from disk/CPU, eliminating re-computation.

# First run: populate cache
python benchmarks/benchmark_serving.py \
  --backend vllm --port 8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 50 --request-rate 2 \
  2>&1 | tee results_lmcache_cold.txt

# Restart server, re-run (cache should be warm from disk)
# Kill and restart the server, then:
python benchmarks/benchmark_serving.py \
  --backend vllm --port 8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 50 --request-rate 2 \
  2>&1 | tee results_lmcache_warm.txt

chunk_size Sweep

for cs in 64 128 256 512 1024; do
  # Update lmcache_config.yaml with chunk_size=$cs
  sed -i "s/chunk_size: .*/chunk_size: $cs/" lmcache_config.yaml
  LMCACHE_CONFIG_FILE=lmcache_config.yaml vllm serve \
    meta-llama/Llama-3.1-8B-Instruct \
    --kv-transfer-config lmcache_config.yaml \
    --port 8010 &
  sleep 30
  python benchmarks/benchmark_serving.py \
    --backend vllm --port 8010 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --num-prompts 100 --request-rate 4 \
    2>&1 | tee results_lmcache_cs${cs}.txt
  kill %1; sleep 5
done

vLLM Disaggregated Setup

# Prefill instance (GPU 0)
CUDA_VISIBLE_DEVICES=0 vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --kv-transfer-config '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer"}'

# Decode instance (GPU 1)
CUDA_VISIBLE_DEVICES=1 vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8001 \
  --kv-transfer-config '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer"}'

SGLang Disaggregated Mode

# SGLang with disaggregation
python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode [prefill] \
  --port 8002

python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode [decode] \
  --port 8003

KV Transfer Time vs Input Length

Measure how KV cache transfer time scales with input sequence length. Longer inputs produce larger KV caches that take more time to transfer between prefill and decode GPUs.

for input_len in 128 256 512 1024 2048 4096; do
  python benchmarks/benchmark_serving.py \
    --backend vllm --port 8001 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --input-len $input_len --output-len 64 \
    --num-prompts 50 --request-rate 2 \
    2>&1 | tee results_pd_input${input_len}.txt
done

Capstone Analysis

Design an experiment comparing standard serving vs disaggregated P/D on a mixed workload with both short and long prefills. When does disaggregation help?