Week 12 — Quantization

Learning Objectives

Key Concepts

Setup & Configuration

# Verify vLLM and lm_eval installations
pip show vllm lm_eval

# Confirm H100 FP8 support (requires CUDA compute capability 8.9+)
python -c "import torch; print(torch.cuda.get_device_capability())"
# Expected: (9, 0) for H100

# BF16 baseline server
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --port 8000 --disable-log-requests &

# FP8 server (native H100 kernel path)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8 \
  --port 8001 --disable-log-requests &

# AWQ-INT4 server (pre-quantized weights required)
# Use an AWQ-quantized model hub checkpoint, e.g.:
vllm serve hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
  --quantization awq \
  --port 8002 --disable-log-requests &

Experiments

for fmt_port in "bf16:8000" "fp8:8001" "awq:8002"; do
  fmt=${fmt_port%%:*}; port=${fmt_port##*:}
  for bs in 1 16 64; do
    python benchmarks/benchmark_throughput.py \
      --backend openai-chat \
      --port $port \
      --model meta-llama/Llama-3.1-8B-Instruct \
      --input-len 512 --output-len 128 \
      --num-prompts $bs \
      2>&1 | tee results_tp_${fmt}_bs${bs}.txt
  done
done

for quant in "none" "fp8" "awq"; do
  if [ $quant = "none" ]; then
    extra_args="--dtype bfloat16"
    model="meta-llama/Llama-3.1-8B-Instruct"
  elif [ $quant = "fp8" ]; then
    extra_args="--quantization fp8"
    model="meta-llama/Llama-3.1-8B-Instruct"
  else
    extra_args="--quantization awq"
    model="hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4"
  fi
  python benchmarks/benchmark_latency.py \
    --model $model $extra_args \
    --batch-size 1 --input-len 512 --output-len 128 \
    2>&1 | tee results_lat_${quant}_bs1.txt
done

for fmt_port in "bf16:8000" "fp8:8001" "awq:8002"; do
  fmt=${fmt_port%%:*}; port=${fmt_port##*:}
  for rate in 1 inf; do
    python benchmarks/benchmark_serving.py \
      --backend vllm --port $port \
      --model meta-llama/Llama-3.1-8B-Instruct \
      --dataset-name sharegpt \
      --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
      --num-prompts 200 --request-rate $rate \
      2>&1 | tee results_serving_${fmt}_rate${rate}.txt
  done
done

# BF16 GSM8K evaluation (100 samples)
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16 \
  --tasks gsm8k \
  --num_fewshot 8 \
  --limit 100 \
  --output_path results_gsm8k_bf16.json

# FP8 GSM8K evaluation
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,quantization=fp8 \
  --tasks gsm8k \
  --num_fewshot 8 \
  --limit 100 \
  --output_path results_gsm8k_fp8.json

# AWQ-INT4 GSM8K evaluation
lm_eval --model vllm \
  --model_args pretrained=hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4,quantization=awq \
  --tasks gsm8k \
  --num_fewshot 8 \
  --limit 100 \
  --output_path results_gsm8k_awq.json

# Compare accuracy across formats
python -c "
import json
for fmt in ['bf16','fp8','awq']:
    d = json.load(open(f'results_gsm8k_{fmt}.json'))
    acc = d['results']['gsm8k']['exact_match,flexible-extract']
    print(f'{fmt}: GSM8K acc = {acc:.4f}')
"

Experiment Results

Hardware

Experiments run on NVIDIA H200 SXM5 141GB, H100 SXM5 80GB HBM3, A100 80GB PCIe, and L40S 48GB (PACE Phoenix cluster) with Llama-3.1-8B-Instruct. H100 has the complete BF16/FP8/AWQ-INT4 sweep; A100/H200/L40S currently have BF16 baselines only.

BF16 Reference — Latency and Throughput Across GPUs

BF16 vs FP8 vs AWQ-INT4 — H100 Quantization Sweep

The same Llama-3.1-8B-Instruct model in three precisions, all measured on the same H100 80GB. The contrast between bs=1 latency (memory-bound) and rr=4 serving (mixed prefill/decode) is the central lesson: quantization that helps decode can hurt prefill.

Figure 1: Left — BF16 offline throughput across 4 GPU classes (H200/H100/A100/L40S), tracking HBM bandwidth. Right — H100 quantization sweep: FP8 strictly improves over BF16 on offline+serving+latency, while AWQ-INT4 wins bs=1 latency but loses badly on offline throughput and serving (4.9× worse TTFT/ITL) due to per-step dequant overhead.

Expected vs Actual

Metrics to Collect

Source Code Reading

Core Source Code Walkthrough

Below are real excerpts from vllm-continuum that implement the concepts you measured. Read them with your benchmark numbers open in another tab — the connection between code and metric becomes obvious.

vllm/model_executor/layers/quantization/awq.py:124 — AWQLinearMethod

class AWQConfig(QuantizationConfig):
    ...
class AWQLinearMethod(LinearMethodBase):
    def apply(self, layer: torch.nn.Module, x: torch.Tensor, bias=None):
        # AWQ stores weights as INT4 packed in INT32, plus per-group scales/zeros
        # The dequantize+matmul is fused into a single Triton/CUTLASS kernel
        ...

AWQ (Activation-aware Weight Quantization) stores weights as 4-bit integers, then dequantizes-on-the-fly inside a fused matmul kernel. The model footprint shrinks ~4x (16GB to 4GB for Llama-8B), HBM bandwidth is the bottleneck for decode, so 4x less weight data = much faster decode. Trade-off: typically 1-3 percentage points GSM8K accuracy loss vs BF16.

Written Analysis — Reference Answers

Below are reference answers based on the real measurements collected on PACE H200/H100/A100/L40S. Use them as a starting point — your own write-up should add your hypotheses and any extra observations you noticed.

Q1: Expected FP8 vs BF16 throughput on H100/H200

Q2: Expected AWQ-INT4 vs BF16 throughput

Q3: When NOT to quantize