Week 0 — PACE Environment Setup

By the end of this page you will have

What you need

SSH access & SSH config

1. Generate a key (one time)

# On your local machine
ssh-keygen -t ed25519 -C "my-name@gatech.edu" -f ~/.ssh/id_ed25519_pace
ssh-copy-id -i ~/.ssh/id_ed25519_pace.pub yourgtid@login-phoenix.pace.gatech.edu

2. Set up SSH config (saves typing for the rest of the semester)

# Add to ~/.ssh/config on your local machine
Host pace
    HostName login-phoenix.pace.gatech.edu
    User yourgtid
    IdentityFile ~/.ssh/id_ed25519_pace
    ServerAliveInterval 60
    ServerAliveCountMax 5
    # ControlMaster reuses one TCP connection across multiple ssh calls
    # — you only have to type your password / 2FA once per ~10 min
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600

mkdir -p ~/.ssh/sockets
chmod 700 ~/.ssh ~/.ssh/sockets

3. Test the connection

ssh pace "hostname && pace-quota"

If you see your storage usage and account balances printed, SSH works.

PACE storage layout

PACE has THREE storage tiers, each with different size, speed, persistence, and access permissions. Putting things in the wrong tier is the #1 cause of beginner pain.

# Conda envs — large, want them permanent
/storage/project/r-rs275-0/yourgtid/.conda_envs/

# Model weights — very large, want them permanent
/storage/project/r-rs275-0/yourgtid/hfcache/hub/

# Project source code — small, want them permanent
/storage/project/r-rs275-0/yourgtid/AI-inference-hands-on-note/

# Datasets — bulky, may regenerate, OK to put in scratch
/storage/scratch1/X/yourgtid/data/

Note: Many labs at GT use a symlink so ~/r-rs275-0 points to the project storage. If your account has it, just use ~/r-rs275-0/... as the path.

Check your quotas

pace-quota
# Shows: home, scratch, project usage and account balances

Conda environments — vLLM and SGLang separately

1. Allocate a CPU node first (DO NOT install on login node)

# Get an interactive 1-hour CPU shell with 16GB RAM
salloc -N1 --cpus-per-task=4 --mem=16G -t 01:00:00 \
       -A gts-rs275-paid -q inferno -p cpu-small
# Wait for prompt to change — you are now on a compute node

2. Create the vLLM environment

module load anaconda3
eval "$(conda shell.bash hook)"

ENV=/storage/project/r-rs275-0/$USER/.conda_envs/vllm-lab
conda create -p $ENV python=3.11 -y
conda activate $ENV

# Install vLLM (pinned to a stable version compatible with this lab)
pip install --no-cache-dir "vllm==0.6.3"
pip install --no-cache-dir accelerate matplotlib pandas

# Verify
python -c "import vllm; print(f'vLLM {vllm.__version__}')"
python -c "import torch; print(f'torch {torch.__version__}')"

3. Create the SGLang environment (separate)

ENV2=/storage/project/r-rs275-0/$USER/.conda_envs/sglang-lab
conda create -p $ENV2 python=3.11 -y
conda activate $ENV2

pip install --no-cache-dir "sglang[all]==0.5.10.post1"
pip install --no-cache-dir matplotlib pandas cachetools

# Verify
python -c "import sglang; print(f'SGLang {sglang.__version__}')"
python -c "import torch; print(f'torch {torch.__version__}')"

4. Exit the salloc session

exit
# You are back on the login node. The envs are saved permanently in project storage.

Downloading model weights

1. Set up Hugging Face authentication

# Get a token from https://huggingface.co/settings/tokens (read access is enough)
# Add to your ~/.bashrc on PACE
export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export HF_HOME=/storage/project/r-rs275-0/$USER/hfcache/hub
export TRANSFORMERS_CACHE=$HF_HOME

# Apply to current shell
source ~/.bashrc

2. Download the models used in these labs

Do this in a CPU node session (huggingface-cli download is network/disk bound, no GPU needed)

salloc -N1 --cpus-per-task=4 --mem=16G -t 02:00:00 \
       -A gts-rs275-paid -q inferno -p cpu-small

module load anaconda3
conda activate /storage/project/r-rs275-0/$USER/.conda_envs/vllm-lab

# Llama-3.1-8B-Instruct (16 GB, used in weeks 1-12, 15)
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct \
    --local-dir $HF_HOME/Llama-3.1-8B-Instruct

# TinyLlama-1.1B (2 GB, used as draft model in week 11)
huggingface-cli download TinyLlama/TinyLlama-1.1B-Chat-v1.0

# FP8 and AWQ quantized variants (week 12)
huggingface-cli download neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8
huggingface-cli download hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4

# Llama-3.1-70B-Instruct (140 GB BF16, week 13 TP, week 14 reference)
# Skip if you don't need 70B — this takes ~30 min and 263 GB on disk
huggingface-cli download meta-llama/Llama-3.1-70B-Instruct \
    --local-dir $HF_HOME/Llama-3.1-70B-Instruct

exit

3. Verify the downloads

ls -la $HF_HOME/
# Llama-3.1-8B-Instruct/  (~16 GB)
# models--TinyLlama--TinyLlama-1.1B-Chat-v1.0/  (HF cache format)
# models--neuralmagic--Meta-Llama-3.1-8B-Instruct-FP8/
# models--hugging-quants--Meta-Llama-3.1-8B-Instruct-AWQ-INT4/
# Llama-3.1-70B-Instruct/  (263 GB)

# Total: ~285 GB — check your project quota allows this
du -sh $HF_HOME

Slurm: GPU types, queues, accounts

Available GPU partitions

Account & queue choice

Slurm command cheat sheet

# Interactive 30-min H100 shell
salloc -N1 --gres=gpu:H100:1 --cpus-per-task=8 --mem=64G -t 00:30:00 \
       -A gts-rs275-paid -q inferno

# Submit a batch job
sbatch myjob.sbatch

# See your jobs
squeue -u $USER --format='%.10i %.20j %.10T %.12M %.20R'

# Cancel a job
scancel JOBID

# See queue depth for a partition
squeue -p gpu-h100 --format='%.8T' | sort | uniq -c

# Check available nodes in a partition
sinfo -p gpu-h100 --format='%P %a %D %T %N'

Your first vLLM job: hello world

Use this template as your starting point. It is the minimum sbatch that will reliably start a vLLM server, send one request, and shut down cleanly.

1. Create the sbatch file

# save as ~/r-rs275-0/AI-inference-hands-on-note/sbatch/hello_vllm.sbatch
#!/bin/bash
#SBATCH --job-name=hello-vllm
#SBATCH --gres=gpu:H100:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=00:30:00
#SBATCH --account=gts-rs275-paid
#SBATCH -q inferno
#SBATCH --output=hello_vllm_%j.out
#SBATCH --error=hello_vllm_%j.err

set -euo pipefail
echo "=== START: $(date) ==="

# Activate environment
module load anaconda3
eval "$(conda shell.bash hook)"
conda activate /storage/project/r-rs275-0/$USER/.conda_envs/vllm-lab

# Use local model cache, force offline (no HF Hub calls)
export HF_HOME=/storage/project/r-rs275-0/$USER/hfcache/hub
export TRANSFORMERS_CACHE=$HF_HOME
export HF_HUB_OFFLINE=1

# Sanity check
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
python -c 'import torch; print(f"torch {torch.__version__}, cuda {torch.cuda.is_available()}")'

# Start the server in the background, write its log to a file
LOGFILE=$PWD/server_$SLURM_JOB_ID.log
python -m vllm.entrypoints.openai.api_server \
    --model $HF_HOME/Llama-3.1-8B-Instruct \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --disable-log-requests \
    > $LOGFILE 2>&1 &
SERVER_PID=$!
echo "Server PID: $SERVER_PID"

# Wait for the /health endpoint to respond (max 5 minutes)
for i in $(seq 1 60); do
    sleep 5
    if ! kill -0 $SERVER_PID 2>/dev/null; then
        echo "SERVER CRASHED"; tail -30 $LOGFILE; exit 1
    fi
    if curl -s http://localhost:8000/health >/dev/null 2>&1; then
        echo "READY after $((i*5))s"; break
    fi
done

# Send a test request
curl -s http://localhost:8000/v1/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "/storage/project/r-rs275-0/'$USER'/hfcache/hub/Llama-3.1-8B-Instruct",
       "prompt": "The capital of France is",
       "max_tokens": 16
     }' | python -m json.tool

# GRACEFUL shutdown — SIGINT, never SIGKILL
echo "=== Stopping server (SIGINT) ==="
kill -2 $SERVER_PID 2>/dev/null || true
wait $SERVER_PID 2>/dev/null || true

echo "=== END: $(date) ==="

2. Submit and watch

# Submit
sbatch hello_vllm.sbatch
# → "Submitted batch job 1234567"

# Watch the queue (Ctrl+C to stop watching)
watch -n 5 squeue -u $USER

# When the job starts running, tail the live output
tail -f hello_vllm_1234567.out

3. What success looks like

=== START: Wed Apr 9 23:00:00 EDT 2026 ===
NVIDIA H100 80GB HBM3, 81559 MiB
torch 2.8.0+cu126, cuda True
Server PID: 12345
READY after 25s
{
    "id": "cmpl-abc123",
    "choices": [{"text": " Paris, located in the north...", ...}],
    "usage": {"completion_tokens": 16, ...}
}
=== Stopping server (SIGINT) ===
=== END: Wed Apr 9 23:00:50 EDT 2026 ===

If you see Paris in the response, your environment is fully working. Proceed to Week 1.

Common errors and fixes

These are the errors we actually hit while building these labs. If you see one, the fix is probably here.

log_file = open("/tmp/server.log", "w")
proc = subprocess.Popen(cmd, stdout=log_file, stderr=subprocess.STDOUT)

export HF_HUB_OFFLINE=1
# And use the absolute local path, not the HF model id
python -m vllm.entrypoints.openai.api_server \
    --model $HF_HOME/Llama-3.1-8B-Instruct \
    ...

pip install --no-cache-dir 'transformers<5'