Vidur: A Large-Scale Simulation Framework for LLM Inference

Problem Statement & Motivation

Optimizing LLM inference deployment is extraordinarily expensive. Providers must navigate a vast configuration space spanning parallelism strategies (TP, PP), scheduling policies (vLLM, Orca, Sarathi-Serve), batch sizes, GPU SKUs (A100, H100), and workload-specific parameters. Each configuration point requires running the actual model on GPUs, costing up to $97K per data point.

Key Observations from the Paper

Challenges in Simulating LLM Inference

Key Contributions

The paper introduces three main components that work together to enable fast, inexpensive LLM inference performance exploration:

System Architecture (Paper Figure 2)

Vidur's architecture has two main phases: Model Onboarding (offline profiling + ML model training) and Simulation Runtime (discrete-event simulation with hierarchical scheduling).

Execution Time Modeling

Vidur's runtime prediction is built on the key insight that LLM operators can be classified into three categories, each with a different prediction strategy. The paper uses Random Forest regression models trained on profiled data, achieving accuracy that prevents error cascading.

Operator Classification & Prediction Strategy

Prefill Attention: The Equivalent Batch Trick

Implementation: ExecutionTime Entity

In the codebase, execution time is decomposed into 20 distinct components, each predicted separately:

# vidur/entities/execution_time.py -- 20 components per iteration
class ExecutionTime:
    def _get_block_execution_time(self) -> float:
        return (
            self._get_attention_layer_execution_time()  # QKV proj + RoPE + KV save + attn + out proj + norm
            + self._get_mlp_layer_execution_time()        # up_proj + down_proj + act + norm + allreduce
            + self._add_time                              # residual connection
        )

    @property
    def model_time(self) -> float:
        block_time = self._get_block_execution_time()
        stage_time = block_time * self._num_layers_per_pipeline_stage
        return (stage_time + self.pipeline_parallel_communication_time) * 1e-3

    @property
    def total_time(self) -> float:
        return self.model_time + self._get_cpu_overhead() * 1e-3
        # CPU overhead = schedule + sampler + prepare_inputs + process_outputs + ray_comm

Random Forest Prediction Pipeline

The sklearn-based predictor in the repo follows this pipeline:

# vidur/execution_time_predictor/sklearn_execution_time_predictor.py
# Derived features for attention prediction:
df["num_tokens"] = df[["prefill_chunk_size", "batch_size"]].max(axis=1)
df["is_decode"] = df["prefill_chunk_size"] == 0
df["prefill_chunk_size_squared"] = df["prefill_chunk_size"] ** 2  # captures quadratic attention cost

# Communication features:
df["num_tokens"] = df["size"] / model_config.embedding_dim / 2  # bytes -> tokens

# Model selection: GridSearchCV with MAPE scorer
# Paper chose Random Forest over MLP and polynomial regression
# because RF captures non-linear CUDA kernel characteristics

Scheduling Policies Compared

Vidur implements five scheduling policies, each under 150 lines of Python. The paper classifies them into prefill-prioritizing and decode-prioritizing categories, with Sarathi-Serve bridging both.

Hierarchical Scheduler Architecture

What-If Analysis: Best Policies by Workload (Paper Section 7.3)

Simulation Accuracy (Paper Figures 3 & 4)

Vidur was validated across four models (LLaMA2-7B, InternLM-20B, LLaMA2-70B, Qwen-72B), three workloads, and both static and dynamic arrival patterns. The baseline is an optimized vLLM fork with CUDA graph support.

Static Workloads (Figure 3)

Dynamic Workloads at 85% Capacity (Figure 4)

Accuracy at Different Load Levels (Appendix Figure 7-8)

Evaluation Results & What-If Analysis

Optimal Configurations Found (Paper Figure 1a & 6)

Vidur-Search explored all 12 model-trace combinations across TP={1,2,4}, PP={1,2,4}, Scheduler={vLLM, Orca+, Sarathi-Serve}, BatchSize={32,64,128,256,512}, and GPU SKU={A100,H100}. The results reveal that no single configuration is universally optimal.

Pareto Frontier Analysis (Paper Figure 5)

Cost Comparison: Simulation vs Actual (Paper Table 2)

Paper Claims vs Repository Implementation

We conducted a thorough comparison of the paper's claims against the open-source codebase at github.com/microsoft/vidur. Here is the detailed analysis:

Core Simulator Engine

Scheduling Policies

Execution Time Prediction

Model Support

Vidur-Search (Configuration Optimizer)

What Is Missing or Different?

Discrete-Event Simulation: Code Deep Dive

The core simulation loop is remarkably concise. The entire engine is 129 lines of Python:

# vidur/simulator.py -- The entire simulation engine
class Simulator:
    def run(self):
        while self._event_queue and not self._terminate:
            _, event = heapq.heappop(self._event_queue)  # get highest priority event
            self._set_time(event._time)                     # advance simulation clock
            new_events = event.handle_event(               # process event, get next events
                self._scheduler, self._metric_store
            )
            self._add_events(new_events)                    # push new events to queue

Event Lifecycle

Memory Management: MemoryPlanner

# vidur/scheduler/utils/memory_planner.py
class MemoryPlanner:
    def get_max_batch_size(self) -> int:
        available = gpu_memory * (1 - margin_fraction)  # typically 80GB * 0.9
        param_memory = 2 * num_parameters_per_device   # FP16
        kv_per_request = (
            2                                          # bytes per float
            * 2                                         # key + value
            * attention_head_dim
            * kv_heads_per_tp_worker
            * max_request_tokens
            * num_layers
        )
        return (available - param_memory) // kv_per_request

Vidur-Bench: Workload Characteristics

The benchmark suite includes three carefully chosen workloads that represent different LLM usage patterns. The key insight is that workload characteristics dramatically affect optimal configuration.

Request Generation in the Repo

The repo implements a flexible request generation framework supporting both synthetic and trace-replay modes:

Use Cases

# Run a single simulation
python -m vidur.main \
  --replica_config_model_name meta-llama/Llama-2-70b-hf \
  --cluster_config_num_replicas 4 \
  --replica_config_num_pipeline_stages 2 \
  --replica_config_tensor_parallel_size 4 \
  --replica_scheduler_config_type sarathi \
  --sarathi_scheduler_chunk_size 512 \
  --vllm_scheduler_batch_size_cap 256 \
  --request_generator_config_type synthetic \
  --synthetic_request_generator_interval_generator_config_type poisson \
  --poisson_request_interval_generator_qps 0.5 \
  --length_generator_config_type trace \
  --trace_request_length_generator_trace_file data/processed_traces/chat1m.csv

# Run Vidur-Search for optimal configuration
python -m vidur.config_optimizer.config_explorer.main \
  --output_dir /tmp/vidur_search \
  --time_limit 600 \
  --scheduling_delay_slo_value 5 \
  --scheduling_delay_slo_quantile 0.99

# Implementing a new scheduler requires 2 methods:
class MyCustomScheduler(BaseReplicaScheduler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._num_running_batches = 0

    def on_batch_end(self, batch: Batch) -> None:
        self._num_running_batches -= 1
        for request in batch.requests:
            if request.completed:
                self.free(request.id)

    def _get_next_batch(self) -> Batch:
        # Your batching logic here
        # Has access to: self._request_queue, self._allocation_map,
        # self.can_allocate(), self.allocate(), self._max_batch_size
        pass

Limitations & Future Work

Future Directions (from Paper Section 4.5 & 9)

Key Takeaways

Repository Structure Summary

Related Work & Context

Vidur is positioned against DNN training simulators and differentiates itself by addressing inference-specific challenges: