Life of an Inference Request in SimAI

1. System Positioning

2. Vidur → SimAI: What Changed and Why

The following table details every dimension where vidur-alibabacloud diverges from the original Microsoft Vidur. These changes enable full-stack network-aware simulation with Prefill-Decode disaggregation support.

3. Five-Component Architecture

SimAI is not a monolith — it is a federation of five repositories, each handling a different layer of the simulation stack. The following diagram shows how data flows between them.

4. Execution Time Breakdown

A request's end-to-end latency decomposes into 5 phases. This section summarizes what parameters affect each phase. For the detailed calculation logic (RandomForest models, backend comparison, per-layer formulas), see the Lifecycle page.

e2e_time = completed_at − arrived_at

Timeline

Non-PD Mode (same replica)

PD Disaggregation Mode

What Affects Each Phase

Typical Timing Reference

Single-request latency at low QPS (no queueing contention), non-PD mode (TP=1, PP=1). The RTX 5090 column is measured with vLLM v0.1 + LLaMA-3.1 8B (bf16). Other columns are estimates based on device specs.

5. New Core Entities

SimAI introduces several new entity classes to model PD disaggregation. These classes extend Vidur's original entity hierarchy with DAG-based task representation and explicit KV cache transfer flows.

5.1 Request with DAG

import networkx as nx

class Request(BaseEntity):
    """A single inference request with PD disaggregation support."""

    def __init__(self, ...):
        self.dag = nx.DiGraph()         # Task dependency graph
        self.prefill_replica_id = None # Assigned prefill replica
        self.decode_replica_id = None  # Assigned decode replica

        # PD communication metrics (populated after KV transfer)
        self.pd_p2p_comm_size = float('inf')
        self.pd_p2p_comm_time = float('inf')

        # Timing breakdown
        self.prefill_e2e = None
        self.decode_e2e = None
        self.prefill_start_timestamp = None
        self.decode_start_timestamp = None

5.2 Replica Types

from enum import IntEnum

class ReplicaType(IntEnum):
    MIXED   = 0  # Co-located: prefill + decode on same replica
    PREFILL = 1  # Dedicated prefill replica
    DECODE  = 2  # Dedicated decode replica

5.3 Node / Task / Flow Hierarchy

class Node(BaseEntity):
    """Base class for Task and Flow."""
    self.start_time: float
    self.end_time: float
    self.duration: float

class Task(Node):
    """Compute operation."""
    pass

class PromptTask(Task):
    """Prefill computation."""
    pass

class TokenTask(Task):
    """Decode computation."""
    pass

class Flow(Node):
    """Data transfer operation."""
    self.size_bytes: int
    self.bandwidth: float

class KVCacheTransferFlow(Flow):
    """KV cache transfer from prefill to decode replica.
    In PD disaggregation mode, this flow is inserted into the
    request DAG between PromptTask and TokenTask nodes."""

    def __init__(self, src_replica_id, dst_replica_id, kv_cache_size):
        self.src_replica_id = src_replica_id
        self.dst_replica_id = dst_replica_id
        self.kv_cache_size = kv_cache_size

5.4 Interconnect Types

class Interconnect:
    """Hardware interconnect abstraction."""

    # Supported interconnect types:
    NVLink     # Intra-node GPU-GPU (e.g., 900 GB/s per direction)
    RDMA       # Inter-node GPU-GPU via RoCEv2/InfiniBand
    Ethernet   # Standard Ethernet (fallback)
    PCIe       # CPU-GPU or NIC-GPU transfer
    DummyLink  # Zero-latency link for testing

    def __init__(self, link_type, bandwidth_bps, latency_us=0):
        self.link_type = link_type
        self.bandwidth_bps = bandwidth_bps
        self.latency_us = latency_us

    def transfer_time(self, size_bytes) -> float:
        """Estimate transfer time in microseconds."""
        return (size_bytes * 8 / self.bandwidth_bps * 1e6
                + self.latency_us)

6. Configuration System

SimAI extends Vidur's configuration with new parameters for PD disaggregation, network simulation, and MoE model support. All parameters are exposed as CLI flags and can be set in config files.

6.1 New Config Parameters

from dataclasses import dataclass, field

@dataclass
class ReplicaConfig:
    # ===== PD Disaggregation =====
    pd_p2p_comm_bandwidth: int = 800      # bps (bits per second)
    pd_p2p_comm_dtype: str = 'float16'    # KV cache data type
    pd_node_ratio: float = 0.5           # P:D ratio (0.5 = 1:1)

    # ===== Network Bandwidth =====
    nvlink_bandwidth: int = 1600          # bps (NVLink per direction)
    rdma_bandwidth: int = 800             # bps (RDMA per NIC)

    # ===== MoE (Mixture of Experts) =====
    expert_model_parallel_size: int = 1  # Expert parallelism degree

    # ===== Simulation Backend =====
    backend: str = "vidur"
    # Choices: "vidur"              - original Vidur (CPU only)
    #          "simai_simulation"   - NS-3 full network sim
    #          "simai_analytical"   - Bus bandwidth estimation
    #          "aicb"               - AICB GPU profiling backend

6.2 Execution Time Predictor Config

@dataclass
class RandomForrestExecutionTimePredictorConfig:
    """Config for the execution time predictor.
    In SimAI, this predictor can delegate to AICB for
    real GPU profiling instead of sklearn RandomForest."""

    backend: str = "vidur"
    # "vidur" - sklearn RandomForest on profiled CSV
    # "aicb"  - AICB AIOB real GPU kernel profiling

    compute_cache_dir: str = "./compute_cache"
    # Directory for cached AICB profiling results

    model_name: str = "deepseek-671B"
    # Target model for profiling

7. Three Simulation Modes

SimAI offers three simulation backends, each trading off speed for fidelity. Choose the right mode based on your exploration stage and available resources.

# Backend flag for analytical mode
--backend simai_analytical

# Backend flag for NS-3 mode
--backend simai_simulation

# Physical mode uses real hardware
--backend physical

8. Example CLI

The following command runs a PD disaggregation simulation for DeepSeek-V3-671B with the SplitWise scheduler and AICB-profiled execution times:

python -m vidur.main \
  --replica_config_model_name deepseek-671B \
  --replica_config_pd_p2p_comm_bandwidth 800 \
  --replica_config_nvlink_bandwidth 1600 \
  --replica_config_pd_node_ratio 0.5 \
  --global_scheduler_config_type split_wise \
  --replica_scheduler_config_type split_wise \
  --random_forrest_execution_time_predictor_config_backend aicb

python -m vidur.main \
  --replica_config_model_name qwen3-moe-235B \
  --replica_config_pd_node_ratio 0.3 \
  --global_scheduler_config_type split_wise \
  --replica_scheduler_config_type split_wise \
  --random_forrest_execution_time_predictor_config_backend vidur \
  --backend simai_analytical

python -m vidur.main \
  --replica_config_model_name deepseek-671B \
  --replica_config_pd_p2p_comm_bandwidth 800 \
  --replica_config_rdma_bandwidth 800 \
  --global_scheduler_config_type split_wise \
  --replica_scheduler_config_type split_wise \
  --backend simai_simulation \
  --network_topology fat_tree

CLI Parameter Reference

9. Key Insights

Appendix: Repository Map

SimAI spans five repositories under the aliyun GitHub organization. The following table maps each component to its repository and primary language.

Appendix: Output Metrics

SimAI produces a rich set of per-request and aggregate metrics. The PD-specific metrics are unique to SimAI and not available in the original Vidur.

class MetricsStore:
    """Central metrics collection for simulation results."""

    def on_request_complete(self, request: Request):
        # Standard Vidur metrics
        self._record("ttft", request.ttft)
        self._record("tbt", request.avg_tbt)
        self._record("e2e_latency", request.e2e_latency)

        # NEW: PD disaggregation metrics
        if request.prefill_e2e is not None:
            self._record("prefill_e2e", request.prefill_e2e)
            self._record("decode_e2e", request.decode_e2e)
            self._record("pd_p2p_comm_time", request.pd_p2p_comm_time)
            self._record("pd_p2p_comm_size", request.pd_p2p_comm_size)

Appendix: Request DAG Example

In PD disaggregation mode, a request produces a DAG with exactly three nodes. The single TokenTask node handles all decode tokens internally through batch iterations (tokens_per_iteration = 1), rather than creating a separate node per token. The KVCacheTransferFlow sits between prefill and decode, creating the critical PD communication dependency.

Appendix: SplitWise Scheduler

The SplitWise scheduler is SimAI's key addition to Vidur's scheduling layer. It operates at two levels: the global scheduler routes requests to the correct replica pool, and the replica scheduler manages batch formation within each pool.

class SplitWiseGlobalScheduler(BaseGlobalScheduler):
    """PD-aware global scheduler.
    Splits replicas into prefill and decode pools."""

    def __init__(self, config, replicas):
        self.pd_node_ratio = config.pd_node_ratio
        n_prefill = int(len(replicas) * self.pd_node_ratio)
        self.prefill_replicas = replicas[:n_prefill]
        self.decode_replicas = replicas[n_prefill:]

    def schedule(self, request: Request) -> int:
        # Route prefill to least-loaded prefill replica
        target = min(
            self.prefill_replicas,
            key=lambda r: r.pending_requests
        )
        request.prefill_replica_id = target.id
        return target.id

class SplitWiseReplicaScheduler(BaseReplicaScheduler):
    """PD-aware replica scheduler.
    Handles batch formation for a single replica type."""

    def _build_batch(self) -> Batch:
        if self.replica_type == ReplicaType.PREFILL:
            # Only schedule prefill requests
            candidates = [r for r in self.waiting
                         if not r.prefill_complete]
        else:
            # Only schedule decode iterations
            candidates = [r for r in self.running
                         if r.prefill_complete]
        return self._form_batch(candidates)

Appendix: AICB Profiling Pipeline

AICB (AI Communication Benchmark) provides two key capabilities to SimAI: workload description generation and real GPU kernel profiling. The profiling pipeline is designed to be run once per model/hardware combination and cached for repeated simulations.

# AICB workload description file for DeepSeek-V3-671B
# Format: op_type  data_size  compute_time_us  comm_type  deps
COMP    0          1250          NONE       -1     # QKV projection (DeepGEMM)
COMP    0          890           NONE       0      # MLA attention (FlashMLA)
COMP    0          420           NONE       1      # Output projection
COMM    8388608   0             ALLREDUCE  2      # TP AllReduce (8MB)
COMP    0          340           NONE       3      # Expert routing + gating
COMP    0          1680          NONE       4      # Sparse expert FFN (DeepGEMM)
COMM    16777216  0             ALLTOALL   5      # EP All-to-All (16MB)
COMP    0          560           NONE       6      # Shared expert FFN
COMM    8388608   0             ALLREDUCE  7      # TP AllReduce (8MB)

Appendix: Supported Network Topologies

The NS-3 backend supports multiple data center network topologies. Choosing the right topology significantly impacts PD communication latency and collective operation performance.

Appendix: SimCCL Algorithm Mapping

SimCCL maps high-level collective communication primitives to concrete point-to-point transfer schedules. The algorithm selection depends on the message size, number of participants, and network topology.

// Simplified Ring AllReduce decomposition (C++)
void RingAllReduce::decompose(
    int num_gpus,
    size_t message_size,
    std::vector<P2PTransfer>& transfers
) {
    size_t chunk_size = message_size / num_gpus;

    // Phase 1: Reduce-Scatter (n-1 steps)
    for (int step = 0; step < num_gpus - 1; step++) {
        for (int gpu = 0; gpu < num_gpus; gpu++) {
            int dst = (gpu + 1) % num_gpus;
            transfers.push_back({
                .src = gpu,
                .dst = dst,
                .size = chunk_size,
                .phase = step
            });
        }
    }

    // Phase 2: AllGather (n-1 steps)
    for (int step = 0; step < num_gpus - 1; step++) {
        for (int gpu = 0; gpu < num_gpus; gpu++) {
            int dst = (gpu + 1) % num_gpus;
            transfers.push_back({
                .src = gpu,
                .dst = dst,
                .size = chunk_size,
                .phase = num_gpus - 1 + step
            });
        }
    }
}

Appendix: Event Types in SimAI

SimAI's discrete-event simulation loop processes the following event types. Events marked with a star are new additions from SimAI (not present in original Vidur).

Repository Structure

SimAI is a multi-repo project composed of 5 git submodules plus top-level orchestration. Below is the complete directory tree with descriptions of each component's role.

Top-Level Structure

SimAI/
├── README.md              ── Project overview, scenarios, setup guide
├── Dockerfile             ── Docker image (nvidia/pytorch base + AICB + Vidur)
├── .gitmodules            ── Submodule config: SimCCL, aicb, ns-3-alibabacloud
├── scripts/
│   └── build.sh           ── Master build: -c analytical | ns3 | phy
├── example/
│   ├── microAllReduce.txt ── Sample AllReduce workload (8 GPUs, TP=8)
│   ├── workload_analytical.txt ── Sample analytical workload
│   └── busbw.yaml        ── Bus bandwidth config (TP/DP/EP/PP per-op BW)
├── docs/
│   ├── Tutorial.md        ── Comprehensive usage tutorial
│   └── SimAI_Intro_Online.pdf ── Presentation slides
│
├── vidur-alibabacloud/    ── ① Inference scheduling simulator (Python)
├── aicb/                  ── ② Workload generation + GPU profiling (Python+CUDA)
├── SimCCL/                ── ③ Collective communication decomposition (C++)
├── astra-sim-alibabacloud/ ── ④ System simulation engine (C++)
└── ns-3-alibabacloud/     ── ⑤ Packet-level network simulator (C++)

① vidur-alibabacloud — Inference Scheduling Simulator

vidur-alibabacloud/vidur/
├── main.py                 ── Entry point (python -m vidur.main)
├── simulator.py            ── DES event loop, manages clock + event queue
├── config/
│   ├── config.py           ── All config dataclasses (ReplicaConfig, PD params)
│   └── model_config.py     ── Model specs (DeepSeek-671B, Qwen3-MoE-235B, etc.)
├── entities/               ── Core data models
│   ├── request.py          ── Request with DAG (nx.DiGraph), PD metadata
│   ├── replica.py          ── ReplicaType: MIXED / PREFILL / DECODE
│   ├── batch.py            ── Batch of requests for co-execution
│   ├── task.py             ── PromptTask (prefill) / TokenTask (decode)
│   ├── flow.py             ── KVCacheTransferFlow (PD disaggregation)
│   ├── node.py             ── Base abstraction for Task/Flow in DAG
│   └── interconnect.py     ── NVLink / RDMA / Ethernet / PCIe link models
├── events/                 ── DES event types
│   ├── request_arrival_event.py ── Request enters system
│   ├── replica_schedule_event.py ── Triggers replica scheduling
│   └── batch_end_event.py  ── Batch execution completes
├── scheduler/
│   ├── global_scheduler/
│   │   ├── splitwise_global_scheduler.py ── PD-aware: P-pool + D-pool routing
│   │   ├── lor_global_scheduler.py ── Least Outstanding Requests
│   │   └── round_robin_global_scheduler.py
│   └── replica_scheduler/
│       ├── splitwise_replica_scheduler.py ── PD-aware per-replica scheduling
│       └── vllm_replica_scheduler.py ── vLLM-style scheduling policy
├── execution_time_predictor/
│   ├── sklearn_execution_time_predictor.py ── RandomForest / AICB backend
│   ├── communication_time_predictor.py    ── SimAI NS-3 / analytical backend
│   └── SimAIWorkload.py   ── Workload file builder for astra-sim
├── request_generator/      ── Synthetic / trace-based / Poisson arrivals
└── metrics/                ── TTFT, TBT, E2E latency, PD breakdown

② aicb — Workload Generation & GPU Profiling

aicb/
├── aicb.py                  ── Main entry point for benchmark execution
├── workload_applyer.py      ── Applies workloads to GPU cluster (runs collectives)
├── workload_generator/
│   ├── SimAI_inference_workload_generator.py ── Inference workload (prefill/decode)
│   ├── SimAI_training_workload_generator.py  ── Training workload files
│   └── mocked_model/
│       ├── MockedModel.py   ── Base class, InferencePhase enum (PREFILL/DECODE)
│       ├── inference/
│       │   ├── MockedDeepSeek.py  ── DeepSeek-V3 MLA + MoE architecture mock
│       │   ├── MockedQwen3Moe.py  ── Qwen3-MoE architecture mock
│       │   ├── MockedQwen3Next.py ── Qwen3-Next (hybrid attention) mock
│       │   ├── AiobDeepSeek.py    ── GPU kernel profiling: FlashMLA, DeepGEMM FP8
│       │   ├── AiobQwen3Moe.py    ── Qwen3-MoE GPU kernel profiling
│       │   └── AiobQwen3Next.py   ── Qwen3-Next GPU kernel profiling
│       └── training/        ── DeepSpeed, Megatron, DeepSeek training mocks
├── utils/
│   ├── utils.py             ── CommType, CommGroup, Strategy enums, compute cache
│   ├── deepgemm_utils.py    ── FP8 GEMM (per_token/per_block quantization)
│   └── timer.py             ── CudaEventTimer for GPU kernel timing
├── workload/
│   └── simAI/model_workload/ ── Pre-generated workload files (.txt)
├── scripts/inference_configs/
│   ├── deepseek_default.json  ── DeepSeek-V3 model config
│   ├── qwen3_moe_default.json ── Qwen3-MoE model config
│   └── qwen3_next_default.json ── Qwen3-Next model config
└── log_analyzer/            ── Result analysis and plotting tools

③ SimCCL — Collective Communication Decomposition

SimCCL's core logic currently lives inside astra-sim-alibabacloud/astra-sim/system/MockNccl* files. The standalone SimCCL repo contains documentation; the full implementation will be released separately.

④ astra-sim-alibabacloud — System Simulation Engine

astra-sim-alibabacloud/
├── CMakeLists.txt          ── Top-level CMake config
├── build/
│   ├── simai_analytical/   ── Build config → bin/SimAI_analytical
│   ├── astra_ns3/          ── Build config → bin/SimAI_simulator
│   └── simai_phy/          ── Build config → bin/SimAI_phynet
├── astra-sim/
│   ├── system/              ── Core simulation (88 files)
│   │   ├── Sys.cc/.hh       ── Main system class, event dispatch
│   │   ├── MockNcclGroup.cc/.h ── NCCL algo decomposition (Ring/Tree/NVLS)
│   │   ├── MockNcclChannel.cc/.h ── SingleFlow, ncclTree, ncclChannelNode
│   │   ├── MockNccl.h       ── Algorithm IDs, base/hw latency tables
│   │   ├── MockNcclQps.h    ── QPS tracking per connection
│   │   ├── SimAiFlowModelRdma.cc ── RDMA flow model
│   │   └── calbusbw.cc      ── Bus bandwidth calculator
│   ├── network_frontend/
│   │   ├── analytical/      ── AnaSim: fast tick-based estimation
│   │   ├── ns3/             ── NS-3 integration (entry.h = main bridge)
│   │   └── phynet/          ── Physical RDMA traffic generation
│   └── workload/
│       ├── Workload.cc/.hh  ── Workload parser (reads .txt workload files)
│       └── Layer.cc/.hh     ── Single compute+comm layer representation
├── inputs/
│   ├── config/
│   │   └── SimAI.conf       ── 60+ params: CC_MODE, PFC, ECN, monitoring
│   ├── topo/
│   │   └── gen_Topo_Template.py ── 5 topologies: Spectrum-X, HPN, DCN+
│   └── ratio/               ── NIC/NVLink/ATA performance ratio CSV

⑤ ns-3-alibabacloud — Packet-Level Network Simulator

ns-3-alibabacloud/
├── simulation/src/point-to-point/model/  ── Core network models (37 files)
│   ├── rdma-hw.cc/.h        ── RDMA NIC: QP management, CC algorithms
│   ├── rdma-queue-pair.cc/.h ── QP state: rate, window, seq tracking
│   ├── switch-node.cc/.h    ── Switch: ECMP hash routing, ECN marking
│   ├── switch-mmu.cc/.h     ── Switch MMU: buffer mgmt, PFC, RED/ECN
│   ├── qbb-net-device.cc/.h ── QBB NIC: PFC pause/resume, WRR scheduling
│   ├── nvswitch-node.cc/.h  ── NVSwitch: intra-node NVLS routing
│   ├── int-header.h         ── INT telemetry header (for HPCC)
│   └── pint.h               ── Probabilistic INT (for HPCC-PINT)
├── simulation/src/network/utils/
│   └── custom-header.cc/.h ── Packet header with IP, port, seq, INT
├── analysis/                ── Post-simulation analysis tools
│   ├── fct_analysis.py      ── Flow Completion Time analysis
│   ├── qlen_analysis.py     ── Queue length analysis
│   ├── bw_analysis.py       ── Bandwidth utilization analysis
│   └── qp_cnp_analysis.py   ── CNP count per QP analysis
└── docs/images/             ── Network topology diagrams