NVIDIA Dynamo
Datacenter-Scale Inference Stack

Three-Plane Architecture

Dynamo is built around three cooperating concerns: a fast Request Plane for token generation, a responsive Control Plane for scaling and placement, and a resilient Storage & Events Plane for KV reuse and failure recovery. This separation allows each concern to evolve and scale independently.

Design Goals

End-to-End Request Flow (Disaggregated Mode)

From the architecture.md design doc -- the nine steps of a disaggregated inference request:

Smart Router: KV-Aware Routing

The KV Router is implemented in Rust (lib/kv-router/src/) and is the critical decision point for every inference request. It tracks two metrics per worker: potential active blocks (decode load) and potential new prefill blocks (tokens requiring fresh computation). These feed a cost function that minimizes redundant prefill while balancing decode load.

Radix Tree KV Cache Indexer

The KVIndexer maintains a global radix (prefix) tree built from worker-reported KV events. Each node in the tree stores a set of worker IDs that have cached that particular block. From lib/kv-router/src/indexer/radix_tree.rs:

// lib/kv-router/src/indexer/radix_tree.rs

pub(crate) struct RadixBlock {
    /// Child blocks, keyed by local block hash
    pub children: FxHashMap<LocalBlockHash, SharedRadixBlock>,
    /// Workers that have this block cached
    pub workers: FxHashSet<WorkerWithDpRank>,
    /// External sequence block hash (None for root)
    pub block_hash: Option<ExternalSequenceBlockHash>,
    /// Recency buffer for frequency-based decisions
    pub recent_uses: VecDeque<Instant>,
}

pub struct RadixTree {
    pub root: SharedRadixBlock,
    /// Per-worker O(1) lookup: worker -> (block_hash -> block)
    pub lookup: FxHashMap<WorkerWithDpRank,
              FxHashMap<ExternalSequenceBlockHash, SharedRadixBlock>>,
    pub expiration_duration: Option<Duration>,
}

The find_matches method traverses the tree for a token sequence, tracking which workers share each prefix depth. Workers "drop out" as we descend where their cached blocks end. The algorithm supports early exit when only a single worker remains, and per-block frequency tracking with TTL-based expiration for approximate mode.

Cost Function and Worker Selection

The core selection logic in lib/kv-router/src/scheduling/selector.rs computes a cost for each worker and selects the minimum:

// lib/kv-router/src/scheduling/selector.rs - DefaultWorkerSelector

let get_score = |worker: WorkerWithDpRank| -> f64 {
    let overlap = overlaps.get(&worker).unwrap_or(&0);
    let prefill_token = prefill_tokens.get(&worker).unwrap_or(&isl);
    let potential_prefill_block = prefill_token as f64 / block_size as f64;
    let decode_block = decode_blocks.get(&worker).unwrap_or(&...);

    // The core cost function:
    let logit = overlap_weight * potential_prefill_block + decode_block;
    logit
};

Scheduling Policies

When router_queue_threshold is set, the router maintains a priority queue with pluggable policies (from lib/kv-router/src/scheduling/policy.rs):

KV Event Transport Modes

KV Block Manager (KVBM)

KVBM is the multi-tier cache manager that extends effective KV cache capacity far beyond GPU HBM. Implemented in Rust (lib/kvbm-physical/, lib/kvbm-logical/, lib/kvbm-common/), it manages a 4-tier memory hierarchy with asynchronous offload/onboard operations coordinated through NIXL.

Block State Machine

From kvbm-design.md, each KV block follows a strict lifecycle managed via RAII handles:

KVBM Internal Architecture

The KvBlockManager<H, D> orchestrates across memory tiers by managing per-backend block pools:

// Conceptual structure from kvbm-design.md

KvBlockManager<H, D> owns:
  - BlockPool<Device>     // GPU-resident blocks (G1)
  - BlockPool<Host>       // CPU pinned-memory blocks (G2)
  - NixlAgent             // Remote communication + memory sharing
  - BlockSetRegistry      // Remote lookup + import/export metadata

Each BlockPool<T> tracks:
  - ActivePool     // Blocks currently in use by sequences
  - InactivePool   // Recycled blocks ready for allocation (free list)

Block memory layout: [num_layers][page_size x inner_dim]
  block_stride = align_up(num_layers * layer_stride, alignment)

Transfer Manager and Data Flows

The TransferManager is an asynchronous orchestrator with per-path queues:

Planner: SLA-Driven Dynamic Scaling

The Planner is Dynamo's autoscaling controller, implemented in Python (components/src/dynamo/planner/). It supports two scaling modes: throughput-based (profiling data + traffic prediction) and load-based (real-time engine metrics + online regression). Both aim to meet TTFT and ITL SLA targets while minimizing total cost of ownership (TCO).

Throughput-Based Scaling Algorithm

Replica Calculation Formulas

From planner-design.md and planner_core.py:

# Prefill replicas (single-batched, linear correction effect)
predicted_load = next_requests * next_isl / interval * min(1, prefill_correction)
prefill_replicas = ceil(predicted_load / interpolated_throughput / gpus_per_engine)

# Decode replicas (correction applied to ITL SLA target)
corrected_itl = target_itl / decode_correction_factor
throughput_per_gpu = decode_interpolator.find_best_throughput_per_gpu(
    itl=corrected_itl,
    context_length=next_isl + next_osl / 2
)
decode_replicas = ceil(next_num_req * next_osl / interval / throughput_per_gpu / gpus_per_engine)

Load Predictor Algorithms

Four predictor implementations in load_predictor.py, all extending BasePredictor:

Load-Based Scaling Mode

Uses ForwardPassMetrics (FPM) from the Dynamo event plane for real-time scaling without profiling data. Three specialized regression models in fpm_regression.py:

Scaling decisions: scale up if ALL engines' estimated latency exceeds SLA; scale down if ALL are below SLA * sensitivity. Only +/-1 per interval with pending-desired guard.

Co-existence of Modes

When both modes are enabled, throughput-based scaling (longer interval) sets a lower bound on replicas while load-based scaling (shorter interval) handles real-time adjustments above that floor. This gives the best of both worlds: capacity planning from profiling data and reactive adaptation from live metrics.

Disaggregated Prefill/Decode Serving

Prefill and decode have fundamentally different compute characteristics: prefill is compute-bound (processes all input tokens in one forward pass) while decode is memory-bandwidth-bound (generates one token at a time). Disaggregating them into specialized GPU pools allows each to scale independently, using optimal tensor-parallel (TP) configurations for each phase.

PrefillRouter Orchestration

The PrefillRouter (from disagg-serving.md) orchestrates the full flow:

Backend-Specific Transfer Metadata

Runtime-Reconfigurable xPyD

Dynamo supports runtime-reconfigurable xPyD (x prefill workers, y decode workers). Workers can be added or removed without downtime:

• Add worker: Registers with discovery service and publishes RuntimeConfig (including KV capacity). Router auto-discovers via discovery service.
• Remove worker: Drains active requests and deregisters from discovery. Router stops routing new requests to it.

NIXL: NVIDIA Interchange Library

NIXL is the data transfer fabric that enables high-speed KV cache movement across workers and memory domains. In Dynamo's Rust codebase (lib/memory/src/nixl.rs, lib/kvbm-physical/src/transfer/executor/nixl.rs), NIXL provides a unified interface over heterogeneous transports (NVLink, InfiniBand/UCX, PCIe, POSIX I/O).

NIXL Architecture in KVBM

// lib/memory/src/nixl.rs

/// Trait for storage types that can be registered with NIXL.
pub trait NixlCompatible {
    /// Returns (ptr, size, mem_type, device_id)
    fn nixl_params(&self) -> (*const u8, usize, MemType, u64);
}

/// NIXL descriptor for memory region registration.
pub struct NixlDescriptor {
    pub addr: u64,       // Base address
    pub size: usize,     // Region size in bytes
    pub mem_type: MemType, // host / device / etc.
    pub device_id: u64,  // GPU index for device memory
}

Typestate Transfer Builder

The NIXL transfer builder in lib/kvbm-physical/src/transfer/executor/nixl.rs uses Rust's typestate pattern for compile-time safety -- all required parameters (source layout, destination layout, block IDs, transfer strategy) must be set before a transfer can be executed:

// lib/kvbm-physical/src/transfer/executor/nixl.rs

pub struct NixlTransferBuilder<'a, TSrc, TDst, TSrcBlocks, TDstBlocks, TStrategy> {
    src: Option<&'a PhysicalLayout>,
    dst: Option<&'a PhysicalLayout>,
    src_block_ids: Option<&'a [BlockId]>,
    dst_block_ids: Option<&'a [BlockId]>,
    strategy: Option<TransferStrategy>,
    // Phantom markers: Unset -> Set at compile time
}

Remote Memory Registration Protocol

From the KVBM design doc, the bidirectional protocol for cross-worker KV transfer:

Agentic Features

From docs/features/agentic_workloads.md: Agentic LLM inference is dominated by KV-cache storage and I/O rather than computation. Without leveraging the predictable structure of agent lifecycles, significant optimizations are left on the table. Dynamo bridges this gap with agentic hints that flow from the harness through the router to the KV cache manager.

nvext API: Agentic Hints

Hints are carried in the request body under nvext on chat completions. The frontend parses them and passes them to the KV router and backends.

Cache Pinning with TTL

From the router config (lib/kv-router/src/scheduling/config.rs), cache control is enabled via router_enable_cache_control. When enabled, after generation completes the router calls a pin_prefix hook (with TTL) to the worker's cache_control service mesh endpoint. Pinned nodes resist eviction in SGLang's HiCache, demoting to host memory rather than being deleted.

// From lib/kv-router/src/scheduling/config.rs

/// Enable cache control (PIN with TTL) via the worker's
/// cache_control service mesh endpoint.
pub router_enable_cache_control: bool,  // default: false

LangChain Integration

Dynamo is supported directly in LangChain via the NVIDIA AI Endpoints integration. Configure the chat model to use the Dynamo endpoint and pass agent hints directly from the LangChain client.

Feature Matrix by Backend

v1.0 Features & Benchmarks

Dynamo 1.0 shipped March 2026 as a production-ready release with strong community adoption. The release spans zero-config deployment, agentic inference, multimodal E/P/D, video generation, and K8s Inference Gateway integration.

What's New in 1.0

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  sla: { ttft: 200.0, itl: 20.0 }
  autoApply: true

Key Benchmarks

Backend Support Matrix

Codebase Map

Key directories and files explored during this analysis of the ai-dynamo/dynamo repository:

Rust Core (Performance-Critical)

Python Components (Extensibility)

Design Documentation