Research digest 5 of 6 · Chapter 3: Splitting the model

研究導讀 5 / 6 · 第 3 章：切分模型

Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI ArchitecturesDeepSeek-V3 的洞見：擴展的挑戰，以及對 AI 架構硬體的反思

ISCA 2025Chenggang Zhao, Chengqi Deng, Chong Ruan, et al. (DeepSeek-AI) · paper論文

After reading, you can explain how DeepSeek-V3 fits MoE all-to-all onto H800 nodes with weak NVLink: node-limited routing, NVLink forwarding, FP8 dispatch and an eight-plane two-tier fat-tree, and what was measured.讀完後，你能說明 DeepSeek-V3 如何把 MoE 的 all-to-all 放進 NVLink 被削弱的 H800 節點：節點受限路由、NVLink 轉送、FP8 dispatch 與八平面兩層 fat-tree，以及實際量到了什麼。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 3 ↗第 3 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

DeepSeek-V3 is a mixture-of-experts (MoE) model with 671B parameters, 37B active per token, trained on 2,048 NVIDIA H800 GPUs whose NVLink is cut to 400 GB/s. Each node has eight 400 Gb/s InfiniBand NICs, and NIC $i$ sits in plane $i$ of an eight-plane, two-tier fat-tree. NVLink is still about 4 times faster than one NIC, so each token goes to experts on at most 4 nodes: it crosses InfiniBand once per node, and NVLink copies it inside the node. Dispatch travels in FP8, combine in BF16. On the full job, the multi-plane network trained as fast as a single-plane multi-rail one (272.80 against 272.52 billion tokens per day). The paper has few ablations and many requests to hardware vendors.

DeepSeek-V3 是一個混合專家（MoE）模型，共 671B 個參數，每個 token 啟用 37B，在 NVLink 被削減到 400 GB/s 的 2,048 顆 NVIDIA H800 上訓練。每個節點有八張 400 Gb/s InfiniBand NIC，NIC $i$ 接在八平面、兩層 fat-tree 的平面 $i$ 。NVLink 仍比一張 NIC 快約 4 倍，所以每個 token 只送往最多 4 個節點上的專家：它對每個節點只跨 InfiniBand 一次，再由 NVLink 在節點內複製。dispatch 用 FP8 傳送，combine 用 BF16。在完整任務上，多平面網路的訓練速度與單平面 multi-rail 網路相同（每天 272.80 對 272.52 billion 個 token）。論文的消融實驗很少，對硬體廠商的要求很多。

Where this paper sits這篇論文的位置

Section 3.7 explains expert parallelism (EP). An expert is a feed-forward block. A router (gate) picks each token's top- $k$ experts. A dispatch all-to-all sends the token's hidden vector to the GPUs of those experts, and a combine all-to-all brings the weighted outputs back. DeepSeek-V3 has 256 routed experts and 1 shared expert per MoE layer; each token uses 8 routed experts plus the shared one. Chapter 3 left open how a production team fits these all-to-alls to a real machine. This paper answers for 2,048 GPUs. It is a co-design paper, not a systems paper: most of it states design choices and requests, and a few tables and figures carry measurements.

第 3.7 節解釋專家平行（EP）。專家是一個前饋區塊。router（gate）為每個 token 選出 top- $k$ 個專家。dispatch all-to-all 把 token 的隱藏向量送到這些專家的 GPU，combine all-to-all 再把加權輸出送回。DeepSeek-V3 每個 MoE 層有 256 個 routed 專家與 1 個共享專家；每個 token 使用 8 個 routed 專家加上共享專家。第 3 章沒有回答生產團隊如何讓這些 all-to-all 配合真實的機器。這篇論文針對 2,048 顆 GPU 給出答案。它是軟硬體協同設計論文，不是系統論文：大部分內容是設計選擇與要求，只有少數表格與圖帶有量測。

The problem, measured問題，用量測說明

The H800 is a Hopper GPU like the H100, but export rules cut its NVLink from 900 GB/s to 400 GB/s (§4.1). §4.3 counts NVLink as 200 GB/s with about 160 GB/s achievable (we read 400 GB/s as both directions; the paper does not say). Each 400 Gb/s NIC carries 50 GB/s on paper and about 40 GB/s in practice. So scale-up is only about 4 times faster than one scale-out path, and this ratio drives the design (figure below).

H800 與 H100 一樣是 Hopper GPU，但出口規定把它的 NVLink 從 900 GB/s 削減到 400 GB/s（§4.1）。§4.3 把 NVLink 算成 200 GB/s，約 160 GB/s 可實際用到（我們把 400 GB/s 理解為雙向合計；論文沒有說明）。每張 400 Gb/s NIC 名目上 50 GB/s，實際約 40 GB/s。所以 scale-up 只比一條 scale-out 路徑快約 4 倍，這個比例驅動整個設計（見下圖）。

One node, drawn from Figure 2 and §4–§5 of the paper. Each GPU shares a PCIe switch (grey) with its own NIC; NIC

i

belongs only to plane

i

. Bandwidths are the paper's stated values.依論文 Figure 2 與 §4–§5 繪製的一個節點。每顆 GPU 與自己的 NIC 共用一個 PCIe 交換器（灰）；NIC

i

只屬於平面

i

。頻寬是論文給的數值。Drawn for this digest from Figure 2, §4.1, §4.3 and §5.1 of the paper.

Two more facts set the budget. Up to 20 of the H800's streaming multiprocessors (SMs) run communication during training, because the GPU itself fills the queue pairs and forwards data between InfiniBand (IB) and NVLink (§4.4.1). And MoE saves compute: the paper counts 250 GFLOPs per training token against 2,448 for the dense LLaMA-3.1 405B (Table 2, sequence length 4,096). It pays in all-to-all bytes instead.

還有兩個事實決定了預算。訓練期間 H800 最多有 20 個串流多處理器（SM）執行通訊，因為由 GPU 自己填寫 queue pair，並在 InfiniBand（IB）與 NVLink 之間轉送資料（§4.4.1）。而 MoE 省下計算：論文算出每個訓練 token 需要 250 GFLOPs，稠密的 LLaMA-3.1 405B 則需要 2,448（Table 2，序列長度 4,096）。代價改由 all-to-all 的位元組支付。

The design設計

1. A parallel layout without tensor parallelism1. 不用張量平行的平行化配置

The team avoids tensor parallelism (TP) in training, because TP's per-layer all-reduces would run on the reduced NVLink (§4.2). Pipeline parallelism (PP) uses DualPipe, which overlaps one micro-batch's compute with another's MoE communication and shrinks bubbles. EP uses DeepEP, the team's open-source all-to-all library. The PP, EP and data-parallel sizes are not in this paper. NVLink carries intra-node EP and forwarding; the eight IB planes carry inter-node EP, pipeline and gradient traffic; a separate RoCE NIC carries storage.

團隊在訓練時不用張量平行（TP），因為 TP 每層的 all-reduce 都要跑在被削弱的 NVLink 上（§4.2）。管線平行（PP）使用 DualPipe，讓一個 micro-batch 的計算與另一個的 MoE 通訊重疊，並縮小氣泡。EP 使用團隊開源的 all-to-all 函式庫 DeepEP。這篇論文沒有給 PP、EP 與資料平行的大小。NVLink 承載節點內 EP 與轉送；八個 IB 平面承載跨節點 EP、管線與梯度流量；另一張 RoCE NIC 承載儲存。

2. Node-limited routing and NVLink forwarding2. 節點受限路由與 NVLink 轉送

§4.3 makes the central argument with an example: 8 nodes, 64 GPUs and 256 routed experts, so 4 experts per GPU. Let $t$ be the time to send one token's hidden vector over IB. If a token's 8 experts sit on 8 nodes, IB costs $8t$ . If they sit on $M$ nodes, the token can cross IB once per node and then be copied over NVLink to the other target GPUs of that node. IB then costs only

§4.3 用一個例子提出核心論點：8 個節點、64 顆 GPU、256 個 routed 專家，所以每顆 GPU 有 4 個專家。設 $t$ 為經 IB 傳送一個 token 隱藏向量的時間。如果一個 token 的 8 個專家分布在 8 個節點，IB 成本是 $8t$ 。如果它們在 $M$ 個節點上，token 可以對每個節點只跨 IB 一次，再經 NVLink 複製到該節點內的其他目標 GPU。這時 IB 成本只有

C_{\mathrm{IB}} = M\,t, \qquad M \le 8

So the IB load depends on $M$ , not on $k$ . DeepSeek-V3 keeps $M$ small on purpose. The 256 routed experts form 8 groups of 32, each group sits on one node, and the router picks a token's experts from at most 4 groups, so at most 4 nodes. This is node-limited routing. With the paper's hidden size of about 7K ( $H$ ), FP8 dispatch ( $b_d = 1$ byte) and BF16 combine ( $b_c = 2$ bytes), the IB bytes per token per MoE layer in the forward pass are

所以 IB 負載取決於 $M$ ，而不是 $k$ 。DeepSeek-V3 刻意讓 $M$ 保持很小。256 個 routed 專家分成 8 組、每組 32 個，每組放在一個節點，router 只從最多 4 組中為 token 選專家，也就是最多 4 個節點。這就是節點受限路由。以論文約 7K 的隱藏維度（ $H$ ）、FP8 dispatch（ $b_d = 1$ 位元組）與 BF16 combine（ $b_c = 2$ 位元組）計算，前向傳遞中每個 token 每個 MoE 層的 IB 位元組數是

B_{\mathrm{IB}} = M \cdot H \cdot (b_d + b_c)

The arithmetic, one question per row (our numbers on the paper's model, $H = 7{,}000$ ):

逐列計算如下（以論文的模型、 $H = 7{,}000$ 計算的我們的數字）：

Question問題	Calculation算式	Result結果
One copy of one token in FP8一個 token 的一份 FP8 副本	$7{,}000 \times 1$	$7$ KB
Dispatch, one IB copy per routed expertdispatch，每個 routed 專家一份 IB 副本	$8 \times 7$	$56$ KB
Dispatch, node-limited, $M = 4$ dispatch，節點受限， $M = 4$	$4 \times 7$	$28$ KB
Combine in BF16, $M = 4$ combine 用 BF16， $M = 4$	$4 \times 7{,}000 \times 2$	$56$ KB
Both, per token per MoE layer, forward pass兩者合計，每個 token 每個 MoE 層，前向傳遞	$28 + 56$	$84$ KBhalf of 168 KB with one copy per expert是每個專家一份副本時 168 KB 的一半

The paper neither measures IB bytes nor ablates node-limited routing. The price is SM work: the receiving GPU runs the NVLink copies, the combine reduces in SMs before its single IB send, and one kernel must balance two bandwidths (§4.4.1). The figure below shows the two cases.

論文既沒有量測 IB 位元組，也沒有做節點受限路由的消融實驗。代價是 SM 的工作：接收端 GPU 執行 NVLink 複製，combine 在唯一一次 IB 傳送前用 SM 做歸約，而且同一個 kernel 必須平衡兩種頻寬（§4.4.1）。下圖畫出兩種情況。

The §4.3 cost model, in which a token on GPU 2 picks 8 experts on nodes A to D. (a) Each expert gets its own IB copy. (b) Each node gets one copy on plane 2, at its GPU 2, which copies it over NVLink to the second target.§4.3 的成本模型，GPU 2 上的一個 token 選了節點 A 到 D 上的 8 個專家。(a) 每個專家各收一份 IB 副本。(b) 每個節點只在平面 2 上、由它的 GPU 2 收一份，再經 NVLink 複製給第二個目標。Drawn for this digest from §4.3 and the Figure 3 caption of the paper.

On the multi-plane network below, forwarding has a second job. Planes share no switch, so a GPU on plane 2 reaches another node only at that node's GPU 2: send on your own plane, then copy over NVLink. Panel 1 computes the model for other $M$ , precisions and loads.

在下文的多平面網路上，轉送還有第二個任務。各平面不共用交換器，所以平面 2 上的 GPU 只能抵達另一個節點的 GPU 2：在自己的平面送出，再經 NVLink 複製。面板 1 讓你用其他的 $M$ 、精度與負載計算這個模型。

Panel 1面板 1

How many IB bytes does one MoE layer send?一個 MoE 層送出多少 IB 位元組？

Nodes a token's 8 experts span,

M

token 的 8 個專家跨越的節點數

M

Delivery inside the node節點內的遞送方式

Dispatch precision

b_d

dispatch 精度

b_d

Combine precision

b_c

combine 精度

b_c

Tokens per GPU per micro-batch每顆 GPU 每個 micro-batch 的 token 數

IB bandwidth per NIC每張 NIC 的 IB 頻寬

IB copies / NVLink copies per token

每個 token 的 IB／NVLink 副本

IB bytes per token,

B_{\mathrm{IB}}

每個 token 的 IB 位元組

B_{\mathrm{IB}}

IB time per GPU, one layer

每顆 GPU 一層的 IB 時間

NVLink time per GPU, one layer

每顆 GPU 一層的 NVLink 時間

Model of §4.3, forward pass of one MoE layer: IB copies per token are $M$ with forwarding and 8 without; NVLink copies are $8 - M$ (each expert on its own GPU, all $M$ nodes remote). $B_{\mathrm{IB}} = \text{copies} \times H \times (b_d + b_c)$ with $H = 7{,}000$ ; IB time is tokens $\times B_{\mathrm{IB}}$ divided by the NIC rate, NVLink time uses 160 GB/s. Defaults: the paper's limit of 4 nodes, FP8 dispatch, BF16 combine, the 4,096 tokens per GPU of Figure 7, and 40 GB/s. These are modelled times, not measurements; the paper overlaps them with compute through DualPipe.

§4.3 的模型，一個 MoE 層的前向傳遞：有轉送時每個 token 的 IB 副本是 $M$ 份，沒有轉送時是 8 份；NVLink 副本是 $8 - M$ 份（每個專家在不同 GPU、 $M$ 個節點都是遠端）。 $B_{\mathrm{IB}} = \text{副本數} \times H \times (b_d + b_c)$ ， $H = 7{,}000$ ；IB 時間是 token 數 $\times B_{\mathrm{IB}}$ 除以 NIC 速率，NVLink 時間用 160 GB/s。預設值：論文的 4 節點上限、FP8 dispatch、BF16 combine、Figure 7 的每顆 GPU 4,096 個 token，以及 40 GB/s。這些是模型推算的時間，不是量測；論文用 DualPipe 讓它們與計算重疊。

3. Fewer bits on the wire3. 線路上更少的位元

Dispatch sends activations in fine-grained FP8, with one scale per 1×128 tile, which halves its bytes against BF16 (§3.2). Combine stays in BF16 for accuracy. The team also built LogFMT-nBit, a logarithmic format per tile. On a dense model of about 7B parameters, 8-bit LogFMT trained better than FP8 E4M3 or E5M2, and 10-bit LogFMT matched a BF16 combine. It was not deployed: encoding and decoding inside the all-to-all kernel added about 50–100% overhead, because GPUs have little log and exp throughput. FP8 training ablations on 16B and 230B DeepSeek-V2 models lost less than 0.25% relative accuracy against BF16 (§2.4).

dispatch 以細粒度 FP8 傳送激活值，每個 1×128 tile 一個縮放係數，位元組比 BF16 少一半（§3.2）。combine 為了準確度維持 BF16。團隊也做了 LogFMT-nBit，一種以 tile 為單位的對數格式。在約 7B 參數的稠密模型上，8 位元 LogFMT 的訓練結果優於 FP8 的 E4M3 或 E5M2，10 位元 LogFMT 與 BF16 combine 相當。它沒有被部署：在 all-to-all kernel 內編解碼會增加約 50–100% 的開銷，因為 GPU 的 log 與 exp 吞吐量很低。在 16B 與 230B 的 DeepSeek-V2 模型上，FP8 訓練的消融實驗相對 BF16 的準確度損失低於 0.25%（§2.4）。

4. An eight-plane, two-tier fat-tree4. 八平面、兩層的 fat-tree

Each node has eight GPUs and eight ConnectX-7 (CX7) 400G IB NICs. GPU-NIC pair $i$ attaches to plane $i$ , and each plane is a separate two-tier fat-tree of 64-port 400G IB switches (§5.1). A ninth NIC per node, 400G RoCE, reaches the 3FS storage network. Planes share no switch, so traffic to another plane crosses its node over NVLink or PCIe and leaves by another NIC: NCCL's PXN does this for collectives, node-limited forwarding for EP. The paper calls this a multi-plane fat-tree (MPFT).

每個節點有八顆 GPU 與八張 ConnectX-7（CX7）400G IB NIC。第 $i$ 對 GPU-NIC 接到平面 $i$ ，每個平面是一個獨立、由 64 埠 400G IB 交換器組成的兩層 fat-tree（§5.1）。每個節點的第九張 NIC 是 400G RoCE，接到 3FS 儲存網路。各平面不共用交換器，所以送往別的平面的流量要在節點內經 NVLink 或 PCIe，再從另一張 NIC 出去：集體通訊由 NCCL 的 PXN 處理，EP 由節點受限的轉送處理。論文稱之為多平面 fat-tree（MPFT）。

Figure 3 of the DeepSeek-V3 paper: eight stacked network planes, each a two-tier fat-tree of spine and leaf switches; below them, nodes whose GPU and NIC columns 0 to 7 attach to the planes, joined by NVLink inside each node — Figure 3 of the paper (design drawing): eight planes, each a two-tier fat-tree. GPU $i$ and NIC $i$ of every node attach to plane $i$ ; NIC 8 is the storage NIC.論文的 Figure 3（設計圖）：八個平面，每個都是兩層 fat-tree。每個節點的 GPU $i$ 與 NIC $i$ 接到平面 $i$ ；NIC 8 是儲存 NIC。Screenshot of Zhao et al., ISCA 2025, Figure 3 (arXiv v2).

Two tiers of 64-port switches serve 2,048 endpoints per plane, so eight planes serve 16,384 GPUs (Table 3 of the paper). Policy and regulatory limits held the deployment to just over 2,000 GPUs. Our check of the table's numbers:

兩層 64 埠交換器每個平面可接 2,048 個端點，所以八個平面可接 16,384 顆 GPU（論文 Table 3）。政策與法規限制讓實際部署只有略多於 2,000 顆 GPU。我們對表中數字的驗算：

Question問題	Calculation算式	Result結果
Endpoints per plane: 64 leaves, 32 ports down each每個平面的端點：64 台 leaf、每台 32 埠向下	$64 \times 32$	$2{,}048$
Switches per plane: leaves plus 32 spines每個平面的交換器：leaf 加 32 台 spine	$64 + 32$	$96$
Eight planes八個平面	$8 \times 2{,}048$ ; $8 \times 96$	$16{,}384$ GPUs顆 GPU; $768$ switches台交換器

The baseline in the paper's tests is a single-plane multi-rail fat-tree (MRFT): rails of the same index still get their own leaves, but all rails meet in shared spines. MPFT removes those shared spines (figure below). Since MPFT is a subset of MRFT, NVIDIA's and NCCL's multi-rail optimizations still apply.

論文測試中的基準線是單平面的多 rail fat-tree（MRFT）：相同編號的 rail 仍有自己的 leaf，但所有 rail 在共用的 spine 交會。MPFT 拿掉了這些共用 spine（見下圖）。因為 MPFT 是 MRFT 的子集，NVIDIA 與 NCCL 的 multi-rail 最佳化仍然適用。

The two tested topologies, drawn with 4 of 8 rails and two nodes A and B. In the upper one all leaves meet in shared spines (S), while in the lower one each plane has its own spines, so GPU 0 of node A reaches plane 3 only through NVLink (teal) and GPU 3's NIC.兩種受測拓樸，只畫 8 條 rail 中的 4 條與 A、B 兩個節點。上方的網路中所有 leaf 在共用 spine（S）交會，下方的網路則讓每個平面有自己的 spine，所以節點 A 的 GPU 0 只能經 NVLink（青綠）與 GPU 3 的 NIC 抵達平面 3。Drawn for this digest from §5.1, Table 3 and Figures 3–4 of the paper.

Table 3 prices the options with the Slim Fly paper's cost model (estimates): 4.39 k$ per endpoint for MPFT and a two-tier fat-tree, 7.5 k$ for three tiers, 4.4 k$ for Slim Fly and 5.8 k$ for Dragonfly. The paper also claims isolation between planes and lower latency than three tiers. Its ideal design (Figure 4 of the paper) gives each NIC one port per plane, with one queue pair spraying packets over all ports. This needs out-of-order placement, which CX7 lacks; ConnectX-8 supports four planes.

Table 3 用 Slim Fly 論文的成本模型為各方案估價（估計值）：MPFT 與兩層 fat-tree 每個端點 4.39 k$，三層 7.5 k$，Slim Fly 4.4 k$，Dragonfly 5.8 k$。論文也主張平面之間的隔離，以及比三層更低的延遲。它的理想設計（論文 Figure 4）讓每張 NIC 每個平面一個埠，由一個 queue pair 把封包噴灑到所有埠。這需要亂序放置，CX7 不支援；ConnectX-8 支援四個平面。

Implementation實作

The paper lists software, not code size. DeepEP implements dispatch and combine with node-limited forwarding and uses IBGDA (InfiniBand GPUDirect Async): the GPU writes work requests and NIC doorbells itself, with no CPU proxy thread (§5.2.3). DeepGEMM provides the FP8 matrix kernels, NCCL with PXN the cross-plane collectives, and 3FS the storage. Online inference runs the EP all-to-all through NIC RDMA to keep SMs free; training spends SMs on forwarding.

論文列出軟體元件，而不是程式碼規模。DeepEP 以節點受限的轉送實作 dispatch 與 combine，並使用 IBGDA（InfiniBand GPUDirect Async）：GPU 自己寫 work request 與 NIC doorbell，不需要 CPU 代理執行緒（§5.2.3）。DeepGEMM 提供 FP8 矩陣 kernel，NCCL 搭配 PXN 處理跨平面集體通訊，3FS 負責儲存。線上推論透過 NIC RDMA 執行 EP all-to-all，讓 SM 保持空閒；訓練則花 SM 在轉送上。

What was measured量到了什麼

To compare topologies, the team re-wired its cluster between MPFT and a single-plane MRFT. Kernel tests ran on 16–128 GPUs; the training comparison used the full 2,048-GPU job. NCCL all-to-all on 32–128 GPUs (Figure 5 of the paper) was almost equal on both: our chart readings give about 52–59 GB/s on 32 GPUs, 46–51 on 64 and 38–48 on 128, for 128 MiB to 16 GiB messages. The paper credits PXN. On 16 GPUs the latency difference was negligible (Figure 6). DeepEP's kernels (figure below) moved more than 40 GB/s per GPU.

為了比較拓樸，團隊把叢集在 MPFT 與單平面 MRFT 之間重新接線。kernel 測試在 16–128 顆 GPU 上進行；訓練比較使用完整的 2,048 顆 GPU 任務。32–128 顆 GPU 的 NCCL all-to-all（論文 Figure 5）在兩者上幾乎相同：我們從圖上讀出，訊息 128 MiB 到 16 GiB 時，32 顆 GPU 約 52–59 GB/s、64 顆約 46–51、128 顆約 38–48。論文歸功於 PXN。在 16 顆 GPU 上，延遲差異可以忽略（Figure 6）。DeepEP 的 kernel（見下圖）每顆 GPU 傳輸超過 40 GB/s。

Figure 7 of the DeepSeek-V3 paper: line chart of algorithm bandwidth in GB/s against GPU count 16, 32, 64 and 128 for dispatch and combine, with values between 41.6 and 58.02 — Figure 7 of the paper, measured on the production multi-plane network: DeepEP dispatch (blue) and combine (orange) algorithm bandwidth, 4,096 tokens per GPU, 16–128 GPUs.論文的 Figure 7，在生產環境的多平面網路上實測：DeepEP dispatch（藍）與 combine（橘）的演算法頻寬，每顆 GPU 4,096 個 token，16–128 顆 GPU。Screenshot of Zhao et al., ISCA 2025, Figure 7 (arXiv v2).

The paper says this nearly saturates the 400 Gb/s NIC. On 32 GPUs the value (58.02 GB/s) exceeds the NIC's 50 GB/s; we read this as algorithm bandwidth also counting bytes that stay on NVLink. Table 4 compares the full training job on the two networks:

論文表示這幾乎用滿了 400 Gb/s 的 NIC。在 32 顆 GPU 上，數值（58.02 GB/s）超過 NIC 的 50 GB/s；我們的解讀是演算法頻寬也計入留在 NVLink 上的位元組。Table 4 比較完整訓練任務在兩種網路上的表現：

Metric (2,048 GPUs, DeepSeek-V3)指標（2,048 顆 GPU，DeepSeek-V3）	MPFT	MRFT
Tokens per day (billions)每天的 token 數（billion）	272.80	272.52
Time per step每步時間	19.926 s	19.946 s
1F1B phase / pipeline bubble1F1B 階段／管線氣泡	13.95 / 2.06 s	14.00 / 2.03 s
TFLOPS per GPU, non-causal / causal每顆 GPU 的 TFLOPS，非因果／因果	432 / 385	432 / 385
MFU vs BF16 peak, non-causal / causal相對 BF16 峰值的 MFU，非因果／因果	43.73% / 38.94%	43.68% / 38.90%

The authors read the differences as noise: MPFT gives up no speed, so its case rests on cost and isolation. The table implies about 62.9 million tokens per step, about 30,700 per GPU (our arithmetic). Table 5 gives CPU-side latency for 64-byte messages:

作者認為這些差異只是雜訊：MPFT 沒有犧牲速度，所以它的理由在於成本與隔離。由表可推得每步約 62.9 million 個 token，每顆 GPU 約 30,700 個（我們的計算）。Table 5 給出 64 位元組訊息的 CPU 端延遲：

Link鏈路	Same leaf同一台 leaf	Across leaves跨 leaf
InfiniBand	2.8 µs	3.7 µs
RoCE	3.6 µs	5.6 µs
NVLink (inside the node)NVLink（節點內）	3.33 µs	–

IB stays the choice for latency-sensitive EP, at a higher price and with 64-port switches against RoCE's common 128. For RoCE, the paper compared routing policies (figure below; network size not stated).

對延遲敏感的 EP 仍選 IB，代價是價格較高，而且交換器是 64 埠，RoCE 常見的是 128 埠。對 RoCE，論文比較了路由策略（見下圖；網路規模未說明）。

Figure 8 of the DeepSeek-V3 paper: two grouped bar charts of reduce-scatter and all-gather bandwidth in GB/s for TP 8, 4 and 2; blue ECMP bars are lowest, orange adaptive-routing and green static-routing bars are equal and reach about 190 at TP 2 — Figure 8 of the paper, measured on a RoCE network: reduce-scatter (left) and all-gather (right) bandwidth for TP groups of 8, 4 and 2 under ECMP (blue), adaptive (orange) and static routing (green). At TP 2, ECMP reaches about half (chart reading: about 97 against 190 GB/s).論文的 Figure 8，在 RoCE 網路上實測：TP 群組為 8、4、2 時，ECMP（藍）、adaptive（橘）與靜態路由（綠）下 reduce-scatter（左）與 all-gather（右）的頻寬。TP 2 時 ECMP 約只有一半（圖上讀數：約 97 對 190 GB/s）。Screenshot of Zhao et al., ISCA 2025, Figure 8 (arXiv v2).

The paper's explanation: training flows have little randomness, so ECMP's hashes stack them on the same links. Static routing avoids conflicts for known destinations but is inflexible, so the authors prefer adaptive routing.

論文的解釋是：訓練流量的隨機性很低，所以 ECMP 的雜湊把它們疊在同一條鏈路上。靜態路由對已知目的地可避免衝突，但缺乏彈性，所以作者偏好 adaptive routing。

Limits and open questions限制與未解問題

Few ablations. No experiment isolates node-limited routing, forwarding or FP8 dispatch. MPFT and MRFT train at the same speed, so MPFT's benefit is cost and isolation, and the cost comes from a model.消融實驗很少。沒有實驗單獨檢驗節點受限路由、轉送或 FP8 dispatch。MPFT 與 MRFT 的訓練速度相同，所以 MPFT 的好處是成本與隔離，而成本來自模型。
Missing layout facts. The paper gives no PP, EP or data-parallel sizes, no share of step time spent communicating (DualPipe hides it in the 13.95 s 1F1B phase), and nothing on expert load balance or remaining skew.缺少配置資訊。論文沒有 PP、EP 與資料平行的大小，沒有通訊占每步時間的比例（DualPipe 把它藏在 13.95 s 的 1F1B 階段內），也沒有談專家負載平衡或剩下的偏斜。
Small kernel tests. Figures 5–7 use 16–128 GPUs; the setups of Figure 8 and Table 5 are not described; NVLink is 400 GB/s in §4.1 and 200 GB/s in §4.3.kernel 測試規模小。Figure 5–7 只用 16–128 顆 GPU；Figure 8 與 Table 5 的設定沒有描述；NVLink 在 §4.1 是 400 GB/s，在 §4.3 是 200 GB/s。
SMs spent on the network. Up to 20 SMs per GPU do forwarding, copies, the combine reduce, layouts and casts: compute the model does not get.花在網路上的 SM。每顆 GPU 最多 20 個 SM 在做轉送、複製、combine 歸約、記憶體配置與型別轉換：這些是模型拿不到的算力。

What the authors ask of the next network作者對下一代網路的要求

Converge scale-up and scale-out (§4.4.2): one NIC or I/O die on both networks; a communication co-processor for forwarding, copies and reduce; hardware broadcast for dispatch and reduce for combine; acquire/release ordering, such as their Region Acquire/Release (§6.4).整合 scale-up 與 scale-out（§4.4.2）：同時接兩種網路的 NIC 或 I/O die；負責轉送、複製與歸約的通訊協同處理器；dispatch 的硬體廣播與 combine 的硬體歸約；acquire/release 排序，例如他們提出的 Region Acquire/Release（§6.4）。
Multi-plane NICs with out-of-order placement, so one queue pair uses all planes (§5.1).多平面 NIC，支援亂序放置，讓一個 queue pair 使用所有平面（§5.1）。
Better RoCE (§5.2.2, §6.3): low-latency switches, adaptive routing instead of ECMP, a virtual output queue per queue pair, and RTT-based or programmable congestion control.更好的 RoCE（§5.2.2、§6.3）：低延遲交換器、以 adaptive routing 取代 ECMP、每個 queue pair 一個虛擬輸出佇列，以及以 RTT 為基礎或可程式化的壅塞控制。
Priorities and in-network help (§4.5.2, §6.5): priorities for EP, TP and KV-cache traffic on NVLink and PCIe; in-network multicast for dispatch and reduction for combine; hardware compression.優先權與網內協助（§4.5.2、§6.5）：在 NVLink 與 PCIe 上為 EP、TP 與 KV cache 流量設優先權；dispatch 的網內多播與 combine 的網內歸約；硬體壓縮。

How it connects與其他內容的關聯

Section 3.7 derives dispatch and combine bytes; this paper cuts them with a routing limit and FP8. Section 9.4 of the Chapter 4 covers all-to-all skew, which this paper does not measure. Balancing and Beyond also sends one copy per host and forwards locally. UBEP rebuilds the EP library for a superpod. Odin attacks the RoCE incast that §5.2.2 names. MixNet changes the topology under the all-to-all; Theseus swaps collective schedules at run time. Rail-only also forwards cross-rail traffic over NVLink, Alibaba HPN also uses planes and two tiers, and MegaScale overlaps communication in production training.

第 3.7 節推導 dispatch 與 combine 的位元組；這篇論文用路由上限與 FP8 削減它們。第 4 章第 9.4 節討論 all-to-all 的偏斜，這篇論文沒有量測它。Balancing and Beyond 也是每台主機只送一份、再在本地轉送。UBEP 為 superpod 重新打造 EP 函式庫。Odin 處理 §5.2.2 提到的 RoCE incast。MixNet 改變 all-to-all 底下的拓樸；Theseus 在執行期替換集體通訊排程。Rail-only 同樣經 NVLink 轉送跨 rail 流量，Alibaba HPN 同樣使用多平面與兩層架構，MegaScale 在生產訓練中讓通訊重疊。

Key numbers關鍵數字

Number數字	What, and where內容與出處
671B / 37B	Total / active parameters per token; 256 routed + 1 shared expert, top-8 (§2.2.1, §4.3)總參數／每個 token 的啟用參數；256 個 routed 加 1 個共享專家，top-8（§2.2.1、§4.3）
2,048	H800 GPUs in the training job; 8 per node, 8 × 400G CX7 IB NICs per node訓練任務的 H800 數；每節點 8 顆、8 張 400G CX7 IB NIC
≈ 160 : 40 GB/s	Achievable NVLink per GPU against effective IB per NIC, stated (§4.3)每顆 GPU 可用的 NVLink 對每張 NIC 的實際 IB，論文所述（§4.3）
$\le 4$ nodes	Node-limited routing: 8 groups of 32 experts, one group per node (§4.3)節點受限路由：8 組、每組 32 個專家，每個節點一組（§4.3）
$\le 20$ SMs	SMs per H800 that run communication during training (§4.4.1)訓練期間每顆 H800 用於通訊的 SM（§4.4.1）
272.80 / 272.52 B	Tokens per day, MPFT / MRFT, full 2,048-GPU job; MFU 43.73% / 43.68% (Table 4)每天 token 數，MPFT／MRFT，完整 2,048 顆 GPU 任務；MFU 43.73%／43.68%（Table 4）
42–58 GB/s	DeepEP dispatch and combine per GPU, 4,096 tokens per GPU, 16–128 GPUs (Figure 7)DeepEP dispatch 與 combine 每顆 GPU，每顆 GPU 4,096 個 token，16–128 顆 GPU（Figure 7）
2.8 / 3.6 µs	64-byte latency, same leaf, IB / RoCE (Table 5)64 位元組延遲，同一台 leaf，IB／RoCE（Table 5）
≈ 2×	Adaptive or static routing over ECMP, RoCE reduce-scatter at TP 2 (Figure 8, chart reading)adaptive 或靜態路由相對 ECMP，RoCE reduce-scatter，TP 2（Figure 8，圖上讀數）

Terms used on this page本頁用到的術語

Expert / router / top- $k$: A feed-forward block of an MoE layer; the gate that scores experts for a token; the number of routed experts it picks (8 here).; MoE 層中的一個前饋區塊；為 token 替專家打分的 gate；它選出的 routed 專家數（此處為 8）。
Dispatch / combine: The all-to-all that sends tokens to experts, and the one that brings the weighted outputs back.; 把 token 送到專家的 all-to-all，以及把加權輸出送回的 all-to-all。
Node-limited routing: A router rule that picks a token's experts from at most 4 nodes.; 讓 token 的專家最多只來自 4 個節點的 router 規則。
NVLink forwarding: Sending a token once per node over IB and copying it to the other target GPUs over NVLink.; 對每個節點只經 IB 送一次 token，再經 NVLink 複製給其他目標 GPU。
MPFT / MRFT: Multi-plane fat-tree: one separate fabric per NIC index. Multi-rail fat-tree: rails with their own leaves but shared spines.; 多平面 fat-tree：每個 NIC 編號一個獨立網路。多 rail fat-tree：rail 有自己的 leaf，但共用 spine。
PXN: NCCL's mechanism that moves data over NVLink to a GPU whose NIC is on the destination's rail or plane.; NCCL 的機制：經 NVLink 把資料移到某顆 GPU，該 GPU 的 NIC 在目的地的 rail 或平面上。
DeepEP / DualPipe: DeepSeek's open-source EP all-to-all library; its pipeline schedule that overlaps MoE communication with compute.; DeepSeek 開源的 EP all-to-all 函式庫；讓 MoE 通訊與計算重疊的管線排程。
IBGDA: InfiniBand GPUDirect Async: the GPU writes work requests and NIC doorbells without a CPU proxy.; InfiniBand GPUDirect Async：GPU 不經 CPU 代理，自己寫 work request 與 NIC doorbell。
LogFMT: The team's logarithmic n-bit format for activations on the wire; tested, not deployed.; 團隊為線路上的激活值設計的 n 位元對數格式；測試過，未部署。
Algorithm bandwidth: Bytes a collective moves per GPU divided by its time, counting intra-node bytes too.; 集體操作每顆 GPU 搬移的位元組除以時間，也計入節點內的位元組。