Research digest 1 of 5 · Chapter 6: Topology

研究導讀 1 / 5 · 第 6 章：拓樸

From ATOP to ZCube: Automated Topology Optimization Pipeline and A Highly Cost-Effective Network Topology for Large Model Training從 ATOP 到 ZCube：自動化拓樸最佳化流程，與一個極具成本效益的大模型訓練網路拓樸

SIGCOMM 2025Zihan Yan, Dan Li, Li Chen, et al. (Tsinghua University, Zhongguancun Laboratory, ByteDance, Harnets.AI) · paper論文

After reading, you can explain how a topology search found ZCube, why two half-speed NIC ports put every GPU pair within two switches, and what a 16-GPU testbed measured.讀完後，你能說明拓樸搜尋如何找到 ZCube、為何兩個半速 NIC 埠讓每對 GPU 最多相隔兩台交換器，以及 16 顆 GPU 的測試平台量到了什麼。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 6 ↗第 6 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

A rail-optimized fat tree (ROFT) needs three switch tiers past a few thousand GPUs, so a pair can be five switches apart. ZCube splits each GPU's 400G NIC into two 200G ports: one to a switch shared by the GPU's group, one to a switch shared by the same-index GPUs of every group, and the two switch layers are fully meshed. Every pair is then at most two switches apart, and 51.2T chips hold 16,384 GPUs with 256 switches instead of 640. ATOP, a multi-objective topology search, kept finding this shape at the knee of the cost–performance frontier. Measured: on 16 H800 GPUs, ZCube(4,2) matched ROFT's NCCL all-reduce and all-to-all bus bandwidth with 48 × 200G links instead of 32 × 400G (25% cheaper in the authors' cost model), and one month of production monitoring found 35% of rail-aligned traffic crossing ROFT spines. The training speedups are simulated, so this page teaches the wiring and its arithmetic.

Rail-optimized fat tree（ROFT）在超過幾千顆 GPU 後需要三層交換器，一對 GPU 可能相隔五台交換器。ZCube 把每顆 GPU 的 400G NIC 拆成兩個 200G 埠：一個接到該 GPU 所屬群組共用的交換器，一個接到各群組中同編號 GPU 共用的交換器，而兩層交換器彼此全連接。於是每一對 GPU 最多相隔兩台交換器，51.2T 晶片用 256 台交換器（而非 640 台）就能容納 16,384 顆 GPU。ATOP 是一個多目標的拓樸搜尋系統，它一再在成本–效能前緣的膝點找到這個形狀。實測：在 16 顆 H800 上，ZCube(4,2) 以 48 × 200G 鏈路（而非 32 × 400G）達到與 ROFT 相同的 NCCL all-reduce 與 all-to-all 匯流排頻寬（依作者的成本模型便宜 25%）；一個月的生產監測發現 35% 的 rail 對齊流量經過 ROFT 的 spine。訓練加速是模擬的，所以本頁教的是接線方式與它的算術。

1. Where this paper sits1. 這篇論文的位置

The Chapter 6, section 1, counts a Clos from the radix $k$ : two tiers hold $k^2/2$ GPUs, three tiers $k^3/4$ , and the longest path grows from 3 to 5 switches. Section 3 explains rail-optimized wiring and PXN. Section 4 shows HPN giving each NIC two 200G ports, both into the same tier, for redundancy. Rail-only (Panel 3 there) deletes the spine. People picked all of these from known shapes. This paper asks whether a search finds better shapes, and what happens when the two NIC ports go to two different switch layers.

第 6.1 節從基數 $k$ 數出 Clos：兩層容納 $k^2/2$ 顆 GPU、三層 $k^3/4$ ，最長路徑從 3 台交換器增加到 5 台。第 3 節解釋 rail-optimized 接線與 PXN。第 4 節說明 HPN 讓每張 NIC 有兩個 200G 埠，兩個都接到同一層，用途是備援。Rail-only（該頁的面板 3）刪掉 spine。這些設計都是人從已知形狀中挑出來的。這篇論文問：搜尋能否找到更好的形狀？兩個 NIC 埠接到兩個不同的交換器層時，會發生什麼？

2. The problem, measured2. 問題，用量測說話

The authors studied training traces from a 4K-GPU production cluster. A run moves through five traffic periods: DP only, PP only, EP only, DP with PP, and DP with EP. Cross-server traffic is rail-aligned: it typically runs between GPUs with the same index, and all-to-all does too because PXN first forwards over NVLink to the same-index GPU. A ROFT gives such a pair one switch only when both servers sit under one rail leaf, but pipeline stages and large DP groups span pods. During one month of monitoring a production ROFT cluster, 35% of rail-aligned traffic traversed spine switches.

作者研究了一個 4K 顆 GPU 生產叢集的訓練紀錄。一次訓練會經過五種流量時期：只有 DP、只有 PP、只有 EP、DP 加 PP、DP 加 EP。跨伺服器的流量是 rail 對齊的：它通常在編號相同的 GPU 之間流動；all-to-all 也是如此，因為 PXN 會先經 NVLink 轉送到同編號的 GPU。ROFT 只有在兩台伺服器位於同一台 rail leaf 下時才讓這樣的配對只經一台交換器，但管線階段與大型 DP 群組會橫跨 pod。在一個月的生產 ROFT 叢集監測中，35% 的 rail 對齊流量經過了 spine 交換器。

Rail

r

in a three-tier ROFT: rail-aligned GPUs under one leaf need 1 switch (A), in another rail group of the pod 3 (B), in another pod 5 (C). The bar is the measured production share: 35% of rail-aligned traffic took paths like B or C.三層 ROFT 中的 rail

r

：同一台 leaf 下 rail 對齊的 GPU 需 1 台交換器（A），在同一 pod 的另一個 rail 群組需 3 台（B），在另一個 pod 需 5 台（C）。長條是實測的生產比例：35% 的 rail 對齊流量走了 B 或 C 這類路徑。Drawn for this primer from §2 of Yan et al., SIGCOMM 2025 (the 35% is measured; the 65% is its complement).

Hashing at every tier still collides under EP all-to-all (shown in simulation), and topologies are redesigned at construction, re-cabling and expansion, where manual design favors symmetric shapes.

每一層的雜湊在 EP 的 all-to-all 下仍會碰撞（以模擬顯示），而拓樸會在新建、重新佈線與擴充時重新設計，人工設計則偏好對稱形狀。

3. The design3. 設計

3.1 ATOP: search a space built from expert intuition3.1 ATOP：在由專家直覺構成的空間中搜尋

Searching adjacency matrices is hopeless: $N$ nodes give $2^{N^2}$ candidates. ATOP describes a topology with 11 types of hyperparameters that encode how experts already think. A topology has layers. Between two layers, block $k$ of layer $i$ connects to block $(k \bmod H_{j}) + 1$ of layer $j$ , and each node opens $E_{ij}$ links by a fixed offset rule. Inside a layer, nodes have coordinates $(X_1,\dots,X_D)$ , and the $m$ -th link of a node in dimension $k$ goes to

搜尋鄰接矩陣是沒有希望的： $N$ 個節點有 $2^{N^2}$ 個候選。ATOP 用 11 種超參數描述拓樸，把專家已有的思考方式寫成參數。拓樸有分層。兩層之間，第 $i$ 層的第 $k$ 個區塊連到第 $j$ 層的第 $(k \bmod H_{j}) + 1$ 個區塊，每個節點以固定的位移規則開出 $E_{ij}$ 條鏈路。層內的節點有座標 $(X_1,\dots,X_D)$ ，節點在第 $k$ 維的第 $m$ 條鏈路連到

X'_t = \Big[\,A_{0t}\,m + \sum_{r=1}^{D} A_{rt}\,X_r + C_t\Big] \bmod S_k

Here $A$ and $C$ are searched coefficients and $S_k$ is the size of dimension $k$ (we drop the superscripts of the paper's Eq. 1). The space contains Clos, fat tree, ROFT, rail-only, HPN, BCube, DCell, HyperX, torus and Dragonfly, plus asymmetric shapes a person would skip.

其中 $A$ 與 $C$ 是被搜尋的係數， $S_k$ 是第 $k$ 維的大小（我們省略了論文 Eq. 1 的上標）。空間涵蓋 Clos、fat tree、ROFT、rail-only、HPN、BCube、DCell、HyperX、環面與 Dragonfly，也涵蓋人會略過的不對稱形狀。

The optimizer, NSGA-II, is an evolutionary algorithm for many objectives. It scores each candidate on 14 objectives of 4 types: completion time of 11 traffic fragments (DP, PP and EP periods, from a flow-level simulator), the ForestColl bound for a 1 GB all-gather, the average path length after one switch fails, and a priced bill of materials. A candidate is Pareto-optimal when no other is as good on every objective and better on one. Stage 1 scores fragments; stage 2 simulates full training only for the Pareto set, which cuts full-scale simulations by 95%. The frontier's knee, where more cost buys little speed, is the most cost-effective design.

最佳化器 NSGA-II 是處理多目標的演化演算法。它以 4 類共 14 個目標為每個候選評分：11 個流量片段的完成時間（DP、PP 與 EP 時期，由流層級模擬器算出）、1 GB all-gather 的 ForestColl 下界、一台交換器故障後的平均路徑長，以及定價的物料清單。若沒有其他候選在每個目標上都一樣好、且某一項更好，該候選就是 Pareto 最佳。第一階段為片段評分；第二階段只對 Pareto 集合模擬完整訓練，使全規模模擬減少 95%。前緣的膝點（再加成本也買不到多少速度之處）就是最具成本效益的設計。

ATOP proposes designs, scores them cheaply, and simulates full training only for the Pareto set. The curve shows the shape of the cost–time frontier, while the bars give the measured run times (§4.1).ATOP 先提出設計、以低成本評分，只對 Pareto 集合模擬完整訓練。曲線呈現成本與時間前緣的形狀，長條則是實測執行時間（§4.1）。Drawn for this primer from §3–§4 of Yan et al., SIGCOMM 2025.

3.2 ZCube: what the search kept finding3.2 ZCube：搜尋一再找到的設計

At 256 to 16K GPUs the knee had a similar shape at every scale, and it was also the most cost-effective design for re-cabling a 4K cluster and for expanding 1K to 4K. The authors named it ZCube(n,k). ZCube(n,1) is one switch with $n$ GPUs. ZCube(n,k+1) takes $n$ copies of ZCube(n,k) and adds $n^k$ level- $k$ switches; each GPU has one port per level. The level- $k$ port of GPU $i$ in copy $j$ goes to port $j$ of level- $k$ switch $i$ , and level- $(k{-}1)$ switch $i$ of every copy connects to level- $k$ switches $in$ to $(i{+}1)n - 1$ (Figure 8 of the paper):

在 256 到 16K 顆 GPU 的每個規模，膝點的形狀都相似；在重新佈線一個 4K 叢集、以及從 1K 擴充到 4K 時，它也是最具成本效益的設計。作者把它命名為 ZCube(n,k)。ZCube(n,1) 是一台接 $n$ 顆 GPU 的交換器。ZCube(n,k+1) 取 $n$ 份 ZCube(n,k)，再加上 $n^k$ 台 level- $k$ 交換器；每顆 GPU 每一層各有一個埠。第 $j$ 份中 GPU $i$ 的 level- $k$ 埠接到 level- $k$ 交換器 $i$ 的第 $j$ 個埠，而每一份中的 level- $(k{-}1)$ 交換器 $i$ 連到 level- $k$ 交換器 $in$ 到 $(i{+}1)n - 1$ （論文 Figure 8）：

Figure 8 (a) and (b) of the ZCube paper: the recursive construction of ZCube(n,k+1) from n copies of ZCube(n,k) plus a new switch level, and the eight-GPU, three-level ZCube(2,3) — Figure 8 (a)–(b) of the paper, a design diagram. (a) $n$ copies of ZCube(n,k) under one new switch level. (b) ZCube(2,3): 8 GPUs with three ports each, three levels of four switches. Panel (c) is not shown.論文 Figure 8 的 (a)–(b)，設計示意圖。(a) $n$ 份 ZCube(n,k) 放在一層新的交換器之下。(b) ZCube(2,3)：8 顆各有三個埠的 GPU，三層各四台交換器。未顯示 (c)。Yan et al., SIGCOMM 2025, Figure 8 (a)–(b) (accepted PDF, ACM open access, re-rendered).

Counting from these rules, ZCube(n,k) has $N$ GPUs, $S$ switches, $L$ links and diameter $D$ (the most switches on any shortest GPU-to-GPU path):

依這些規則計數，ZCube(n,k) 有 $N$ 顆 GPU、 $S$ 台交換器、 $L$ 條鏈路，直徑為 $D$ （任一對 GPU 最短路徑上最多經過的交換器數）：

N = n^{k}, \qquad S = k\,n^{k-1}

L = (2k-1)\,n^{k}, \qquad D = k

The paper proves $D = k$ by induction. $L$ is our count, $k$ links per GPU plus $n^k$ between adjacent levels; it gives the paper's 48 for ZCube(4,2) and 49,152 for ZCube(128,2). Edge levels need $2n$ ports per switch and middle levels $3n$ : an asymmetric design that a fat-tree habit would not choose.

論文以歸納法證明 $D = k$ 。 $L$ 是我們數的：每顆 GPU $k$ 條鏈路，加上相鄰兩層之間 $n^k$ 條；它得到論文中 ZCube(4,2) 的 48 與 ZCube(128,2) 的 49,152。頭尾兩層的交換器需要 $2n$ 個埠，中間層需要 $3n$ 個：這是照 fat tree 習慣不會選的不對稱設計。

ZCube(n,2) reads best as a grid: groups in rows, indices in columns. Each row has a group switch (level 0), each column an index switch (level 1), and every group switch links to every index switch. A GPU in the same row or the same column is one switch away. Any other GPU is two switches away, by two disjoint paths: own group switch, then the destination's index switch; or own index switch, then the destination's group switch.

ZCube(n,2) 最好看成網格：群組排成列、編號排成行。每一列有一台群組交換器（level 0），每一行有一台編號交換器（level 1），每台群組交換器都連到每台編號交換器。同一列或同一行的 GPU 相隔一台交換器。其他任何 GPU 都相隔兩台，而且有兩條不相交的路徑：先到自己的群組交換器，再到目的地的編號交換器；或先到自己的編號交換器，再到目的地的群組交換器。

ZCube(3,2) with each GPU labelled (group, index), where grey lines are GPU ports and the G–I mesh appears only for the two links in use. From GPU 0,0 a GPU in the same row or column is one switch away, while any other GPU is two switches away by either of two disjoint paths, shown in red.ZCube(3,2) 中每顆 GPU 標為（群組, 編號），灰線是 GPU 的埠，G–I 之間的全連接只畫出用到的兩條。從 GPU 0,0 出發，同列或同行的 GPU 只隔一台交換器，其他 GPU 則隔兩台，可走紅色標出的兩條不相交路徑中的任一條。Drawn for this primer from §5.1 of Yan et al., SIGCOMM 2025.

3.3 Why it is cheaper: one chip, twice the reach3.3 為何更便宜：同一顆晶片，觸及範圍加倍

At 200G a 51.2T chip has 256 ports instead of 128, and every ZCube switch also carries GPUs, so none is a pure spine. At 16,384 GPUs, one question per row:

以 200G 計，51.2T 晶片有 256 個埠而不是 128 個，而且每台 ZCube 交換器都接 GPU，沒有一台是純 spine。16,384 顆 GPU 時，逐列計算如下：

Question問題	Calculation算式	Result結果
ROFT at 400G: radix 128; does a two-tier Clos fit?ROFT 以 400G 計：基數 128；兩層 Clos 放得下嗎？	$128^2 / 2$	$8{,}192$ GPUs: no, three tiers顆 GPU：不夠，要三層
ROFT switches: leaves, spines, coresROFT 交換器：leaf、spine、core	$256 + 256 + 128$	$640$
ROFT links: host, leaf–spine, spine–coreROFT 鏈路：主機、leaf–spine、spine–core	$3 \times 16{,}384$	$49{,}152$ at 400G條 400G
ZCube at 200G: $2n = 256$ ports, so $n = 128$ ZCube 以 200G 計： $2n = 256$ 個埠，所以 $n = 128$	$128^2$	$16{,}384$ GPUs顆 GPU
ZCube switches and linksZCube 交換器與鏈路	$2 \times 128;\ 3 \times 128^2$	$256;\ 49{,}152$ at 200G條 200G
Switches saved against ROFT (640) and rail-only or HPN (384)相對 ROFT（640）與 rail-only 或 HPN（384）省下的交換器	$1 - 256/640;\ 1 - 256/384$	$60\%;\ 33\%$

The main text states the same 33–60% fewer switches; Appendix K, which is not peer-reviewed, tabulates them. The diameter falls from 5 to 2. ZCube(42,4) on 128-port switches holds 3,111,696 GPUs at diameter 4, against 524,288 at diameter 5 for a three-tier fat tree (analysis). The authors' packet-level simulations at 1K–16K H100 GPUs report 3–7% faster training and 26–46% lower list-price network cost; this page does not show them.

正文寫的也是交換器少 33–60%；未經同儕審查的附錄 K 列出這些數量。直徑從 5 降到 2。128 埠交換器上的 ZCube(42,4) 在直徑 4 下容納 3,111,696 顆 GPU，三層 fat tree 則是直徑 5 下 524,288 顆（分析）。作者在 1K–16K 顆 H100 的封包層級模擬中報告訓練快 3–7%、以牌價計網路成本低 26–46%；本頁不展示這些數字。

Panel 1面板 1

Build ZCube(n,2) and a ROFT from the same switch chip用同一顆交換晶片建 ZCube(n,2) 與 ROFT

Size

n

(switch chip)大小

n

（交換晶片）

Destination group目的群組

Destination index目的編號

GPUs,

n^2

GPU 數，

n^2

ZCube: switches · links

ZCube：交換器 · 鏈路

ROFT: switches · links

ROFT：交換器 · 鏈路

Link capacity, ZCube / ROFT

鏈路總容量，ZCube / ROFT

ZCube path from GPU 0,0

ZCube 從 GPU 0,0 的路徑

Same pair in the ROFT

同一對在 ROFT 中

ZCube: GPUs 1 / 2 switches away

ZCube：相隔 1 / 2 台的 GPU

ROFT: GPU 0's rail partners behind a spine

ROFT：GPU 0 的同 rail 夥伴中需經 spine 者

Default: the paper's testbed, ZCube(4,2) on four servers using 4 GPUs each against the ROFT of its Figure 11 (8 switches, 32 × 400G). The chip settings compare ZCube(n,2) with $2n$ ports at 200G against a non-blocking three-tier ROFT of radix $n$ at 400G ( $5n$ switches, $3n^2$ links) on 8-GPU servers; at 51.2T this is the 256 vs 640 of the table above. The drawing shows at most four groups and four indices. ROFT hops follow Chapter 6's rail-optimized counting, with PXN for cross-rail pairs. The spine tile counts GPU pairs, not bytes: placement keeps most bytes local, and the measured traffic share in production was 35%.

預設：論文的測試平台，四台伺服器（各用 4 顆 GPU）上的 ZCube(4,2)，對照其 Figure 11 的 ROFT（8 台交換器、32 × 400G）。晶片選項比較以 200G 計有 $2n$ 個埠的 ZCube(n,2)，與以 400G 計基數 $n$ 的非阻塞三層 ROFT（ $5n$ 台交換器、 $3n^2$ 條鏈路），伺服器為 8 GPU；51.2T 時即上表的 256 對 640。圖中最多畫出四個群組與四個編號。ROFT 的跳數依第 6 章的 rail-optimized 計數，跨 rail 的配對用 PXN。spine 格計算的是 GPU 配對而非位元組：擺放讓大多數位元組留在本地，生產中實測的流量比例是 35%。

Stop and think停下來想一想

At 51.2T, set the destination to group 5, index 0, then index 1: the ZCube path grows from one switch to two, never three. Why can the ROFT be shorter for some pairs yet longer in the worst case?

在 51.2T 下，把目的設成群組 5、編號 0，再改成編號 1：ZCube 的路徑從一台交換器變成兩台，永遠不會到三台。為什麼 ROFT 對某些配對更短，最差情況卻更長？

Figure 11 of the ZCube paper: the testbed wiring of a rail-optimized fat tree with 400G links and of ZCube(4,2) with 200G links, each on four servers of four GPUs — Figure 11 of the paper, the testbed wiring. (a) ROFT: 400G NICs to rail leaves, 4 leaves meshed with 4 spines, 32 links. (b) ZCube(4,2): one 200G port to the server's group switch, one to an index switch, 48 links. The paper's L1 and L2 are our levels 0 and 1.論文 Figure 11，測試平台的接線。(a) ROFT：400G NIC 接 rail leaf，4 台 leaf 與 4 台 spine 全連接，32 條鏈路。(b) ZCube(4,2)：一個 200G 埠接伺服器的群組交換器，一個接編號交換器，48 條鏈路。論文的 L1、L2 即本頁的 level 0 與 1。Yan et al., SIGCOMM 2025, Figure 11 (accepted PDF, ACM open access, re-rendered).

5. What was measured5. 量到了什麼

The paper's main measured numbers come from three places: the production study of §2, the search run time, and the testbed. On the testbed, ZCube(4,2) and ROFT reach the same NCCL bus bandwidth for all-reduce and all-to-all at every size from 1 MB to 16 GB (Figure 12), with different bills:

論文主要的實測數字來自三處：§2 的生產研究、搜尋的執行時間，以及測試平台。在測試平台上，ZCube(4,2) 與 ROFT 從 1 MB 到 16 GB 的每個大小，all-reduce 與 all-to-all 的 NCCL 匯流排頻寬都相同（Figure 12），但物料清單不同：

Question問題	Calculation算式	Result結果
ROFT links: GPU–leaf plus leaf–spineROFT 鏈路：GPU–leaf 加 leaf–spine	$16 + 16$	$32 \times 400$ Gb/s12.8 Tb/s12.8 Tb/s
ZCube(4,2) links: two ports per GPU plus the 4 × 4 meshZCube(4,2) 鏈路：每顆 GPU 兩個埠，加 4 × 4 全連接	$2 \times 16 + 4 \times 4$	$48 \times 200$ Gb/s9.6 Tb/s9.6 Tb/s
Link capacity, ZCube against ROFT鏈路總容量，ZCube 對 ROFT	$9.6 \div 12.8$	$0.75$ 25% less; the cost model also gives 25%少 25%；成本模型也得到 25%

Both fabrics use 8 switches and give each GPU 400 Gb/s. The 25% cost saving is the authors' cost model applied to this bill, not a measured quantity.

兩個網路都用 8 台交換器，每顆 GPU 都有 400 Gb/s。25% 的成本節省是作者的成本模型套用到這份清單的結果，不是實測量。

Figure 12 of the ZCube paper: NCCL all-reduce and all-to-all bus bandwidth from 1 MB to 16 GB on ROFT and ZCube(4,2); the paired bars are equal at every size — Figure 12 of the paper, measured on the 16-GPU H800 testbed (static routes, NCCL 2.21.5, mean and standard deviation of 100 runs). At every size the paired bars are equal for both collectives.論文 Figure 12，於 16 顆 H800 的測試平台實測（靜態路由、NCCL 2.21.5、100 次的平均與標準差）。每個大小下，兩種集體通訊的成對長條都等高。Yan et al., SIGCOMM 2025, Figure 12 (accepted PDF, ACM open access, re-rendered).

The result shows that the two-port wiring loses no bandwidth for these collectives on 16 GPUs. It does not test PP, the source of the simulated speedup, nor ECMP or scale. The search time is a measured run time: 6.5, 10.6, 25.4 and 71.2 hours for 256, 1K, 4K and 16K GPUs on 256 cores, for up to $10^5$ topologies, which the authors compare with the months a data-center build takes.

這個結果顯示：在 16 顆 GPU 上，雙埠接線對這兩種集體通訊沒有損失頻寬。它沒有測 PP（模擬加速的來源），也沒有測 ECMP 或規模。搜尋時間是實測的執行時間：256、1K、4K 與 16K 顆 GPU 分別為 6.5、10.6、25.4 與 71.2 小時，使用 256 個核心，最多 $10^5$ 個拓樸；作者拿它與資料中心建置所需的數個月相比。

6. Limits and open questions6. 限制與未解的問題

Scale tested. Hardware evidence is 16 GPUs on static routes; all ZCube results at 1K GPUs or more are simulation, analysis or a cost model. Users chose ZCube only for small-scale deployment and testing.測試規模。硬體證據是 16 顆 GPU 加靜態路由；1K 顆 GPU 以上的 ZCube 結果都是模擬、分析或成本模型。使用者只選了 ZCube 做小規模部署與測試。
Hardware assumptions. A multi-port NIC or several NICs per GPU, 256-port switches at 200G, cheap 200G optics, and all NICs of a server on one group switch.硬體假設。每顆 GPU 一張多埠 NIC 或多張 NIC、以 200G 計 256 埠的交換器、便宜的 200G 光模組，且一台伺服器的 NIC 全接到同一台群組交換器。
Traffic assumptions. All evaluated traffic is rail-aligned with PXN on. MoE all-to-all is simulated only in this rail-aligned form, and shared hosts are untested (both are HPN's reasons to reject rail-only).流量假設。所有評估的流量都是 rail 對齊且開啟 PXN。MoE 的 all-to-all 只以這種 rail 對齊的形式模擬，共享主機則沒有測試（兩者都是 HPN 拒絕 rail-only 的理由）。
Routing differs by stage: ECMP in the search, packet spraying in the final simulation, static routes on the testbed. A pair has at most two minimal paths.各階段的路由不同：搜尋用 ECMP，最終模擬用封包噴灑，測試平台用靜態路由。一對 GPU 最多只有兩條最短路徑。
Cost and review. Prices are list prices (FS.com, Colfax). The appendix (fault tolerance, simulator validation, multi-tenant and heterogeneous cases, the 16K bill) is not peer-reviewed.成本與審查。價格是牌價（FS.com、Colfax）。附錄（容錯、模擬器驗證、多租戶與異質情境、16K 的清單）未經同儕審查。

7. How it connects7. 與其他內容的關聯

Chapter 6: section 1 (Panel 2) gives the $k^2/2$ and $k^3/4$ counts that half-speed ports beat; section 3 the rail-optimized wiring whose spine ZCube replaces; section 4 HPN's dual-ToR, the other use of a second port.第 6 章：第 1 節（面板 2）給出半速埠所超越的 $k^2/2$ 與 $k^3/4$ 計數；第 3 節是 ZCube 取代其 spine 的 rail-optimized 接線；第 4 節是 HPN 的 dual-ToR，也就是第二個埠的另一種用法。
Rail-only deletes the spine; ZCube keeps any-to-any reach at two switches. Alibaba HPN splits the NIC the same way for redundancy. MegaScale describes ByteDance's fabric; five co-authors are from ByteDance.Rail-only 刪掉 spine；ZCube 以兩台交換器保留任意對任意的觸及。Alibaba HPN 以同樣方式拆分 NIC 做備援。MegaScale 描述字節跳動的網路；共同作者中有五位來自字節跳動。
BST reaches a cheaper two-level static fabric by combinatorial design instead of search, and puts the sparsity in the spine wiring instead of the NIC. SimAI supplies ATOP's workloads.BST 以組合設計而非搜尋得到更便宜的兩層靜態網路，並把稀疏性放在 spine 接線而非 NIC。SimAI 提供 ATOP 的工作負載。

Key numbers關鍵數字

Number	數字	Meaning and setup
35%	Rail-aligned traffic crossing spines (one month, production ROFT; measured)	經過 spine 的 rail 對齊流量（一個月，生產 ROFT；實測）
48 × 200G vs 32 × 400G	Testbed links at equal NCCL bus bandwidth, ZCube(4,2) vs ROFT (16 H800; measured)	NCCL 匯流排頻寬相同時的測試平台鏈路，ZCube(4,2) 對 ROFT（16 顆 H800；實測）
25%	Testbed hardware saving (authors' cost model)	測試平台的硬體節省（作者的成本模型）
6.5 / 10.6 / 25.4 / 71.2 h	ATOP search, 256 / 1K / 4K / 16K GPUs, 256 cores (measured run time)	ATOP 搜尋，256 / 1K / 4K / 16K 顆 GPU，256 核心（實測執行時間）
256 vs 640	Switches for 16,384 GPUs, ZCube(128,2) vs ROFT, 51.2T (analysis)	16,384 顆 GPU 的交換器數，ZCube(128,2) 對 ROFT，51.2T（分析）
2 vs 5	Diameter in switches, ZCube(n,2) vs three-tier ROFT (analysis)	以交換器計的直徑，ZCube(n,2) 對三層 ROFT（分析）

Terms used on this page本頁用到的術語

ATOP: Automated Topology Optimization Pipeline: hyperparameter model, NSGA-II optimizer and two-stage simulator evaluation.; 自動化拓樸最佳化流程：超參數模型、NSGA-II 最佳化器與兩階段模擬評估。
ZCube(n,k): A recursive topology with $n^k$ GPUs, $k$ ports per GPU and $k$ switch levels of $n^{k-1}$ switches; diameter $k$ .; 遞迴拓樸，有 $n^k$ 顆 GPU、每顆 $k$ 個埠、 $k$ 層各 $n^{k-1}$ 台交換器；直徑 $k$ 。
ROFT: Rail-optimized fat tree: GPU $r$ of every server goes to the leaf of rail $r$ .; Rail-optimized fat tree：每台伺服器的 GPU $r$ 接到 rail $r$ 的 leaf。
Rail-aligned traffic: Traffic between GPUs with the same local index on different servers.; 不同伺服器上本地編號相同的 GPU 之間的流量。
Pareto-optimal / knee: No other design is as good on every objective and better on one / where more cost buys little speed.; 沒有其他設計在每個目標都一樣好且某一項更好 / 再加成本也買不到多少速度之處。
Bus bandwidth: nccl-tests' measure that normalizes a collective's speed so it compares with link speed.; nccl-tests 的指標，把集體通訊的速度正規化，使它能與鏈路速度比較。