Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion ParametersRail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters

Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, Naader Hasani · arXiv 2023 / IEEE Hot Interconnects 2024

≈ 35 min · self-contained digest≈ 35 分鐘 · 不必打開論文也能讀懂Primer chapters: 03, 05對應入門頁章節：03, 05Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

This short paper asks a dangerous question: if we work out where the bytes actually go during LLM training, do we need an any-to-any network at all? The authors computed, from Megatron-LM's parallel layout, the traffic of GPT models up to one trillion parameters on up to 3,072 GPUs, and found that more than 75% of all bytes are tensor-parallel traffic that never leaves the server's NVSwitch domain, over 99% of GPU pairs exchange nothing, and the traffic that does cross the network (pipeline and data parallel) flows only between GPUs with the same local rank. They propose deleting the spine layer of the standard Clos and keeping only one full-bisection switch per "rail" (per local rank). Traffic that must cross rails first hops over NVLink inside a server to a GPU on the right rail. For dense models the paper's analytical model predicts unchanged training time, and the network costs 38–77% less and uses 37–75% less power, worth $117–234 million and 1.7–6.9 MW for a 32K- or 64K-GPU cluster. The authors are honest about the limits: by their analytical estimate, mixture-of-experts all-to-all slows by 8.2–11.2%, and replacing a single failed GPU is harder (their §IV-C). Alibaba's HPN rejected the idea for two reasons of its own: MoE traffic, and hosts shared by tenants.

這篇短論文問了一個危險的問題：如果我們算出 LLM 訓練期間位元組實際上流向哪裡，我們到底需不需要任意對任意的網路？作者依 Megatron-LM 的平行配置，計算了最多一兆參數的 GPT 模型在最多 3,072 顆 GPU 上的流量，發現超過 75% 的位元組是從不離開伺服器 NVSwitch 範圍的張量平行流量，超過 99% 的 GPU 配對之間毫無交換，而真正跨越網路的流量（管線與資料平行）只在相同本地 rank 的 GPU 之間流動。他們提議刪掉標準 Clos 的 spine 層，只保留每個「rail」（每個本地 rank）一台完整對分的交換器。必須跨 rail 的流量先在伺服器內透過 NVLink 跳到正確 rail 上的 GPU。對稠密模型，論文的分析模型預測訓練時間不變，網路成本減少 38–77%、功耗減少 37–75%，對 32K 或 64K GPU 的叢集相當於 1.17–2.34 億美元與 1.7–6.9 MW。作者對限制很坦白：依他們的分析估計，混合專家的 all-to-all 慢 8.2–11.2%，更換單一故障 GPU 也更難（論文 §IV-C）。阿里 HPN 拒絕此想法則是出於自己的兩個理由：MoE 流量，以及由多個租戶共用的主機。

Reading plan (35 minutes)閱讀計畫（35 分鐘）

§II gives the vocabulary (HB domain, rail, rail-optimized). §III is the traffic analysis and the reason the paper is cited; spend most of your time on Figures 2–5. §IV is the proposal, including the MoE and failure discussion, and §V the performance and cost model. Then re-read the "why not rail-only at tier 2" paragraph and Table 4 of the HPN paper; the two papers are a debate.

§II 給出詞彙（HB domain、rail、rail-optimized）。§III 是流量分析，也是這篇被引用的原因；大部分時間花在 Figures 2–5。§IV 是提案，包含 MoE 與故障的討論，§V 是效能與成本模型。接著重讀 HPN 論文「為何第二層不用 rail-only」那段與 Table 4；這兩篇是一場辯論。

1. Vocabulary1. 詞彙

A high-bandwidth (HB) domain is a group of GPUs joined by an interconnect with terabits per second of internal bandwidth. The usual one is a DGX H100 server: 8 GPUs connected through NVSwitches with 3.6 Tb/s of non-blocking bandwidth. A DGX GH200 extends the NVSwitch fabric over 256 GPUs at the same per-GPU bandwidth. Outside the HB domain, each GPU has one ConnectX-7 NIC at 400 Gb/s; this is the NIC domain. The gap between the two is roughly 9×, and the whole paper lives in that gap. A rail is the set of GPUs that have the same local rank in every HB domain: all the GPU 0s form rail 0, all the GPU 1s form rail 1, and so on; an HB domain of size $K$ produces $K$ rails. In the standard rail-optimized cluster (Figure 1), each rail has its own leaf switch, so same-rail traffic crosses one switch, and a spine layer joins all the rail switches into one full-bisection Clos so that any GPU can reach any other.

高頻寬（HB）domain 是一群由內部頻寬達每秒數兆位元的互連連接的 GPU。常見的是一台 DGX H100 伺服器：8 顆 GPU 透過 NVSwitch 連接，有 3.6 Tb/s 的非阻塞頻寬。DGX GH200 把 NVSwitch 網路以相同的每 GPU 頻寬延伸到 256 顆 GPU。HB domain 之外，每顆 GPU 有一張 400 Gb/s 的 ConnectX-7 NIC；這是 NIC domain。兩者差距約 9 倍，整篇論文就活在這個差距裡。rail 是在每個 HB domain 中本地 rank 相同的 GPU 集合：所有 GPU 0 構成 rail 0、所有 GPU 1 構成 rail 1，以此類推；大小 $K$ 的 HB domain 產生 $K$ 條 rail。在標準的 rail-optimized 叢集（Figure 1）中，每條 rail 有自己的 leaf 交換器，同 rail 流量只跨一台交換器，再由 spine 層把所有 rail 交換器接成一個完整對分的 Clos，讓任何 GPU 都能到達任何其他 GPU。

2. Where the bytes go2. 位元組流向哪裡

The authors ran Megatron-LM training of GPT models with 145.6B, 310.1B, 539.6B and 1T parameters on up to 384 DGX A100 servers (3,072 GPUs), with tensor parallelism inside each server and pipeline and data parallelism across servers, and computed the bytes exchanged between every pair of GPUs during an iteration (an analysis of the parallel layout, not a hardware trace). Three findings:

作者用 Megatron-LM 在最多 384 台 DGX A100 伺服器（3,072 顆 GPU）上訓練 145.6B、310.1B、539.6B 與 1T 參數的 GPT 模型，張量平行在伺服器內、管線與資料平行在伺服器之間，並計算一次迭代中每一對 GPU 之間交換的位元組（這是對平行配置的分析，而非硬體上的追蹤紀錄）。三個發現：

Tensor parallelism dominates. Figure 2a: TP is more than 75% of all bytes transmitted. All of it stays inside the HB domain by construction, because TP groups are placed within a server.張量平行主導。Figure 2a：TP 超過全部傳輸位元組的 75%。依設計它全部留在 HB domain 內，因為 TP 群組被放在一台伺服器內。
Almost nobody talks to almost anybody. Figure 2b: over 99% of GPU pairs exchange no data at all, and fewer than 0.04% of pairs carry TP traffic. The traffic matrix is extremely sparse.幾乎沒有人跟幾乎任何人說話。Figure 2b：超過 99% 的 GPU 配對完全不交換資料，少於 0.04% 的配對承載 TP 流量。流量矩陣極度稀疏。
The volumes differ by 50×. Figure 3, the GPT-1T heatmap: pairs inside an HB domain exchange about 300 GB per iteration; pairs across the NIC domain exchange about 6 GB. The dense squares on the diagonal are TP; the lighter stripes are data-parallel all-reduce; the thin lines far from the diagonal are pipeline send/receive between stages.流量差 50 倍。Figure 3，GPT-1T 的熱圖：HB domain 內的配對每次迭代交換約 300 GB；跨 NIC domain 的配對約 6 GB。對角線上的深色方塊是 TP；較淺的條紋是資料平行的 all-reduce；遠離對角線的細線是階段之間的管線 send/receive。

Traffic heatmap GPT-1T — Figure 3. GPT-1T traffic between GPU pairs on a log color scale, computed from the parallel layout: the diagonal blocks (TP, inside a server) are ~300 GB; the off-diagonal stripes (DP, PP) are a few GB.Figure 3。GPT-1T 在 GPU 配對之間的流量（依平行配置計算），對數色階：對角方塊（TP，伺服器內）約 300 GB；離對角線的條紋（DP、PP）只有幾 GB。Wang et al., 2023.

Then they repeat the exercise with a larger HB domain. On 16 DGX GH200 systems ( $K = 256$ ), with data parallelism arranged hierarchically so that most of the all-reduce happens inside the HB domain, the data-parallel traffic that actually crosses the NIC domain is 0.8% of the total; the hierarchical all-reduce sends 98% of its bytes over NVSwitch and 2% over the network. The key structural observation follows: the traffic that must leave an HB domain is pipeline point-to-point traffic and the remainder of data-parallel collectives, and both are between GPUs with the same local rank. GPU $k$ in one server talks to GPU $k$ in another. By construction, this traffic never needs to cross rails.

接著他們用更大的 HB domain 重做一次。在 16 台 DGX GH200 系統（ $K = 256$ ）上，資料平行以階層方式安排、讓大部分 all-reduce 在 HB domain 內完成，真正跨 NIC domain 的資料平行流量只占總量的 0.8%；階層式 all-reduce 把 98% 的位元組送過 NVSwitch、2% 送過網路。關鍵的結構觀察隨之而來：必須離開 HB domain 的流量是管線的點對點流量與資料平行集體操作的剩餘部分，而兩者都在本地 rank 相同的 GPU 之間。一台伺服器的 GPU $k$ 與另一台的 GPU $k$ 通訊。依設計，這些流量永遠不需要跨 rail。

What the heatmap depends on熱圖依賴什麼

The picture is for one placement: TP inside the server, DP and PP outside, with ranks assigned so that partners share a local rank. If a job used TP across two servers, or if the scheduler assigned ranks carelessly, cross-rail traffic would appear. The rail-only argument is therefore an argument about the traffic that a well-placed dense LLM produces, not about LLM training in general. HPN's authors make exactly this point.

這張圖對應一種配置：TP 在伺服器內、DP 與 PP 在外、rank 指派使通訊夥伴共享本地 rank。若任務把 TP 跨兩台伺服器，或排程器隨便指派 rank，跨 rail 流量就會出現。所以 rail-only 的論證是關於「配置良好的稠密 LLM 所產生的流量」，而不是一般的 LLM 訓練。HPN 的作者正是指出這一點。

3. The proposal: delete the spine3. 提案：刪掉 spine

If no traffic needs to cross rails, the spine layer that connects the rail switches to each other is carrying nothing useful. The rail-only design (Figure 6) keeps one full-bisection network per rail and removes everything above it. The ports that were spine uplinks on each rail switch are turned around into downlinks to more GPUs, so each rail switch serves more servers. In the rare case that two GPUs on different rails must communicate, the data takes two steps: first across the HB domain over NVLink to the GPU on the destination's rail, then across that rail's switch. NCCL already supports this pattern under the name PXN (PCIe × NVLink), so no new software is needed. Because NVLink is about 9× faster than the NIC, the first hop is cheap.

如果沒有流量需要跨 rail，那麼把 rail 交換器彼此連接的 spine 層就沒有承載任何有用的東西。Rail-only 設計（Figure 6）為每條 rail 保留一個完整對分的網路，並移除其上的一切。每台 rail 交換器原本當 spine 上行的埠轉過來當下行、接更多 GPU，所以每台 rail 交換器服務更多伺服器。在罕見的情況下，若兩顆不同 rail 的 GPU 必須通訊，資料走兩步：先在 HB domain 內透過 NVLink 到目的地 rail 上的 GPU，再跨那條 rail 的交換器。NCCL 已經以 PXN（PCIe × NVLink）之名支援這種模式，不需要新軟體。因為 NVLink 比 NIC 快約 9 倍，第一跳很便宜。

Rail-only proposal — Figure 6. One switch (or one full-bisection network) per rail; nothing above. GPU $k$ of every server connects only to rail switch $k$ .Figure 6。每條 rail 一台交換器（或一個完整對分網路）；上面什麼都沒有。每台伺服器的 GPU $k$ 只接到 rail 交換器 $k$ 。Wang et al., 2023.

Drawn from §IV: a cross-rail transfer becomes an NVLink hop inside the source server plus one same-rail network hop; the rail-0 switch cannot reach GPU 5 at all.依 §IV 繪製：跨 rail 傳輸變成來源伺服器內的一次 NVLink 跳，加一次同 rail 的網路跳；rail-0 交換器完全無法到達 GPU 5。Drawn for this primer from Wang et al., 2023.

4. Does it cost performance?4. 會損失效能嗎？

For dense LLMs the authors argue there is no loss, because the traffic that rail-only cannot carry directly did not appear in the analysis. Their simulations also quantify how much the HB domain matters: growing it from $K = 1$ (no NVLink) to $K = 8$ cuts GPT-146B iteration time by 43.3%, and growing from 8 to 256 cuts another 30.6%; GPT-1T stops improving above about 32 GPUs per domain because the remaining time is computation (Amdahl's law). Against an idealized machine where all $N$ GPUs share one NVLink fabric, a rail-only cluster with $K = 256$ is within 0.9% (GPT-1T) and 4.1% (GPT-146B) in iteration time. For mixture-of-experts models the story changes: expert parallelism sends every token's activations to the GPU that holds its expert, an all-to-all among GPUs that sit on different rails. In their MoE-1.3B example with 128 experts, all-to-all is 27% of total traffic. On rail-only, the all-to-all must take the two-step path, and the paper estimates analytically, for an evenly spread all-to-all, that the completion time of that collective grows by 8.2% on DGX A100 and 11.2% on DGX H100.

對稠密 LLM，作者主張沒有損失，因為 rail-only 無法直接承載的流量在分析中本來就不存在。他們的模擬也量化了 HB domain 的重要性：從 $K = 1$ （沒有 NVLink）增加到 $K = 8$ ，GPT-146B 的迭代時間減少 43.3%；從 8 到 256 再減 30.6%；GPT-1T 在每個 domain 約 32 顆 GPU 以上就不再改善，因為剩下的時間是計算（Amdahl 定律）。相對於一台所有 $N$ 顆 GPU 共享單一 NVLink 網路的理想機器， $K = 256$ 的 rail-only 叢集迭代時間差距在 0.9%（GPT-1T）與 4.1%（GPT-146B）以內。對混合專家模型故事就不同了：專家平行把每個 token 的激活值送到持有其專家的 GPU，是不同 rail 上 GPU 之間的 all-to-all。在他們 128 個專家的 MoE-1.3B 例子中，all-to-all 占總流量 27%。在 rail-only 上 all-to-all 必須走兩步路徑，論文以分析方式（假設 all-to-all 平均分散）估計，該集體操作的完成時間在 DGX A100 上增加 8.2%，在 DGX H100 上增加 11.2%。

5. Counting the savings5. 計算節省

The cost model counts switch ports and transceivers: $694 per 400G switch port and $199 per transceiver, 18 W per port and 9 W per transceiver (the paper's footnote). Removing the spine removes its switches and also the transceivers at both ends of every leaf-to-spine link, which is why the savings exceed the share of switches removed. Table II evaluates clusters of 32,768 and 65,536 GPUs with switch radices of 64, 128 and 256. At 32,768 GPUs and radix 128, rail-only needs 256 switches where the rail-optimized Clos needs 1,280, a 77% cost saving; at radix 64 the saving is 38%. Across the configurations, network cost falls by 38–77% and network power by 37–75%, which the authors translate to $117–234 million and 1.7–6.9 MW. The saving does not grow steadily with the radix: for 32,768 GPUs Table II gives 38%, 77% and 62% at radix 64, 128 and 256, and for 65,536 GPUs 38%, 38% and 77%, because the number of tiers each design needs changes at different radices.

成本模型計算交換器埠與光模組：每個 400G 交換器埠 694 美元、每個光模組 199 美元，每埠 18 W、每個光模組 9 W（論文腳註）。移除 spine 不只移除其交換器，也移除每條 leaf 到 spine 鏈路兩端的光模組，這就是為什麼節省比例超過移除的交換器比例。Table II 評估 32,768 與 65,536 顆 GPU 的叢集、交換器埠數 64、128 與 256。在 32,768 顆 GPU 與埠數 128 時，rail-only 需要 256 台交換器，rail-optimized Clos 需要 1,280 台，成本節省 77%；埠數 64 時節省 38%。跨所有配置，網路成本下降 38–77%、網路功耗下降 37–75%，作者換算為 1.17–2.34 億美元與 1.7–6.9 MW。節省並不隨埠數穩定增加：32,768 顆 GPU 時 Table II 在埠數 64、128、256 下分別是 38%、77%、62%，65,536 顆 GPU 時是 38%、38%、77%，因為兩種設計需要的層數在不同埠數時改變。

6. The honest limits6. 誠實的限制

Link and switch failures behave the same in both designs: the GPUs behind a failed switch or link are unavailable either way, and rail-only has fewer switches to fail; operators can add redundant rail switches. A failed server is replaced by a healthy one with no change in connectivity. A single failed GPU is the hard case. In a Clos, the job can move that rank to any idle GPU anywhere. In rail-only, the replacement must sit on the same rail, and it must be in an HB domain with spare NVLink capacity for the forwarding; if the domain is fully occupied, there is no such GPU. The authors propose migrating a whole HB domain when domains are small, and, for large domains such as $K = 256$ , adding optical switches between GPUs and rail switches so that a small set of GPUs can be swapped. They leave the analysis to future work. This limitation, together with multi-tenant hosts where one tenant cannot forward through another tenant's GPU, is what HPN's authors cite when they keep an any-to-any tier 2 even though Table 4 of their paper shows a rail-only tier 2 would hold 120K GPUs per pod.

鏈路與交換器故障在兩種設計中表現相同：故障交換器或鏈路後面的 GPU 無論如何都不可用，而 rail-only 可能故障的交換器更少；營運者可以增加備援 rail 交換器。伺服器故障用健康的伺服器取代，連接性不變。單顆 GPU 故障是困難的情況。在 Clos 中任務可以把那個 rank 移到任何地方的任何閒置 GPU。在 rail-only 中，替代者必須在同一條 rail 上，而且必須在有多餘 NVLink 容量供轉送的 HB domain 內；如果 domain 已滿，就沒有這樣的 GPU。作者提議在 domain 小的時候遷移整個 HB domain，對 $K = 256$ 之類的大 domain 則在 GPU 與 rail 交換器之間加入光交換器，讓一小組 GPU 可以被替換。他們把分析留給未來工作。這個限制，加上多租戶主機中一個租戶不能透過另一個租戶的 GPU 轉送，就是 HPN 作者保留任意對任意第二層時所引用的理由——儘管他們論文的 Table 4 顯示 rail-only 的第二層每個 pod 可容納 120K 顆 GPU。

7. Critique and context7. 評論與脈絡

The traffic analysis is the lasting contribution: the heatmap in Figure 3, computed rather than measured, is reproduced in talks and papers across the field, and it taught everyone to ask which bytes truly need to cross the network before drawing a topology. The proposal is weaker than the analysis. It depends on a placement discipline (TP inside the server, partners on the same rail) that the paper assumes rather than enforces, the performance claims against a rail-optimized Clos rest on the traffic analysis rather than on a head-to-head simulation, and the failure story is left open. The cost model is also simple: it prices ports and transceivers but not cabling, fiber reach or operational complexity. The debate with HPN is the best way to read it: rail-only is right about dense models; for MoE it pays a detour that it puts at 8.2–11.2% of the all-to-all, and it does not suit shared hosts. HPN judged that price too high. Later, DeepSeek-V3 trained an MoE model on eight network planes that never meet, with each token limited to 4 nodes and forwarded over NVLink, as fast as on a network whose rails meet (Chapter 3, section 7). Meta's topology-aware scheduler and Llama 3's [TP, CP, PP, DP] order are a middle path, keeping any-to-any hardware but placing work so that it behaves rail-only in practice.

流量分析是持久的貢獻：Figure 3 的熱圖（計算而非量測）在整個領域的演講與論文中被反覆重現，它教會所有人在畫拓樸之前先問哪些位元組真的需要跨越網路。提案比分析弱。它依賴一種論文假設而非強制的配置紀律（TP 在伺服器內、夥伴在同一條 rail），對 rail-optimized Clos 的效能宣稱建立在流量分析而非正面對決的模擬上，故障的故事也留白。成本模型也很簡單：它計價埠與光模組，但不計線纜、光纖距離或營運複雜度。與 HPN 的辯論是讀它最好的方式：rail-only 對稠密模型是對的；對 MoE 它要付出繞路的代價，依它自己的估計是 all-to-all 慢 8.2–11.2%，而且它不適合共享主機。HPN 認為這個代價太高。後來 DeepSeek-V3 在八個互不相連的網路平面上訓練 MoE 模型，每個 token 最多送到 4 個節點、再經 NVLink 轉送，速度與 rail 彼此相連的網路相同（第 3.9 節）。Meta 的拓樸感知排程器與 Llama 3 的 [TP, CP, PP, DP] 順序是一條中間路線：保留任意對任意的硬體，但把工作配置得在實務上表現成 rail-only。

Key numbers關鍵數字

Number	數字	Meaning
3.6 Tb/s vs 400 Gb/s	HB-domain bandwidth per GPU (NVSwitch) vs. NIC bandwidth; ~9× gap	每顆 GPU 的 HB domain 頻寬（NVSwitch）vs. NIC 頻寬；約 9 倍差距
3,072	GPUs (384 DGX A100) in the analyzed Megatron-LM configurations	分析的 Megatron-LM 配置中的 GPU 數（384 台 DGX A100）
>75%	Share of all bytes that are tensor-parallel traffic	張量平行流量占全部位元組的比例
>99% / <0.04%	GPU pairs with no traffic / pairs carrying TP traffic	沒有流量的 GPU 配對 / 承載 TP 流量的配對
~300 GB vs ~6 GB	Per-iteration traffic between pairs inside an HB domain vs. across the NIC domain (GPT-1T)	每次迭代 HB domain 內配對 vs. 跨 NIC domain 配對的流量（GPT-1T）
0.8% / 98%–2%	DP share of traffic over the NIC at $K = 256$ ; hierarchical all-reduce split NVSwitch/network	$K = 256$ 時 DP 經 NIC 的流量比例；階層式 all-reduce 在 NVSwitch/網路的分配
43.3% / 30.6%	GPT-146B iteration-time cut from $K = 1 \to 8$ and $K = 8 \to 256$	GPT-146B 迭代時間在 $K = 1 \to 8$ 與 $K = 8 \to 256$ 的減少
0.9% / 4.1%	Gap to an ideal all-NVLink machine at $K = 256$ (GPT-1T / GPT-146B)	$K = 256$ 時相對理想全 NVLink 機器的差距（GPT-1T / GPT-146B）
$694 / $199 / 18 W / 9 W	Assumed cost and power per 400G switch port / per transceiver	假設的每個 400G 交換器埠 / 每個光模組的成本與功耗
256 vs 1,280	Switches needed, rail-only vs. Clos, 32,768 GPUs at radix 128 (77% saving)	32,768 顆 GPU、埠數 128 時所需交換器數，rail-only vs. Clos（節省 77%）
38–77% / 37–75%	Network cost / power reduction; $117–234M and 1.7–6.9 MW	網路成本 / 功耗減少；1.17–2.34 億美元與 1.7–6.9 MW
27% / 8.2–11.2%	All-to-all share of MoE-1.3B traffic / its slowdown on rail-only (A100 / H100)	MoE-1.3B 流量中 all-to-all 的比例 / 在 rail-only 上的減速（A100 / H100）

Terms used in this paper本篇用到的術語

HB domain: A set of GPUs with terabit-scale internal bandwidth (NVSwitch); 8 in a DGX H100, 256 in a DGX GH200.; 內部頻寬達兆位元級（NVSwitch）的一組 GPU；DGX H100 有 8 顆，DGX GH200 有 256 顆。
NIC domain: The network outside the HB domain, reached through each GPU's 400 Gb/s NIC.; HB domain 之外、透過每顆 GPU 的 400 Gb/s NIC 到達的網路。
Rail: All GPUs with the same local rank across HB domains; a domain of size $K$ gives $K$ rails.; 跨 HB domain 中本地 rank 相同的所有 GPU；大小 $K$ 的 domain 給出 $K$ 條 rail。
Rail-optimized: One leaf switch per rail plus a spine joining all rails into a full-bisection Clos.; 每條 rail 一台 leaf 交換器，加上把所有 rail 接成完整對分 Clos 的 spine。
Rail-only: One full-bisection network per rail and no spine; cross-rail traffic is forwarded through the HB domain first.; 每條 rail 一個完整對分網路、沒有 spine；跨 rail 流量先經 HB domain 轉送。
PXN: NCCL's PCIe × NVLink mode: send through a same-server GPU on the destination's rail, then over the network.; NCCL 的 PCIe × NVLink 模式：先透過同伺服器、位於目的 rail 的 GPU，再經網路送出。
Hierarchical all-reduce: Reduce inside each HB domain first, exchange the small result across domains, then broadcast back; 98% of bytes stay on NVSwitch.; 先在每個 HB domain 內歸約，再在 domain 之間交換小結果，最後廣播回去；98% 的位元組留在 NVSwitch 上。