Alibaba HPN: A Data Center Network for Large Language Model TrainingAlibaba HPN: A Data Center Network for Large Language Model Training

Reading plan (50 minutes)閱讀計畫（50 分鐘）

This page follows the paper's own order. §2 is the evidence; §3 is the whole design in one figure; §4–§6 are the three mechanisms (dual-ToR, single-chip tier 1, dual-plane tier 2); §7–§8 cover growing past one pod and the front end; §9–§10 give results and lessons. The lessons are the part most people skip and the part most worth reading.

本頁依論文自己的順序。§2 是證據；§3 是把整個設計放進一張圖；§4–§6 是三個機制（dual-ToR、單晶片第一層、dual-plane 第二層）；§7–§8 談超出一個 pod 的擴展與前端網路；§9–§10 是結果與教訓。教訓是多數人跳過、卻最值得讀的部分。

1. Five measurements, five arguments1. 五個量測，五個論點

The authors begin not with a design but with data from their own production clusters, one figure per argument.

作者不是從設計開始，而是從自家生產叢集的數據開始，一張圖一個論點。

Figure 2: traffic is bursty. They traced one host's NICs during a training job. Each NIC (2×200 Gb/s) periodically jumps to its full 400 Gb/s and then goes idle. This is the shape of a training step: compute, then a burst of gradient exchange, then compute again. Figure 3: there are very few connections. A host in a normal cloud workload has thousands of flows; a training host has tens. Together these two facts break equal-cost multipath (ECMP) routing. A switch hashes each flow's header to choose an uplink; with tens of flows at line rate, two landing on one link is common and the collision halves both. In a three-tier network the same header is hashed at three successive switches, so an unlucky pair of flows can be stuck together at every tier. The paper calls this hash polarization, and the authors say it caused multiple production incidents before HPN.

Figure 2：流量是突發的。他們追蹤一台主機的 NIC 在訓練任務期間的行為。每張 NIC（2×200 Gb/s）週期性地跳到滿額的 400 Gb/s，然後閒置。這就是一個訓練步的形狀：計算、一波梯度交換、再計算。Figure 3：連線非常少。一般雲端工作負載的主機有數千條流；訓練主機只有幾十條。這兩個事實合起來讓等價多路徑（ECMP）路由失效。交換器對每條流的標頭做雜湊來選上行鏈路；幾十條線速的流，兩條落在同一條鏈路很常見，碰撞讓兩條都減半。在三層網路中，同一個標頭會在三台連續的交換器被雜湊，所以一對倒楣的流可能在每一層都被卡在一起。論文稱之為 hash polarization，作者說在 HPN 之前它造成了多起生產事故。

Figure 4: failures are expensive. Jobs checkpoint every two to four hours. A crash therefore loses hours of GPU time plus the restart. Figure 5: failures are routine. In their fleet, 0.057% of NIC-to-ToR links and about 0.051% of ToR switches fail each month, and 5,000 to 60,000 link flaps (a link going down and up within seconds) happen every day. At a cluster of 15,000 GPUs this means a failure somewhere is a near-daily event. In a classic single-ToR design, each host hangs off one rack switch, so one ToR failure takes down every host in the rack and the job with it. Figure 6: most jobs are not huge. 96.3% of their LLM training jobs use fewer than 1,000 GPUs. If a single network tier could hold 1,000 GPUs, nearly every job would run without ever touching a second tier.

Figure 4：故障很昂貵。任務每兩到四小時存一次檢查點，所以一次崩潰損失數小時的 GPU 時間再加上重啟。Figure 5：故障是常態。在他們的機群裡，每月有 0.057% 的 NIC 到 ToR 鏈路與約 0.051% 的 ToR 交換器故障，每天發生 5,000 到 60,000 次鏈路抖動（鏈路在數秒內斷掉又恢復）。在 15,000 顆 GPU 的叢集，這表示某處故障幾乎每天都會發生。傳統的單 ToR 設計中每台主機只掛在一台機櫃交換器上，一台 ToR 故障就讓整個機櫃的主機連同任務一起倒下。Figure 6：多數任務不算巨大。96.3% 的 LLM 訓練任務使用少於 1,000 顆 GPU。如果單一網路層能容納 1,000 顆 GPU，幾乎每個任務都不必碰到第二層。

From these, §2.4 states three goals. Scale: 15,000 GPUs in one pod now, with a path to 100,000. Performance: fewer hops and fewer hash decisions between any two GPUs. Reliability: no single ToR failure may stop a job.

由此 §2.4 提出三個目標。規模：現在一個 pod 容納 15,000 顆 GPU，並有通往 100,000 顆的路。效能：任意兩顆 GPU 之間更少的跳數與更少的雜湊決策。可靠性：任何單一 ToR 故障都不得讓任務停止。

2. The design in one figure2. 一張圖裡的設計

Figure 7 shows everything. Read it from the host outward.

Figure 7 畫出了一切。從主機往外讀。

The host has 8 GPUs and 9 NICs. NIC 0 serves the front-end network. NICs 1–8 serve the back end, one per GPU, each with two 200 Gb/s ports, so every GPU has 400 Gb/s into the network and a host has 3.2 Tb/s. The two ports of a NIC go to two different ToR switches. This is the dual-ToR of §4.

主機有 8 顆 GPU 與 9 張 NIC。NIC 0 服務前端網路。NIC 1–8 服務後端，每顆 GPU 一張，每張有兩個 200 Gb/s 埠，所以每顆 GPU 有 400 Gb/s 進入網路，一台主機有 3.2 Tb/s。一張 NIC 的兩個埠接到兩台不同的 ToR 交換器。這就是 §4 的 dual-ToR。

The segment is tier 1: 16 ToR switches and the 136 hosts under them. Each ToR is a 51.2 Tb/s single-chip switch with 128 ports of 400 Gb/s. HPN splits 68 of those ports into 136 ports of 200 Gb/s facing hosts: 128 active and 8 backup. The other 60 ports face up. Because each NIC reaches two ToRs, a pair of ToRs serves 128 GPUs, and the 16 ToRs of a segment form 8 such pairs. The port budget of one ToR and of the segment, one question per row:

segment 是第一層：16 台 ToR 交換器與其下的 136 台主機。每台 ToR 是一顆 51.2 Tb/s 的單晶片交換器，有 128 個 400 Gb/s 埠。HPN 把其中 68 個埠拆成 136 個面向主機的 200 Gb/s 埠：128 個使用中、8 個備援。其餘 60 個埠朝上。因為每張 NIC 連到兩台 ToR，一對 ToR 服務 128 顆 GPU，一個 segment 的 16 台 ToR 組成 8 對。單台 ToR 與整個 segment 的埠預算，逐列計算如下：

So one ToR is close to non-blocking at 1.067:1, and a segment holds 1,024 active GPUs plus 64 spares that can take over when a host fails.

所以單台 ToR 以 1.067:1 接近非阻塞，一個 segment 容納 1,024 顆使用中的 GPU，外加 64 顆在主機故障時可以接手的備用 GPU。

The pod is tier 2: 15 segments joined by aggregation switches, split into two planes (§6). Each aggregation switch uses 120 ports down toward ToRs and 8 ports up toward a core tier; that 15:1 oversubscription toward the core is deliberate. A pod holds 15,000 GPUs, which the authors note covers every job they have ever run. The core (tier 3) joins pods for jobs that one day need more (§7). The front end is a separate, non-blocking network for storage, management and inference (§8).

pod 是第二層：15 個 segment 由聚合交換器連接，並切成兩個平面（§6）。每台聚合交換器用 120 個埠向下接 ToR、8 個埠向上接核心層；朝核心的 15:1 超額認購是刻意的。一個 pod 容納 15,000 顆 GPU，作者指出這涵蓋了他們跑過的每一個任務。核心層（第三層）連接各 pod，供將來需要更多的任務使用（§7）。前端是另一個獨立的非阻塞網路，供儲存、管理與推論使用（§8）。

3. Dual-ToR without stacking3. 不堆疊的 dual-ToR

Connecting each host to two rack switches is an old idea. The usual way to do it, called stacking, vPC or M-LAG depending on the vendor, wires the two switches together with a direct link and makes them behave as one logical switch toward the host. Alibaba's failure records say this is exactly where the risk lives: more than 40% of critical failures in their traditional data centers came from the stacked design. Figure 8a shows the mechanism. If ToR1's data plane fails but its control plane still answers, ToR2 sees a healthy peer that is not forwarding. The two switches now disagree about which one is primary. To avoid forwarding packets inconsistently, ToR2 shuts itself down, and now both switches are dark. Software upgrades are a second hazard: vendors promise in-service upgrades, but that promise assumes the old and new versions differ only slightly, and 70% of Alibaba's ToR upgrades did not meet that assumption.

把每台主機接到兩台機櫃交換器是老想法。常見做法——依廠商叫 stacking、vPC 或 M-LAG——用一條直接鏈路把兩台交換器接在一起，讓它們對主機表現為一台邏輯交換器。阿里的故障記錄說風險正好就在這裡：他們傳統資料中心超過 40% 的重大故障來自堆疊設計。Figure 8a 畫出機制。若 ToR1 的資料平面故障但控制平面仍有回應，ToR2 看到的是一個健康卻不轉送的同儕。兩台交換器對誰是主要者產生分歧。為了避免不一致的轉送，ToR2 把自己關掉，於是兩台都暗了。軟體升級是第二個風險：廠商承諾不中斷服務的升級，但那個承諾假設新舊版本差異很小，而阿里 70% 的 ToR 升級不符合這個假設。

HPN's answer (Figure 8b) is to make the two ToRs fully independent. There is no link between them and no shared state. The host does the work instead: it bonds its two NIC ports into one logical interface with one IP and one MAC address, and the host sends its ARP announcements to both ToRs; each ToR converts what it learns into a /32 host route that it advertises upward with BGP, so each one knows, on its own, that this host is reachable through it. The host itself runs no BGP. Because both ports of a NIC share the same RDMA queue-pair context, the upper layers never notice which port a packet used. When a link or a ToR fails, the host simply stops using that port. In the §9 experiment, which failed one NIC–ToR link during LLaMa-7B training on 256 GPUs, throughput fell 6.25% (one of sixteen 200G ports on a host), and throughput returned to normal the moment the link was repaired; in the single-ToR baseline the job halted, and if repair took more than two minutes it could not recover at all. A link flap that stalled single-ToR training for more than nine seconds was negligible under dual-ToR.

HPN 的答案（Figure 8b）是讓兩台 ToR 完全獨立。它們之間沒有鏈路、沒有共享狀態。工作改由主機來做：主機把兩個 NIC 埠綁成一個邏輯介面、一個 IP、一個 MAC，並把 ARP 通告送給兩台 ToR；每台 ToR 把學到的資訊轉成一條 /32 主機路由，再用 BGP 往上宣告，所以每台 ToR 各自知道這台主機可透過自己到達。主機本身不跑 BGP。因為一張 NIC 的兩個埠共用同一個 RDMA queue-pair 上下文，上層永遠不會察覉封包用了哪個埠。鏈路或 ToR 故障時，主機就是不再使用那個埠。在 §9 的實驗中（在 256 顆 GPU 上訓練 LLaMa-7B 時弄斷一條 NIC–ToR 鏈路），吞吐下降 6.25%（主機 16 個 200G 埠中的一個），鏈路修復的瞬間吞吐就恢復正常；在單 ToR 基線中任務停擺，若修復超過兩分鐘就完全無法恢復。一次讓單 ToR 訓練停擺超過九秒的鏈路抖動，在 dual-ToR 下影響可忽略。

4. Tier 1: one chip, rail-optimized4. 第一層：一顆晶片，rail-optimized

Why a single chip. A rack switch can be a modular chassis with many switching chips joined internally, or a fixed box with one chip. Alibaba's fleet data favors the single chip: they run 32.6× more single-chip switches than multi-chip ones, yet the multi-chip switches produced 3.77× more critical hardware failures in total. The 51.2 Tb/s chip, first available in early 2023, was the first single chip large enough to serve 1,024 GPUs in one tier.

為何用單晶片。機櫃交換器可以是內部由多顆交換晶片連接的模組化機箱，也可以是只有一顆晶片的固定盒子。阿里的機群數據偏向單晶片：他們運行的單晶片交換器數量是多晶片的 32.6 倍，多晶片交換器卻總共產生 3.77 倍的重大硬體故障。2023 年初問世的 51.2 Tb/s 晶片是第一顆大到足以在一層內服務 1,024 顆 GPU 的單晶片。

The price is heat. The 51.2T chip draws 45% more power than the previous 25.6T generation, but its maximum junction temperature is unchanged at 105 °C. With a standard vapor-chamber heat sink, the chip exceeded that limit at full load and would throttle or shut down. The team redesigned the vapor chamber, placing more wick pillars at the center of the chip where the heat is densest, which carried away 15% more heat and kept the chip under the limit. It is unusual for a networking paper to include a heat-sink design; it is here because without it the topology does not exist.

代價是熱。51.2T 晶片的功耗比上一代 25.6T 高 45%，但最高接面溫度不變，仍是 105 °C。用標準的均熱板散熱器，晶片在滿載時會超過上限而降頻或關機。團隊重新設計均熱板，在晶片中央熱最密集處放更多吸液柱，多帶走 15% 的熱，讓晶片維持在上限之下。網路論文裡放散熱器設計並不常見；它出現在這裡是因為沒有它拓樸就不存在。

Rail-optimized wiring. Within a segment, GPU $k$ of every host connects to the same ToR pair (rail $k$). Traffic between GPU 3 on host A and GPU 3 on host B therefore crosses one switch. Traffic between GPU 3 and GPU 5 on different hosts first moves inside host A over NVLink to GPU 5, then crosses rail 5's switch. NVLink inside a host is 4–9× faster than the NIC, so this detour is cheap. The payoff is that the 16 ToRs collectively serve 1,024 GPUs with a single switch hop for the common case, and tier 1 scale multiplies by 8 (Table 2).

Rail-optimized 接線。在一個 segment 內，每台主機的 GPU $k$ 都接到同一對 ToR（rail $k$）。主機 A 的 GPU 3 與主機 B 的 GPU 3 之間的流量因此只跨一台交換器。不同主機的 GPU 3 與 GPU 5 之間的流量，先在主機 A 內經 NVLink 移到 GPU 5，再跨 rail 5 的交換器。主機內的 NVLink 比 NIC 快 4–9 倍，所以這段繞路很便宜。回報是 16 台 ToR 合起來以單一交換器跳數服務 1,024 顆 GPU 的常見情況，第一層規模乘以 8（Table 2）。

5. Tier 2: dual-plane5. 第二層：dual-plane

The problem dual-ToR creates. Consider traffic flowing down toward one NIC. It arrives from any of 60 aggregation switches and must converge onto one of the NIC's two ToRs. In a normal Clos (Figure 12a), each aggregation switch hashes to decide which ToR; with so few flows the hash is unfair. Figure 13a shows the two ToR ports toward the same NIC during a production GPT-3 175B training job: one carried three times the load of the other. The busy port's queue sat at 267 KB continuously while the other sat at 3 KB.

Dual-ToR 製造的問題。想像流向某一張 NIC 的下行流量。它從 60 台聚合交換器中的任何一台到來，必須收斂到該 NIC 兩台 ToR 之一。在一般的 Clos（Figure 12a），每台聚合交換器雜湊決定走哪台 ToR；流這麼少，雜湊就不公平。Figure 13a 顯示生產環境 GPT-3 175B 訓練任務中朝同一張 NIC 的兩個 ToR 埠：一個承載另一個三倍的負載。忙的那個埠佇列持續停在 267 KB，另一個停在 3 KB。

The structural fix. Dual-plane (Figure 12b) assigns the two ToRs of every pair to two separate groups. ToR1 of every pair connects its 60 uplinks only to the 60 aggregation switches of plane 1; ToR2 connects only to plane 2. Now a packet that leaves a source ToR in plane 1 can only arrive at the destination's plane-1 ToR. Once a flow enters a ToR, its path through the pod is completely determined. There is no hash at the aggregation layer because there is no choice to make. After deployment, the two ports toward a NIC carried equal load (Figure 13b), the average queue length fell to 20 KB, a 91.8% reduction, and an ablation attributes up to 71.6% of the cross-segment performance gain to dual-plane. A side effect is reliability: with 60 aggregation switches per plane, losing one leaves 59 to carry the plane.

結構性修法。Dual-plane（Figure 12b）把每一對的兩台 ToR 分到兩個不同的群組。每對的 ToR1 把 60 條上行只接到平面 1 的 60 台聚合交換器；ToR2 只接平面 2。現在，從平面 1 的來源 ToR 出去的封包只可能到達目的地的平面 1 ToR。一條流進入 ToR 後，它穿越 pod 的路徑就完全確定。聚合層沒有雜湊，因為沒有選擇可做。部署後，朝一張 NIC 的兩個埠負載相等（Figure 13b），平均佇列長度降到 20 KB，減少 91.8%；消融實驗把跨 segment 效能提升中最多 71.6% 歸因於 dual-plane。附帶效果是可靠性：每個平面有 60 台聚合交換器，少一台還有 59 台撐住該平面。

The choices that remain. Inside a plane, a source ToR still has 60 uplinks to choose from. HPN moves this choice to the host and the collective library. Using RePaC, a tool that can read the exact hash result each switch would compute, the host finds a set of five-tuples whose paths are fully disjoint and opens one RDMA connection per path. Because of dual-plane, the search only has to consider the 60 links of one ToR, which Table 1 says is one to two orders of magnitude cheaper than in a three-tier network. Then, for every message, the collective library keeps a counter of bytes currently queued on each connection and sends on the connection with the smallest counter: a congested path drains slowly, so its counter stays high and it is avoided. With four AllReduce jobs running at once on 512 GPUs, this raised collective performance by up to 34.7%. The authors note why they did not use switch-side schemes such as flowlet switching or packet spraying: those were unavailable or unproven at their scale, and host-side transport changes are impossible when the RDMA transport is in NIC hardware.

剩下的選擇。在一個平面內，來源 ToR 仍有 60 條上行可選。HPN 把這個選擇移到主機與集體通訊函式庫。利用能讀出每台交換器精確雜湊結果的工具 RePaC，主機找出一組路徑完全不相交的五元組，並為每條路徑開一條 RDMA 連線。由於 dual-plane，搜尋只需考慮一台 ToR 的 60 條鏈路，Table 1 指出這比三層網路便宜一到兩個數量級。接著，對每一則訊息，集體通訊函式庫為每條連線維護一個目前排隊位元組數的計數器，並用計數器最小的連線發送：壅塞的路徑排空得慢，計數器居高不下，就會被避開。在 512 顆 GPU 上同時跑四個 AllReduce 任務時，這讓集體通訊效能提升最多 34.7%。作者說明為何不用交換器端的方案如 flowlet switching 或封包噴灑：那些在他們的規模上不可得或未經驗證，而 RDMA 傳輸層在 NIC 硬體裡時，主機端的傳輸層改動也不可能。

6. Beyond one pod, and the front end6. 超出一個 pod，以及前端

Tier 3. To connect pods, HPN adds core switches with 15:1 oversubscription from the aggregation layer. This is acceptable because of how the parallelism is placed. Table 3 gives the traffic of a GPT-3 175B job, without stating over what period, with TP=8, PP=8, DP=512 (32K GPUs): tensor parallelism moves 560 MB in AllReduce/AllGather, data parallelism 5.5 GB in AllReduce, and pipeline parallelism only 6 MB in point-to-point send/receive. The scheduler is made to place a job so that only pipeline traffic, the smallest and least bandwidth-sensitive, crosses pods. The core tier keeps the dual-plane split, and each core switch uses a per-port rule, "traffic for pod $i$ arriving on port $j$ always leaves on port $k$", instead of a five-tuple hash, falling back to hashing only when port $k$ fails.

第三層。為了連接各 pod，HPN 加入核心交換器，從聚合層往上是 15:1 的超額認購。這可以接受，是因為平行化的擺放方式。Table 3 給出 TP=8、PP=8、DP=512（32K 顆 GPU）的 GPT-3 175B 任務的流量（沒有說明是多長時間內的量）：張量平行在 AllReduce/AllGather 中搬 560 MB，資料平行在 AllReduce 中搬 5.5 GB，管線平行只在點對點 send/receive 中搬 6 MB。排程器被設計成只讓最小、對頻寬最不敏感的管線流量跨 pod。核心層保留 dual-plane 的切分，每台核心交換器用每埠規則「從埠 $j$ 進來、往 pod $i$ 的流量一律從埠 $k$ 出去」取代五元組雜湊，只在埠 $k$ 故障時退回雜湊。

Where storage lives. The back end offers 3.2 Tb/s per host versus 400 Gb/s on the front end, which makes it tempting to put storage there. The authors rejected this for three reasons: customers' datasets and container images live in other data centers and cannot reach the back end without a proxy; when they tried injecting storage traffic into the back end, training performance fluctuated; and storage servers would consume ToR ports that could otherwise serve GPUs. So storage stays on the front end, which is built non-blocking at 2×200 Gb/s per host so the same machines can also serve inference, whose traffic is front-end heavy.

儲存放在哪裡。後端每台主機有 3.2 Tb/s，前端只有 400 Gb/s，讓人很想把儲存放在後端。作者基於三個理由拒絕：客戶的資料集與容器映像放在其他資料中心，沒有代理就無法到達後端；他們嘗試把儲存流量注入後端時，訓練效能出現波動；儲存伺服器會消耗原本可服務 GPU 的 ToR 埠。所以儲存留在前端，前端以每台主機 2×200 Gb/s 建成非阻塞，讓同樣的機器也能服務以前端流量為主的推論。

7. Results7. 結果

The headline experiment moved a production training job of more than 2,300 GPUs from the previous network, DCN+, to HPN. Under DCN+ the job spanned 19 segments; under HPN it fit in 3. End-to-end training throughput rose 14.9%, and cross-segment traffic fell 37% on average. For a cluster costing billions, the authors note, 14.9% is a very large number. In microbenchmarks, AllReduce on 448 GPUs improved by up to 59.3%, because a 1,024-GPU segment has no contention between tiers. The reliability experiments (§3 above) used Llama-7B on 256 GPUs with injected link failures and flaps. Over eight months of production, HPN saw no single-point failure.

主要實驗把一個超過 2,300 顆 GPU 的生產訓練任務從舊網路 DCN+ 搬到 HPN。在 DCN+ 下任務橫跨 19 個 segment；在 HPN 下只需 3 個。端到端訓練吞吐提高 14.9%，跨 segment 流量平均減少 37%。作者指出，對一座價值數十億美元的叢集，14.9% 是非常大的數字。微基準測試中，448 顆 GPU 的 AllReduce 改善最多 59.3%，因為 1,024 顆 GPU 的 segment 在層與層之間沒有競爭。可靠性實驗（上面 §3）在 256 顆 GPU 上用 Llama-7B 並注入鏈路故障與抖動。八個月的生產運行中，HPN 沒有出現任何單點故障。

8. Lessons from operating it8. 營運它學到的教訓

One pod fits one building. Every Alibaba Cloud data-center building is limited to 18 MW, which is about 15,000 GPUs. HPN's pod size was chosen to match. The consequence is that all fibers stay inside one building and are under 100 m long, which allows multi-mode optical transceivers at 70% lower cost than single-mode. The next generation is planned around 102.4 Tb/s chips, and new buildings have had their power budgets raised to match.

一個 pod 剛好一棟樓。阿里雲每棟資料中心建築的上限是 18 MW，約等於 15,000 顆 GPU。HPN 的 pod 大小就是依此選定。結果是所有光纖都留在同一棟樓內、長度低於 100 m，可以使用比單模便宜 70% 的多模光模組。下一代規劃圍繞 102.4 Tb/s 晶片，新建築的電力預算也已相應提高。

Links can be broken in one direction. A production incident came from a link whose optical signal was bad in the NIC-to-ToR direction but fine in the other. The ToR detected it and tried to tell the NIC through Link Fault Signaling, but a NIC firmware bug ignored the message. The NIC kept sending on a link it believed healthy, and packets were lost at scale. Such a fault almost never shows up in a testbed. Dual-ToR turned it from a crash into a slowdown.

鏈路可能只壞一個方向。一次生產事故來自一條在 NIC 到 ToR 方向光訊號不良、另一方向卻正常的鏈路。ToR 偵測到並嘗試透過 Link Fault Signaling 通知 NIC，但 NIC 韌體的錯誤忽略了訊息。NIC 持續在它以為健康的鏈路上發送，大量封包遺失。這種故障幾乎不會在測試環境出現。Dual-ToR 讓它從崩潰變成減速。

HPN is hard to wire. Rail-optimized plus dual-plane means thousands of cables with exact required endpoints. In the early builds, on-site staff made many mistakes, which showed up as mysterious performance problems. The team now sends in-band network telemetry (INT) probes that record the switch ID and port ID at every hop and compares them against the blueprint before any end-to-end test.

HPN 很難接線。Rail-optimized 加 dual-plane 表示數千條線纜都有精確要求的兩端。早期建置時現場人員犯了很多錯，表現為莫名的效能問題。團隊現在在任何端到端測試之前，先送出帶內網路遙測（INT）探測封包，記錄每一跳的交換器 ID 與埠 ID，並與藍圖比對。

Why not rail-only at tier 2? Meta's and MIT's work argue that LLM traffic never needs to cross rails, so the aggregation layer could be pruned to connect only same-rail ToRs. Table 4 shows that a rail-only tier 2 would let a pod hold more than 120,000 GPUs. HPN rejected it anyway, for two reasons. First, the assumption holds for today's dense models only because their traffic was carefully arranged to hold; mixture-of-experts models generate all-to-all traffic between experts that inherently sit on different rails. Second, in serverless use a host is shared by several tenants who each own some NICs, so a tenant cannot relay traffic across the host over NVLink. HPN therefore keeps an any-to-any tier 2 and grows with tier 3.

為何第二層不用 rail-only？Meta 與 MIT 的工作主張 LLM 流量從不需要跨 rail，所以聚合層可以修剪成只連接同 rail 的 ToR。Table 4 顯示 rail-only 的第二層可以讓一個 pod 容納超過 120,000 顆 GPU。HPN 仍然拒絕，理由有二。第一，這個假設對今日的稠密模型成立，只是因為流量被精心安排成如此；混合專家模型在專家之間產生 all-to-all 流量，而專家本質上落在不同 rail。第二，在 serverless 用法中一台主機由多個租戶共享、各擁有部分 NIC，租戶無法透過 NVLink 在主機內中繼流量。因此 HPN 保留任意對任意的第二層，並以第三層成長。

9. Critique and context9. 評論與脈絡

HPN is the most complete published topology for LLM training, and its strength is that every choice is backed by a fleet measurement. Its weaknesses are the ones the authors admit: the design is tied to one chip generation and one building size; dual-plane removes the hash at tier 2 but the 60-way choice at tier 1 still needs a host-side system (RePaC plus the disjoint-path search) that the paper only sketches; and the whole structure assumes a scheduler that places pipeline traffic across pods and everything else inside. Compared with Meta's RoCE paper, which attacks the same low-entropy problem with more flows and a smarter hash, HPN attacks it with wiring. Compared with the rail-only proposal, HPN is the conservative answer, trading scale per pod for MoE and multi-tenant generality. Alibaba's later work (SimAI at NSDI 2025 and collective-communication papers at SIGCOMM 2025) takes HPN's structure as given and optimizes on top of it.

HPN 是已發表的 LLM 訓練拓樸中最完整的一個，其強處在於每個選擇都有機群量測支撐。弱點正是作者承認的那些：設計綁定於一個晶片世代與一種建築尺寸；dual-plane 拿掉第二層的雜湊，但第一層 60 選 1 的選擇仍需要一個論文只勾勒了輪廓的主機端系統（RePaC 加不相交路徑搜尋）；整個結構假設排程器會把管線流量放在跨 pod、其餘全放在 pod 內。相較於 Meta 的 RoCE 論文用更多的流與更聰明的雜湊對付同一個低熵問題，HPN 用接線對付它。相較於 rail-only 提案，HPN 是保守的答案，用每個 pod 的規模換取對 MoE 與多租戶的通用性。阿里後續的工作（NSDI 2025 的 SimAI、SIGCOMM 2025 的集體通訊論文）把 HPN 的結構視為既定，在其上做最佳化。

Key numbers關鍵數字

Terms used in this paper本篇用到的術語

Hash polarization

When the same few flows are hashed to the same links at successive switch tiers, so a collision at one tier repeats at the next.

同樣的少數流在連續的交換器層被雜湊到相同鏈路，使一層的碰撞在下一層重複。

Dual-ToR (non-stacked)

Each NIC's two ports connect to two independent rack switches with no link or shared state between them; the host sends its ARP announcements to both.

每張 NIC 的兩個埠接到兩台之間沒有鏈路也沒有共享狀態的獨立機櫃交換器；主機各自向兩台送出 ARP 通告。

Segment

HPN's tier 1: 16 ToRs and 136 hosts, 1,024 active GPUs plus 64 backup.

HPN 的第一層：16 台 ToR 與 136 台主機，1,024 顆使用中 GPU 加 64 顆備援。

Pod

HPN's tier 2: 15 segments joined by two planes of 60 aggregation switches each; 15,000 GPUs.

HPN 的第二層：15 個 segment 由兩個各 60 台聚合交換器的平面連接；15,000 顆 GPU。

Dual-plane

Splitting the aggregation layer so that each ToR of a pair reaches only one plane; a flow's path is fixed once it enters a ToR.

把聚合層切開，使每對 ToR 中的每一台只到達一個平面；流進入 ToR 後路徑即固定。

Rail-optimized

GPU $k$ of every host wires to the same ToR pair; cross-rail traffic detours over NVLink inside the host first.

每台主機的 GPU $k$ 接到同一對 ToR；跨 rail 流量先在主機內經 NVLink 繞路。

RePaC

Alibaba's tool that reproduces each switch's exact hash result, letting a host pick five-tuples that follow disjoint paths.

阿里能重現每台交換器精確雜湊結果的工具，讓主機挑出走不相交路徑的五元組。

INT probe

An in-band network telemetry packet that records switch and port IDs at each hop; HPN uses it to detect miswiring.

記錄每一跳交換器與埠 ID 的帶內網路遙測封包；HPN 用它偵測接錯線。