Design research: Astral and Pegasus設計研究：Astral 與 Pegasus

The four designs on this page were published in 2023 and 2024. Two newer papers from one operator, Tencent, test this page's conclusions in production: Astral (SIGCOMM 2025, with Nanjing University) for a dedicated training fabric, and Pegasus (SIGCOMM 2026, with Tsinghua University) for a multi-tenant bare-metal cloud. Read together, they show this page's thesis at work: one company chose differently when its constraints changed.

本頁的四種設計發表於 2023 與 2024 年。同一個營運者騰訊的兩篇較新論文，在生產環境中檢驗本頁的結論：Astral（SIGCOMM 2025，與南京大學合作）處理專用的訓練網路，Pegasus（SIGCOMM 2026，與清華大學合作）處理多租戶的裸機雲。兩篇合讀，正好展示本頁的論點：同一家公司在限制條件改變時做了不同的選擇。

What Astral attacks. Section 13.1 and Section 13.7 treat the top tier as the place to save money: Meta oversubscribes 1:7 between pods and HPN 15:1 above its pod. Step 1 of the decision procedure treats an oversubscribed top tier from 15K to about 24K GPUs as the cheap choice. Astral shows that it is not the only one. By the paper's account, Tencent runs 128K GPUs in two Pods with no oversubscribed tier, and the design extends to 524,288 GPUs, the $k^3/4$ limit of a three-tier fat tree of 128-port switches (Section 6.1).

Astral 處理的問題。第 13.1 節與第 13.7 節把頂層當成省錢的地方：Meta 在 pod 之間以 1:7 超額認購，HPN 在 pod 之上用 15:1。決策程序的第 1 步把 15K 到約 24K 顆 GPU 時超額認購的頂層當成便宜的選擇。Astral 證明它不是唯一的選擇。依論文所述，騰訊以兩個 Pod 運行 128K 顆 GPU，沒有任何一層超額認購，而且設計可延伸到 524,288 顆 GPU，也就是 128 埠交換器三層 fat tree 的 $k^3/4$ 上限（第 6.1 節）。

How it differs, axis by axis. (a) Reachability: Astral's second tier joins one rail per group of switches, which is the option HPN priced at more than 120K GPUs per pod and rejected. A full-bandwidth Core then restores the cross-rail paths that rail-only gives up. So Astral is neither any-to-any at tier 2 nor rail-only; it is a fifth pattern (Figure R1). (b) Buffers: shallow 51.2 Tb/s single chips at every tier, with DCQCN and PFC, like HPN and unlike Meta's deep-buffer spines. (c) Load balance: the host picks UDP source ports with the switch's own hash, and a controller reassigns them when 5-second ECN counters show congestion. This sits between Meta's answer (more flows) and HPN's answer (no choice at tier 2). (d) Redundancy: HPN's dual-ToR, every NIC on two ToRs.

逐軸比較差異。(a) 可達性：Astral 的第二層讓每組交換器只連一條 rail，正是 HPN 算過每個 pod 可超過 120K 顆 GPU、卻否決的選項。接著由全頻寬的 Core 補回 rail-only 放棄的跨 rail 路徑。所以 Astral 既不是第二層任意對任意，也不是 rail-only；它是第五種模式（圖 R1）。(b) 緩衝：每一層都用淺緩衝的 51.2 Tb/s 單晶片，加 DCQCN 與 PFC，與 HPN 相同、與 Meta 的深緩衝 spine 不同。(c) 負載平衡：主機以交換器自己的雜湊挑選 UDP 來源埠，當每 5 秒的 ECN 計數顯示壅塞時，由控制器重新指派。這介於 Meta 的答案（更多的流）與 HPN 的答案（第二層沒有選擇）之間。(d) 備援：HPN 的 dual-ToR，每張 NIC 接兩台 ToR。

What Pegasus attacks. Pegasus rents GPU servers to tenants who run their own collective libraries, so the co-design of Section 13.7 (a tuned NCCL, placement, a controller) is not available. Tencent's previous cluster used a controller that re-paths congested flows; in the cloud it needed about 95 s to relieve congestion, too slow for inference. Pegasus moves the transport into its own NIC: per-packet spraying over plain switch ECMP, receiver credits that bound the bytes in flight (a hardware form of Meta's clear-to-send, Section 13.6), and receiver-side ordering for write-with-immediate. One DPU manages all eight NICs of a server; by the paper's arithmetic, one DPU per NIC would add 6.5–13.7% to server power. This is a fourth answer on axis (c): fix load balance in the NIC. Pegasus does not describe its topology, so Figure R1 cannot place it.

Pegasus 處理的問題。Pegasus 把 GPU 伺服器租給自己執行集體通訊函式庫的租戶，所以第 13.7 節的共同設計（調校過的 NCCL、配置、控制器）都用不上。騰訊先前的叢集用控制器替壅塞的流重新選路；在雲端環境中它要約 95 s 才能紓解壅塞，對推論來說太慢。Pegasus 把傳輸層搬進自家的 NIC：在一般交換器 ECMP 上逐封包噴灑、以接收端 credit 限制在途位元組（Meta clear-to-send 的硬體版，第 13.6 節），以及在接收端處理 write-with-immediate 的順序。一台 DPU 管理伺服器的全部八張 NIC；依論文的算術，每張 NIC 配一台 DPU 會讓伺服器功耗增加 6.5–13.7%。這是軸 (c) 的第四種答案：在 NIC 裡修負載平衡。Pegasus 沒有描述它的拓樸，所以圖 R1 無法擺放它。

What they agree on. MoE all-to-all is the traffic that decides the design. The rail-only paper measured MoE all-to-all 8.2–11.2% slower without a spine; HPN rejected rail-only for MoE and shared hosts; Astral measured all-to-all up to 52% worse with tier 3 at 16:1 (training up to 3%); Pegasus built its credits for all-to-all incast. Placement matters too: HPN gained 14.9% by moving a job from 19 segments to 3, and Astral measured 19%–37% lower all-to-all when 1K GPUs were spread over 32 Pods. Pegasus is the case where placement is not the operator's to choose, which is why it fixes balance in the NIC. And every operator automates diagnosis: Astral located fail-stop and fail-hang failures up to 12× and 25× faster, in minutes.

它們一致的地方。MoE 的 all-to-all 是決定設計的流量。Rail-only 論文量到沒有 spine 時 MoE all-to-all 慢 8.2–11.2%；HPN 因 MoE 與共享主機而否決 rail-only；Astral 量到第三層為 16:1 時 all-to-all 最多差 52%（訓練最多 3%）；Pegasus 的 credit 正是為 all-to-all 的 incast 而設計。配置也很重要：HPN 把一個任務從 19 個 segment 移到 3 個而快了 14.9%，Astral 量到 1K 顆 GPU 分散到 32 個 Pod 時 all-to-all 低 19%–37%。Pegasus 則是營運者無法決定配置的情況，所以它改在 NIC 裡修平衡。每個營運者也都把診斷自動化：Astral 讓 fail-stop 與 fail-hang 的定位最多快 12 倍與 25 倍，降到數分鐘。

What to question. Astral's network tests do not say where they ran, its 3% training loss has no stated model or scale, and the paper does not price 1:1 against 15:1 or 1:7; at full build its servers alone would draw about 0.5 GW. Its appendix measured pipeline traffic across data centers hundreds of km apart: no loss at 8:1 oversubscription, 4.6% at 32:1. Pegasus's production gains compare two different clusters, the older one and the Pegasus one, so the NIC, the load balancing and the congestion control all change at once; this is not a controlled test. Its percentages match $1 - \text{old}/\text{new}$ in the paper's figure, so +53.2% means that the old cluster reached about 47% of Pegasus's throughput, a ratio of about 2.14×. Pegasus runs 8K GPUs against a target of more than 100K. Finally, Astral's descriptions of MegaScale's top tier and of Meta's cluster scale differ from this site's guided readings; the matrix in Section 13.1 follows the readings.

值得質疑的地方。Astral 的網路測試沒有說明在哪裡進行，3% 的訓練損失沒有說明模型或規模，論文也沒有把 1:1 與 15:1 或 1:7 的價格相比；全部建滿時光伺服器就約 0.5 GW。它的附錄量測了相距數百公里的資料中心之間的管線流量：8:1 超額認購時無損失，32:1 時 4.6%。Pegasus 的生產增益比較的是兩個不同的叢集（舊叢集與 Pegasus 叢集），NIC、負載平衡與壅塞控制同時改變；這不是受控測試。從論文的圖看，它的百分比相當於 $1 - \text{舊}/\text{新}$，所以 +53.2% 表示舊叢集只達到 Pegasus 吞吐的約 47%，也就是約 2.14 倍的差距。Pegasus 目前運行 8K 顆 GPU，目標則超過 100K。最後，Astral 對 MegaScale 頂層與 Meta 叢集規模的描述與本站導讀不同；第 13.1 節的矩陣以導讀為準。