RDMA over Ethernet for Distributed AI Training at Meta ScaleRDMA over Ethernet for Distributed AI Training at Meta Scale

Reading plan (45 minutes)閱讀計畫（45 分鐘）

This page is written so that you can understand the paper without opening it. If you do open it, read in this order: §2.3–2.4 (what the traffic looks like), §3 (hardware), §4 (routing, the heart of the paper), §5 (transport), §6 (experience). Skip §7 (related work).

這一頁的寫法讓你不打開論文也能理解它。如果你要打開論文，建議順序：§2.3–2.4（流量長什麼樣）、§3（硬體）、§4（路由，論文的核心）、§5（傳輸層）、§6（經驗）。§7（相關工作）可略過。

1. The situation Meta was in1. Meta 當時的處境

By 2020, Meta's production models (ranking, recommendation, content understanding, and later LLMs such as Llama) no longer fit on one server. Training had to be distributed across many servers, which means the gradients of every step had to be exchanged between GPUs over a network. The existing data-center network carried web traffic: millions of small TCP flows that average out smoothly. Training traffic is the opposite. Meta had two choices for the GPU network: InfiniBand, the usual choice in high-performance computing, or Ethernet with RDMA (RoCEv2). They chose RoCEv2 for an operational reason: it uses the same Clos topology, switches, monitoring tools and engineers that Meta already had, while still offering the RDMA "verbs" interface that the collective library (NCCL) expects. The first clusters were small: a few racks of GPUs connected in a star to one big switch, running RoCEv1, which is not routable across IP subnets. These could not grow or survive a switch failure, so Meta moved to a routed Clos fabric with RoCEv2, which carries RDMA inside UDP/IP packets.

到 2020 年，Meta 的生產模型（排序、推薦、內容理解，後來的 Llama 等 LLM）已經放不進單一台伺服器。訓練必須分散到許多伺服器上，也就是每一步的梯度都得透過網路在 GPU 之間交換。既有的資料中心網路承載的是網頁流量：數百萬條小 TCP 流，平均起來很平滑。訓練流量恰好相反。GPU 網路 Meta 有兩個選擇：高效能運算常用的 InfiniBand，或 Ethernet 上的 RDMA（RoCEv2）。他們基於營運理由選了 RoCEv2：它用的是 Meta 已經有的 Clos 拓樸、交換器、監控工具與工程師，同時仍提供集體通訊函式庫（NCCL）所需的 RDMA「verbs」介面。最早的叢集很小：幾個機櫃的 GPU 以星狀接到一台大交換器，跑的是無法跨 IP 子網路由的 RoCEv1。這種架構無法成長、也撐不過交換器故障，所以 Meta 改為路由式 Clos 網路加 RoCEv2——它把 RDMA 包在 UDP/IP 封包裡。

2. What the traffic looks like2. 流量長什麼樣

The authors sampled about 30,000 training jobs from one quarter of 2023 and looked at which collective operations they use. Models trained with classic data parallelism use AllReduce (average the gradients) and AlltoAll (exchange embedding lookups in recommendation models). Models trained with fully sharded data parallelism (FSDP) use AllGather and ReduceScatter instead. Nearly all jobs use a multiple of 8 GPUs, because a server holds 8. Table 2 of the paper gives the most important measurement: on 128 GPUs, an AllReduce opens about 4 RDMA connections (queue pairs, QPs) per GPU, and an AlltoAll about 15. A web server would have thousands. The authors name three properties that follow from this. Low entropy: the packet headers that a switch hashes (source and destination IP, source and destination UDP port, protocol) take only a handful of distinct values. Burstiness: a flow is either sending at full line rate or silent, switching on and off at millisecond scale, because all GPUs reach the communication phase of a step at the same moment. Elephant flows: each burst can fill a 400 Gb/s NIC by itself. Every design decision in the paper is a reaction to these three words.

作者抽樣了 2023 年某一季約 30,000 個訓練任務，看它們使用哪些集體操作。用傳統資料平行的模型使用 AllReduce（平均梯度）與 AlltoAll（推薦模型中交換 embedding 查找）；用完全分片資料平行（FSDP）的模型則改用 AllGather 與 ReduceScatter。幾乎所有任務的 GPU 數都是 8 的倍數，因為一台伺服器裝 8 顆。論文 Table 2 給出最重要的量測：在 128 顆 GPU 上，一個 AllReduce 每顆 GPU 約開 4 條 RDMA 連線（queue pair，QP），AlltoAll 約 15 條。一台網頁伺服器會有數千條。作者由此歸納三個特性。低熵：交換器拿來雜湊的封包標頭（來源與目的 IP、來源與目的 UDP 埠、協定）只有少數幾種值。突發：一條流要嘛以滿線速發送、要嘛沉默，以毫秒尺度開關，因為所有 GPU 同時到達一步的通訊階段。大象流：每一次突發本身就能塞滿一張 400 Gb/s 的 NIC。論文中每一個設計決策都是對這三個詞的反應。

3. The hardware and the topology3. 硬體與拓樸

The server. Meta's training server is called Grand Teton. It holds 8 GPUs joined by NVSwitch, so any GPU can talk to any other inside the server at high speed, and it has 8 RDMA NICs, one per GPU. Each GPU therefore has its own 400 Gb/s path into the network. The server also has front-end NICs for storage and management, but those are a different network.

伺服器。Meta 的訓練伺服器叫 Grand Teton。它裝 8 顆由 NVSwitch 連接的 GPU，伺服器內任何 GPU 都能高速與其他 GPU 通訊，另有 8 張 RDMA NIC、每顆 GPU 一張。因此每顆 GPU 都有自己 400 Gb/s 的網路路徑。伺服器也有供儲存與管理用的前端 NIC，但那是另一個網路。

Two networks. Every training rack connects to two independent networks. The front end (FE) is the ordinary data-center fabric: rack switches, fabric switches, and the storage warehouse that feeds training data, receives checkpoints and collects logs. The back end (BE) is the dedicated RoCE fabric that carries only GPU-to-GPU traffic. Separating them means that a burst of checkpoint writes can never pause a collective, and that the back end can be upgraded and tuned on its own schedule.

兩個網路。每個訓練機櫃接到兩個獨立的網路。前端（FE）是一般的資料中心網路：機櫃交換器、fabric 交換器，以及提供訓練資料、接收檢查點、蒐集日誌的儲存倉庫。後端（BE）是只承載 GPU 對 GPU 流量的專用 RoCE 網路。分開的好處是：一波檢查點寫入永遠不會讓集體通訊暫停，而且後端可以依自己的時程升級與調校。

The AI Zone. The back end is a two-stage Clos. The leaf is the rack training switch (RTSW), which connects to the GPUs over short copper cables. The spine is the cluster training switch (CTSW), a modular chassis with deep packet buffers that are divided statically among its ports. Every RTSW connects to every CTSW over single-mode fiber with 400G transceivers. Within an AI Zone the network is non-blocking: every GPU can send at full rate to any other GPU in the zone at the same time.

AI Zone。後端是一個兩層 Clos。leaf 是機櫃訓練交換器（RTSW），用短銅纜接 GPU。spine 是叢集訓練交換器（CTSW），是一台模組化機箱，有深的封包緩衝區並靜態分配給各埠。每台 RTSW 都以單模光纖與 400G 光模組接到每台 CTSW。在一個 AI Zone 內網路是非阻塞的：每顆 GPU 都能同時以全速送給 zone 內任何其他 GPU。

Beyond one zone. Large language models need more GPUs than one zone holds. Meta added an aggregator training switch (ATSW) layer that joins the CTSWs of several zones in one building. The links between zones are oversubscribed on purpose: there is less bandwidth across zones than inside them, and traffic across them is balanced with plain ECMP. To make this acceptable, Meta changed the job scheduler. It learns where each server sits in the topology and assigns the ranks of a job so that the cut between zones carries as little traffic as possible (a minimum-cut placement). In practice this means that the heavy collectives stay inside a zone and only lighter traffic crosses.

超出一個 zone。大型語言模型需要的 GPU 比一個 zone 多。Meta 加入一層聚合訓練交換器（ATSW），把同一棟建築內多個 zone 的 CTSW 連起來。zone 之間的鏈路刻意超額認購：跨 zone 的頻寬比 zone 內少，跨 zone 流量以一般 ECMP 平衡。為了讓這可以接受，Meta 修改了任務排程器。它學習每台伺服器在拓樸中的位置，並指派任務的 rank，使跨 zone 的切割承載最少流量（最小割配置）。實務上這表示重的集體通訊留在 zone 內，只有較輕的流量跨越。

4. Routing: five stages4. 路由：五個階段

Section 4 is the core of the paper. It is a history of how Meta routed training flows across the many equal paths of the Clos, and why each approach failed or was kept.

§4 是論文的核心。它是 Meta 如何在 Clos 的眾多等價路徑上為訓練流選路的歷史，以及每種方法為何失敗或被保留。

1. Stage 1: ECMP. The default. Each switch hashes the five-tuple of a packet and picks an uplink; all packets of a flow follow the same path. Because of low entropy, flows collided, and training ran well below the network's capacity. Meta rejected this quickly.第一階段：ECMP。預設做法。每台交換器對封包五元組做雜湊並挑一條上行；同一條流的所有封包走同一條路。由於低熵，流互相碰撞，訓練遠低於網路容量。Meta 很快就放棄它。
2. Stage 2: Path pinning. Instead of hashing, each flow was assigned a fixed path based on the destination "slice", the index of the RTSW downlink that leads to the destination host. If every rack were fully used by one job and no link ever failed, this spreads flows perfectly. In production neither condition held. A rack was often partly allocated, so only one of its two hosts used the uplinks, leaving some uplinks idle and others overloaded; the paper measured up to more than 30% slower training. And when an uplink or a CTSW failed, the affected flows fell back to ECMP and collided with the pinned flows.第二階段：路徑固定。不做雜湊，而是依目的地「slice」（通往目的主機的 RTSW 下行埠索引）為每條流指派固定路徑。如果每個機櫃都被同一個任務完整使用、而且鏈路從不故障，這能把流分得很完美。生產環境中兩個條件都不成立。機櫃常被部分分配，只有兩台主機之一在用上行鏈路，導致某些上行閒置、另一些過載；論文量到訓練最多慢了超過 30%。而當某條上行或某台 CTSW 故障，受影響的流會退回 ECMP，並與固定路徑的流碰撞。
3. Stage 3: Double the uplinks. The short-term cure was to give each RTSW twice as much uplink bandwidth as downlink bandwidth (1:2 under-subscription). Collisions still happened but no longer slowed anything, because every uplink had spare capacity. It worked, and it cost twice the network. The authors call it an expensive mitigation and kept looking.第三階段：上行加倍。短期解法是讓每台 RTSW 的上行頻寬是下行的兩倍（1:2 欠額認購）。碰撞仍然發生，但不再拖慢任何東西，因為每條上行都有餘裕。它有效，但網路成本加倍。作者稱之為昂貴的緩解，並繼續尋找。
4. Stage 4: Enhanced ECMP with QP scaling. Two changes together. On the switch, the hash now includes the RoCE destination queue-pair number, read from inside the packet with the switch ASIC's user-defined-field feature. In the collective library, "QP scaling" opens several queue pairs per connection so that one logical connection becomes several flows with different destination QPs. Meta tried two ways to use the extra QPs: split each message across them (which produces smaller messages and more ACKs) or post whole messages to them in round-robin order. Round-robin was better. The result was up to 40% faster AllReduce than plain ECMP. Ranking jobs used 4 QPs; LLM jobs used 16. In simulation, moving from 4 to 32 QPs improved the worst case from 20% to 52% of the theoretical optimum, but the average stayed about 40% above the optimum: hashing is still probabilistic.第四階段：Enhanced ECMP 加 QP scaling。兩個改動一起做。交換器端，雜湊現在包含 RoCE 的目的 queue-pair 編號，透過交換器 ASIC 的使用者自定義欄位功能從封包內讀出。集體通訊函式庫端，「QP scaling」為每條連線開數個 queue pair，讓一條邏輯連線變成數條目的 QP 不同的流。Meta 試了兩種用法：把每則訊息切到各 QP（訊息變小、ACK 變多），或把整則訊息輪流投到各 QP。輪流較好。結果是 AllReduce 比一般 ECMP 最多快 40%。排序任務用 4 個 QP；LLM 任務用 16 個。模擬中從 4 個增加到 32 個 QP，最差情況從理論最佳值的 20% 改善到 52%，但平均仍比最佳值差約 40%：雜湊終究是機率性的。
5. Stage 5: Centralized traffic engineering (TE). A controller that knows all flows and assigns each one a path, like a planner instead of a dice roll. On a 16-uplink test the utilization of links under E-ECMP swung between 40% and 90%, while TE held them near 80%. On 128 GPUs, TE beat E-ECMP by 5–10% for AllReduce and AlltoAll. Meta deployed TE inside AI Zones but not across the data center: the gain did not justify a controller that must react to every failure and job change at that scale. The paper's last routing idea is flowlet switching, where the switch may move a flow to another path during a natural gap in its packets; with a 256–512 µs gap threshold, reordering fell below one packet per second, and the authors expect it to match TE at lower operating cost.第五階段：集中式流量工程（TE）。一個知道所有流並為每一條指派路徑的控制器，像規劃者而不是擲骰子。在 16 條上行的測試中，E-ECMP 下鏈路使用率在 40% 到 90% 之間擺盪，TE 則維持在 80% 附近。在 128 顆 GPU 上，TE 的 AllReduce 與 AlltoAll 比 E-ECMP 快 5–10%。Meta 在 AI Zone 內部署了 TE，但沒有跨整個資料中心：在那個規模下，收益不值得維護一個必須對每次故障與任務變動做反應的控制器。論文最後的路由想法是 flowlet switching：交換器可以在一條流的封包出現自然空檔時把它移到另一條路徑；以 256–512 µs 的空檔門檻，亂序降到每秒不到一個封包，作者預期它能以更低的營運成本達到與 TE 相當的效果。

5. Transport: running 400G without DCQCN5. 傳輸層：在 400G 不用 DCQCN

Background. RoCE NICs recover from a lost packet with go-back-N: they resend the lost packet and everything after it. A small loss rate therefore destroys throughput, so the network must not drop packets. Ethernet achieves this with priority flow control (PFC): when a switch buffer fills past a threshold, the switch sends a pause frame upstream and the upstream port stops sending until told to resume. PFC is a blunt tool. It pauses whole ports, not individual flows, and the pauses spread backward hop by hop. The usual remedy is a congestion-control algorithm, DCQCN, that marks packets with ECN when a queue grows and makes the sender slow down before PFC is needed.

背景。RoCE NIC 以 go-back-N 從丟包恢復：重送遺失的封包以及其後的所有封包。因此很小的丟包率就會摧毀吞吐，網路必須不丟包。Ethernet 用優先權流量控制（PFC）做到這點：交換器緩衝超過門檻時向上游送出暫停幀，上游埠停止發送直到被告知恢復。PFC 是鈍器：它暫停整個埠而非單一條流，而且暫停會一跳一跳往回擴散。常見的補救是壅塞控制演算法 DCQCN，在佇列增長時用 ECN 標記封包，讓發送端在需要 PFC 之前就減速。

At 200G. Meta ran DCQCN with a deliberately relaxed ECN threshold of 5 MB. Tuning the thresholds tighter, as the DCQCN paper recommends, gained about 3% in AlltoAll tests but made PFC pauses two to three times worse, because the senders slowed down too much and the bursts piled up elsewhere. They kept the relaxed setting.

在 200G。Meta 以刻意放寬的 5 MB ECN 門檻跑 DCQCN。依 DCQCN 論文建議把門檻調緊，AlltoAll 測試只多了約 3%，PFC 暫停卻惡化兩到三倍，因為發送端減速過頭、突發堆到別處。他們保留了寬鬆的設定。

At 400G. The NIC firmware's DCQCN implementation changed, introduced bugs in counting congestion notifications, and the default settings reduced performance. Meta made the unusual decision to run without DCQCN at all. Their 400G clusters then ran for more than a year on PFC alone, and the paper reports that in four years of operation, a CTSW spine has never had to send persistent pause frames down to the RTSWs. Two mechanisms make this safe. First, the deep CTSW buffers absorb transient bursts. Second, the collective library's own protocol limits how much data can be in flight.

在 400G。NIC 韌體的 DCQCN 實作改了版，在計數壅塞通知時引入錯誤，預設參數反而降低效能。Meta 做了一個不尋常的決定：完全不用 DCQCN。他們的 400G 叢集之後只靠 PFC 運行超過一年，論文並報告四年營運中 CTSW spine 從未需要持續向 RTSW 送出暫停幀。兩個機制讓這樣是安全的。第一，CTSW 的深緩衝吸收瞬間突發。第二，集體通訊函式庫自己的協定限制了在途資料量。

Receiver-driven traffic admission. NCCL moves data in two stages with a handshake. The sender's GPU threads copy data from the compute buffer into one of a fixed number of channel buffers. The sender's CPU proxy thread may post the RDMA write only after the receiver has sent a clear-to-send (CTS) message that names the size and destination. The receiver's GPU threads copy the data out of the channel buffer, and the CPU proxies on both sides recycle it; only then does the receiver send another CTS. Because the number of channel buffers is fixed, the amount of unacknowledged data on the network is bounded, and the receiver, not the sender, decides when data may enter the network. Meta tuned the number and size of the channel buffers per job size and collective type, and configured the switches to put CTS packets in a high-priority queue so that a tiny control message is never stuck behind a megabyte of data. In a 16-to-1 incast test (16 senders to one receiver), no PFC back-pressure was observed.

接收端驅動的流量許可。NCCL 以兩階段加交握來搬資料。發送端的 GPU 執行緒把資料從計算緩衝區複製到數量固定的 channel 緩衝區之一。發送端的 CPU 代理執行緒必須等接收端送出標明大小與目的地的 clear-to-send（CTS）訊息後，才能提交 RDMA 寫入。接收端的 GPU 執行緒把資料從 channel 緩衝區複製出去，兩側的 CPU 代理回收該緩衝區；之後接收端才送下一個 CTS。因為 channel 緩衝區數量固定，網路上未確認的資料量有上限，而且是由接收端而非發送端決定資料何時可以進入網路。Meta 針對任務規模與集體操作類型調校 channel 緩衝區的數量與大小，並設定交換器把 CTS 封包放進高優先權佇列，讓極小的控制訊息永遠不會被一 MB 的資料堵住。在 16 對 1 的 incast 測試（16 個發送端對 1 個接收端）中，沒有觀察到 PFC 反壓。

6. What production taught them6. 生產環境教會他們什麼

Co-tuning more than doubled performance. Out of the box, RoCE collectives ran at well under half of what the hardware could do. The largest single cause was round-trip time: NCCL's defaults assume a round trip under 10 µs, typical of one rack, but the measured production RTT across the fabric was about 22 µs. With that RTT, the default channel buffers were too small to keep a 400G link busy, so the link idled between handshakes. Increasing the buffers, prioritizing CTS (its 90th-percentile delay fell from 43 µs to 4 µs), and tuning the CTSW's per-port credits (up to 15% for small messages) together gave more than 2× in many cases.

共同調校讓效能提升超過兩倍。開箱狀態下，RoCE 集體通訊的效能遠低於硬體能力的一半。最大的單一原因是往返時間：NCCL 的預設假設往返低於 10 µs（典型的單機櫃值），但實測跨網路的生產 RTT 約 22 µs。在這個 RTT 下，預設的 channel 緩衝區太小、塞不滿 400G 鏈路，鏈路在交握之間閒置。加大緩衝區、讓 CTS 優先（其第 90 百分位延遲從 43 µs 降到 4 µs）、調校 CTSW 的每埠 credit（小訊息最多 +15%），合起來在許多情況下帶來超過 2 倍的提升。

Topology changes mattered as much as routing. Moving the RTSW from 1:1 to 1:2 uplink bandwidth gave a large improvement. The final design settled at 1:1.125, enough headroom to tolerate two failed uplinks without slowing a job, at much lower cost than 1:2.

拓樸的改動和路由一樣重要。把 RTSW 的上行從 1:1 改為 1:2 帶來很大改善。最終設計定在 1:1.125——足以容忍兩條上行鏈路故障而不拖慢任務，成本又遠低於 1:2。

Observability. The team monitors out-of-sequence packet counters (a sign of loss or reordering), link flaps, and local ACK timeouts on the NICs. A PFC watchdog fires if a port stays paused for 200 ms. Buffer alarms fire when a CTSW buffer passes 80%. Periodic reachability pings run between all hosts. Two troubleshooting stories show why: a CTSW software upgrade increased port-to-port latency and silently slowed training until the monitoring caught it; and an outdated assumption about SRAM buffer carving caused tail drops whenever more than half the buffer was in use.

可觀測性。團隊監看 NIC 上的亂序封包計數器（丟包或亂序的徵兆）、鏈路抖動與本地 ACK 逾時。某個埠持續暫停 200 ms 就觸發 PFC watchdog。CTSW 緩衝超過 80% 就觸發緩衝警報。所有主機之間定期執行可達性 ping。兩個除錯故事說明原因：一次 CTSW 軟體升級增加了埠對埠延遲，悄悄拖慢訓練，直到監控抓到；另一次是關於 SRAM 緩衝切割的過時假設，讓緩衝使用超過一半時就出現尾端丟包。

7. Critique and context7. 評論與脈絡

What the paper does not do: it does not fix the root cause of ECMP's weakness, which is that the network balances flows rather than packets. QP scaling and E-ECMP raise the odds of a good hash; they do not guarantee one. Running without DCQCN depends on deep spine buffers and on NCCL's handshake, so it may not transfer to shallow-buffer switches or to other communication libraries, which the authors acknowledge. The cross-zone oversubscription relies on the scheduler getting placement right. Later work moves in the directions the paper points to: Alibaba HPN removes the tier-2 hash structurally with dual-plane wiring; the Llama 3 paper confirms this same design running 24K GPUs with 16 flows per GPU pair and no DCQCN; and Meta's 2026 SIGCOMM paper replaces the transport for 100K+ GPU clusters with packet spraying and selective retransmission, which is what flowlet switching and the IRN paper foreshadow.

論文沒做到的事：它沒有解決 ECMP 弱點的根本原因——網路平衡的是流而不是封包。QP scaling 與 E-ECMP 提高了雜湊結果良好的機率，但不保證。不用 DCQCN 依賴深的 spine 緩衝與 NCCL 的交握，所以未必能移植到淺緩衝交換器或其他通訊函式庫，作者自己也承認。跨 zone 的超額認購則仰賴排程器把配置做對。後續工作朝論文指出的方向前進：Alibaba HPN 用 dual-plane 接線從結構上消除第二層雜湊；Llama 3 論文證實同樣的設計以每對 GPU 16 條流、不用 DCQCN 運行 24K 顆 GPU；Meta 2026 年的 SIGCOMM 論文則為十萬顆以上 GPU 的叢集換上封包噴灑與選擇性重傳的傳輸層——這正是 flowlet switching 與 IRN 論文所預示的。

Key numbers關鍵數字

Terms used in this paper本篇用到的術語

AI Zone

One non-blocking two-stage Clos of RTSWs and CTSWs; the unit inside which any GPU can reach any other at full rate.

由 RTSW 與 CTSW 組成的一個非阻塞兩層 Clos；在此單位內任何 GPU 都能以全速到達任何其他 GPU。

RTSW / CTSW / ATSW

Rack, cluster and aggregator training switches: leaf, spine, and the layer that joins zones.

機櫃、叢集與聚合訓練交換器：leaf、spine，以及連接各 zone 的那一層。

Queue pair (QP)

An RDMA connection (a send queue and a receive queue). Switches see each QP as a separate flow only if they hash on the QP number.

一條 RDMA 連線（一個發送佇列加一個接收佇列）。只有在交換器對 QP 編號做雜湊時，每個 QP 才會被視為不同的流。

QP scaling

Opening several QPs per logical connection in NCCL to create more flows for the switches to spread.

在 NCCL 中為每條邏輯連線開數個 QP，製造更多流讓交換器分散。

E-ECMP

Enhanced ECMP: hashing on the destination QP field in addition to the five-tuple.

Enhanced ECMP：除了五元組之外，也對目的 QP 欄位做雜湊。

Clear-to-send (CTS)

The receiver's message in NCCL's protocol that permits the sender to post an RDMA write into a specific channel buffer.

NCCL 協定中接收端送出的訊息，允許發送端把 RDMA 寫入提交到特定的 channel 緩衝區。

Flowlet

A burst of packets within a flow separated from the next burst by a gap; switches may re-route at the gap without reordering.

一條流中由空檔隔開的一段封包突發；交換器可以在空檔時改路徑而不造成亂序。