Guided reading 7 of 9

論文導讀 7 / 9

RDMA over Commodity Ethernet at ScaleRDMA over Commodity Ethernet at Scale

Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitendra Padhye, Marina Lipshteyn · ACM SIGCOMM 2016

≈ 40 min · self-contained digest≈ 40 分鐘 · 不必打開論文也能讀懂Primer chapters: 07對應入門頁章節：07Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

Eight years before the AI-cluster papers, Microsoft deployed RDMA over Converged Ethernet (RoCEv2) across Azure data centers to cut the latency and CPU cost of TCP at 40 Gb/s, and then wrote down everything that went wrong. RoCE needs a lossless network, which Ethernet provides with priority flow control (PFC): a switch whose buffer is filling tells the upstream port to stop. To make PFC work on a routed network, Microsoft moved the priority marking from the VLAN tag to the IP DSCP field. Then came the failures. A NIC that restarted a whole message on every loss drove goodput to zero at a 0.4% drop rate (fixed with go-back-N). Flooded packets created a buffer-dependency cycle across four switches and deadlocked the network (fixed by making such packets droppable). One broken NIC emitted pause frames forever and froze thousands of servers (fixed with watchdogs at 100 ms and 200 ms). A receiver whose address-translation cache missed became a source of back-pressure for everyone upstream (fixed with 2 MB pages). In production, RoCE cut 99th-percentile latency from 700 µs to 90 µs, but the paper's last question, "do we really need a lossless network?", is the one IRN and the modern transports answer.

在 AI 叢集論文之前八年，微軟在 Azure 資料中心部署了 RDMA over Converged Ethernet（RoCEv2），以降低 40 Gb/s 下 TCP 的延遲與 CPU 成本，然後把所有出錯的事寫下來。RoCE 需要無損網路，Ethernet 以優先權流量控制（PFC）提供：緩衝快滿的交換器告訴上游埠停止。為了讓 PFC 在路由式網路上運作，微軟把優先權標記從 VLAN 標籤移到 IP 的 DSCP 欄位。接著是故障。一張每次丟包都重頭傳整則訊息的 NIC，在 0.4% 丟包率下讓 goodput 歸零（以 go-back-N 修正）。被廣播氾濫的封包在四台交換器之間形成緩衝依賴迴圈，鎖死了網路（讓此類封包可丟棄來修正）。一張壞掉的 NIC 永無止境地發暫停幀，凍結了數千台伺服器（以 100 ms 與 200 ms 的 watchdog 修正）。位址轉換快取失誤的接收端成為所有上游的反壓來源（以 2 MB 分頁修正）。在生產環境中，RoCE 把第 99 百分位延遲從 700 µs 降到 90 µs，但論文最後的問題「我們真的需要無損網路嗎？」正是 IRN 與現代傳輸層所回答的。

Reading plan (40 minutes)閱讀計畫（40 分鐘）

§1–§2 give the motivation and the network; §3 the DSCP change; §4 is the heart, four failure stories, each with a trigger, a mechanism and a fix; §5 the production measurements; §6–§7 the lessons and the open questions. Every lossless-network decision in the Meta, Alibaba and ByteDance papers is a response to §4, so read it slowly.

§1–§2 是動機與網路；§3 是 DSCP 的改動；§4 是核心，四個故障故事，各有觸發、機制與修法；§5 是生產量測；§6–§7 是教訓與開放問題。Meta、阿里與字節跳動論文中每一個關於無損網路的決策都是對 §4 的回應，所以慢慢讀。

1. Why Microsoft wanted RDMA1. 微軟為何想要 RDMA

At 40 Gb/s, TCP was becoming expensive. Sending at line rate over 8 TCP connections used 6% of a 32-core server's CPU time and receiving used 12%; kernel processing added latency that could reach tens of milliseconds under load; and any packet drop forced a timeout or fast retransmit. RDMA moves data directly between application memory on two machines with the NIC doing the transport in hardware, so the CPU is bypassed and latency is a few microseconds. RDMA came from InfiniBand, but Microsoft could not use InfiniBand: there were no InfiniBand routers in production, so it could not scale across subnets, and it is not Ethernet-compatible. RoCEv2 encapsulates RDMA in Ethernet/IPv4/UDP, so it runs on the existing fabric. That fabric is a multi-tier Clos: 20–40 servers per ToR, tens of ToRs per leaf layer, tens to hundreds of spines, mostly 40 Gb/s links. The ToR and leaf switches have shallow buffers of 9 MB or 12 MB, which will matter.

在 40 Gb/s，TCP 變得昂貴。用 8 條 TCP 連線以線速發送，耗掉一台 32 核伺服器 6% 的 CPU 時間，接收則耗 12%；核心處理增加的延遲在負載下可達數十毫秒；任何丟包都迫使逾時或快速重傳。RDMA 由 NIC 以硬體執行傳輸層，在兩台機器的應用程式記憶體之間直接搬資料，繞過 CPU，延遲只有幾微秒。RDMA 源自 InfiniBand，但微軟不能用 InfiniBand：生產環境沒有 InfiniBand 路由器，無法跨子網擴展，也與 Ethernet 不相容。RoCEv2 把 RDMA 封裝在 Ethernet/IPv4/UDP 中，所以能在既有網路上運行。那個網路是多層 Clos：每台 ToR 20–40 台伺服器、每個 leaf 層數十台 ToR、數十到數百台 spine，大多是 40 Gb/s 鏈路。ToR 與 leaf 交換器的緩衝很淺，只有 9 MB 或 12 MB，這一點之後很重要。

Why lossless. The RDMA transport in a RoCE NIC was designed for InfiniBand, where losses were rare, so it recovers from loss very inefficiently. The network must therefore not drop packets. Ethernet's tool for this is priority flow control: when a switch's ingress buffer for a priority class passes a threshold, it sends a PAUSE frame to the upstream port, which stops sending that class until a RESUME. Because packets already in flight keep arriving after the pause is sent, the switch must reserve headroom for each lossless class on each port. With 9–12 MB of buffer, Microsoft could afford headroom for only two lossless classes out of the eight PFC allows: one for latency-sensitive traffic and one for bulk transfers. Congestion control (DCQCN, by overlapping authors) runs on top to slow senders before PFC is needed.

為何無損。RoCE NIC 中的 RDMA 傳輸層是為 InfiniBand 設計的，那裡丟包罕見，所以它從丟包恢復的效率極差。網路因此不能丟包。Ethernet 的工具是優先權流量控制：當交換器某個優先權類別的入口緩衝超過門檻，它向上游埠送出 PAUSE 幀，上游停止發送該類別直到收到 RESUME。因為暫停送出後在途封包仍會持續抵達，交換器必須為每個埠的每個無損類別保留headroom。只有 9–12 MB 的緩衝，微軟只負擔得起 PFC 允許的八個類別中兩個無損類別的 headroom：一個給延遲敏感流量、一個給大量傳輸。壅塞控制（DCQCN，作者有重疊）在其上運行，在需要 PFC 之前先讓發送端減速。

2. Move the priority from VLAN to DSCP2. 把優先權從 VLAN 移到 DSCP

The PFC standard identifies the priority class of a packet by the 3-bit PCP field in the VLAN tag. Two things broke in Microsoft's network. First, servers are provisioned by PXE boot over the network before they have any VLAN configuration, and a switch port in trunk mode (required for VLAN tags) will not exchange untagged packets with a booting NIC, so provisioning failed. Second, the network forwards at layer 3, and there is no standard way to carry a VLAN priority across an IP subnet boundary. The fix is small: put the priority in the DSCP field of the IP header, mapping DSCP value $i$ to PFC priority $i$ . Switches classify on DSCP, server ports stay in access mode, and the marking survives routing. The PAUSE frame itself is unchanged in either design. This one-paragraph change is what made RoCEv2 deployable on an existing routed fabric at all.

PFC 標準以 VLAN 標籤中 3 位元的 PCP 欄位識別封包的優先權類別。在微軟的網路中有兩件事壞了。第一，伺服器在有任何 VLAN 設定之前就透過網路 PXE 開機配置，而 trunk 模式的交換器埠（VLAN 標籤所需）不會與正在開機的 NIC 交換無標籤封包，配置因此失敗。第二，網路在第三層轉送，沒有標準方法把 VLAN 優先權帶過 IP 子網邊界。修法很小：把優先權放進 IP 標頭的 DSCP 欄位，把 DSCP 值 $i$ 對應到 PFC 優先權 $i$ 。交換器依 DSCP 分類，伺服器埠維持 access 模式，標記在路由後仍保留。兩種設計中 PAUSE 幀本身不變。這一段的改動就是讓 RoCEv2 能在既有路由式網路上部署的關鍵。

3. Four failures3. 四個故障

3.1 Transport livelock3.1 傳輸層活鎖

Even a lossless network loses packets occasionally, from bit errors or bugs. The team tested resilience by dropping a deterministic 1 in 256 packets (0.4%). Application goodput fell to zero while the link ran at full rate. The NIC implemented go-back-0: on a NACK it restarted the whole message from its first packet, so a message longer than 256 packets could never complete. The fix is go-back-N: restart from the first lost packet. This still wastes up to one round-trip's worth of bandwidth ( $\mathrm{RTT} \times C$ ) per loss, but it cannot livelock. Note the historical irony: the go-back-N that the IRN paper later criticizes as wasteful was itself the improvement over go-back-0.

即使無損網路也偶爾會因位元錯誤或錯誤程式丟包。團隊以確定性地每 256 個封包丟 1 個（0.4%）來測試韌性。應用程式 goodput 降到零，鏈路卻以全速運行。NIC 實作的是 go-back-0：收到 NACK 就從訊息的第一個封包重頭開始，所以超過 256 個封包的訊息永遠無法完成。修法是 go-back-N：從第一個遺失的封包重新開始。這每次丟包仍會浪費最多一個往返的頻寬（ $\mathrm{RTT} \times C$ ），但不會活鎖。注意歷史的諷刺：IRN 論文後來批評為浪費的 go-back-N，本身就是對 go-back-0 的改進。

3.2 PFC deadlock3.2 PFC 死鎖

PFC can deadlock: if switch A is paused by B, B by C, and C by A, nothing ever drains. A Clos network with up-down routing (packets go up toward the spine, then down, never up again) should be deadlock-free because buffer dependencies cannot form a cycle. The deadlock Microsoft found came from a corner case in layer-2 forwarding. A switch keeps an ARP table (IP → MAC, 4-hour timeout) and a MAC table (MAC → port, 5-minute timeout). When a destination's ARP entry existed but its MAC entry had expired, as happens when a server dies, the switch flooded the packet out of every port, including ports that violate up-down routing. Flooded lossless packets to dead servers filled queues on two ToRs (T0, T1) and two leaves (La, Lb) in a loop; each switch paused the next, and the cycle froze. The fix is not to prevent flooding, which is impossible, but to make such packets droppable: lossless packets whose ARP entry is incomplete are dropped instead of flooded, so they can never hold a lossless buffer. The general lesson is that broadcast and multicast must never be put in a lossless class.

PFC 可能死鎖：如果交換器 A 被 B 暫停、B 被 C 暫停、C 又被 A 暫停，就永遠排不空。採上下路由（封包往 spine 上去、再下來、永不再上去）的 Clos 網路應該不會死鎖，因為緩衝依賴無法形成迴圈。微軟發現的死鎖來自第二層轉送的邊角情況。交換器有 ARP 表（IP → MAC，4 小時逾時）與 MAC 表（MAC → 埠，5 分鐘逾時）。當目的地的 ARP 項目存在但 MAC 項目已過期——伺服器死掉時就會這樣——交換器把封包氾濫到每個埠，包括違反上下路由的埠。送往死伺服器的無損氾濫封包在兩台 ToR（T0、T1）與兩台 leaf（La、Lb）之間形成迴圈填滿佇列；每台暫停下一台，迴圈就凍住了。修法不是防止氾濫（不可能），而是讓這類封包可丟棄：ARP 項目不完整的無損封包被丟棄而非氾濫，所以永遠不會占住無損緩衝。一般性的教訓是：廣播與多播絕不能放進無損類別。

Drawn from §4.2: flooding sends lossless packets along links that up-down routing never uses, closing a cycle of buffer dependencies among two ToRs and two leaves.依 §4.2 繪製：氾濫讓無損封包走上下路由從不使用的鏈路，在兩台 ToR 與兩台 leaf 之間形成緩衝依賴的迴圈。Drawn for this primer from Guo et al., SIGCOMM 2016.

3.3 The NIC pause storm3.3 NIC 暫停風暴

A bug in one NIC's receive pipeline left it unable to process incoming packets. Its receive buffer filled, so it did what PFC says: it sent PAUSE frames, continuously. Its ToR paused that port, the ToR's own buffers filled, the ToR paused its leaves, and the pause spread across the network. One broken card stopped thousands of servers. This is the pause storm, and it is a property of PFC itself: back-pressure propagates without limit. Two watchdogs contain it. In the NIC, a micro-controller stops generating pauses if the receive pipeline has been stalled for 100 ms while pausing. In the ToR, if a server-facing port keeps receiving pauses and cannot drain, the switch disables lossless mode on that port and drops that server's lossless packets, re-enabling after the pauses have stopped for 200 ms. Meta's 2024 paper reports a PFC watchdog at the same 200 ms.

一張 NIC 接收管線的錯誤讓它無法處理進來的封包。它的接收緩衝填滿，於是它做了 PFC 規定的事：持續送出 PAUSE 幀。它的 ToR 暫停了那個埠，ToR 自己的緩衝填滿，ToR 再暫停它的 leaf，暫停就這樣擴散到整個網路。一張壞卡讓數千台伺服器停擺。這就是暫停風暴，而它是 PFC 本身的性質：反壓無限制地傳播。兩個 watchdog 把它圍住。在 NIC 端，微控制器在接收管線暫停期間停滯達 100 ms 時停止產生暫停。在 ToR 端，若面向伺服器的埠持續收到暫停且無法排空，交換器在該埠關閉無損模式並丟棄該伺服器的無損封包，在暫停停止 200 ms 後重新啟用。Meta 2024 年的論文報告的 PFC watchdog 用的正是同樣的 200 ms。

3.4 The slow receiver3.4 慢接收端

A NIC translates the virtual addresses in RDMA requests to physical addresses using a memory translation table (MTT), of which it caches only 2K entries. With 4 KB pages, 2K entries cover 8 MB. A receiver whose working set is larger misses the cache constantly, each miss slows the receive pipeline, the buffer fills, and the NIC sends pauses. A perfectly healthy server with a large memory footprint becomes a source of back-pressure for everyone upstream of it. Two mitigations: 2 MB huge pages, so the same 2K entries cover far more memory and misses are rare; and dynamic buffer sharing on the switch so that one slow port does not consume the headroom of others. The general form of this lesson is uncomfortable: in a lossless network, any slow endpoint is a network problem.

NIC 用記憶體轉換表（MTT）把 RDMA 請求中的虛擬位址轉成物理位址，但它只快取 2K 筆項目。用 4 KB 分頁，2K 筆涵蓋 8 MB。工作集更大的接收端不斷快取失誤，每次失誤拖慢接收管線，緩衝填滿，NIC 送出暫停。一台完全健康、但記憶體足跡大的伺服器，變成所有上游的反壓來源。兩個緩解：2 MB 大分頁，讓同樣 2K 筆項目涵蓋多得多的記憶體、失誤變罕見；交換器上的動態緩衝共享，讓一個慢埠不會吃掉其他埠的 headroom。這個教訓的一般形式令人不安：在無損網路中，任何慢的端點都是網路問題。

4. What production showed4. 生產環境顯示了什麼

Latency. In a production service, RDMA's 99th-percentile latency was 90 µs against 700 µs for TCP, with TCP spiking to several milliseconds; RDMA's 99.9th percentile was about 200 µs. Throughput. A test across two podsets of 576 servers each (24 ToRs and 4 leaves per podset, 6:1 oversubscription at the ToR and 3:2 at the leaf) reached 3.0 Tb/s aggregate with zero drops and near-zero CPU, but that was 60% of the 5.12 Tb/s capacity. The ceiling was ECMP hash collisions, not PFC: the same low-entropy problem the AI papers meet eight years later. Latency under load. In a testbed, when bulk traffic started, RDMA's 99th percentile rose from 50 µs to 400 µs and the 99.9th from 80 µs to 800 µs, while TCP in a separate queue was unaffected. The authors' verdict: RDMA is not a panacea for achieving both low latency and high throughput. Incidents. Besides the NIC pause storm, a switch buffer misconfiguration (a dynamic-threshold parameter changed from 1/16 to 1/64 on a new switch model) caused PFC to propagate to thousands of servers until it was reverted. Rollout was staged: lab, test clusters, ToR-only PFC, then podset, then spine.

延遲。在一個生產服務中，RDMA 的第 99 百分位延遲是 90 µs，TCP 是 700 µs，且 TCP 會飆到數毫秒；RDMA 的第 99.9 百分位約 200 µs。吞吐。跨兩個各 576 台伺服器的 podset（每個 podset 24 台 ToR、4 台 leaf，ToR 6:1、leaf 3:2 超額認購）的測試達到 3.0 Tb/s 總吞吐、零丟包、CPU 近乎零，但那只是 5.12 Tb/s 容量的 60%。上限來自 ECMP 雜湊碰撞而非 PFC：正是 AI 論文八年後遇到的同一個低熵問題。負載下的延遲。在測試環境中，大量流量啟動時 RDMA 的第 99 百分位從 50 µs 升到 400 µs、第 99.9 百分位從 80 µs 升到 800 µs，而在獨立佇列中的 TCP 不受影響。作者的結論：RDMA 不是同時達到低延遲與高吞吐的萬靈丹。事故。除了 NIC 暫停風暴，一次交換器緩衝設定錯誤（新交換器型號上動態門檻參數從 1/16 改為 1/64）讓 PFC 傳播到數千台伺服器，直到改回來。部署是分階段的：實驗室、測試叢集、僅 ToR 的 PFC、再 podset、最後 spine。

5. Lessons and open questions5. 教訓與開放問題

The authors' own lessons: deadlock, livelock and pause propagation all happened despite careful design analysis, so "a design works in theory" is not enough; and most bugs were in NICs, not switches, because the NIC holds the complex RDMA transport with limited resources. Their open questions set the research agenda for the next decade. PFC's 300 m reach confines RoCEv2 to one spine layer. ECMP's 60% ceiling suggests per-packet or multipath routing adapted to lossless networks. Topologies that break the up-down property may deadlock in ways not yet understood. Low latency and high throughput together remain unsolved. And, given programmable NIC hardware, "do we really need a lossless network to get the benefits of RoCEv2?" IRN answered that one two years later.

作者自己的教訓：儘管經過仔細的設計分析，死鎖、活鎖與暫停傳播都發生了，所以「設計在理論上可行」並不夠；而且大多數錯誤在 NIC 而非交換器，因為 NIC 以有限資源承載複雜的 RDMA 傳輸層。他們的開放問題為接下來十年設定了研究議程。PFC 的 300 m 距離限制把 RoCEv2 侷限在單一 spine 層。ECMP 的 60% 上限暗示需要為無損網路改造的逐封包或多路徑路由。打破上下性質的拓樸可能以尚未理解的方式死鎖。低延遲與高吞吐兼得仍未解決。還有，既然有可程式化的 NIC 硬體，「我們真的需要無損網路才能得到 RoCEv2 的好處嗎？」IRN 兩年後回答了這一題。

6. How the AI papers answer this one6. AI 論文如何回應這一篇

Meta's PFC watchdog, buffer alarms and receiver-driven admission are direct descendants of §4: contain pause propagation, watch buffers, and bound bytes in flight so pauses are rare. HPN's decision to keep storage off the back-end network avoids exactly the mixed-traffic and slow-receiver problems described here. Meta's deep-buffer spines are the opposite of Microsoft's 9–12 MB switches, and that difference is why Meta could afford to drop DCQCN while Microsoft could not. IRN takes the closing question literally: with selective retransmission instead of go-back-N and a cap on bytes in flight, PFC becomes unnecessary, and the deadlock, storm and slow-receiver problems disappear with it. The modern transports (Ultra Ethernet, Meta's 2026 design) follow IRN's path.

Meta 的 PFC watchdog、緩衝警報與接收端驅動的許可是 §4 的直系後代：圍堵暫停傳播、監看緩衝、限制在途位元組讓暫停罕見。HPN 把儲存留在後端網路之外的決定，正好避開這裡描述的混合流量與慢接收端問題。Meta 的深緩衝 spine 與微軟 9–12 MB 的交換器相反，這個差異就是 Meta 能放棄 DCQCN 而微軟不能的原因。IRN 把結尾的問題當真：用選擇性重傳取代 go-back-N、再加上在途位元組上限，PFC 就不再必要，死鎖、風暴與慢接收端的問題也隨之消失。現代的傳輸層（Ultra Ethernet、Meta 2026 年的設計）走的是 IRN 的路。

Key numbers關鍵數字

Number	數字	Meaning
6% / 12%	CPU time to send / receive at 40 Gb/s over TCP on a 32-core server	32 核伺服器以 TCP 在 40 Gb/s 發送 / 接收的 CPU 時間
9 or 12 MB / 2 of 8	Shallow ToR and leaf buffers / lossless priorities affordable with headroom	淺的 ToR 與 leaf 緩衝 / 扣除 headroom 後負擔得起的無損優先權數
1/256 → 0	Injected drop rate that drove goodput to zero under go-back-0	注入的丟包率，在 go-back-0 下讓 goodput 歸零
4 h vs 5 min	ARP vs. MAC table timeouts whose mismatch caused flooding and the deadlock	ARP 與 MAC 表逾時，其不一致造成氾濫與死鎖
100 ms / 200 ms	NIC watchdog stall threshold / ToR watchdog re-enable delay	NIC watchdog 停滯門檻 / ToR watchdog 重新啟用延遲
$2\text{K} \times 4\,\text{KB} = 8\,\text{MB}$	MTT cache coverage behind the slow-receiver pauses; fixed with 2 MB pages	慢接收端暫停背後的 MTT 快取涵蓋範圍；以 2 MB 分頁修正
90 vs 700 µs	99th-percentile latency, RDMA vs. TCP, in production	生產環境第 99 百分位延遲，RDMA vs. TCP
3.0 Tb/s / 60%	Aggregate throughput over 1,152 servers / share of capacity, limited by ECMP	1,152 台伺服器的總吞吐 / 占容量比例，受 ECMP 限制
50 → 400 µs	RDMA 99th-percentile latency before and during bulk traffic	RDMA 第 99 百分位延遲在大量流量前與期間
1/16 → 1/64	Buffer-threshold misconfiguration that spread PFC to thousands of servers	把 PFC 擴散到數千台伺服器的緩衝門檻設定錯誤
300 m	PFC reach that confines RoCEv2 to a single spine layer	把 RoCEv2 侷限在單一 spine 層的 PFC 距離

Terms used in this paper本篇用到的術語

PFC: Priority flow control: a per-port, per-priority PAUSE/RESUME mechanism that makes an Ethernet class lossless.; 優先權流量控制：逐埠、逐優先權的 PAUSE/RESUME 機制，使某個 Ethernet 類別無損。
Headroom: Buffer reserved per port and priority for the packets still arriving after a PAUSE is sent.; 每個埠與優先權為送出 PAUSE 後仍在抵達的封包所保留的緩衝。
DSCP: The 6-bit Differentiated Services field in the IP header, used here to carry the PFC priority across routed hops.; IP 標頭中 6 位元的 Differentiated Services 欄位，此處用來把 PFC 優先權帶過路由跳。
Go-back-0 / go-back-N: Loss recovery that restarts a message from its first packet / from the first lost packet.; 從訊息第一個封包 / 從第一個遺失封包重新開始的丟包恢復。
Up-down routing: Clos routing in which a packet goes up toward the spine and then only down; it prevents cyclic buffer dependencies.; 封包往 spine 上去後只往下的 Clos 路由；它防止緩衝依賴形成迴圈。
Pause storm: Unbounded propagation of PAUSE frames from one stuck port through the fabric.; PAUSE 幀從一個卡住的埠無限制地傳播到整個網路。
MTT: Memory translation table: the NIC's virtual-to-physical address map, cached in 2K entries.; 記憶體轉換表：NIC 的虛擬到物理位址對應，快取 2K 筆。