Transport research: congestion control for collectives, and reliability without PFC傳輸層研究：為集體操作設計的壅塞控制，以及不靠 PFC 的可靠性
Odin, DCP, Falcon and Stellar's multipath transport.Odin、DCP、Falcon，以及 Stellar 的多路徑傳輸。
This page assumes Chapters 1–8. Every term it uses is defined there; the glossary on the contents page links to each definition. The papers here are recent and most of their results come from small testbeds or simulation, so read each number with its label.
本頁假設你已讀過第 1–8 章。這裡用到的每個名詞都在那些章節中定義過；目錄頁的詞彙表連到每個定義。這些論文都很新，多數結果來自小型測試平台或模擬，所以請連同標籤一起讀每個數字。
Terms from later chapters used here (1)本頁用到的後面章節名詞（1 個）
- Simulation, packet-level simulation模擬、封包層級模擬 Ch. 12 Running a model of the workload and network in software; a packet-level simulator follows every packet through every queue.以軟體執行工作負載與網路的模型；封包層級模擬器追蹤每個封包經過每個佇列。
This page built RoCE's recipe from its parts: go-back-N makes loss expensive, PFC prevents loss at a price, and DCQCN keeps PFC rare if the switch marks before it pauses. Four recent papers test that recipe at AI scale. Each keeps a different part of it, as the map shows.
本頁把 RoCE 的配方拆開來看：go-back-N 讓丟包代價高昂，PFC 以代價換來不丟包，而只要交換器先標記、後暫停，DCQCN 就能讓 PFC 很少觸發。四篇近期論文在 AI 規模下檢驗這份配方，各自保留了其中不同的部分，如下圖所示。
Congestion control for collective patterns. Odin (SIGCOMM 2026, Central South University and ByteDance) keeps PFC and single paths and changes only congestion control, for the MoE all-to-all in which every switch port is busy at once. Many small queues then add up to a large total buffer occupancy, which lowers the dynamic pause threshold of Section 7.3 and Section 7.4 for every port together. With hundreds of QPs on one NIC, each QP's calculated rate is far above its real share, so DCQCN's cuts act late. Odin adds a 1-bit buffer flag from the switch and a NIC-wide oversubscription ratio to an HPCC-style controller.
為集體通訊模式設計的壅塞控制。Odin（SIGCOMM 2026，中南大學與字節跳動）保留 PFC 與單一路徑，只改壅塞控制，目標是讓交換器每個埠同時忙碌的 MoE all-to-all。這時許多小佇列加總成很大的總緩衝占用，同時壓低每個埠在第 7.3 節與第 7.4 節的動態暫停門檻。一張 NIC 上有數百個 QP 時，每個 QP 的計算速率遠高於真實份額，所以 DCQCN 的降速生效得很晚。Odin 在 HPCC 式的控制器上加入交換器送來的 1 位元緩衝旗標，以及整張 NIC 的超額訂閱比值。
Reliability without PFC. Two papers continue Section 7.6, where IRN showed that selective repair makes PFC unnecessary. DCP (SIGCOMM 2025, HKUST and Huawei) fixes what IRN still gets wrong: a selective-repeat NIC mistakes reordering for loss, and a lost last packet or a lost retransmission waits for a timeout. DCP's switches cut an overflowing packet to its 57-byte header and forward the header in a separate queue that cannot overflow, so every loss arrives as an exact notice and the sender resends only that packet. Falcon (SIGCOMM 2025, Google) needs nothing from the switches. It is a complete hardware transport for general-purpose Ethernet: delay-based congestion control with separate fabric and NIC windows, a SACK bitmap with time-based loss detection (RACK-TLP), and a few flows per connection on different paths.
不靠 PFC 的可靠性。兩篇論文延續第 7.6 節：IRN 已說明選擇性修復讓 PFC 變得不必要。DCP（SIGCOMM 2025，香港科技大學與華為）修正 IRN 仍然做錯的地方：選擇性重傳的 NIC 會把亂序誤認為丟包，而遺失最後一個封包或遺失重傳封包時只能等逾時。DCP 的交換器把溢出的封包修剪成 57 位元組的標頭，經由一個不會溢出的獨立佇列轉送，所以每次丟包都以精確的通知到達，發送端只重送那一個封包。Falcon（SIGCOMM 2025，Google）不需要交換器提供任何東西。它是給一般用途 Ethernet 的完整硬體傳輸層：以延遲為基礎、網路與 NIC 視窗分開的壅塞控制，搭配時間式丟包偵測（RACK-TLP）的 SACK 位圖，以及每條連線分走不同路徑的少數幾條流。
Multipath in production. Stellar (SIGCOMM 2025, Alibaba Cloud) is mostly about RDMA virtualization for secure containers; its section 7 is the transport. Alibaba's own FPGA RNIC sprays each connection over 128 paths without per-path state, shares one congestion window among them, and resends a lost packet on another path after a 250 µs timeout. It also fills a gap that Section 8.5 states: a production paper with numbers for per-packet spreading of training traffic.
生產環境中的多路徑。Stellar（SIGCOMM 2025，阿里雲）主要談安全容器的 RDMA 虛擬化；它的第 7 節才是傳輸層。阿里自製的 FPGA RNIC 把每條連線噴灑到 128 條路徑上，不保留逐路徑狀態，所有路徑共用一個壅塞視窗，並在 250 µs 逾時後從另一條路徑重送遺失的封包。它也補上了第 8.5 節指出的缺口：一篇有數字的生產環境論文，談訓練流量的逐封包分散。
| Paper論文 | Problem, mechanism and measured result問題、機制與實測結果 |
|---|---|
| Odin SIGCOMM 2026 CSU, ByteDance中南大學、字節跳動 | Problem: in MoE all-to-all, many small queues fill a shared buffer and hundreds of QPs per NIC inflate rates. Mechanism: a switch TBO bit turns off additive increase, a NIC ratio scales rates, compensation fills idle links. Measured: 8 nodes, ConnectX-5 at 40 Gb/s, one Tofino 1, CC in DPDK: TBO −78.7% against DCQCN, throughput 1.75× HPCC's. All larger results are NS-3.問題：MoE all-to-all 中許多小佇列填滿共享緩衝，每張 NIC 上數百個 QP 讓速率膨脹。機制：交換器的 TBO 位元關閉加性增加，NIC 比值縮放速率，補償填滿閒置鏈路。實測：8 個節點、ConnectX-5 跑 40 Gb/s、一台 Tofino 1、CC 在 DPDK：TBO 比 DCQCN 低 78.7%，吞吐量是 HPCC 的 1.75 倍。更大規模的結果都是 NS-3。 |
| DCP SIGCOMM 2025 HKUST, Huawei香港科技大學、華為 | Problem: selective-repeat NICs mistake reordering for loss and need timeouts for tail losses. Mechanism: switches trim to 57-byte headers in a lossless control queue; the sender resends exactly the lost packet; counters replace bitmaps. Measured: 2 P4 switches, 16 FPGA NICs at 100 Gb/s: loss recovery 1.6–72× better than ConnectX-5 at 0.01–5% loss; AllToAll −42% and AllReduce −33%, but with adaptive routing for DCP and ECMP for ConnectX-5.問題：選擇性重傳的 NIC 把亂序誤認為丟包，尾端丟包要靠逾時。機制：交換器修剪成 57 位元組標頭、放進無損的控制佇列；發送端精確重送遺失的封包；以計數器取代位圖。實測：2 台 P4 交換器、16 張 100 Gb/s 的 FPGA NIC：在 0.01–5% 丟包下，丟包恢復比 ConnectX-5 好 1.6–72 倍；AllToAll −42%、AllReduce −33%，但 DCP 用自適應路由，ConnectX-5 用 ECMP。 |
| Falcon SIGCOMM 2025 | Problem: RoCE needs a lossless, special-purpose network, while general Ethernet drops and reorders. Mechanism: a hardware transport with delay-based CC (fabric and NIC windows), a SACK bitmap with RACK-TLP, a few flows per connection, Push/Pull transactions. Measured: 32 machines, Intel E2100 200 Gb/s IPU against ConnectX-7 RoCE: 5–65% more goodput under random loss; RoCE go-back-N up to 20× lower under reordering; 120 Mops. The multipath results are simulation.問題：RoCE 需要無損的專用網路，一般 Ethernet 卻會丟包與亂序。機制：硬體傳輸層：以延遲為基礎的 CC（網路與 NIC 兩個視窗）、搭配 RACK-TLP 的 SACK 位圖、每條連線少數幾條流、Push/Pull 交易。實測：32 台機器，Intel E2100 200 Gb/s IPU 對 ConnectX-7 RoCE：隨機丟包下 goodput 高 5–65%；亂序下 RoCE 的 go-back-N 最多低 20 倍；120 Mops。多路徑的結果是模擬。 |
| Stellar SIGCOMM 2025 Alibaba Cloud阿里雲 | Problem: single-path RDMA hashes unevenly on HPN, and SR-IOV limits secure containers. Mechanism: an FPGA 400G RNIC sprays each connection over 128 paths with one shared window and a 250 µs timeout. Measured: 1,024 GPUs against ConnectX-7: training +6% on average, up to 14%, with random rank placement, and +0.72% with locality-aware placement. A 512-GPU AllReduce reaches the RNIC's full 50 GB/s with 128 paths.問題：單一路徑的 RDMA 在 HPN 上雜湊不均，SR-IOV 又限制安全容器。機制：FPGA 400G RNIC 把每條連線噴灑到 128 條路徑，共用一個視窗，逾時 250 µs。實測：1,024 顆 GPU 對 ConnectX-7：隨機排列 rank 時訓練平均 +6%、最多 +14%，依區域性排列時 +0.72%。512 顆 GPU 的 AllReduce 以 128 條路徑達到 RNIC 滿載的 50 GB/s。 |
Where they agree. Go-back-N is the root problem: selective repair with out-of-order placement is the minimum for multipath. One ECN bit per 50 µs is too little signal, so the papers use delay from hardware timestamps (Falcon), INT plus a switch-wide buffer bit (Odin), or ECN plus RTT (Stellar). And NIC state is the binding limit: DCP replaces bitmaps with per-message counters, Stellar shares one window across 128 paths (per-path state would allow only 4), and Falcon carves its NIC buffers. Where they differ. Stellar sprays blindly; Falcon rejects spraying for networks whose paths differ; DCP relies on adaptive routing in the switches; Odin argues that path choice hardly helps all-to-all on a non-oversubscribed Clos, where every downlink is full. Loss detection differs too: an explicit header (DCP), a timer near one RTT (Falcon), a 250 µs timeout (Stellar).
共識。go-back-N 是根本問題：選擇性修復加上亂序放置，是多路徑的最低要求。每 50 µs 一個 ECN 位元的訊號太少，所以這些論文改用硬體時間戳的延遲（Falcon）、INT 加上交換器層級的緩衝位元（Odin），或 ECN 加 RTT（Stellar）。NIC 的狀態是真正的限制：DCP 以每則訊息的計數器取代位圖，Stellar 讓 128 條路徑共用一個視窗（逐路徑狀態只容得下 4 條），Falcon 則切分 NIC 的緩衝。分歧。Stellar 盲目噴灑；Falcon 認為路徑不同質的網路不該噴灑；DCP 依賴交換器的自適應路由；Odin 主張在無超額訂閱的 Clos 上，每條下行鏈路都已滿載，路徑選擇幾乎幫不了 all-to-all。丟包偵測也不同：明確的標頭（DCP）、約一個 RTT 的計時（Falcon）、250 µs 的逾時（Stellar）。
Open questions. Only Stellar measures AI training at scale, and Falcon runs no GPU job. Odin alone targets all-to-all, and its evidence at scale is simulation. Congestion control across many paths is still open, and so is coexistence with go-back-N traffic that still needs PFC.
未解的問題。只有 Stellar 量測了大規模的 AI 訓練，Falcon 沒有跑任何 GPU 任務。只有 Odin 針對 all-to-all，而它大規模的證據是模擬。跨多條路徑的壅塞控制仍未解決，與仍需 PFC 的 go-back-N 流量共存也是。