Research digest 1 of 4 · Chapter 7: Transport

研究導讀 1 / 4 · 第 7 章：傳輸層

Odin: Rethinking Congestion Control under All-to-All TrafficOdin：重新思考 All-to-All 流量下的壅塞控制

SIGCOMM 2026Wanchun Jiang, Yujie Hu, Kai Wang, …, Jianxi Ye, Huichen Dai (Central South University, ByteDance) · paper論文

After reading, you can explain why MoE all-to-all traffic defeats per-port congestion control on lossless RoCE, and how Odin's buffer flag, NIC ratio and compensation step change an HPCC-style controller.讀完後，你能說明 MoE 的 all-to-all 流量為何讓無損 RoCE 上逐埠的壅塞控制失效，以及 Odin 的緩衝旗標、NIC 比值與補償步驟如何改變 HPCC 式的控制器。

≈ 11 min · 2 interactive panels · 5-question quiz≈ 11 分鐘 · 2 個互動面板 · 5 題測驗Chapter 7 ↗第 7 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

In a Mixture-of-Experts (MoE) all-to-all (A2A), every node sends to every other node at once. Odin, from Central South University and ByteDance, shows how this defeats per-port congestion control (CC) on lossless RoCE. Many small queues add up to a large total buffer occupancy (TBO), which lowers the dynamic PFC threshold, so a switch can pause before it marks. Hundreds of queue pairs (QPs) share one NIC, so each QP's calculated rate is far above its real share, and its feedback comes only about every 50 µs. Odin adds three rules to an HPCC-style controller: a 1-bit TBO flag turns off additive increase, a NIC oversubscription ratio scales rates toward the NIC's capacity, and a compensation step fills idle links. On an 8-node, 40 Gb/s testbed, Odin cut TBO by 78.7% against DCQCN and raised throughput 1.75× against HPCC. Everything larger is simulation.

在 Mixture-of-Experts（MoE）的 all-to-all（A2A）中，每個節點同時送資料給其他每個節點。來自中南大學與字節跳動的 Odin 說明這如何擊敗無損 RoCE 上逐埠的壅塞控制（CC）。許多小佇列加總成很大的總緩衝占用（TBO），壓低了動態 PFC 門檻，所以交換器可能先暫停、後標記。數百個 queue pair（QP）共用一張 NIC，每個 QP 的計算速率遠高於它的真實份額，而它的回饋大約每 50 µs 才來一次。Odin 在 HPCC 式的控制器上加了三條規則：1 位元的 TBO 旗標關閉加性增加；NIC 超額訂閱比值把速率縮向 NIC 的容量；補償步驟填滿閒置的鏈路。在 8 個節點、40 Gb/s 的測試平台上，Odin 的 TBO 比 DCQCN 低 78.7%，吞吐量是 HPCC 的 1.75 倍。更大規模的結果全是模擬。

Where this paper sits這篇論文的位置

Section 7.4 showed that DCQCN works only if a switch marks before it pauses. The pause point is dynamic: the free shared buffer divided by a parameter, here $\beta_{\mathrm{DT}}$ , because Odin uses $\beta$ for its AI step. The derivation assumed one hot port. Section 5 showed ByteDance's MegaScale hit by PFC pauses under large all-to-all; Odin, with ByteDance co-authors, offers a likely reason, although the paper itself cites MegaScale only for its multi-rail topology.

第 7.4 節說明，DCQCN 只有在交換器先標記、後暫停時才有效。暫停點是動態的：剩餘共享緩衝除以一個參數，本頁寫成 $\beta_{\mathrm{DT}}$ ，因為 Odin 把 $\beta$ 用作 AI 步長。那個推導假設只有一個熱埠。第 5 節提到字節跳動的 MegaScale 在大規模 all-to-all 下飽受 PFC 暫停之苦；Odin 有字節跳動的共同作者，它提供了一個可能的原因，不過論文本身引用 MegaScale 只是為了它的多 rail 拓樸。

HPCC in brief. Each switch writes in-band telemetry (INT) into data packets: queue length, bytes sent, link rate. The ACK carries it back, and the sender computes the normalized load of the busiest hop,

HPCC 簡述。每台交換器把帶內遙測（INT）寫進資料封包：佇列長度、已送位元組、鏈路速率。ACK 把它帶回，發送端計算最忙那一跳的正規化負載，

\psi = \frac{\mu \tau + q}{b \, \tau}

with link rate $b$ , base RTT $\tau$ , queue $q$ and throughput $\mu$ . On every ACK it scales its rate toward a target utilization $\eta = 0.95$ (MIMD) and adds a small additive increase (AI) $\beta$ for fairness:

其中 $b$ 是鏈路速率、 $\tau$ 是基本 RTT、 $q$ 是佇列、 $\mu$ 是吞吐量。每個 ACK 到達時，發送端把速率朝目標使用率 $\eta = 0.95$ 乘法調整（MIMD），再加上一個小的加性增加（AI） $\beta$ 以求公平：

R_i \leftarrow R_i \cdot \frac{\eta}{\psi} + \beta

Odin keeps this rule (its Eq. 3). What was missing before: every signal on the site measures one port or one path, so none sees a sum of many small queues, or a rate that the NIC cannot send.

Odin 保留這條規則（它的 Eq. 3）。之前缺少的是：本站的每一種訊號都只量一個埠或一條路徑，所以看不到許多小佇列的總和，也看不到 NIC 送不出去的速率。

The problem: what all-to-all does to a switch and a NIC問題：all-to-all 對交換器與 NIC 做了什麼

Unlike incast, an A2A loads every ingress and every egress port at once (Chapter 3). The authors report large TBO in ByteDance's production clusters but give no production numbers. Almost all their evidence is NS-3 simulation, so this page gives the mechanisms and the paper's arithmetic, not the simulated curves.

與 incast 不同，A2A 同時壓滿每一個入口埠與出口埠（第 3 章）。作者表示 A2A 在字節跳動的生產叢集中造成很大的 TBO，但沒有給任何生產環境的數字。他們的證據幾乎全是 NS-3 模擬，所以本頁講的是機制與論文的算術，而不是模擬的曲線。

1. Many small queues fill one shared buffer1. 許多小佇列填滿同一個共享緩衝

CC starts every QP at line rate, so the first packets collide. Appendix A takes the worst case: in each time slice, $N-1$ ingress ports send one packet each to one egress port, which forwards one; the next slice hits the next port. The queues end at 0, 0, 1, …, $N-2$ packets (figure below), so the total, in packets, is

CC 讓每個 QP 從線速開始，所以第一批封包會相撞。Appendix A 取最差情況：每個時間片， $N-1$ 個入口埠各送一個封包到同一個出口埠，而它只轉送一個；下一個時間片換下一個埠。各佇列最後停在 0、0、1、…、 $N-2$ 個封包（見下圖），所以以封包計的總量是

\mathrm{TBO} = \frac{(N-1)(N-2)}{2}

This is the paper's Eq. 7: each queue is small, but the sum grows with $N^2$ . Our arithmetic from it:

這是論文的 Eq. 7：每個佇列都不大，總和卻隨 $N^2$ 成長。以下是我們依它所做的算術：

Question (our arithmetic from Eq. 7)問題（依 Eq. 7 的我方算術）	Calculation算式	Result結果
Packets queued after one round, 128 ports128 個埠在一輪之後排隊的封包	$127 \times 126 \div 2$	$8{,}001$ packets個封包
Bytes at 4 KB per packet每個封包 4 KB 時的位元組	$8{,}001 \times 4{,}096$	$\approx 32.8$ MB40% of the 82 MB shared buffer82 MB 共享緩衝的 40%
Mean queue per egress port每個出口埠的平均佇列	$32.8\,\text{MB} \div 128$	$\approx 256$ KB
Same formula for the 8-port testbed同一公式用在 8 埠測試平台	$7 \times 6 \div 2$	$21$ packets個封包

The worst case in Appendix A for 8 ports, in which port

k

takes its 7-to-1 burst in slice

k

, then drains one packet per slice, so every queue stays small while the total reaches 21 packets, as the analysis of Appendix A gives.依 Appendix A 的最差情況畫出 8 個埠，第

k

個埠在第

k

個時間片承受 7 對 1 的突發，之後每片排出一個封包，因此每個佇列都很小，而合計為 21 個封包，與 Appendix A 的分析一致。Drawn for this digest from Jiang et al., SIGCOMM 2026, Appendix A (Eq. 7).

Why the sum matters. Under the dynamic threshold, every port pauses when its ingress counter exceeds $(B - \mathrm{TBO})/\beta_{\mathrm{DT}}$ for a shared buffer $B$ , while ECN marks at a fixed $K$ per egress queue. In our simplified model, every port holds $q = \mathrm{TBO}/N$ at ingress and at egress, so a port pauses when

總和為何要緊。在動態門檻下，共享緩衝為 $B$ 時，每個埠的入口計數器一超過 $(B - \mathrm{TBO})/\beta_{\mathrm{DT}}$ 就暫停；ECN 則在每個出口佇列固定的 $K$ 標記。在我們的簡化模型中，每個埠在入口與出口都存著 $q = \mathrm{TBO}/N$ ，所以一個埠在下列條件下暫停：

q \gt \frac{B - Nq}{\beta_{\mathrm{DT}}} \iff q \gt q_{\text{pause}} = \frac{B}{N + \beta_{\mathrm{DT}}}

So an ECN threshold above it never marks first. For 82 MB, 128 ports and Chapter 7's value of 8 (Odin does not give its own), that is about 603 KB per port, near 77 MB of TBO. The paper's simulation saw PFC on almost every port and ECN on only a few.

所以高於它的 ECN 門檻永遠不會先標記。以 82 MB、128 個埠，加上第 7 章的值 8（Odin 沒有給自己的值），這是每埠約 603 KB，TBO 接近 77 MB。論文的模擬看到 PFC 幾乎在每個埠觸發，ECN 只在少數幾個埠標記。

Panel 1面板 1

Many warm ports: does the switch pause before it marks?許多溫熱的埠：交換器會先暫停、後標記嗎？

Ports

N

埠數

N

Packet size封包大小

Shared buffer

B

共享緩衝

B

Dynamic threshold

\beta_{\mathrm{DT}}

動態門檻

\beta_{\mathrm{DT}}

ECN threshold

K

ECN 門檻

K

TBO as a share of

B

TBO 占

B

的比例

Start-up TBO, Eq. 7

起始 TBO，Eq. 7

Queue per port, TBO / N

每埠佇列，TBO / N

Pause point now

目前的暫停點

PFC fires at a queue of

PFC 觸發時的佇列

First signal as TBO grows

TBO 增長時先出現的訊號

Odin's flag threshold, B / 16

Odin 的旗標門檻，B / 16

Model (ours): every port holds the same queue $q = \mathrm{TBO}/N$ at ingress and at egress; ECN marks when $q \gt K$ ; PFC pauses when $q \gt (B - \mathrm{TBO})/\beta_{\mathrm{DT}}$ . Defaults are the paper's simulated switch (128 ports, 82 MB shared, 4 KB packets) with its lowered DCQCN threshold of 100 KB; $\beta_{\mathrm{DT}} = 8$ is Chapter 7's value, an assumption. The start-up TBO is Eq. 7, the idealized worst case. Raise $K$ above about 600 KB, and the switch pauses before any mark. Then raise $N$ : the start-up TBO grows with $N^2$ and passes $B$ near 200 ports.

模型（我方）：每個埠在入口與出口都存著相同的佇列 $q = \mathrm{TBO}/N$ ； $q \gt K$ 時 ECN 標記； $q \gt (B - \mathrm{TBO})/\beta_{\mathrm{DT}}$ 時 PFC 暫停。預設值是論文模擬的交換器（128 埠、82 MB 共享緩衝、4 KB 封包），加上論文調低後的 DCQCN 門檻 100 KB； $\beta_{\mathrm{DT}} = 8$ 取自第 7 章，是一個假設。起始 TBO 用 Eq. 7，即理想化的最差情況。把 $K$ 調到約 600 KB 以上，交換器就會在任何標記之前暫停。再把 $N$ 調高：起始 TBO 隨 $N^2$ 成長，在 200 埠左右超過 $B$ 。

A low $K$ is not enough. With DCQCN's thresholds lowered to 50 and 100 KB, the simulated TBO stayed large, because DCQCN raises its rate on a timer between sparse marks.

低的 $K$ 並不夠。DCQCN 的門檻調低到 50 與 100 KB 後，模擬中的 TBO 仍然很大，因為 DCQCN 在稀疏的標記之間依計時器自行提高速率。

2. Rate inflation: the calculated rate is not the real rate2. 速率膨脹：計算速率不是真實速率

With 5 QPs to each of 126 peers, a NIC carries 630 QPs. Each starts at a calculated rate of 400 Gb/s, but they share 400 Gb/s: rate inflation. A cut does nothing until the calculated rate falls below the real share, and each QP gets feedback only once per packet. From §2.2.2:

對 126 個對端各開 5 個 QP，一張 NIC 就承載 630 個 QP。每個 QP 以 400 Gb/s 的計算速率開始，但它們共用 400 Gb/s：這就是速率膨脹。在計算速率低於真實份額之前，降速毫無作用，而每個 QP 每送一個封包才得到一次回饋。依 §2.2.2：

Question (630 QPs on a 400 Gb/s NIC)問題（400 Gb/s NIC 上有 630 個 QP）	Calculation算式	Result結果
Real share of one QP一個 QP 的真實份額	$400 \div 630$	$\approx 0.635$ Gb/scalculated: 400 Gb/s計算速率：400 Gb/s
Time between packets, and so between ACKs封包間隔，也就是 ACK 間隔	$4{,}096 \times 8 \div (635 \times 10^{6})$	$\approx 52$ µsthe paper rounds to 50 µs; base RTT 8 µs論文取整為 50 µs；基本 RTT 8 µs
DCQCN halvings to get below the shareDCQCN 要減半幾次才低於份額	$\lceil \log_2 630 \rceil$	$10$ at most one per feedback每次回饋最多一次
Time for those halvings這些減半所需的時間	$10 \times 52$ µs	$\approx 0.5$ msabout 65 base RTTs約 65 個基本 RTT

The arithmetic above plotted on a log scale shows that even with one halving per ACK, the calculated rate (red) needs ten ACKs to reach the real share (green), while ACKs (upper ticks) arrive six base RTTs (lower ticks) apart.把上面的算術畫成對數尺度的圖，可見即使每個 ACK 都減半一次，計算速率（紅）也要十個 ACK 才降到真實份額（綠），而 ACK（上排刻度）彼此相隔六個基本 RTT（下排刻度）。Drawn for this digest from §2.2.2 of Jiang et al., SIGCOMM 2026.

With one QP per pair (Appendix D.1), this effect is much weaker for DCQCN in simulation: the QP count drives it.

每對只用一個 QP 時（Appendix D.1），這個效應對 DCQCN 在模擬中明顯減弱：驅動它的是 QP 數量。

3. Pre-emption at the NIC: a rate cut that makes TBO grow3. NIC 上的搶占：讓 TBO 增長的降速

Suppose that one egress port, SPort, is congested. Its QPs cut their calculated rates (①), but their real rates fall only once the calculated rates drop below the share (②). The NIC bandwidth that they free goes at once to sibling QPs that are still inflated (③). SPort drains (④) while the other queues grow (⑤). Each ingress port still receives 400 Gb/s, so TBO grows faster just when CC acts.

假設一個出口埠 SPort 壅塞了。往它的 QP 調降計算速率（①），但真實速率要等計算速率低於份額才會下降（②）。它們釋出的 NIC 頻寬立刻被仍然膨脹的兄弟 QP 拿走（③）。SPort 排空（④），其他佇列卻在增長（⑤）。每個入口埠仍然收到 400 Gb/s，所以正好在 CC 動作的時候，TBO 長得更快。

A redrawing of Figure 3 of the paper, in which outlined bars are calculated rates and filled bars real rates, while the dark segments are the share that siblings take (③) and the queue growth that follows (⑤).重畫論文的 Figure 3，外框條是計算速率，實心條是真實速率，而深色區段是兄弟 QP 拿走的份額（③）與隨後的佇列增長（⑤）。Drawn for this digest from Jiang et al., SIGCOMM 2026, §2.2.2 and Figure 3.

4. Ingress–egress mismatch4. 入口與出口不匹配

Existing CC has what the paper calls Feature 1: when a QP finishes, no surviving QP slows down. A2A finishes fastest only if every ingress and egress link stays full, and Theorem 1 (Appendix B, from Hall's marriage theorem) shows that some finish orders make this impossible under Feature 1. So CC for A2A must watch the ingress side. Figure 4 of the paper shows all three problems.

現有的 CC 有論文所說的 Feature 1：一個 QP 結束時，沒有任何存活的 QP 會減速。A2A 只有在每條入口與出口鏈路都保持滿載時才完成得最快，而 Theorem 1（Appendix B，由 Hall 婚配定理推得）證明在 Feature 1 之下，某些結束順序讓這件事不可能發生。所以給 A2A 用的 CC 必須看入口端。論文的 Figure 4 畫出全部三個問題。

Figure 4 of the Odin paper: senders 1 to N feed a switch whose buffer holds a large total occupancy, with callouts for large TBO, interference on rate adjustment at the senders, and two kinds of ingress–egress mismatch — Figure 4 of the paper: ① large TBO in the shared buffer, ② interference among the QPs of one sender, ③ the two mismatches, idle ingress against full egress and idle egress against full ingress. A problem illustration, not data.論文的 Figure 4：① 共享緩衝中很大的 TBO；② 同一發送端各 QP 之間的干擾；③ 兩種不匹配：閒置入口遇上滿載出口，以及閒置出口遇上滿載入口。問題示意圖，不是數據。Screenshot of Jiang et al., SIGCOMM 2026, Figure 4 (p. 1249).

The design設計

Odin extends Eq. 3 on every ACK (the paper's Algorithm 1) with three inputs: per-hop INT, a 1-bit TBO flag from each switch, and one NIC-wide sum of calculated rates.

Odin 在每個 ACK 上擴充 Eq. 3（論文的 Algorithm 1），使用三個輸入：每一跳的 INT、每台交換器的 1 位元 TBO 旗標，以及整張 NIC 的計算速率總和。

Figure 6 of the Odin paper: the sender's basic MIMD plus AI rate adjustment, a TBO suppression block that sets the AI factor to zero, a NIC oversubscription block with proactive allocation and compensation, and a switch that sets the TBO signal in the INT header — Figure 6 of the paper. Start at the switch, which sets the flag in the INT header when TBO passes the threshold. At the sender, the red block turns AI off; the blue-green block compares $U_{\mathrm{NIC}}$ with $\psi$ to choose allocation or compensation. A design diagram.論文的 Figure 6。從交換器開始：TBO 超過門檻時，它在 INT 標頭中設定旗標。在發送端，紅色方塊關閉 AI；藍綠色方塊比較 $U_{\mathrm{NIC}}$ 與 $\psi$ ，選擇分配或補償。設計圖。Screenshot of Jiang et al., SIGCOMM 2026, Figure 6 (p. 1251).

1. TBO suppression: one bit from the switch1. 抑制 TBO：交換器送來的一個位元

Each switch counts its TBO and sets the flag in INT when it exceeds

每台交換器計算自己的 TBO，超過下列門檻時就在 INT 中設定旗標：

\mathrm{Th}_{\mathrm{TBO}} = \alpha_{\mathrm{TBO}} \cdot B

with $\alpha_{\mathrm{TBO}} = 1/16$ : 5.125 MB for 82 MB. If an ACK carries the flag and $\psi \lt 1.2$ , the sender sets $\beta = 0$ and AI stops. Above 1.2, a few hot ports more likely made the TBO, and MIMD with AI should clear them fairly; a flag with mild per-port load is the A2A signature. Without AI, a bottleneck's standing queue goes toward zero instead of growing with the flow count, so TBO drains, then hovers near the threshold as AI returns.

其中 $\alpha_{\mathrm{TBO}} = 1/16$ ：82 MB 時是 5.125 MB。若 ACK 帶著旗標且 $\psi \lt 1.2$ ，發送端就令 $\beta = 0$ ，AI 停止。高於 1.2 時，TBO 比較可能來自少數熱埠，應由帶 AI 的 MIMD 公平清除；有旗標而每埠負載輕微，才是 A2A 的特徵。沒有 AI 時，瓶頸的常駐佇列趨近於零，而不是隨流數增長，於是 TBO 排空，之後隨 AI 恢復而在門檻附近徘徊。

2. Proactive allocation: make the calculated rate real2. 主動分配：讓計算速率成真

The NIC smooths the sum of its QPs' calculated rates into the NIC oversubscription ratio,

NIC 把所有 QP 計算速率的總和平滑成 NIC 超額訂閱比值：

U_{\mathrm{NIC}} \leftarrow \gamma \cdot \frac{\sum_i R_i}{B_{\mathrm{NIC}}} + (1 - \gamma) \, U_{\mathrm{NIC}}

with NIC bandwidth $B_{\mathrm{NIC}}$ and $\gamma = 0.1$ ; a value above 1 means rate inflation. When $U_{\mathrm{NIC}} \gt 1$ and the NIC dominates ( $1 \lt \psi \lt U_{\mathrm{NIC}}$ ), each ACK also applies

其中 $B_{\mathrm{NIC}}$ 是 NIC 頻寬， $\gamma = 0.1$ ；大於 1 就表示速率膨脹。當 $U_{\mathrm{NIC}} \gt 1$ 且 NIC 的影響大於網路（ $1 \lt \psi \lt U_{\mathrm{NIC}}$ ）時，每個 ACK 還會套用

R_i \leftarrow R_i \cdot \left( \frac{\gamma}{U_{\mathrm{NIC}}} + 1 - \gamma \right)

This is Eq. 5. It moves each rate a fraction $\gamma$ of the way toward its share. All QPs on the NIC get the same factor, so the fairness among them is kept, and later network reactions act on rates that the NIC can really send.

這是 Eq. 5。它讓每個速率朝自己的份額移動 $\gamma$ 的比例。NIC 上所有 QP 都乘上同一個係數，所以彼此的公平性不變，之後對網路的反應作用的都是 NIC 真的送得出去的速率。

3. Compensation: break Feature 13. 補償：打破 Feature 1

Two branches add rate on purpose. If the NIC is full but the path has idle capacity ( $\psi \lt 1$ ), Odin applies Eq. 5 and then adds $\beta_{\text{init}}$ , so the QP takes idle egress bandwidth. If $U_{\mathrm{NIC}} \lt 0.9$ , every QP adds $\beta_{\text{init}}$ (Eq. 6). This breaks Feature 1 at some cost in short-term fairness. On classic traffic the NIC is rarely full, so the first branch rarely fires; the second adds the same step to every QP, doubling AI without hurting fairness, and Odin acts like HPCC. Panel 2 runs one ACK through them.

有兩個分支刻意增加速率。若 NIC 已滿而路徑還有閒置容量（ $\psi \lt 1$ ），Odin 先套用 Eq. 5 再加上 $\beta_{\text{init}}$ ，讓這個 QP 拿走閒置的出口頻寬。若 $U_{\mathrm{NIC}} \lt 0.9$ ，每個 QP 都加上 $\beta_{\text{init}}$ （Eq. 6）。這以一些短期公平為代價打破 Feature 1。在傳統流量上 NIC 很少滿載，所以第一個分支很少觸發；第二個分支讓每個 QP 加上同樣的步長，等於把 AI 加倍而不損公平，Odin 的行為就像 HPCC。面板 2 讓一個 ACK 走過這些規則。

Panel 2面板 2

One ACK through Odin's rules (Algorithm 1)一個 ACK 走過 Odin 的規則（Algorithm 1）

Path load

\psi

路徑負載

\psi

NIC ratio

U_{\mathrm{NIC}}

NIC 比值

U_{\mathrm{NIC}}

TBO flag in the ACKACK 中的 TBO 旗標

EWMA factor

\gamma

EWMA 係數

\gamma

AI step in Eq. 3

Eq. 3 的 AI 步長

MIMD factor

\eta / \psi

MIMD 係數

\eta / \psi

NIC factor, Eq. 5

NIC 係數，Eq. 5

Compensation

補償

Rate after this ACK, from 400 Gb/s

此 ACK 之後的速率，從 400 Gb/s 起算

Rule that runs

執行的規則

The map follows Algorithm 1 line by line: Eq. 3 always runs first (with $\beta = 0$ when the flag is set and $\psi \lt 1.2$ ), then Eq. 4 updates $U_{\mathrm{NIC}}$ , then the branch for the region of the dot. The default is the paper's start-up case: 630 QPs at 400 Gb/s on a 400 Gb/s NIC give $U_{\mathrm{NIC}} \approx 630$ ; $\psi = 1.1$ is our example of mild congestion, and $\gamma = 0.1$ , $\eta = 0.95$ are the paper's values. The rate tile leaves out $\beta$ and $\beta_{\text{init}}$ , whose value the paper does not give. The horizontal axis is linear up to 2 and logarithmic beyond. Try $\psi = 0.8$ with $U_{\mathrm{NIC}} = 1.5$ , then $U_{\mathrm{NIC}} = 0.7$ .

這張圖逐行依照 Algorithm 1：Eq. 3 一定先執行（旗標已設定且 $\psi \lt 1.2$ 時 $\beta = 0$ ），接著 Eq. 4 更新 $U_{\mathrm{NIC}}$ ，然後執行圓點所在區域的分支。預設值是論文的起始情況：630 個 QP 各以 400 Gb/s 擠在一張 400 Gb/s 的 NIC 上，得到 $U_{\mathrm{NIC}} \approx 630$ ； $\psi = 1.1$ 是我們舉的輕微壅塞例子， $\gamma = 0.1$ 、 $\eta = 0.95$ 是論文的值。速率那一格不計 $\beta$ 與 $\beta_{\text{init}}$ ，論文沒有給它們的值。橫軸在 2 以下是線性、2 以上是對數。試試 $\psi = 0.8$ 配 $U_{\mathrm{NIC}} = 1.5$ ，再試 $U_{\mathrm{NIC}} = 0.7$ 。

Implementation實作

A DPDK prototype on ConnectX-5 NICs parses ACK feedback, runs Algorithm 1 and paces each QP. A P4 program on a Tofino keeps one TBO variable, compares it in one ALU and writes the flag into reserved INT space. No line counts are given. The cost per ACK uses a SiFive 7 RISC-V model: 1, 3 and 16 cycles per compare or add, multiply and divide.

ConnectX-5 NIC 上的 DPDK 原型解析 ACK 回饋、執行 Algorithm 1，並對每個 QP 做 pacing。Tofino 上的 P4 程式用一個變數記錄 TBO，以一個 ALU 比較，並把旗標寫進預留的 INT 空間。論文沒有給程式碼行數。每個 ACK 的成本採用 SiFive 7 RISC-V 模型：比較或加減、乘法、除法各 1、3、16 個週期。

Question (analysis, per ACK)問題（分析，每個 ACK）	Calculation算式	Result結果
HPCC, two-layer Clos with 3 hops: 25 compares or adds, 9 multiplies, 8 dividesHPCC，3 跳的兩層 Clos：25 次比較或加減、9 次乘、8 次除	$25 \times 1 + 9 \times 3 + 8 \times 16$	$180$ cycles個週期
Odin's additions: 7 compares or adds, 3 multiplies, 1 divideOdin 增加的：7 次比較或加減、3 次乘、1 次除	$7 \times 1 + 3 \times 3 + 1 \times 16$	$32$ cycles個週期about 18% more多約 18%

A deployment needs a programmable CC engine in the NIC, such as BlueField-3 DOCA-PCC or ConnectX-8, and switches that export their buffer counter. Switches keep that counter for PFC but do not expose it today.

部署需要 NIC 中可程式化的 CC 引擎（例如 BlueField-3 DOCA-PCC 或 ConnectX-8），以及能匯出緩衝計數器的交換器。交換器為 PFC 維護這個計數器，但目前不對外提供。

What was measured量了什麼

Eight Dell Precision 5820 workstations with ConnectX-5 NICs share one Tofino 1 switch with PFC on; CC runs in DPDK, with the NIC's DCQCN off. Each NIC sends at most 40 Gb/s, and the base RTT is under 10 µs. All 8 nodes start one A2A together.

八台 Dell Precision 5820 工作站以 ConnectX-5 NIC 共用一台開著 PFC 的 Tofino 1 交換器；CC 跑在 DPDK 中，NIC 的 DCQCN 關閉。每張 NIC 最多送 40 Gb/s，基本 RTT 低於 10 µs。8 個節點同時開始一次 A2A。

Figure 8 of the Odin paper: throughput in Gb/s and TBO in MB over 65 ms for DCQCN, HPCC and Odin on the 8-node testbed — Figure 8 of the paper: throughput (dashed, left axis) and TBO (solid, right axis) over 65 ms. DCQCN's TBO climbs toward 2 MB, HPCC's throughput swings, and Odin keeps both steady. Measured on the testbed.論文的 Figure 8：65 ms 內的吞吐量（虛線，左軸）與 TBO（實線，右軸）。DCQCN 的 TBO 爬向 2 MB，HPCC 的吞吐量大幅擺盪，Odin 兩者都很穩定。測試平台實測。Screenshot of Jiang et al., SIGCOMM 2026, Figure 8 (p. 1254).

DCQCN builds about 2 MB of TBO through its timer-driven increase. HPCC keeps TBO near zero but loses throughput, because it reacts strongly to small queues. Odin keeps both in check. These are the paper's only hardware numbers:

DCQCN 因依計時器提高速率而累積約 2 MB 的 TBO。HPCC 讓 TBO 接近零卻損失吞吐量，因為它對很小的佇列反應強烈。Odin 兩者都控制住了。以下是論文僅有的硬體數字：

Metric指標	Odin againstOdin 相對於	Result結果
Total buffer occupancy總緩衝占用	DCQCN	−78.7%
Throughput吞吐量	HPCC	1.75×

The 512-node, asynchronous, prefill–decode, mixed AllReduce, MoE-trace, AlexNet and WebSearch results are NS-3, most with SimAI traffic, so this site does not report them.

512 節點、非同步、prefill–decode、混合 AllReduce、MoE 訓練紀錄、AlexNet 與 WebSearch 的結果都是 NS-3 模擬，其中多數使用 SimAI 流量，所以本站不列出它們。

Limits and open questions限制與未解的問題

The paper's own. Matching is done only at the NIC; fabric hops are future work. Compensation trades short-term fairness. With static buffer allocation, PFC-before-ECN disappears, but large TBO remains. And switches must export a counter that they now keep internally.

論文自己承認的。匹配只在 NIC 上進行，網路中的各跳留待未來。補償犧牲短期公平。使用靜態緩衝分配時，PFC 先於 ECN 的現象消失，但大的 TBO 仍在。還有，交換器必須匯出一個目前只在內部維護的計數器。

What to question. The hardware test is 8 nodes at 40 Gb/s, where start-up TBO is only 21 packets, with CC in DPDK and no deployment. The AI step, the switch's dynamic-threshold parameter and the baselines' 400 Gb/s thresholds are not given in numbers (Figure 2 draws DCQCN's near 0.7 and 1.6 MB), and PFC-before-ECN depends on the second. NCCL's clear-to-send caps bytes in flight (Section 7.5); if the simulated traffic ignores it, 630 QPs at line rate may overstate TBO. Rate inflation scales with the chosen 5 to 10 QPs per pair. There is no measured comparison with Meta's deep-buffer spines or with lossy transports.

該追問的。硬體測試是 8 個節點、40 Gb/s，起始 TBO 只有 21 個封包，CC 跑在 DPDK 中，也沒有部署。AI 步長、交換器的動態門檻參數，以及各基準在 400 Gb/s 下的門檻都沒有以數字給出（Figure 2 畫出 DCQCN 的門檻，約 0.7 與 1.6 MB），而 PFC 先於 ECN 取決於其中第二項。NCCL 的 clear-to-send 會限制在途位元組（第 7.5 節）；若模擬的流量忽略它，630 個 QP 從線速開始可能高估 TBO。速率膨脹隨所選的每對 5 到 10 個 QP 而放大。論文也沒有以實驗與 Meta 的深緩衝 spine 或有損傳輸比較。

How it connects與其他內容的關聯

Try the dynamic threshold in Chapter 7's Panel 3 and DCQCN's line-rate start in its Panel 4; the DCQCN reading explains the missing slow start where rate inflation begins. The Meta RoCE reading covers QP scaling and the deep-buffer spines that absorb leaf TBO, and Alibaba HPN runs HPCC. In this chapter, DCP, Falcon and Stellar make loss cheap and add multipath instead. Odin argues that load balancing hardly helps A2A on a non-oversubscribed Clos, where every ToR downlink is full; compare Stellar's spraying gains and Section 8.5.

可以在第 7 章的面板 3 試試動態門檻，在面板 4 看 DCQCN 從線速開始；DCQCN 導讀說明它為何沒有慢啟動，速率膨脹正從這裡開始。Meta RoCE 導讀談 QP 擴增以及吸收 leaf TBO 的深緩衝 spine，阿里 HPN 執行的正是 HPCC。在本章中，DCP、Falcon 與 Stellar 則讓丟包變便宜並加入多路徑。Odin 主張在沒有超額訂閱的 Clos 上，每條 ToR 下行鏈路都已滿載，負載平衡幾乎幫不了 A2A；可對照 Stellar 的噴灑收益與第 8.5 節。

Key numbers關鍵數字

Number數字	What it is是什麼	Setup and kind設定與類型
−78.7%	TBO, Odin against DCQCNTBO，Odin 相對 DCQCN	8 × ConnectX-5 at 40 Gb/s, one Tofino 1, CC in DPDK; testbed8 張 ConnectX-5、40 Gb/s、一台 Tofino 1、CC 在 DPDK；測試平台
1.75×	Throughput, Odin against HPCC吞吐量，Odin 相對 HPCC	Same testbed同一測試平台
180 + 32	Cycles per ACK: HPCC, and Odin's addition每個 ACK 的週期數：HPCC，以及 Odin 增加的	SiFive 7 model, 3 hops; analysisSiFive 7 模型、3 跳；分析
$(N-1)(N-2)/2$	Worst-case start-up TBO in packets; ≈ 32.8 MB at 128 ports and 4 KB, our arithmetic最差情況的起始 TBO（封包數）；128 埠、4 KB 時約 32.8 MB，我方算術	One switch; analysis一台交換器；分析
0.635 vs 400 Gb/s; 50 vs 8 µs	One QP's real share against its calculated rate; its ACK interval against the base RTT一個 QP 的真實份額對計算速率；它的 ACK 間隔對基本 RTT	630 QPs per NIC, 4 KB; analysis每張 NIC 630 個 QP、4 KB；分析
1/16; 0.1; 1.2; 0.9	$\alpha_{\mathrm{TBO}}$ ; $\gamma$ ; the $\psi$ gate; the compensation gate $\alpha_{\mathrm{TBO}}$ ； $\gamma$ ； $\psi$ 閘門；補償閘門	Odin's settingsOdin 的設定值

Terms used on this page本頁用到的術語

All-to-all (A2A): Every participant sends a different block to every other one; the core of MoE expert dispatch.; 每個參與者各送一塊不同的資料給其他每個參與者；MoE 專家分派的核心。
Total buffer occupancy (TBO): All bytes queued in a shared-buffer switch, summed over its ports.; 共享緩衝交換器中各埠排隊位元組的總和。
Rate inflation: A calculated rate far above the NIC share that a QP really gets.; 計算速率遠高於 QP 實際分到的 NIC 份額。
$\psi$ / MIMD + AI: HPCC's load of the busiest hop / scaling by $\eta/\psi$ plus a small fairness step $\beta$ .; HPCC 對最忙一跳的負載 / 以 $\eta/\psi$ 縮放再加上求公平的小步長 $\beta$ 。
$U_{\mathrm{NIC}}$: Smoothed sum of a NIC's calculated rates over its bandwidth.; 一張 NIC 計算速率總和的平滑值除以其頻寬。
Feature 1: Existing CC never slows a surviving QP when another QP finishes.; 現有 CC 在別的 QP 結束時，從不讓存活的 QP 減速。