Research digest 2 of 4 · Chapter 7: Transport

研究導讀 2 / 4 · 第 7 章：傳輸層

Revisiting RDMA Reliability for Lossy Fabrics重新檢視有損網路上的 RDMA 可靠性

SIGCOMM 2025Wenxue Li, Xiangzhou Liu, Yunxuan Zhang et al. (HKUST, Huawei) · paper論文

After reading, you can explain why RoCE needed a lossless fabric, what breaks when packets drop or reorder, and how DCP's trimmed headers and precise retransmission fix it, with IRN as the reference.讀完後，你能說明 RoCE 為何需要無損網路、封包遺失或亂序時哪裡會壞掉，以及 DCP 如何以修剪後的標頭與精確重傳修好它，並以 IRN 作為對照。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 7 ↗第 7 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

RoCE NICs recover from loss with go-back-N, so RoCE fabrics use PFC to stay lossless. IRN showed that selective retransmission removes that need, but an IRN-style NIC still takes reordered packets for lost ones, and it needs a timeout when a last packet or a retransmission is lost. DCP (HKUST and Huawei) makes each switch keep the header of any packet it cannot queue: the switch cuts off the payload and sends the 57-byte header through a control queue that cannot overflow. The receiver bounces the header to the sender, which resends exactly that packet. Every packet then arrives exactly once, so the receiver counts packets per message instead of keeping a bitmap. On 16 FPGA NICs and two P4 switches at 100 Gb/s, DCP recovered from 0.01–5% loss 1.6–72× more efficiently than a ConnectX-5. AllToAll ran up to 42% faster and AllReduce 33%, but DCP also used adaptive routing against the ConnectX-5's ECMP, so DCP alone does not earn that gain. The NIC needed 1.7% more logic and 1.1% more memory.

RoCE 網卡以 go-back-N 從丟包恢復，所以 RoCE 網路用 PFC 保持無損。IRN 證明選擇性重傳可以移除這個需求，但 IRN 式網卡仍會把亂序的封包當成遺失，而且最後一個封包或某個重傳遺失時需要逾時。DCP（香港科技大學與華為）讓每台交換器保留它無法排入佇列之封包的標頭：交換器切掉載荷，把 57 位元組的標頭經由一個不會溢出的控制佇列送出。接收端把標頭反彈給發送端，發送端就精確地重送那一個封包。於是每個封包恰好到達一次，接收端改為逐訊息計數封包，不必維護位圖。在 16 張 FPGA 網卡與兩台 P4 交換器、100 Gb/s 的測試平台上，DCP 在 0.01–5% 丟包下的恢復效率是 ConnectX-5 的 1.6–72 倍。AllToAll 最多快 42%、AllReduce 快 33%，但 DCP 同時使用自適應路由，ConnectX-5 用的是 ECMP，所以這個增益不能全算在 DCP 身上。網卡多用了 1.7% 的邏輯與 1.1% 的記憶體。

1. Where this paper sits1. 這篇論文的位置

§2 of the Chapter 7 explains why RoCE needs a lossless fabric. The receiver accepts packets only in sequence-number (PSN) order, and the sender uses go-back-N, which resends up to one bandwidth-delay product (BDP) per loss (Panel 2 there). PFC hides this at the price of headroom, pause storms and deadlocks (§3). §6 and the IRN guided reading give the first fix: keep out-of-order packets, track them in a bitmap, and resend only the gaps. Three problems remain.

第 7 章的 §2 說明 RoCE 為何需要無損網路。接收端只依封包序號（PSN）順序接受封包，發送端用 go-back-N，每次丟包最多重送一個頻寬延遲乘積（BDP）（該頁面板 2）。PFC 把問題遮住，代價是 headroom、暫停風暴與死鎖（§3）。§6 與 IRN 導讀給出第一個修法：保留亂序封包、用位圖追蹤、只重送缺口。仍剩下三個問題。

Reordering looks like loss. IRN sends a NACK for every out-of-order arrival. With per-packet load balancing (spraying, or adaptive routing in the switches), packets arrive out of order without any loss, and each NACK can start a spurious retransmission.亂序看起來像丟包。IRN 對每次亂序到達都送出 NACK。在逐封包負載平衡（噴灑，或交換器中的自適應路由）下，即使沒有丟包，封包也會亂序到達，而每個 NACK 都可能引發一次多餘的重傳。
Some losses wait for a timer. A lost last packet has no later packet to trigger a NACK. IRN resends each gap once, so a lost retransmission is silent too. Both end in a retransmission timeout (RTO).有些丟包要等計時器。遺失的最後一個封包後面沒有封包能觸發 NACK。IRN 每個缺口只重送一次，所以遺失的重傳也不會發出訊號。兩者最後都要等重傳逾時（RTO）。
State. The bitmap must cover one BDP of packets per queue pair (QP).狀態。位圖必須為每個 queue pair（QP）涵蓋一個 BDP 的封包。

The figure runs two short traces through the three kinds of NIC.

下圖讓三種網卡處理兩段簡短的封包序列。

Drawn from §2.2 of the paper. Left: packets 4 and 5 swap paths, and nothing is lost. Go-back-N resends five packets, IRN resends packet 4, which was only late, and DCP resends nothing. Right: the last packet is lost. Go-back-N and IRN need a timer (IRN uses 100 µs or about 320 µs), as they do for a lost retransmission; DCP learns of the loss from the trimmed header.依論文 §2.2 繪製。左：封包 4 與 5 換了路徑，沒有任何遺失。go-back-N 重送五個封包，IRN 重送只是遲到的封包 4，DCP 什麼都不重送。右：最後一個封包遺失。go-back-N 與 IRN 需要計時器（IRN 用 100 µs 或約 320 µs），遺失的重傳也一樣；DCP 則從修剪後的標頭得知遺失。Drawn for this digest from Li et al., SIGCOMM 2025, §2.2 and §4.1, and the IRN guided reading.

DCP's answer: if the network never loses a header, the receiver always knows which PSN is missing. The NIC then never guesses from arrival order and never waits for a timer.

DCP 的答案：如果網路從不丟失標頭，接收端就總是知道缺了哪個 PSN。網卡因此不必從到達順序猜測，也不必等待計時器。

2. The problem, measured2. 問題，用量測說話

The paper's measurements of IRN (its Figures 1 and 2) are NS-3 simulations, so only their findings appear here. With adaptive routing and no loss at all, IRN still resent packets; under incast, its flows met repeated timeouts.

論文對 IRN 的量測（圖 1 與圖 2）是 NS-3 模擬，所以這裡只說明它們的發現。在自適應路由下、完全沒有丟包時，IRN 仍會重送封包；在 incast 下，它的流一再遇到逾時。

The paper's own analysis of PFC is a distance limit (its Table 1). A paused link still delivers the bytes already on the fiber, about 5 µs per km each way, so the headroom bounds the cable length (Section 7.3, Panel 3):

論文自己對 PFC 的分析是一個距離上限（表 1）。被暫停的鏈路仍會送達已在光纖上的位元組（每公里單向約 5 µs），所以 headroom 限制了線纜長度（第 7.3 節、面板 3）：

L = \frac{\text{buffer}}{\text{bandwidth} \times \text{one-hop delay per km} \times 2}

L = \frac{\text{緩衝}}{\text{頻寬} \times \text{每公里單跳延遲} \times 2}

For Tomahawk 3, with 0.5 MB per port per 100 Gb/s:

以 Tomahawk 3 為例，每埠每 100 Gb/s 有 0.5 MB：

Question問題	Calculation算式	Result結果
One lossless queue: 0.5 MB ( $4.096 \times 10^{6}$ bits, as Table 1 counts it), 100 Gb/s, 5 µs per km, out and back一個無損佇列：0.5 MB（依表 1 的算法為 $4.096 \times 10^{6}$ 位元）、100 Gb/s、每公里 5 µs、來回	$4.096 \times 10^{6} \div (100 \times 10^{9} \times 5 \times 10^{-6} \times 2)$	$\approx 4.1$ km
Eight lossless queues share it八個無損佇列分用	$4.096 \div 8$	$\approx 0.51$ km512 m in Table 1表 1 的 512 m
Headroom for the testbed's 10 km link (50 µs one-hop delay)測試平台 10 km 鏈路（單跳 50 µs）需要的 headroom	$100 \times 10^{9} \times 50 \times 10^{-6} \times 2 \div 8$	$\approx 1.25$ MB2.5 times the 0.5 MB是 0.5 MB 的 2.5 倍

Newer chips reach even less, because they have less buffer per 100 Gb/s. A loss-tolerant NIC needs no headroom at all.

較新的晶片因為每 100 Gb/s 的緩衝更少，距離更短。能容忍丟包的網卡則完全不需要 headroom。

Drawn from Table 1 of the paper, an analysis of datasheet buffers. One lossless queue reaches at most 5.08 km, eight reach at most 634 m, and the newest chips (Tomahawk 5, Spectrum-4) reach least. The red line is the 10 km link that DCP ran without PFC.依論文表 1 繪製，數值是以規格書緩衝推算的分析。一個無損佇列最遠 5.08 km，八個最遠 634 m，最新的晶片（Tomahawk 5、Spectrum-4）最短。紅線是 DCP 不用 PFC 運行的 10 km 鏈路。Drawn for this digest from Li et al., SIGCOMM 2025, Table 1 and Eq. 1.

3. The design3. 設計

DCP has a switch part and a NIC part. Figure 3 of the paper follows one lost packet: ① the switch trims it; ② the receiver bounces the header; ③ the sender resends; ④ the receiver places the packet; ⑤ it acknowledges. Reliability stays separate from congestion control and load balancing.

DCP 有交換器部分與網卡部分。論文圖 3 追蹤一個遺失的封包：① 交換器修剪它；② 接收端反彈標頭；③ 發送端重送；④ 接收端放置封包；⑤ 接收端回送確認。可靠性與壅塞控制、負載平衡保持分離。

Figure 3 of the DCP paper: sender with packet retransmission, receiver with packet reception and tracking, and a switch with packet trimming and a WRR scheduler; coloured arrows for data, header-only, retransmitted data and ACK packets, numbered 1 to 5 — Figure 3 of the paper, a design diagram (no data). Black: the data packet. Red: the trimmed header-only (HO) packet, to the receiver ② and back to the sender. Blue: the retransmission ③ ④. Yellow: the ACK ⑤. Two switch queues share one WRR scheduler ①.論文圖 3，設計示意圖（不是資料）。黑色：資料封包。紅色：修剪後只剩標頭的 HO 封包，送到接收端 ② 再反彈回發送端。藍色：重傳 ③ ④。黃色：ACK ⑤。交換器的兩個佇列共用一個 WRR 排程器 ①。Li et al., SIGCOMM 2025, Figure 3 (camera-ready PDF, re-rendered).

3.1 Trim, do not drop3.1 修剪而不丟棄

A 2-bit tag in the IP ToS field marks other traffic (00), DCP data including retransmissions (10), header-only (HO) packets (11) and DCP ACKs (01). Each egress port has a data queue and a control queue. Above a data-queue threshold, 00 and 01 packets are dropped, but a 10 packet loses its payload, becomes 11 and enters the control queue, which the switch serves with enough weight that it should not drop (a design goal, not a guarantee; see the limits below). This is packet trimming. The header is 57 bytes: Ethernet 14, IP 20, UDP 8, BTH 12, and a 3-byte message sequence number (MSN).

IP ToS 欄位中 2 位元的標記區分其他流量（00）、含重傳的 DCP 資料（10）、只有標頭的 HO 封包（11）與 DCP 的 ACK（01）。每個出口埠有一個資料佇列與一個控制佇列。資料佇列高於門檻時，00 與 01 封包被丟棄，10 封包則被切掉載荷、改為 11，進入控制佇列；交換器給這個佇列足夠的排程權重，讓它原則上不丟包（這是設計目標，不是保證；見下方的限制）。這就是封包修剪。標頭是 57 位元組：Ethernet 14、IP 20、UDP 8、BTH 12，再加 3 位元組的訊息序號（MSN）。

One DCP egress port, drawn from §4.2 of the paper. Below the threshold a packet is queued normally. Above it, only the 57-byte header (bottom row, to scale) enters the control queue. WRR serves the control queue with weight

w

and the data queue with 1. The paper does not state its threshold or weight.依論文 §4.2 繪製的一個 DCP 出口埠。低於門檻時封包正常排隊。高於門檻時，只有 57 位元組的標頭（下排，按比例）進入控制佇列。WRR 以權重

w

服務控制佇列、以 1 服務資料佇列。論文沒有給出所用的門檻與權重。Drawn for this digest from Li et al., SIGCOMM 2025, §4.2 and Figure 4.

The weighted round-robin (WRR) weight must keep the control queue from overflowing in the worst case. Take radix $N$ , port bandwidth $B$ , and $r$ , the size of a data packet over its header. In an $(N-1)$ -to-1 incast where every packet is trimmed, headers arrive at $B(N-1)/r$ . The control queue drains at $Bw/(1+w)$ . Draining at least as fast as headers arrive gives

加權輪詢（WRR）的權重必須確保控制佇列在最差情況下不溢出。令埠數為 $N$ 、埠頻寬為 $B$ 、 $r$ 為資料封包與其標頭的大小比。在每個封包都被修剪的 $(N-1)$ 對 1 incast 中，標頭以 $B(N-1)/r$ 到達。控制佇列以 $Bw/(1+w)$ 排空。排空速度至少等於到達速度，得出

w \ge \frac{N-1}{r-(N-1)}, \qquad r \gt N-1

No weight is enough when $r \le N-1$ . There the paper relies on simulation, and a coarse timeout (§3.3) covers a lost header. Worked with the paper's formula:

當 $r \le N-1$ 時，任何權重都不夠。那時論文以模擬佐證，遺失的標頭則由粗粒度逾時（§3.3）處理。以論文的公式計算：

Question問題	Calculation算式	Result結果
Ratio $r$ , 1 KB data packet比值 $r$ ，1 KB 資料封包	$1{,}024 \div 57$	$\approx 18$
Minimum weight, radix 16最小權重，埠數 16	$15 \div (18 - 15)$	$\approx 5$ used only while headers wait只在有標頭等待時使用
Margin, radix 22, 1 KB packets餘裕，埠數 22、1 KB 封包	$18 - 21$	$\lt 0$ no weight is enough任何權重都不夠
Minimum weight, radix 64, 4 KB packets最小權重，埠數 64、4 KB 封包	$63 \div (4{,}096 / 57 - 63)$	$\approx 7.1$

Large packets make the guarantee easy; with 1 KB packets it fails from radix 19 upward.

大封包讓保證容易成立；1 KB 封包時，從埠數 19 起就不成立。

3.2 Bounce the header, resend one packet3.2 反彈標頭，只重送一個封包

The receiver swaps the addresses and QP numbers of an HO packet and returns it. A switch cannot, because a reliable-connection packet names only the destination QP. The sender's NIC writes the (MSN, PSN) pair into a per-QP RetransQ in host memory. Two PCIe round trips per header (work request, then payload) would give about 4 Gb/s (the paper's estimate). So DCP reads up to 16 entries at once, caches work requests and fetches only the payload. Retransmissions use the congestion window (DCQCN in the prototype), so a burst of headers cannot bypass congestion control.

接收端把 HO 封包的位址與 QP 編號互換後送回。交換器做不到，因為可靠連線（RC）封包只指明目的 QP。發送端網卡把（MSN, PSN）寫進主機記憶體中逐 QP 的 RetransQ。每個標頭兩次 PCIe 往返（先抓工作請求、再抓載荷）只能達到約 4 Gb/s（論文的估算）。所以 DCP 一次讀取最多 16 筆項目、快取工作請求，只抓取載荷。重傳會消耗壅塞視窗（原型中是 DCQCN），所以一陣標頭無法繞過壅塞控制。

The panel compares the three schemes on one flow.

下面的面板在一條流上比較三種方案。

Panel 1面板 1

One flow under loss: go-back-N, selective repeat, DCP丟包下的一條流：go-back-N、選擇性重傳、DCP

Packet loss rate封包丟失率

Link rate鏈路速率

Packet size (MTU)封包大小（MTU）

Round-trip time (µs)往返時間（µs）

Retransmission timeout (µs)重傳逾時（µs）

Message size訊息大小

Scheme highlighted強調的方案

BDP

W

(packets)

BDP

W

（封包數）

Long-flow goodput, highlighted

長流 goodput（強調的方案）

Selective: chance a message needs an RTO

選擇性重傳：一則訊息需要 RTO 的機率

p99 message time ÷ loss-free time

p99 訊息時間 ÷ 無丟包時間

Figure 10, nearest measured rate

圖 10，最接近的量測點

A model, not the paper's. Long-flow goodput: go-back-N wastes one BDP per loss, ${G \approx 1/(1+pW)}$ , as in Panel 2 of Chapter 7; selective repeat resends one packet per loss, but a lost retransmission stalls the window until the RTO, ${G \approx 1/(1 + p + p^2 R\,(\mathrm{RTO}-\mathrm{RTT}))}$ with $R$ packets per µs; DCP resends one packet and bounces one 57 B header per loss, ${G \approx 1/(1 + p(1 + 57/\mathrm{MTU}))}$ . Message time, from 500 random trials per point: go-back-N pays one BDP per loss and an RTO for a lost last packet; selective repeat pays one RTT when anything is lost, and an RTO for a lost last packet or a lost retransmission; DCP pays one RTT per round of trimming and never an RTO. Defaults: 100 Gb/s, 1 KB packets, the paper's 10 µs intra-DC RTT (its Table 3), IRN's 320 µs $\mathrm{RTO}_{\text{high}}$ , a 1 MB message. Markers (100 Gb/s only): the points of Figure 10, read from the plot and divided by the loss-free goodput of about 90 Gb/s. They lie below the curves, because the NICs pay costs that this model leaves out and the paper does not break down. The model reproduces the order of the schemes, not the measured values.

這是模型，不是論文的。長流 goodput：go-back-N 每次丟包浪費一個 BDP， ${G \approx 1/(1+pW)}$ ，與第 7 章的面板 2 相同；選擇性重傳每次丟包重送一個封包，但遺失的重傳會讓視窗停住直到 RTO， ${G \approx 1/(1 + p + p^2 R\,(\mathrm{RTO}-\mathrm{RTT}))}$ ， $R$ 為每 µs 的封包數；DCP 每次丟包重送一個封包並反彈一個 57 B 的標頭， ${G \approx 1/(1 + p(1 + 57/\mathrm{MTU}))}$ 。訊息時間取自每點 500 次隨機試驗：go-back-N 每次丟包付一個 BDP，最後一個封包遺失時付一個 RTO；選擇性重傳只要有丟包就付一個 RTT，最後一個封包或某個重傳遺失時付一個 RTO；DCP 每一輪修剪付一個 RTT，永遠不付 RTO。預設值：100 Gb/s、1 KB 封包、論文的資料中心內 10 µs RTT（其表 3）、IRN 的 320 µs $\mathrm{RTO}_{\text{high}}$ 、1 MB 的訊息。標記（僅 100 Gb/s）：圖 10 的點，從圖上讀出，再除以約 90 Gb/s 的無丟包 goodput。它們落在曲線之下，因為網卡付出了本模型沒有計入、論文也沒有細分的代價。模型重現的是方案的優劣順序，不是量測值。

3.3 Place every packet, count instead of track3.3 放置每個封包，以計數取代追蹤

Out-of-order packets are cheap only if each goes straight to its address. DCP puts the RETH in every WRITE packet, as IRN does, and adds a send sequence number (SSN) so that a SEND finds its receive request. Only truly trimmed packets are resent, so each packet arrives exactly once, and one counter per message is enough: the message is complete when the count equals its packet count. The expected MSN (eMSN) then advances and an ACK goes out. State per message falls from $n$ bits to about $\log_2 n$ bits.

只有當每個封包都能直接寫到它的位址時，亂序封包才便宜。DCP 像 IRN 一樣在每個 WRITE 封包中放入 RETH，並加入發送序號（SSN），讓 SEND 找到它的接收請求。只有真正被修剪的封包會重送，所以每個封包恰好一次到達，每則訊息一個計數器就夠了：計數等於訊息的封包數時，訊息完成。接著預期 MSN（eMSN）前進並送出 ACK。每則訊息的狀態從 $n$ 位元降到約 $\log_2 n$ 位元。

Drawn from Table 3 of the paper (analysis). Linked 128-bit chunks save memory when reordering is rare, but reaching the

n

-th chunk takes

n

pointer steps (the paper's Figure 7). With NCCL's 8 outstanding messages per QP, DCP's counters alone take about 16 B.依論文表 3 繪製（分析）。串接的 128 位元區塊在亂序很少時省記憶體，但要到第

n

個區塊得走

n

步指標（論文圖 7）。以 NCCL 每個 QP 8 則未完成訊息計，DCP 光是計數器約 16 B。Drawn for this digest from Li et al., SIGCOMM 2025, Table 3 and §4.5.

If a header is lost after a failure, a per-message timeout resends the whole message with a new retry number (sRetryNo). The paper admits that bitmap-free tracking "faces tremendous challenges in real-world deployment"; header-based retransmission also works with a bitmap.

如果標頭在故障後遺失，逐訊息的逾時會以新的重試編號（sRetryNo）重送整則訊息。論文承認無位圖追蹤在實際部署上面臨巨大挑戰；以標頭觸發的重傳也能搭配位圖使用。

4. Implementation4. 實作

DCP-Switch is a P4 program: trimming mirrors a copy cut to header length and drops the original, WRR runs at egress, and adaptive routing at ingress picks the shortest egress queue. DCP-RNIC is the authors' own go-back-N FPGA NIC, changed and run with DCQCN on an AMD Alveo U250 (PCIe Gen3 ×16, 100 GbE, 300 MHz). No lines of code are given. The cost (Table 4; share of the FPGA in parentheses):

DCP-Switch 是一個 P4 程式：修剪以鏡像產生截到標頭長度的副本並丟棄原封包，WRR 在出口執行，入口的自適應路由挑選佇列最短的出口。DCP-RNIC 是作者自己的 go-back-N FPGA 網卡加以修改，搭配 DCQCN，跑在 AMD Alveo U250 上（PCIe Gen3 ×16、100 GbE、300 MHz）。論文沒有給程式碼行數。成本（表 4；括號內為占 FPGA 的比例）：

Resource	資源	Go-back-N NIC	go-back-N 網卡	DCP-RNIC
LUTs (logic)	LUT（邏輯）	66k (5.4%)	67k (5.5%)	+1.7%
Registers	暫存器	102k (3.5%)	103k (3.6%)	+0.4%
Block RAM (memory)	Block RAM（記憶體）	408 (20%)	412 (20%)	+1.1%

In perftest, DCP-RNIC matched the go-back-N NIC in throughput and latency; the paper gives the values only in a figure.

在 perftest 中，DCP-RNIC 的吞吐與延遲和 go-back-N 網卡相當；論文只在圖中給出數值。

5. What was measured5. 量到了什麼

The testbed has two P4 switches with 8 FPGA NICs each, 100 Gb/s links, and 8 parallel links between the switches. The baseline is an NVIDIA ConnectX-5 with go-back-N. Its PFC setting, the MTU, the threshold, the WRR weight and the timeout are not stated.

測試平台有兩台 P4 交換器，各接 8 張 FPGA 網卡，鏈路 100 Gb/s，交換器之間有 8 條平行鏈路。基準是使用 go-back-N 的 NVIDIA ConnectX-5。它的 PFC 設定、MTU、門檻、WRR 權重與逾時都沒有說明。

Loss recovery. One long flow crosses the switches, which drop 0.01–5% of packets. DCP packets are trimmed and ConnectX-5 packets dropped, so the whole DCP design is compared with go-back-N.

丟包恢復。一條長流穿過交換器，交換器丟棄 0.01–5% 的封包。DCP 的封包被修剪、ConnectX-5 的封包被丟棄，所以比較的是整個 DCP 設計與 go-back-N。

Figure 10 of the DCP paper: line chart of goodput in Gb/s against loss rate from 0 to 5% for CX5 and DCP; CX5 drops from about 90 to near 0, DCP from about 90 to about 52 — Figure 10 of the paper, measured: goodput of one long flow against the drop rate (x axis not to scale). Both NICs start near 90 Gb/s. The ConnectX-5 (red) loses about 40% at 0.01% and nearly everything at 5%; DCP (blue) keeps about half the link at 5%. The text states the gap as 1.6× to 72×.論文圖 10，實測：一條長流的 goodput 對丟包率（x 軸不按比例）。兩張網卡起初都接近 90 Gb/s。ConnectX-5（紅）在 0.01% 時少了約 40%，在 5% 時幾乎歸零；DCP（藍）在 5% 時仍保有約一半的鏈路。內文給出的差距是 1.6 到 72 倍。Li et al., SIGCOMM 2025, Figure 10 (camera-ready PDF, re-rendered).

These points are the panel's markers. They fall faster than the model, but in the same order, and the gap widens with loss.

這些點就是面板中的標記。它們比模型下降得快，但順序相同，差距也隨丟包擴大。

Unequal paths. On two paths with capacity ratios of 1:1, 1:4 and 1:10, with adaptive routing, DCP's goodput stayed stable and the ConnectX-5's fell sharply. The text gives no numbers.

不等的路徑。在容量比為 1:1、1:4、1:10 的兩條路徑上搭配自適應路由，DCP 的 goodput 保持穩定，ConnectX-5 的則大幅下降。內文沒有給數字。

AI collectives. Four groups of 4 NICs ran AllReduce or AllToAll at once (verbs API, OpenMPI). DCP cut completion time by up to 33% and 42%. But DCP ran with adaptive routing and the ConnectX-5 with ECMP, so the gain cannot be credited to DCP's reliability alone.

AI 集體通訊。4 組、每組 4 張網卡同時執行 AllReduce 或 AllToAll（verbs API、OpenMPI）。DCP 讓完成時間最多縮短 33% 與 42%。但 DCP 搭配自適應路由，ConnectX-5 搭配 ECMP，所以這個增益不能只歸功於 DCP 的可靠性機制。

Figure 12 of the DCP paper: two bar charts of completion time in ms for groups 1 to 4; (a) AllReduce, DCP about 40 versus CX5 about 53 to 60; (b) AllToAll, DCP about 20 versus CX5 about 29 to 33 — Figure 12 of the paper, measured: completion time of each group. DCP (light blue) is near 40 ms for AllReduce and near 20 ms for AllToAll; the ConnectX-5 (orange) is slower in every group. Caution: the bars differ in load balancing as well as in loss recovery.論文圖 12，實測：每個群組的完成時間。DCP（淺藍）AllReduce 約 40 ms、AllToAll 約 20 ms；ConnectX-5（橘）在每個群組都較慢。注意：兩組長條除了丟包恢復，負載平衡也不同。Li et al., SIGCOMM 2025, Figure 12 (camera-ready PDF, re-rendered).

Long haul. Over 10 km of fiber (50 µs one-hop delay), one DCP flow ran stably at about 85 Gb/s. No baseline ran on this link; by the arithmetic of §2, its PFC headroom would not fit.

長距離。在 10 km 光纖（單跳 50 µs）上，一條 DCP 流穩定運行在約 85 Gb/s。這條鏈路上沒有基準測試；依 §2 的算術，它需要的 PFC headroom 放不下。

Test	測試	Measured result (testbed, 100 Gb/s)	實測結果（測試平台，100 Gb/s）
Loss recovery, 0.01–5% drops	丟包恢復，丟包 0.01–5%	DCP goodput 1.6× to 72× that of the ConnectX-5	DCP 的 goodput 是 ConnectX-5 的 1.6 到 72 倍
Unequal paths, 1:1 to 1:10	不等路徑，1:1 到 1:10	DCP stable; ConnectX-5 falls sharply	DCP 穩定；ConnectX-5 大幅下降
AllReduce / AllToAll, 4 × 4 NICs	AllReduce / AllToAll，4 × 4 張網卡	−33% / −42% completion time, DCP + AR vs ConnectX-5 + ECMP	完成時間 −33% / −42%，DCP + AR 對 ConnectX-5 + ECMP
10 km link	10 km 鏈路	About 85 Gb/s, one DCP flow, no baseline	約 85 Gb/s，一條 DCP 流，無基準

Simulation only: the NS-3 comparisons with IRN and MP-RDMA, cross-data-center runs, heavy incast, control-queue robustness and the RACK-TLP comparison. This page does not report their numbers.

僅有模擬：與 IRN、MP-RDMA 的 NS-3 比較、跨資料中心實驗、嚴重 incast、控制佇列的穩健性，以及與 RACK-TLP 的比較。本頁不引用它們的數字。

6. Limits and open questions6. 限制與未解的問題

Switch support. Every switch must trim, keep a second queue and run WRR. A hop that cannot trim leaves only the coarse timeout.交換器支援。每台交換器都必須能修剪、有第二個佇列並執行 WRR。無法修剪的一跳只剩粗粒度逾時可用。
A gap in the guarantee. The WRR bound needs $r \gt N-1$ ; beyond it the paper relies on simulation, which lost 0.16% of header-only packets at a 255-to-1 incast without congestion control (Table 5); a lost header falls back to a timeout.保證有缺口。WRR 的上界需要 $r \gt N-1$ ；超出時論文只靠模擬，而模擬在沒有壅塞控制、255 對 1 的 incast 下仍丟了 0.16% 的純標頭封包（Table 5）；遺失的標頭只能靠逾時恢復。
Scale and baseline. 16 FPGA NICs at 100 Gb/s against a ConnectX-5; no 400 Gb/s NIC, no production use, and no AI run of DCP with ECMP.規模與基準。16 張 100 Gb/s 的 FPGA 網卡對上 ConnectX-5；沒有 400 Gb/s 網卡、沒有生產部署，也沒有 DCP 搭 ECMP 的 AI 實驗。
Verbs and congestion control. READ and atomics are not covered. DCQCN cannot separate signals from different paths, and the paper leaves congestion control for per-packet balancing to future work. In its simulations, DCP without congestion control had the worst tail under heavy incast.Verbs 與壅塞控制。沒有涵蓋 READ 與 atomic。DCQCN 無法區分來自不同路徑的訊號，論文把適用於逐封包平衡的壅塞控制留待未來。在它的模擬中，沒有壅塞控制的 DCP 在嚴重 incast 下尾端最差。
State numbers. Table 3's 320 B per QP is not derived. One BDP at 400 Gb/s and 10 µs is about 490 packets of 1 KB, about 61 B of bitmap.狀態數字。表 3 的每 QP 320 B 沒有推導。400 Gb/s、10 µs 的一個 BDP 約 490 個 1 KB 封包，只需約 61 B 的位圖。

7. How it connects7. 與其他內容的關聯

IRN: DCP keeps IRN's direct placement and replaces its NACKs, timers and bitmaps. Of the three hard problems in deep-dive §6, DCP addresses state and reordering; co-existence stays open.IRN：DCP 保留 IRN 的直接放置，並取代它的 NACK、計時器與位圖。第 7.6 節的三個難題中，DCP 處理了狀態與亂序；共存仍未解決。
Falcon (same conference) asks nothing of switches, detects loss by time (RACK-TLP) and uses a few congestion-aware flows. DCP's simulations argue that time-based detection costs about one RTT.Falcon（同一屆會議）對交換器沒有要求，依時間偵測丟包（RACK-TLP），並使用少數幾條感知壅塞的流。DCP 的模擬主張依時間偵測要多花約一個 RTT。
Stellar sprays over 128 paths and repairs loss with a 250 µs RTO, the timer dependence of DCP's second problem. Odin keeps PFC instead.Stellar 噴灑到 128 條路徑，並以 250 µs 的 RTO 修復丟包，正是 DCP 第二個問題所說的對計時器的依賴。Odin 則保留 PFC。
Section 8.5: spraying needs a NIC that can tell reordering from loss, not one that only tolerates reordering.第 8.5 節：噴灑需要能分辨亂序與丟包的網卡，而不只是能容忍亂序的網卡。

Key numbers關鍵數字

Number	數字	Meaning and setup
1.6×–72×	Loss-recovery goodput, DCP vs ConnectX-5, 0.01–5% drops (testbed)	丟包恢復 goodput，DCP 對 ConnectX-5，丟包 0.01–5%（測試平台）
−33% / −42%	AllReduce / AllToAll time, DCP + AR vs ConnectX-5 + ECMP (testbed)	AllReduce / AllToAll 時間，DCP + AR 對 ConnectX-5 + ECMP（測試平台）
≈85 Gb/s	One DCP flow over 10 km (testbed)	一條 DCP 流跨 10 km（測試平台）
+1.7% / +1.1%	FPGA logic / memory vs a go-back-N NIC (prototype)	FPGA 邏輯 / 記憶體，相對 go-back-N 網卡（原型）
57 B; $w \ge \tfrac{N-1}{r-N+1}$	Trimmed header; WRR weight for a lossless control queue (analysis)	修剪後的標頭；無損控制佇列所需的 WRR 權重（分析）
4.1 km / 512 m	PFC reach, Tomahawk 3, 1 / 8 lossless queues (analysis)	PFC 可達距離，Tomahawk 3，1 / 8 個無損佇列（分析）
320 B / 32 B	Receiver state per QP, bitmap vs DCP (analysis)	每 QP 的接收端狀態，位圖對 DCP（分析）

Terms used on this page本頁用到的術語

Packet trimming / HO packet: A switch cuts the payload of a packet it cannot queue and forwards the 57-byte header-only (HO) packet.; 交換器切掉無法排隊之封包的載荷，轉送 57 位元組、只有標頭的 HO 封包。
Control queue / WRR: A second egress queue for HO packets, served with weight $w$ against the data queue's 1.; 給 HO 封包的第二個出口佇列，以權重 $w$ 對資料佇列的 1 服務。
RetransQ: Per-QP queue of (MSN, PSN) entries in host memory, drained under the congestion window.; 主機記憶體中逐 QP 的（MSN, PSN）項目佇列，在壅塞視窗內消化。
MSN / eMSN / SSN: Message sequence number / the receiver's expected MSN / send sequence number for SENDs.; 訊息序號 / 接收端預期的 MSN / SEND 用的發送序號。
Exactly-once delivery: Each packet arrives once, because only truly lost packets are resent; counters can replace bitmaps.; 只有真正遺失的封包會重送，所以每個封包只到達一次；計數器得以取代位圖。
Adaptive routing / spurious retransmission: Per-packet choice of the least-loaded port / resending a packet that was only late.; 逐封包選擇負載最輕的埠 / 重送一個只是遲到的封包。