Revisiting Network Support for RDMA (IRN)Revisiting Network Support for RDMA（IRN）

Radhika Mittal, Alexander Shpiner, Aurojit Panda, Eitan Zahavi, Arvind Krishnamurthy, Sylvia Ratnasamy, Scott Shenker · ACM SIGCOMM 2018

≈ 40 min · self-contained digest≈ 40 分鐘 · 不必打開論文也能讀懂Primer chapters: 06, 07對應入門頁章節：06, 07Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

Microsoft's 2016 paper ended by asking whether RoCE really needs a lossless network. IRN answers no. The authors observe that RoCE needs PFC only because its NIC recovers from a lost packet with go-back-N, resending everything after the loss, which collapses under even light loss. They make two changes to the NIC. First, efficient loss recovery: the receiver keeps out-of-order packets and sends a NACK naming the gap, the sender tracks acknowledgments in a bitmap and retransmits only what is missing, and a short timeout protects the tail of small transfers. Second, BDP-FC: a static cap so that a connection never has more than one bandwidth-delay product of bytes in flight, about 120 KB in their setup, which bounds queueing and loss without any feedback. In simulation, IRN without PFC beats RoCE with PFC by 6–83% across scenarios and by 2.8–3.7× when neither has congestion control; adding PFC to IRN makes it worse, while removing PFC from RoCE makes RoCE 1.5–3× worse. The NIC changes cost 3–10% of its cache and a few percent of an FPGA. The conclusion, "the need for PFC is an artifact of current RoCE NIC designs rather than a fundamental requirement," is the intellectual root of packet spraying, Ultra Ethernet and the transports in the 2026 AI-cluster papers.

微軟 2016 年的論文以「RoCE 真的需要無損網路嗎」作結。IRN 的答案是不需要。作者觀察到 RoCE 需要 PFC 只因為它的 NIC 以 go-back-N 從丟包恢復——重送丟失點之後的一切——即使輕微丟包也會崩潰。他們對 NIC 做兩個改動。第一，高效的丟包恢復：接收端保留亂序封包並送出指名缺口的 NACK，發送端用位圖追蹤確認並只重傳缺少的部分，一個短逾時保護小型傳輸的尾端。第二，BDP-FC：一個靜態上限，讓一條連線的在途位元組永不超過一個頻寬延遲乘積，在他們的設定中約 120 KB，不需任何回饋就限制了排隊與丟包。模擬中，不用 PFC 的 IRN 在各種情境下勝過用 PFC 的 RoCE 6–83%，在兩者都沒有壅塞控制時勝 2.8–3.7 倍；給 IRN 加上 PFC 反而變差，而把 PFC 從 RoCE 拿掉會讓 RoCE 差 1.5–3 倍。NIC 的改動只花 3–10% 的快取與 FPGA 的幾個百分點。結論「對 PFC 的需求是當前 RoCE NIC 設計的產物，而非根本要求」，是封包噴灑、Ultra Ethernet 以及 2026 年 AI 叢集論文中新傳輸層的思想根源。

Reading plan (40 minutes)閱讀計畫（40 分鐘）

§1–§2 pose the question and explain go-back-N and iWARP; §3 is the design, two mechanisms; §4 is the evaluation, four comparisons that together make the argument; §5–§6 are the implementation, which is where the paper earns its credibility with hardware people. Read §4 slowly: the logic of the four comparisons is the paper.

§1–§2 提出問題並解釋 go-back-N 與 iWARP；§3 是設計，兩個機制；§4 是評估，四個比較合起來構成論證；§5–§6 是實作，這篇論文在硬體人面前的可信度就來自這裡。慢慢讀 §4：四個比較的邏輯就是這篇論文。

1. The real reason RoCE needs PFC1. RoCE 需要 PFC 的真正原因

RDMA was designed for InfiniBand, where the fabric is lossless by construction, so the InfiniBand transport never needed good loss recovery. It uses go-back-N: the receiver discards any packet that arrives out of order and sends a NACK; the sender then retransmits every packet after the last one acknowledged. RoCE inherited this transport unchanged and so inherited the requirement for a lossless fabric, which Ethernet supplies with PFC. PFC brings head-of-line blocking, congestion spreading, unfairness and occasional deadlock (see the Microsoft and DCQCN pages). The one alternative that existed, iWARP, put a complete TCP stack on the NIC with extra layers to map TCP's byte stream onto RDMA messages; vendors and operators found it more complex, more expensive and slower (in the authors' own benchmark the iWARP NIC had about 3× the latency and a quarter of the packet rate of the RoCE NIC), and it lost the market. IRN's thesis is that you do not need all of TCP, only two of its ideas, and that those two fit in a NIC.

RDMA 是為 InfiniBand 設計的，那裡的網路依構造就是無損的，所以 InfiniBand 的傳輸層從不需要好的丟包恢復。它用 go-back-N：接收端丟棄任何亂序到達的封包並送出 NACK；發送端接著重傳最後一個被確認之後的每個封包。RoCE 原封不動地繼承了這個傳輸層，也因此繼承了對無損網路的需求，Ethernet 以 PFC 供應。PFC 帶來隊頭阻塞、壅塞擴散、不公平與偶發的死鎖（見微軟與 DCQCN 兩頁）。當時存在的唯一替代方案 iWARP 把完整的 TCP 堆疊放在 NIC 上，再加額外的層把 TCP 的位元組流對應到 RDMA 訊息；廠商與營運者發現它更複雜、更貴、更慢（在作者自己的基準測試中 iWARP NIC 的延遲約是 RoCE NIC 的 3 倍、封包速率只有四分之一），於是輸掉了市場。IRN 的論點是：你不需要整個 TCP，只需要它的兩個想法，而這兩個想法放得進一張 NIC。

2. Change one: efficient loss recovery2. 改動一：高效的丟包恢復

The receiver stops discarding out-of-order packets. When one arrives, it keeps it and sends a NACK that carries two sequence numbers: the cumulative acknowledgment (the next packet it still expects) and the sequence number of the packet that just arrived, which acts as a one-entry selective acknowledgment. The sender keeps a bitmap of which packets have been acknowledged, cumulatively or selectively. On a NACK or a timeout it enters loss recovery, retransmits the packet at the cumulative-ACK point, and uses the bitmap to resend only the gaps, leaving everything already received alone; it exits recovery when a cumulative ACK passes the recovery point. For losses at the tail of a transfer, where no later packet will arrive to trigger a NACK, there are two timeouts: a large $\mathrm{RTO}_{\text{high}}$ (about 320 µs in their setup, the maximum RTT with one congested link) in general, and a small $\mathrm{RTO}_{\text{low}}$ (100 µs) used only when $N = 3$ or fewer packets are in flight, so that a short message is not stuck waiting for a long timer while a long transfer is not disturbed by spurious short ones. The memory cost is small: five bitmaps of one bandwidth-delay product each, 128 bits apiece, 640 bits per queue pair.

接收端不再丟棄亂序封包。亂序封包到達時它保留下來，並送出帶兩個序號的 NACK：累積確認（它仍在等待的下一個封包）與剛到達封包的序號——後者相當於一筆的選擇性確認。發送端維護一張位圖，記錄哪些封包已被累積或選擇性確認。收到 NACK 或逾時時它進入丟包恢復，重傳累積確認點的封包，並用位圖只重送缺口、不碰已收到的部分；當累積確認越過恢復點時退出恢復。對傳輸尾端的丟包——之後不會再有封包到達來觸發 NACK——有兩個逾時：一般情況用大的 $\mathrm{RTO}_{\text{high}}$ （他們的設定中約 320 µs，即有一條壅塞鏈路時的最大 RTT），而只在在途封包 $N = 3$ 個以下時用小的 $\mathrm{RTO}_{\text{low}}$ （100 µs），讓短訊息不會卡著等長計時器、長傳輸也不會被誤觸的短計時器干擾。記憶體成本很小：五張各一個頻寬延遲乘積大小的位圖，每張 128 位元，每個 queue pair 640 位元。

Drawn from §3.1, the figure shows how the same single loss costs eight retransmissions under go-back-N and one under IRN. The red box is the lost packet, grey outlines mark packets that arrived but were discarded, and orange boxes are the retransmissions.依 §3.1 繪製，同樣一次丟包在 go-back-N 下要重傳八個封包，在 IRN 下只要一個。紅框是遺失的封包，灰色外框是已送達但被丟棄的封包，橘色框則是重傳的封包。Drawn for this primer from Mittal et al., SIGCOMM 2018.

3. Change two: BDP-FC3. 改動二：BDP-FC

The second change is a static limit on how much data a queue pair may have in flight: at most one bandwidth-delay product (BDP) of the longest path, divided by the MTU to get a packet count. In their default setup, 40 Gb/s links with 2 µs of propagation delay per link and a six-hop longest path give a BDP of 120 KB, about 110 MTU-sized packets. BDP-FC is not congestion control: it reacts to nothing and never changes. Its job is to make it impossible for a single connection to put more than one path's worth of data into the network, which bounds how much queueing any one flow can cause and therefore bounds loss, so that the loss recovery of §2 is rarely exercised. Over-estimating the BDP, the authors note, still delivers most of the benefit. Real congestion control, DCQCN or Timely, can be layered on top and in the evaluation both are. The idea reappears in Meta's 2024 paper as receiver-driven admission: a fixed number of channel buffers per connection is exactly a fixed bytes-in-flight cap, enforced by the collective library instead of the NIC.

第二個改動是對一個 queue pair 在途資料量的靜態限制：最多一個最長路徑的頻寬延遲乘積（BDP），除以 MTU 得到封包數。在他們的預設設定中，40 Gb/s 鏈路、每條鏈路 2 µs 傳播延遲、六跳的最長路徑，BDP 為 120 KB，約 110 個 MTU 大小的封包。BDP-FC 不是壅塞控制：它不對任何事反應、也永不改變。它的工作是讓單一連線不可能往網路放進超過一條路徑容量的資料，從而限制任何一條流能造成的排隊量、因此也限制了丟包，使 §2 的丟包恢復很少被動用。作者指出，高估 BDP 仍能得到大部分好處。真正的壅塞控制——DCQCN 或 Timely——可以疊在其上，評估中兩者都有疊加。這個想法在 Meta 2024 年的論文中以接收端驅動許可的形式重現：每條連線固定數量的 channel 緩衝區正是固定的在途位元組上限，只是由集體通訊函式庫而非 NIC 來執行。

Scale it to an AI cluster把它放大到 AI 叢集

Redo the BDP at 400 Gb/s with Meta's measured 22 µs RTT:

以 400 Gb/s 與 Meta 實測的 22 µs RTT 重算 BDP：

Question問題	Calculation算式	Result結果
One BDP at 400 Gb/s and 22 µs RTT, converted from bits to bytes400 Gb/s、22 µs RTT 下的一個 BDP，由位元換算成位元組	$400 \times 10^{9} \times 22 \times 10^{-6} \div 8$	$1.1$ MB
Packets of 4 KB in that BDP這個 BDP 裝得下的 4 KB 封包數	$1.1 \times 10^{6} \div 4{,}096$	$\approx 270$ packets 個封包

One BDP is now 1.1 MB, about 270 packets of 4 KB. A 128-bit bitmap no longer suffices, and the paper's §6.2 already shows FPGA usage rising at 100 Gb/s. This is the engineering reason modern transports (Ultra Ethernet, Amazon's SRD, Meta's 2026 design) redesign the state tracking rather than simply porting IRN.

一個 BDP 現在是 1.1 MB，約 270 個 4 KB 封包。128 位元的位圖已不夠，論文的 §6.2 也已顯示 FPGA 用量在 100 Gb/s 時上升。這就是現代傳輸層（Ultra Ethernet、Amazon 的 SRD、Meta 2026 年的設計）重新設計狀態追蹤、而非直接移植 IRN 的工程原因。

4. The four comparisons4. 四個比較

The evaluation is in simulation with detailed NIC models: a 54-server three-tier fat-tree of 45 six-port switches with full bisection bandwidth, 40 Gb/s links, 2 µs per link, ECMP, 70% load, and a heavy-tailed workload in which about half the flows are single packets of 32 B–1 KB and about 15% are 200 KB–3 MB transfers. Switch buffers are 2× BDP (240 KB) per port with a PFC threshold of 220 KB. Three metrics are reported: average slowdown (completion time relative to an unloaded network), average flow completion time, and tail (99th-percentile) completion time. The argument is built from four comparisons.

評估以具詳細 NIC 模型的模擬進行：54 台伺服器、由 45 台六埠交換器組成的三層胖樹、完整對分頻寬、40 Gb/s 鏈路、每條鏈路 2 µs、ECMP、70% 負載，以及重尾工作負載——約一半的流是 32 B–1 KB 的單封包、約 15% 是 200 KB–3 MB 的傳輸。交換器緩衝每埠 2 倍 BDP（240 KB），PFC 門檻 220 KB。報告三個指標：平均減速（相對於無負載網路的完成時間）、平均流完成時間，以及尾端（第 99 百分位）完成時間。論證由四個比較構成。

IRN without PFC vs. RoCE with PFC, no congestion control on either. IRN is 2.8–3.7× better on all three metrics. Loss recovery plus BDP-FC alone beat a lossless fabric.不用 PFC 的 IRN vs. 用 PFC 的 RoCE，兩者都沒有壅塞控制。IRN 在三個指標上好 2.8–3.7 倍。單憑丟包恢復加 BDP-FC 就勝過無損網路。
Both with congestion control (Timely or DCQCN). IRN remains up to 1.5–2.2× better. Across every scenario tested (link speeds, topologies, workloads, buffer sizes) IRN without PFC beats RoCE with PFC by 6% to 83%.兩者都有壅塞控制（Timely 或 DCQCN）。IRN 仍好 1.5–2.2 倍。在測試的每種情境（鏈路速度、拓樸、工作負載、緩衝大小）中，不用 PFC 的 IRN 都勝過用 PFC 的 RoCE 6% 到 83%。
Add PFC to IRN. Without congestion control, IRN gets 1.5–2× worse (up to 2.4× in the worst scenario); with congestion control the change is tiny in the default case (best gain under 1%, loss about 3.4%), though some scenarios degrade more. PFC does not help IRN; it hurts it, because pausing spreads congestion that IRN would otherwise absorb as a few cheap retransmissions.給 IRN 加上 PFC。沒有壅塞控制時 IRN 變差 1.5–2 倍（最差情境 2.4 倍）；有壅塞控制時預設情況變化極小（最好多不到 1%，最差少約 3.4%），某些情境劣化較多。PFC 對 IRN 沒有幫助、反而有害，因為暫停會擴散 IRN 本可用幾次便宜重傳吸收的壅塞。
Remove PFC from RoCE. RoCE gets 1.5–3× worse without congestion control, and with congestion control PFC still improves RoCE by 1.35–3.5×. RoCE needs PFC; IRN does not. The difference between them is only the NIC.把 PFC 從 RoCE 拿掉。沒有壅塞控制時 RoCE 變差 1.5–3 倍，有壅塞控制時 PFC 仍讓 RoCE 好 1.35–3.5 倍。RoCE 需要 PFC；IRN 不需要。兩者的差別只在 NIC。

Sensitivity checks hold the trends at 10, 40 and 100 Gb/s (at 100 Gb/s IRN's metrics are 0.39–0.66 of RoCE+PFC's), on 54-, 128- and 250-server topologies, with a uniform 500 KB–5 MB workload, with buffers from 60 KB to 480 KB (smaller buffers make PFC worse), and with $\mathrm{RTO}_{\text{high}}$ doubled or quadrupled or $N$ raised to 10 or 15. The sentence the paper is remembered for: "the need for PFC is an artifact of current RoCE NIC designs rather than a fundamental requirement."

敏感度檢查在 10、40 與 100 Gb/s（100 Gb/s 時 IRN 的指標是 RoCE+PFC 的 0.39–0.66）、54、128 與 250 台伺服器的拓樸、均勻的 500 KB–5 MB 工作負載、60 KB 到 480 KB 的緩衝（緩衝越小 PFC 越糟），以及 $\mathrm{RTO}_{\text{high}}$ 加倍或四倍、 $N$ 提高到 10 或 15 的情況下都維持趨勢。這篇論文被記住的那句話：「對 PFC 的需求是當前 RoCE NIC 設計的產物，而非根本要求。」

5. What out-of-order delivery costs the NIC5. 亂序投遞讓 NIC 付出什麼

RDMA semantics assume packets arrive in order, so accepting them out of order requires changes to the wire protocol and the NIC's state. For an RDMA Write, the RETH header carrying the destination address normally appears only in the first packet of a message; IRN puts it in every packet so that each one can be placed directly into memory without knowing its predecessors. For Send and Write-with-immediate, which consume a receive work-queue entry at the responder, IRN adds a per-WQE sequence number (recv_WQE_SN) so the right entry is matched and completions are signaled in order. For Read and Atomic requests arriving out of order, the responder parks them in a Read WQE buffer indexed by a read_WQE_SN. The added per-queue-pair state is 52 bits on each side (104 total), plus 56 more at the responder for Read tracking (160); each 64-byte work-queue entry grows by 3 bytes. Direct placement is what keeps this cheap: buffering out-of-order packets in NIC memory would cost about 1 KB per packet (110 MB for 1,000 flows), while tracking them with bitmaps costs a few bits each. In total IRN adds about 3–10% to a NIC's cache for a few thousand queue pairs and tens of thousands of WQEs, even at 100 Gb/s. An FPGA prototype on a Xilinx Kintex UltraScale XCKU060 used 1.35% of flip-flops and 4.01% of LUTs with no block RAM or DSP, added at most 16.5 ns of latency per module, and its slowest module ran at 45.45 million packets per second, about 372 Gb/s for MTU-sized packets. The paper notes that Mellanox was implementing a version of IRN in its next NIC release.

RDMA 語意假設封包依序到達，所以接受亂序需要修改線上協定與 NIC 的狀態。對 RDMA Write，攜帶目的位址的 RETH 標頭通常只出現在訊息的第一個封包；IRN 把它放進每個封包，讓每一個都能在不知道前面封包的情況下直接放進記憶體。對會在回應端消耗一筆接收工作佇列項目的 Send 與 Write-with-immediate，IRN 加入每 WQE 的序號（recv_WQE_SN），以配對正確的項目並依序發出完成通知。對亂序到達的 Read 與 Atomic 請求，回應端把它們停放在以 read_WQE_SN 索引的 Read WQE 緩衝區。每個 queue pair 新增的狀態為兩側各 52 位元（共 104），回應端再加 56 位元供 Read 追蹤（共 160）；每筆 64 位元組的工作佇列項目增加 3 位元組。直接放置是讓這一切便宜的關鍵：把亂序封包緩衝在 NIC 記憶體中每個封包約需 1 KB（1,000 條流需 110 MB），用位圖追蹤則每個只需幾個位元。總計 IRN 為數千個 queue pair 與數萬筆 WQE 增加約 3–10% 的 NIC 快取，即使在 100 Gb/s 也是如此。在 Xilinx Kintex UltraScale XCKU060 上的 FPGA 原型使用 1.35% 的正反器與 4.01% 的 LUT、不用 block RAM 或 DSP，每個模組最多增加 16.5 ns 延遲，最慢的模組每秒處理 4,545 萬個封包，MTU 大小的封包約 372 Gb/s。論文指出 Mellanox 正在下一版 NIC 中實作 IRN 的某個版本。

6. Critique and where it leads6. 評論與去向

IRN's evidence is simulation, on a small fat-tree with 40 Gb/s links and a web-style heavy-tailed workload, not production, and not the synchronized elephant flows of training. Its loss recovery still assumes a single path per connection, and its bitmaps are sized for a 40 Gb/s BDP. What matters is the direction. Once a NIC tolerates reordering and recovers from loss selectively, a switch may spray the packets of one flow across many paths, which removes the ECMP low-entropy problem at its root instead of adding hash inputs. Amazon's SRD transport did this in production; the Ultra Ethernet Transport specification requires selective retransmission and multipath spraying; Meta's 2026 SIGCOMM paper describes such a transport for clusters beyond 100K GPUs. Meta's 2024 receiver-driven admission is BDP-FC implemented in the collective library. Read IRN as the hinge between two eras: "make the network lossless" (Microsoft 2016, DCQCN, and the 2024 AI papers that still run PFC) and "make the endpoint tolerant" (everything after).

IRN 的證據是模擬——在 40 Gb/s 鏈路的小型胖樹上、網頁式的重尾工作負載——不是生產環境，也不是訓練那種同步的大象流。它的丟包恢復仍假設每條連線單一路徑，位圖也是依 40 Gb/s 的 BDP 設計。重要的是方向。一旦 NIC 能容忍亂序並選擇性地從丟包恢復，交換器就可以把一條流的封包噴灑到許多路徑上，從根本上消除 ECMP 的低熵問題，而非增加雜湊輸入。Amazon 的 SRD 傳輸層在生產環境中做到了這點；Ultra Ethernet 傳輸層規格要求選擇性重傳與多路徑噴灑；Meta 2026 年的 SIGCOMM 論文描述了為十萬顆以上 GPU 叢集設計的這類傳輸層。Meta 2024 年的接收端驅動許可就是在集體通訊函式庫中實作的 BDP-FC。把 IRN 當成兩個時代之間的樞紐來讀：「讓網路無損」（微軟 2016、DCQCN，以及仍在跑 PFC 的 2024 年 AI 論文）與「讓端點能容忍」（其後的一切）。

Key numbers關鍵數字

Number	數字	Meaning
3× / 4×	iWARP NIC latency / packet-rate penalty vs. RoCE NIC in the authors' benchmark	作者基準測試中 iWARP NIC 相對 RoCE NIC 的延遲 / 封包速率劣勢
100 µs / ~320 µs / $N = 3$	$\mathrm{RTO}_{\text{low}}$ / $\mathrm{RTO}_{\text{high}}$ / packets-in-flight threshold for using the low timer	$\mathrm{RTO}_{\text{low}}$ / $\mathrm{RTO}_{\text{high}}$ / 使用短計時器的在途封包門檻
$5 \times 128 = 640$ bits	Bitmap state per queue pair	每個 queue pair 的位圖狀態
120 KB ≈ 110 pkts	BDP-FC cap: 40 Gb/s, 2 µs per link, 6 hops	BDP-FC 上限：40 Gb/s、每鏈路 2 µs、6 跳
54 / 45 / 240 KB / 220 KB	Servers / switches in the simulated fat-tree; per-port buffer; PFC threshold	模擬胖樹的伺服器 / 交換器數；每埠緩衝；PFC 門檻
2.8–3.7×	IRN (no PFC) over RoCE (PFC) with no congestion control	無壅塞控制時 IRN（無 PFC）相對 RoCE（有 PFC）
1.5–2.2× / 6–83%	IRN's advantage with Timely or DCQCN / across all scenarios	有 Timely 或 DCQCN 時 IRN 的優勢 / 所有情境的範圍
1.5–2× worse	Effect of adding PFC to IRN (no congestion control)	給 IRN 加上 PFC 的效果（無壅塞控制）
1.5–3× worse	Effect of removing PFC from RoCE (no congestion control)	把 PFC 從 RoCE 拿掉的效果（無壅塞控制）
104 / 160 bits; 3 B	Added per-QP state (requester+responder / with Read tracking); added bytes per WQE	每 QP 新增狀態（請求端+回應端 / 含 Read 追蹤）；每 WQE 新增位元組
1 KB vs a few bits	Per out-of-order packet: buffering in the NIC vs. direct placement with bitmaps	每個亂序封包：在 NIC 中緩衝 vs. 以位圖直接放置
3–10%	Extra NIC cache for IRN state	IRN 狀態所需的額外 NIC 快取
1.35% / 4.01% / 16.5 ns / 372 Gb/s	FPGA flip-flops / LUTs / max added latency per module / sustainable rate	FPGA 正反器 / LUT / 每模組最大新增延遲 / 可維持速率

Terms used in this paper本篇用到的術語

Go-back-N: Loss recovery that discards out-of-order packets and retransmits everything after the last acknowledged one.; 丟棄亂序封包、並重傳最後一個已確認封包之後所有封包的丟包恢復。
Selective retransmission: Keeping out-of-order packets and resending only the missing ones, tracked by a bitmap at the sender.; 保留亂序封包、只重送缺少的部分，由發送端的位圖追蹤。
NACK (IRN): A negative acknowledgment carrying the cumulative ACK and the sequence number of the out-of-order packet that triggered it.; 帶有累積確認與觸發它的亂序封包序號的負向確認。
$\mathrm{RTO}_{\text{low}}$ / $\mathrm{RTO}_{\text{high}}$: Two retransmission timers: short when few packets are in flight (tail losses), long otherwise.; 兩個重傳計時器：在途封包少時（尾端丟包）用短的，否則用長的。
BDP-FC: A static cap of one bandwidth-delay product on bytes in flight per queue pair; bounds queueing without feedback.; 每個 queue pair 在途位元組一個頻寬延遲乘積的靜態上限；不需回饋即限制排隊。
RETH: The RDMA extended transport header carrying the remote address; IRN adds it to every Write packet.; 攜帶遠端位址的 RDMA 擴充傳輸標頭；IRN 把它加進每個 Write 封包。
WQE: Work-queue entry: one posted send, receive or read request on a queue pair.; 工作佇列項目：queue pair 上一筆已提交的 send、receive 或 read 請求。
Timely: Google's RTT-based congestion control, used alongside DCQCN as the congestion-control layer in IRN's evaluation.; Google 基於 RTT 的壅塞控制，在 IRN 的評估中與 DCQCN 一起作為壅塞控制層。