Research digest 3 of 4 · Chapter 7: Transport

研究導讀 3 / 4 · 第 7 章：傳輸層

Falcon: A Reliable, Low Latency Hardware TransportFalcon：可靠、低延遲的硬體傳輸層

SIGCOMM 2025Arjun Singhvi, Nandita Dukkipati, Prashant Chandra et al. (Google) · paper論文

After reading, you can explain how the Falcon NIC transport recovers from loss and reordering without PFC or switch help, separates fabric from host congestion, and avoids the connection-count cliff of RoCE NICs.讀完後，你能說明 Falcon 網卡傳輸層如何不靠 PFC 或交換器協助就從丟包與亂序恢復、區分網路與主機端的壅塞，並避開 RoCE 網卡在連線數上的效能懸崖。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 7 ↗第 7 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

Falcon is Google's transport in NIC hardware for general-purpose data-center Ethernet: oversubscribed, shared by tenants, allowed to drop packets, with no PFC and no special switch features. It carries RDMA and NVMe. The paper argues that RoCE fits such networks badly: go-back-N recovery with selective repeat for only some operations, no multipath support, and congestion control beside the datapath. Falcon combines a 128-bit ACK bitmap with time-based loss detection, Swift-style delay control with one window for the fabric and one for the receiving host, several flows per connection on different paths, and carved NIC resources. In an Intel E2100 200 Gb/s IPU it costs 5–6% of the die and 3–4% of the power. On 32 machines against ConnectX-7 RoCE, RoCE lost up to 3.4× of its goodput under random loss and Falcon delivered 5–65% more; under reordering, RoCE go-back-N fell up to 20×. At 5000:1 incast, Falcon's p99 was about 2× ideal while RoCE lost 13% of the bandwidth. Falcon reached 120 Mops with 12 QPs and showed no latency jump at 100K connections, where the ConnectX-7 became about 3× slower. Its multipath and RACK-TLP results are simulation.

Falcon 是 Google 為通用資料中心 Ethernet 做在網卡硬體中的傳輸層：網路超額訂閱、多租戶共用、允許丟包，沒有 PFC，也沒有特殊的交換器功能。它承載 RDMA 與 NVMe。論文主張 RoCE 不適合這種網路：go-back-N 恢復、只有部分操作有選擇性重傳、不支援多路徑，壅塞控制也在資料路徑之外。Falcon 結合 128 位元的 ACK 位圖與依時間的丟包偵測、Swift 式的延遲壅塞控制（網路一個視窗、接收端主機一個視窗）、每條連線在不同路徑上的多條 flow，以及切分好的網卡資源。整合進 Intel E2100 200 Gb/s IPU 的成本是 5–6% 的晶片面積與 3–4% 的功耗。在 32 台機器上對比 ConnectX-7 的 RoCE：隨機丟包下 RoCE 的 goodput 最多掉 3.4 倍，Falcon 多出 5–65%；亂序下 RoCE 的 go-back-N 最多低 20 倍。在 5000:1 incast 下，Falcon 的 p99 約為理想值的 2 倍，RoCE 則損失 13% 的頻寬。Falcon 以 12 個 QP 達到 120 Mops，在 100K 條連線時延遲沒有跳升，ConnectX-7 則慢了約 3 倍。它的多路徑與 RACK-TLP 結果來自模擬。

1. Where this paper sits1. 這篇論文的位置

The Chapter 7 builds RoCE's problem in steps. §2 shows that go-back-N wastes up to one bandwidth-delay product (BDP) per loss, as a model in Panel 2. §5 introduces Swift, which reacts to measured delay instead of ECN marks. §6 asks for selective retransmission, a cap on bytes in flight, and multipath. Falcon is a complete answer in NIC hardware, and it supplies what Chapter 7 lacks: measured goodput of a current RoCE NIC under loss and reordering.

第 7 章一步步建立 RoCE 的問題。§2 以面板 2 的模型說明 go-back-N 每次丟包最多浪費一個頻寬延遲乘積（BDP）。§5 介紹 Swift，它依實測延遲而不是 ECN 標記反應。§6 要求選擇性重傳、在途位元組上限與多路徑。Falcon 是做在網卡硬體中的完整答案，也補上第 7 章缺少的東西：現役 RoCE 網卡在丟包與亂序下的實測 goodput。

The paper names three gaps in RoCE. Loss recovery is go-back-N, with selective repeat only for Writes and Read Responses. There is no protocol support for multipath, so operators use PFC to avoid loss. Congestion control is an add-on fed by out-of-band probes (NVIDIA's RTTCC), and timeouts and errors are handled in firmware, which causes performance cliffs. Falcon targets a general-purpose network with unequal paths and plain switches. So it rejects receiver-driven credit, which assumes no oversubscription, per-packet spraying, which assumes equal paths, and switch features such as the trimming that DCP needs.

論文指出 RoCE 的三個缺口。丟包恢復是 go-back-N，只有 Write 與 Read Response 有選擇性重傳。協定不支援多路徑，所以營運者用 PFC 避免丟包。壅塞控制是外掛的，依賴帶外探測（NVIDIA 的 RTTCC），逾時與錯誤又在韌體中處理，造成效能懸崖。Falcon 的目標是路徑品質不一、交換器很普通的通用網路。所以它不採用假設沒有超額訂閱的接收端驅動 credit、假設路徑相同的逐封包噴灑，也不採用 DCP 所需的修剪這類交換器功能。

2. The problem, measured2. 問題，用量測說話

§2 of the paper argues the gaps in words; its measurements of RoCE come in §6.1. A ConnectX-7 running RoCE with RTTCC faces Falcon, both set to 200 Gb/s. One host sends many 8 KB operations on one QP, and the switch drops a random share of forward packets.

論文 §2 以文字說明這些缺口；它對 RoCE 的量測在 §6.1。執行 RoCE 與 RTTCC 的 ConnectX-7 對上 Falcon，兩者都設為 200 Gb/s。一台主機在一個 QP 上送出許多 8 KB 的操作，交換器隨機丟棄一部分前向封包。

Figure 10 of the Falcon paper: four line charts of goodput in Gb/s against drop rate 0 to 1% for Write, Send, Read Response and Read Request, comparing Falcon, RoCE Go-Back-N and RoCE Selective Repeat — Figure 10 of the paper, measured on the testbed (200 Gb/s, 8 KB operations on one QP, 0–1% random drops). Falcon (green) stays flat. RoCE go-back-N (yellow) and selective repeat (blue) fall for (a) Writes, (b) Sends and (c) Read Responses; for Sends the two RoCE modes are the same, because selective repeat does not cover Sends. (d) Dropped Read Requests cost little.論文圖 10，在測試平台上實測（200 Gb/s、單一 QP 上的 8 KB 操作、0–1% 隨機丟包）。Falcon（綠）保持平坦。RoCE 的 go-back-N（黃）與選擇性重傳（藍）在 (a) Write、(b) Send、(c) Read Response 都下降；Send 的兩種 RoCE 模式相同，因為選擇性重傳不涵蓋 Send。(d) 丟棄 Read Request 的代價很小。Singhvi et al., SIGCOMM 2025, Figure 10 (camera-ready PDF, re-rendered).

RoCE lost up to 3.4× of its goodput. Selective repeat beat go-back-N by 5–95% for Writes and Read Responses, but not for Sends or Read Requests: RoCE lacks the NIC resources to hold those out of order, so it drops them. Even where selective repeat helped, Falcon delivered 5–65% more, because one Falcon ACK bitmap reports several lost packets per round trip. Dropped Read Requests cost little, because the bottleneck is the reverse direction.

RoCE 的 goodput 最多掉 3.4 倍。選擇性重傳在 Write 與 Read Response 上比 go-back-N 好 5–95%，在 Send 與 Read Request 上則沒有：RoCE 缺少網卡資源來保存亂序的這些封包，只好丟棄。即使在選擇性重傳有幫助之處，Falcon 仍多出 5–65%，因為一個 Falcon ACK 位圖能在一次往返內回報多個遺失的封包。丟棄 Read Request 代價很小，因為瓶頸在反方向。

Reordering hurts RoCE without any loss. With 0–25% of Writes reordered, Falcon stayed flat, RoCE selective repeat lost 5–52%, and go-back-N lost up to 20×, because it resends the whole window. A third mode, RoCE adaptive routing (400 Gb/s, 16 KB Writes), was the slowest under loss: packet captures showed no fast signal such as a NACK. The paper notes that this mode is designed for lossless fabrics.

即使沒有丟包，亂序也會傷害 RoCE。當 0–25% 的 Write 被重排時，Falcon 保持平坦，RoCE 選擇性重傳少了 5–52%，go-back-N 最多少 20 倍，因為它重送整個視窗。第三種模式 RoCE 自適應路由（400 Gb/s、16 KB Write）在丟包下最慢：封包擷取顯示沒有 NACK 之類的快速訊號。論文指出這個模式是為無損網路設計的。

3. The design3. 設計

Falcon sits in the NIC under the upper-layer protocols (ULPs). A Transaction Layer gives each ULP a request–response interface and manages NIC resources. A Packet Delivery Layer handles reliability, congestion control and multipath. The Falcon Adaptive Engine (FAE), software on Arm cores in the NIC, computes windows, timeouts and path choices, and the hardware enforces them.

Falcon 位於網卡中、上層協定（ULP）之下。Transaction Layer 為每個 ULP 提供請求–回應介面並管理網卡資源。Packet Delivery Layer 處理可靠性、壅塞控制與多路徑。Falcon Adaptive Engine（FAE）是網卡內 Arm 核心上的軟體，負責計算視窗、逾時與路徑選擇，再由硬體執行。

Figure 2 of the Falcon paper: block diagram of a NIC containing Falcon, with RDMA, NVM Express and custom ULP boxes and their ULP mapping layers, a Transaction Layer, a Packet Delivery Layer connected to the network, and a Falcon Adaptive Engine — Figure 2 of the paper, a design diagram (no data). RDMA, NVMe and a custom ULP map onto one Transaction Layer (resources, ordering, scheduling). Below it, the Packet Delivery Layer handles reliability, multipathing and congestion-control gating. The Falcon Adaptive Engine on the left runs the congestion-control and multipath algorithms.論文圖 2，設計示意圖（不是資料）。RDMA、NVMe 與自訂 ULP 對應到同一個 Transaction Layer（資源、排序、排程）。其下的 Packet Delivery Layer 處理可靠性、多路徑與壅塞控制閘控。左側的 Falcon Adaptive Engine 執行壅塞控制與多路徑演算法。Singhvi et al., SIGCOMM 2025, Figure 2 (camera-ready PDF, re-rendered).

3.1 Reliability: a bitmap, RACK and a tail probe3.1 可靠性：位圖、RACK 與尾端探測

Each packet carries a PSN. Each ACK carries a timestamp and a 128-bit receive bitmap of the packets that have arrived, and the receiver accepts packets out of order. The sender decides what is lost by time, not by order (RACK). A packet sent before the most recently delivered one, and older than a RACK timeout, is presumed lost and resent. So RACK waits about one RTT for reordering before it resends anything. A lost last packet produces no later delivery; the tail loss probe (TLP) resends the lowest unacknowledged PSN after a quiet period, and the ACK it returns lets RACK repair the rest. The FAE sets the RTO, the RACK timeout and the TLP threshold.

每個封包帶一個 PSN。每個 ACK 帶一個時間戳記與一個 128 位元的接收位圖，標出已到達的封包，接收端接受亂序封包。發送端依時間而不是依順序判斷遺失（RACK）。比最近一個已送達封包更早送出、且超過 RACK 逾時的封包，就被認定遺失並重送。所以 RACK 在重送之前會等約一個 RTT，容許亂序。遺失的最後一個封包後面沒有送達的封包；尾端遺失探測（TLP）在一段安靜期後重送最小的未確認 PSN，它換回的 ACK 讓 RACK 修復其餘部分。FAE 設定 RTO、RACK 逾時與 TLP 門檻。

One caution: RACK-TLP was not in the testbed silicon, which used an earlier out-of-order-distance threshold (OOO-D). The measured loss results above are therefore for that older heuristic; the RACK-TLP comparison is simulation only.

要注意：測試平台的晶片沒有 RACK-TLP，它用的是較早的亂序距離門檻（OOO-D）。所以上面的實測丟包結果屬於那個舊方法；與 RACK-TLP 的比較只有模擬。

3.2 Two windows from four timestamps3.2 四個時間戳記、兩個視窗

Falcon's congestion control follows Swift (Section 7.5) but splits the delay into two signals. Four timestamps give the fabric delay without synchronized clocks: $t_1$ when the packet leaves, $t_2$ when it reaches the receiving NIC, $t_3$ when the ACK leaves, $t_4$ when the ACK arrives. Each difference uses one clock, and the receiver's dwell time drops out:

Falcon 的壅塞控制沿用 Swift（第 7.5 節），但把延遲拆成兩個訊號。四個時間戳記不需同步時鐘就能得到網路延遲： $t_1$ 封包離開、 $t_2$ 到達接收端網卡、 $t_3$ ACK 離開、 $t_4$ ACK 回到發送端。每個差值都只用一個時鐘，接收端的停留時間被扣除：

D_{\text{fabric}} = (t_4 - t_1) - (t_3 - t_2)

The fabric window fcwnd reacts to $D_{\text{fabric}}$ . A second window, ncwnd, reacts to how full the receiving NIC's buffer is, a level the receiver reports in each ACK. That buffer fills when the host is slow (PCIe, IOMMU, memory) or when ordered connections hold out-of-order packets. The sender uses the smaller window:

網路視窗 fcwnd 依 $D_{\text{fabric}}$ 反應。第二個視窗 ncwnd 依接收端網卡緩衝的滿度反應，接收端在每個 ACK 中回報這個水位。主機變慢（PCIe、IOMMU、記憶體）或有序連線保存亂序封包時，這個緩衝就會變滿。發送端使用較小的視窗：

W = \min(\text{fcwnd}, \text{ncwnd})

Drawn from §4.2 of the paper.

t_4 - t_1

is read on the sender's clock and

t_3 - t_2

on the receiver's, so no clock sync is needed; their difference is the time spent in the fabric and drives fcwnd. The receiver's buffer level, returned in the ACK, drives ncwnd toward a target. The smaller window wins.依論文 §4.2 繪製。

t_4 - t_1

在發送端的時鐘上讀取，

t_3 - t_2

在接收端的時鐘上讀取，所以不需同步時鐘；兩者之差是在網路中花的時間，驅動 fcwnd。ACK 帶回的接收端緩衝水位驅動 ncwnd 趨向目標。較小的視窗生效。Drawn for this digest from Singhvi et al., SIGCOMM 2025, §4.2.

One delay signal cannot tell a congested fabric from a slow host; two windows can. On the testbed, when a receiver's PCIe link was cut from 200 to about 100 Gb/s, Falcon converged to about 100 Gb/s almost at once, and ncwnd fell within about 5 RTTs. RoCE converged about 2× slower and took 6× longer to ramp back up. At 5000:1 fabric incast (5 clients writing 1 MB per QP to one server), Falcon kept mean and median latency near ideal, p99 about 2× ideal and per-QP goodput variance under 1%, at near line rate. RoCE lost 13% of the bandwidth.

單一延遲訊號分不出是網路壅塞還是主機太慢；兩個視窗分得出來。在測試平台上，接收端的 PCIe 鏈路從 200 降到約 100 Gb/s 時，Falcon 幾乎立刻收斂到約 100 Gb/s，ncwnd 在約 5 個 RTT 內下降。RoCE 收斂慢約 2 倍，恢復速率要多花 6 倍時間。在 5000:1 的網路 incast 下（5 台用戶端對一台伺服器每個 QP 寫 1 MB），Falcon 的平均與中位延遲接近理想、p99 約為理想的 2 倍、每個 QP 的 goodput 差異低於 1%，總量接近線速。RoCE 損失 13% 的頻寬。

3.3 Several flows per connection3.3 每條連線多條 flow

A Falcon connection keeps a list of flows. Each flow has its own IPv6 flow label, which switches include in their ECMP hash, so flows can take different paths. Each flow has its own fabric window, and the next packet goes to the flow with the largest open window, $\text{fcwnd}_f - \text{unacked}_f$ . PSNs are shared across flows, and RACK runs per flow, so reordering between paths does not look like loss. PLB moves a flow off a congested path by changing its label; PRR changes the label after a failure.

Falcon 的連線維護一份 flow 清單。每條 flow 有自己的 IPv6 flow label，交換器把它納入 ECMP 雜湊，所以不同 flow 可以走不同路徑。每條 flow 有自己的網路視窗，下一個封包交給開放視窗最大的 flow，即 $\text{fcwnd}_f - \text{unacked}_f$ 。各 flow 共用 PSN，RACK 則逐 flow 執行，所以路徑之間的亂序不會被當成遺失。PLB 以更換標籤把 flow 移離壅塞路徑；PRR 在故障後更換標籤。

Drawn from §4.3 of the paper. Each flow of one connection has its own IPv6 flow label, so ECMP hashing puts it on its own path. The flow on the congested spine has the smallest open window and gets the fewest packets. Four flows are drawn; the paper does not state how many a connection uses.依論文 §4.3 繪製。同一條連線的每條 flow 有自己的 IPv6 flow label，所以 ECMP 雜湊把它放到自己的路徑上。走壅塞 spine 的 flow 開放視窗最小，分到的封包也最少。圖中畫了四條 flow；論文沒有說明一條連線用幾條。Drawn for this digest from Singhvi et al., SIGCOMM 2025, §4.3.

This is a middle way between ECMP (one path per connection) and spraying (every packet on its own path). All multipath results in the paper come from simulation, so this page does not report them.

這是介於 ECMP（每條連線一條路徑）與噴灑（每個封包各走一條路徑）之間的折衷。論文的多路徑結果全部來自模擬，所以本頁不引用。

3.4 Transactions and NIC resources3.4 交易與網卡資源

The Transaction Layer turns ULP operations into transactions of at most one MTU: a Push sends data (WRITE, SEND), a Pull fetches it (READ, atomics). A request sequence number (RSN) orders transactions, and the PSN serves only reliability. So a connection can be ordered (IB Verbs), unordered, or weakly ordered, and Pull requests and responses use separate PSN spaces.

Transaction Layer 把 ULP 操作轉成最多一個 MTU 的交易：Push 送出資料（WRITE、SEND），Pull 取回資料（READ、atomic）。請求序號（RSN）負責交易排序，PSN 只負責可靠性。所以連線可以是有序的（IB Verbs）、無序的或弱有序的，Pull 的請求與回應也使用不同的 PSN 空間。

The NIC splits buffers and contexts into transmit and receive, and each into requests and responses. Above a threshold, the receive side admits only the head-of-line request, the one the connection needs next. Per-connection Xon/Xoff stops a ULP at a dynamic threshold $T_c = \alpha_c \cdot \text{free resources}$ , where $\alpha_c$ shrinks for slow connections. On the testbed, 500 slow cross-rack flows slowed fast intra-rack flows by 63× with no backpressure, 37× with a static threshold, and about 3× with the dynamic one.

網卡把緩衝與 context 分成傳送與接收，各自再分成請求與回應。超過門檻時，接收端只接納隊頭的請求，也就是連線下一個需要的那個。逐連線的 Xon/Xoff 在動態門檻 $T_c = \alpha_c \cdot \text{空閒資源}$ 處讓 ULP 停下，其中慢連線的 $\alpha_c$ 較小。在測試平台上，500 條跨機櫃的慢流讓機櫃內的快流在沒有反壓時慢 63 倍、靜態門檻時慢 37 倍、動態門檻時只慢約 3 倍。

The isolation test in §6.2 of the paper, measured on the testbed, with the three values stated in the text. Slow flows that hold NIC resources hurt unrelated fast flows, while a threshold that shrinks for slow connections keeps the damage near 3×.論文 §6.2 的隔離測試，在測試平台上實測，三個數值見於內文。占住網卡資源的慢流會傷害無關的快流，而對慢連線縮小的門檻能把傷害壓在約 3 倍。Drawn for this digest from Singhvi et al., SIGCOMM 2025, §4.6 and §6.2 (Figure 24).

4. Implementation4. 實作

Falcon is an IP block on the NIC chip; the authors chose an ASIC over an FPGA for cost and performance. It is integrated in the Intel E2100 200 Gb/s IPU and scaled to 400 and 800 Gb/s, at 5–6% of the die area and 3–4% of the power. The process node, clock and lines of code are not stated. Reordering needs an on-NIC buffer of about one BDP; at the paper's 50 µs RTT:

Falcon 是網卡晶片上的一個 IP 區塊；作者因成本與效能選擇 ASIC 而非 FPGA。它整合在 Intel E2100 200 Gb/s IPU 中，並擴展到 400 與 800 Gb/s，代價是 5–6% 的晶片面積與 3–4% 的功耗。論文沒有說明製程、時脈與程式碼行數。重排需要約一個 BDP 的網卡緩衝；以論文的 50 µs RTT 計：

Question問題	Calculation算式	Result結果
One BDP at 200 Gb/s, in units of $2^{20}$ bytes200 Gb/s 的一個 BDP，以 $2^{20}$ 位元組為單位	$200 \times 10^{9} \times 50 \times 10^{-6} \div 8 \div 2^{20}$	$\approx 1.2$ MB
The same at 400 and 800 Gb/s400 與 800 Gb/s 時	$1.19 \times 2$ , $1.19 \times 4$	$\approx 2.4$ , $4.8$ MBthe paper's three values即論文的三個數值
Packets in one 200 Gb/s BDP, if packets are 4 KB200 Gb/s 的一個 BDP 有幾個封包（若封包為 4 KB）	$200 \times 10^{9} \times 50 \times 10^{-6} \div 8 \div 4{,}096$	$\approx 305$ more than the 128 the bitmap tracks多於位圖追蹤的 128 個

The packet buffer takes 1–2 mm², under 1% of the die, and can spill to on-NIC DRAM. Connection state lives in an on-chip cache. At warehouse scale the active connections far exceed it and the miss rate nears 100%, so Falcon provisions enough bandwidth from the memory that holds the state and adds a larger second-level cache shared across the IPU. Errors and timeouts are handled in hardware. One Arm Neoverse-N1 core ran the FAE at a stable 20 million events per second; real applications peaked at 8 million.

封包緩衝占 1–2 mm²，不到晶片的 1%，並可溢出到網卡上的 DRAM。連線狀態放在晶片上的快取中。在倉儲規模下，活躍連線遠超過快取，失誤率接近 100%，所以 Falcon 從存放狀態的記憶體配置足夠的頻寬，並加入在 IPU 內共用的更大第二層快取。錯誤與逾時在硬體中處理。一個 Arm Neoverse-N1 核心以穩定的每秒 2,000 萬個事件執行 FAE；真實應用的峰值是每秒 800 萬個。

5. What was measured5. 量到了什麼

The testbed has 32 machines with 200 Gb/s Falcon NICs and 400 Gb/s NVIDIA ConnectX-7 NICs running RoCE with RTTCC, in go-back-N, selective-repeat and adaptive-routing modes. Switch models, MTU, base RTT and the ConnectX-7's PFC setting are not stated.

測試平台有 32 台機器，配備 200 Gb/s 的 Falcon 網卡與 400 Gb/s 的 NVIDIA ConnectX-7 網卡；後者以 go-back-N、選擇性重傳與自適應路由模式執行 RoCE 與 RTTCC。論文沒有說明交換器型號、MTU、基本 RTT 與 ConnectX-7 的 PFC 設定。

Test (testbed)	測試（測試平台）	Measured result	實測結果
Random loss, 0–1%, 8 KB ops, 200 Gb/s	隨機丟包 0–1%，8 KB 操作，200 Gb/s	RoCE goodput down up to 3.4×; Falcon 5–65% above RoCE	RoCE goodput 最多降 3.4 倍；Falcon 比 RoCE 高 5–65%
Reordering, 0–25% of Writes	重排 0–25% 的 Write	RoCE SR 5–52% lower; RoCE GBN up to 20× lower; Falcon flat	RoCE SR 低 5–52%；RoCE GBN 最多低 20 倍；Falcon 平坦
Fabric incast up to 5000:1	網路 incast 最高 5000:1	Falcon p99 ≈2× ideal, per-QP variance <1%; RoCE −13% bandwidth	Falcon p99 約理想的 2 倍，每 QP 差異 <1%；RoCE 頻寬 −13%
Receiver PCIe 200 → ≈100 Gb/s	接收端 PCIe 200 → 約 100 Gb/s	ncwnd reacts within ≈5 RTTs; RoCE 2× slower, 6× slower ramp-up	ncwnd 約 5 個 RTT 內反應；RoCE 慢 2 倍，恢復慢 6 倍
Op rate, 8 B Writes	操作速率，8 B Write	20 Mops on one QP; 120 Mops with 12 QPs	單一 QP 20 Mops；12 個 QP 120 Mops
Unloaded Writes, 1 B–2 MB	無負載 Write，1 B–2 MB	Median and p99 within 1% of ideal	中位數與 p99 都在理想值的 1% 內
1 to 100K connections	1 到 100K 條連線	Falcon nearly flat; ConnectX-7 ≈3× slower above 100K	Falcon 幾乎平坦；ConnectX-7 在 100K 以上慢約 3 倍

The connection cliff. A closed loop of single 8 B RDMA Reads picks connections at random, so each lookup stresses the connection cache. Throughput tests can hide misses behind many outstanding operations; this test cannot. A miss goes to on-NIC DRAM in Falcon and to host memory in the ConnectX-7.

連線懸崖。一次只有一個未完成的 8 B RDMA Read 以閉迴路隨機挑選連線，所以每次查找都考驗連線快取。吞吐測試可以把失誤藏在許多未完成的操作後面；這個測試不行。失誤時，Falcon 去網卡上的 DRAM 取狀態，ConnectX-7 則去主機記憶體。

Figure 21 of the Falcon paper: line chart of latency increase from 0 to 3 against number of QPs from 1 to 100,000 on a log axis; CX-7 rises to about 2.8, Falcon stays below about 1.3 — Figure 21 of the paper, measured on the testbed: latency increase against the number of QPs (log scale). The ConnectX-7 (blue) jumps between 5K and 10K QPs and nears 3× at 100K; the text states about 3×. Falcon (green) rises only slightly.論文圖 21，在測試平台上實測：延遲增加倍數對 QP 數（對數尺度）。ConnectX-7（藍）在 5K 與 10K 個 QP 之間跳升，到 100K 時接近 3 倍；內文給出約 3 倍。Falcon（綠）只略微上升。Singhvi et al., SIGCOMM 2025, Figure 21 (camera-ready PDF, re-rendered).

The panel models that difference: the same cache, but a cheap or an expensive miss.

下面的面板為這個差別建模：同樣的快取，但失誤的代價便宜或昂貴。

Panel 1面板 1

Connections vs the NIC's state cache連線數與網卡的狀態快取

Active connections

N

活躍連線數

N

Contexts the cache holds

C

快取容納的 context 數

C

Miss cost, state in host memory

k_h

失誤代價，狀態在主機記憶體

k_h

Miss cost, state on the NIC

k_n

失誤代價，狀態在網卡上

k_n

Miss ratio

m = 1 - C/N

失誤率

m = 1 - C/N

Latency, host-memory spill

延遲，溢出到主機記憶體

Latency, on-NIC spill

延遲，溢出到網卡記憶體

Misses per second at 120 Mops

120 Mops 時每秒失誤數

Measured (Figure 21)

實測（圖 21）

A model, not the paper's. Connections are used uniformly at random; a cache of $C$ contexts hits with probability $C/N$ ; each miss adds $k$ times the unloaded latency, so latency is $1 + k\,m$ times the unloaded value. The paper states neither the cache size nor the miss cost. Defaults are fitted to Figure 21: $C = 6{,}000$ sits between the 5K and 10K points where the ConnectX-7 curve jumps; $k_h = 2$ gives $1 + 2 \times 0.94 \approx 2.9$ at 100K, near the measured 3×; $k_n = 0.3$ gives about 1.3, near the plotted Falcon point. The misses-per-second tile multiplies the measured 120 Mops by $m$ : the rate the memory behind the cache must serve, which is why the paper provisions miss bandwidth. Markers: ConnectX-7 ≈3× (stated in the text) and Falcon ≈1.3× (read from the plot), both at 100K.

這是模型，不是論文的。連線以均勻隨機的方式被使用；容納 $C$ 個 context 的快取命中機率為 $C/N$ ；每次失誤增加 $k$ 倍的無負載延遲，所以延遲是無負載值的 $1 + k\,m$ 倍。論文沒有給出快取大小與失誤代價。預設值依圖 21 擬合： $C = 6{,}000$ 位於 ConnectX-7 曲線跳升的 5K 與 10K 兩點之間； $k_h = 2$ 在 100K 時給出 $1 + 2 \times 0.94 \approx 2.9$ ，接近實測的 3 倍； $k_n = 0.3$ 給出約 1.3，接近圖上 Falcon 的點。每秒失誤數的方塊把實測的 120 Mops 乘上 $m$ ：這是快取背後的記憶體必須服務的速率，也是論文配置失誤頻寬的原因。標記：ConnectX-7 約 3 倍（內文所述）與 Falcon 約 1.3 倍（由圖讀出），都在 100K。

Applications were compared with TCP, not RoCE: MPI AllToAll (4 B) 4.3× and AllReduce (64 KB) 5.5× faster on 32 nodes × 192 processes, GROMACS 2.4× the best TCP result, WRF 32% shorter, and NVMe-over-Falcon flash at 97.1% (read) and 91.0% (write) of local-SSD bandwidth. The multipath results and the RACK-TLP comparison are simulation and are not reported here.

應用是與 TCP 而非 RoCE 比較：在 32 個節點 × 192 個行程上，MPI AllToAll（4 B）快 4.3 倍、AllReduce（64 KB）快 5.5 倍；GROMACS 是 TCP 最佳結果的 2.4 倍；WRF 縮短 32%；NVMe-over-Falcon 的快閃儲存達到本機 SSD 頻寬的 97.1%（讀）與 91.0%（寫）。多路徑結果與 RACK-TLP 比較是模擬，這裡不引用。

6. Limits and open questions6. 限制與未解的問題

Unequal NICs. Falcon ran at 200 Gb/s and the ConnectX-7 at 400 Gb/s (set to 200 Gb/s for the loss test). The ConnectX-7's PFC, MTU and RTTCC settings are not stated.網卡不對等。Falcon 跑 200 Gb/s，ConnectX-7 跑 400 Gb/s（丟包測試時設為 200 Gb/s）。ConnectX-7 的 PFC、MTU 與 RTTCC 設定都沒有說明。
The tested chip is not the described design. RACK-TLP and multipath were absent from the testbed silicon; their benefits rest on Google's simulator.測試的晶片不是論文描述的設計。測試平台的晶片沒有 RACK-TLP 與多路徑；它們的效益只建立在 Google 的模擬器上。
No AI-training run. The only ML result is a simulation; MPI collectives are compared with TCP; there is no GPU experiment.沒有 AI 訓練實驗。唯一的 ML 結果是模擬；MPI 集體通訊是與 TCP 比較；沒有 GPU 實驗。
Missing parameters. Flows per connection, PLB and PRR thresholds, Swift's target delay, RACK and TLP timers, $\alpha$ , and the cache size.缺少的參數。每條連線的 flow 數、PLB 與 PRR 門檻、Swift 的目標延遲、RACK 與 TLP 計時器、 $\alpha$ ，以及快取大小。
Bitmap reach. 128 bits track 128 packets, fewer than one BDP at 200 Gb/s and 50 µs with 4 KB packets (table in §4); the paper says only that 128 bits worked for its use cases.位圖的範圍。128 位元只能追蹤 128 個封包，少於 200 Gb/s、50 µs、4 KB 封包時的一個 BDP（§4 的表）；論文只說 128 位元對它的用途已經足夠。
Headline and deployment. The abstract's largest speedup over ConnectX-7 is not traced to an experiment in the text, so this page does not use it. The paper gives no fleet-scale production data.主打數字與部署。摘要中對 ConnectX-7 的最大加速倍數在內文找不到對應的實驗，所以本頁不採用。論文也沒有給出全機群規模的生產資料。

7. How it connects7. 與其他內容的關聯

DCP makes the opposite bet: every switch trims, and routing is per packet. DCP's simulations argue that time-based detection such as RACK delays recovery by about one RTT.DCP 押了相反的注：每台交換器都修剪、路由逐封包進行。DCP 的模擬主張 RACK 這類依時間的偵測會讓恢復延遲約一個 RTT。
Stellar sprays each connection over 128 paths with one shared window and a 250 µs RTO; Falcon uses a few congestion-aware flows with a window each. Odin keeps PFC and cites Falcon as a multipath transport.Stellar 把每條連線噴灑到 128 條路徑，共用一個視窗、RTO 250 µs；Falcon 則用少數幾條感知壅塞、各有視窗的 flow。Odin 保留 PFC，並把 Falcon 列為一種多路徑傳輸層。
Section 7.5: Falcon is Swift in NIC hardware with the delay split into fabric and host parts; MegaScale's RTT-plus-ECN controller is the RoCE-side relative. §2 and the IRN guided reading: RoCE's selective repeat descends from IRN; Falcon adds time-based detection and a tail probe.第 7.5 節：Falcon 是做在網卡硬體中、把延遲拆成網路與主機兩部分的 Swift；MegaScale 的 RTT 加 ECN 控制器是 RoCE 這一側的近親。§2 與 IRN 導讀：RoCE 的選擇性重傳源自 IRN；Falcon 再加上依時間的偵測與尾端探測。
Section 8.5: flows with their own flow labels are a third option between ECMP and spraying. Chapter 9: NVMe-over-Falcon flash is storage traffic on the general-purpose network.第 8.5 節：各有 flow label 的多條 flow 是介於 ECMP 與噴灑之間的第三種選擇。第 9 章：NVMe-over-Falcon 的快閃儲存是通用網路上的儲存流量。

Key numbers關鍵數字

Number	數字	Meaning and setup
up to 3.4×; 5–65%	RoCE goodput loss under random drops; Falcon's gain over RoCE (testbed, 200 Gb/s)	隨機丟包下 RoCE 的 goodput 損失；Falcon 相對 RoCE 的增益（測試平台，200 Gb/s）
up to 20×	RoCE go-back-N goodput below Falcon with 0–25% reordering (testbed)	0–25% 重排時 RoCE go-back-N 的 goodput 低於 Falcon 的倍數（測試平台）
≈2× ideal; −13%	Falcon p99 at 5000:1 incast; RoCE bandwidth loss (testbed)	5000:1 incast 時 Falcon 的 p99；RoCE 的頻寬損失（測試平台）
120 Mops	Op rate with 12 QPs, 8 B Writes (testbed)	12 個 QP、8 B Write 的操作速率（測試平台）
≈3× vs flat	ConnectX-7 vs Falcon latency at 100K connections (testbed)	100K 條連線時 ConnectX-7 對 Falcon 的延遲（測試平台）
5–6% / 3–4%	Die area / power of the Falcon block in the Intel E2100 IPU	Falcon 區塊在 Intel E2100 IPU 中的晶片面積 / 功耗
63× / 37× / ≈3×	Fast-flow slowdown with no / static / dynamic backpressure (testbed)	沒有 / 靜態 / 動態反壓時快流的變慢倍數（測試平台）

Terms used on this page本頁用到的術語

RACK / TLP: Loss inferred by time once a later-sent packet is delivered / a probe that resends the lowest unacknowledged PSN after a quiet period.; 當較晚送出的封包已送達時依時間推斷遺失 / 安靜一段時間後重送最小未確認 PSN 的探測。
fcwnd / ncwnd: Fabric window driven by fabric delay / NIC window driven by the receiver's buffer level; the smaller one applies.; 由網路延遲驅動的網路視窗 / 由接收端緩衝水位驅動的網卡視窗；取較小者。
Flow label / PLB / PRR: IPv6 field that ECMP hashes, one per flow / re-path a congested flow / re-path after a failure.; ECMP 會雜湊的 IPv6 欄位，每條 flow 一個 / 為壅塞的 flow 換路徑 / 故障後換路徑。
PSN / RSN: Packet sequence number for reliability / request sequence number for ordering transactions.; 負責可靠性的封包序號 / 負責交易排序的請求序號。
Dynamic threshold: Per-connection limit $T_c = \alpha_c \cdot$ free resources, smaller for slow connections.; 逐連線的上限 $T_c = \alpha_c \cdot$ 空閒資源，慢連線較小。
Connection cliff: The latency jump when active connections exceed the NIC's on-chip state cache.; 活躍連線超過網卡晶片上的狀態快取時出現的延遲跳升。