Research digest 2 of 2 · Chapter 13: Designs compared

研究導讀 2 / 2 · 第 13 章：設計比較

Pegasus: A Data Center Network for Bare-Metal AI CloudPegasus：裸機 AI 雲的資料中心網路

SIGCOMM 2026 (Experience track)Xianneng Zou, Yadong Liu, Yiran Zhang et al., Yong Jiang, Yachen Wang (Tsinghua University, Tencent, Nanjing University, Shanghai Jiao Tong University) · paper論文

After reading, you can explain how Tencent's bare-metal AI cloud moves load balancing and congestion control into its own NIC, why one DPU manages eight NICs, and what its evidence shows.讀完後，你能說明騰訊的裸機 AI 雲如何把負載平衡與壅塞控制搬進自研 NIC、為何一台 DPU 管理八張 NIC，以及它的證據能說明什麼。

≈ 16 min · 1 interactive panel · 5-question quiz≈ 16 分鐘 · 1 個互動面板 · 5 題測驗Chapter 13 ↗第 13 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

Pegasus is Tencent's network for a bare-metal AI cloud. Tenants rent whole GPU servers and run their own collective library, so the provider cannot tune NCCL, place the ranks or wait for a slow controller. Pegasus makes three choices together. One self-built DPU per server manages eight self-built RNICs; one DPU per NIC would add 6.5–13.7% to server power, by the paper's arithmetic. The RNICs spray every packet over plain switch ECMP and restore order at the receiver. They also run receiver-granted credits with a one-batch overdraft, which lifted a four-RNIC testbed from 323–383 Gb/s to about 385 Gb/s. Pegasus serves 8K GPUs. Against Tencent's previous AI cluster, a different cluster in which the NIC, the load balancing and the congestion control all changed at once, training ran about 1.05× to 2.14× as fast (the paper prints +4.8% to +53.2% in its own convention).

Pegasus 是騰訊裸機 AI 雲的網路。租戶租下整台 GPU 伺服器、執行自己的集體通訊函式庫，所以業者無法調校 NCCL、無法安排 rank 的位置，也不能等待緩慢的控制器。Pegasus 同時做了三個選擇。每台伺服器用一台自研 DPU 管理八張自研 RNIC；依論文的算術，若每張 NIC 都配一台 DPU，伺服器功耗會增加 6.5–13.7%。RNIC 在一般交換器 ECMP 上逐封包噴灑，並在接收端恢復順序。RNIC 也執行由接收端授予的 credit，並允許透支一個 batch，讓四張 RNIC 的測試床從 323–383 Gb/s 提高到約 385 Gb/s。Pegasus 目前服務 8K 顆 GPU。與騰訊先前的 AI 叢集相比（那是另一個叢集，NIC、負載平衡與壅塞控制同時都不同），訓練速度約為舊叢集的 1.05 到 2.14 倍（論文以自己的換算方式印成 +4.8% 到 +53.2%）。

1. Where this paper sits1. 這篇論文的位置

The Chapter 13 treats load balance as axis (c) in its section 4: fix it in software (Meta's QP scaling), in wiring (HPN's dual-plane) or in circuits (TPU v4). Its section 6 lists "bound the bytes in flight" as a point on which all designs agree. Every answer there assumes one owner, who can change the collective library, place the ranks and run a controller. Astral, the previous digest, comes from the same operator and keeps that model: stock NCCL, source ports chosen through the switch hash, and a controller driven by 5-second ECN counters.

第 13 章在第 4 節把負載平衡當成軸 (c)：用軟體修（Meta 的 QP scaling）、用接線修（HPN 的 dual-plane），或用電路修（TPU v4）。它的第 6 節把「限制在途位元組」列為所有設計一致的做法。那裡的每個答案都假設只有一個擁有者，能修改集體通訊函式庫、安排 rank 的位置並運行控制器。上一篇導讀 Astral 來自同一個營運者，也維持這個模式：原版 NCCL、透過交換器雜湊挑選來源埠，以及由每 5 秒 ECN 計數驅動的控制器。

A bare-metal cloud has no such owner. The tenant controls the server's software, NCCL included, and the provider must still isolate each tenant's virtual network (VPC) without a hypervisor. Panel 7 of Chapter 13 has inputs for this case: multi-tenant hosts "Yes", custom collective library "No". Pegasus is Tencent's answer. It adds a fourth answer on axis (c), fix load balance in the NIC, and a fifth way to bound the bytes in flight, receiver credits in NIC hardware. The Chapter 8 (section 5) explains why spraying needs a NIC that accepts out-of-order packets; the Chapter 7 (section 6) presents receiver credits as the step after NCCL's clear-to-send.

裸機雲沒有這樣的擁有者。租戶掌控伺服器上的軟體（包括 NCCL），而業者仍必須在沒有 hypervisor 的情況下隔離每個租戶的虛擬網路（VPC）。第 13 章的面板 7 正好有這種情況的輸入：多租戶主機「是」、自製集體通訊函式庫「否」。Pegasus 是騰訊的答案。它在軸 (c) 加上第四種答案：在 NIC 裡修負載平衡；也加上限制在途位元組的第五種做法：NIC 硬體中的接收端 credit。第 8 章（第 5 節）說明噴灑為何需要能接受亂序封包的 NIC；第 7 章（第 6 節）把接收端 credit 描述成 NCCL clear-to-send 之後的下一步。

2. The problem, measured2. 問題，用量測說話

The paper names two obstacles to copying existing practice. The first is power. Bare-metal clouds usually put a DPU (a NIC with its own Arm cores and FPGA or ASIC) in the server and pass all tenant RDMA through it. A GPU server has eight RNICs, and a commodity DPU draws up to about 150 W. By the paper's arithmetic, one DPU per RNIC adds 6.5–13.7% to server power, and so costs as many GPUs under a fixed site power limit (section 3.1 checks this).

論文指出照搬現有做法有兩個障礙。第一是功耗。裸機雲通常在伺服器裡放一台 DPU（帶有自己的 Arm 核心與 FPGA 或 ASIC 的 NIC），讓租戶的所有 RDMA 都經過它。一台 GPU 伺服器有八張 RNIC，而市售 DPU 最多耗電約 150 W。依論文的算術，每張 RNIC 配一台 DPU 會讓伺服器功耗增加 6.5–13.7%，在固定的站點電力上限下等於少放同樣比例的 GPU（第 3.1 節會驗算）。

The second is traffic. Tencent's previous AI cluster used CX7 RNICs, tuned DCQCN, topology-aware placement, and traffic engineering (TE): a controller that watches the largest flows and moves them. That works when one owner co-designs the workload. Production data from the cloud show that tenant traffic does not cooperate. In inference, 68% of collective messages are 1 MB, while training messages cluster at a few sizes of tens to hundreds of MB. Measured every 10 s at the ToR–NIC link, a training task holds about 70 Gb/s, an inference task decays from about 45 to 12 Gb/s over 3 hours, and one tenant's task alternates between about 45 Gb/s and idle. A co-designed training network has a block-diagonal traffic matrix; the cloud's is scattered, with more load in the core.

第二是流量。騰訊先前的 AI 叢集使用 CX7 RNIC、調校過的 DCQCN、拓樸感知的配置，以及流量工程（TE）：一個監看最大的流並搬移它們的控制器。只要有一個擁有者共同設計工作負載，這套做法就行得通。雲端的生產數據顯示租戶的流量並不配合。推論中 68% 的集體通訊訊息是 1 MB，訓練訊息則集中在數十到數百 MB 的幾個大小。在 ToR 到 NIC 的鏈路上每 10 秒量一次，訓練任務維持約 70 Gb/s，推論任務在 3 小時內從約 45 Gb/s 降到 12 Gb/s，某個租戶的任務則在約 45 Gb/s 與閒置之間交替。共同設計的訓練網路流量矩陣呈區塊對角；雲端的則四散，核心層的負載更高。

The controller paid for this. In the cloud, congestion often comes from several medium flows that overlap for a short time. At a 10-second granularity, the largest average flow is not necessarily the flow that causes the incast. TE moved the wrong flow, waited for the next statistics, and tried again. Relief took about 95 s (Figure 1). That is acceptable for a training job of several weeks, and it violates inference latency targets.

控制器為此付出代價。在雲端，壅塞常來自幾條中等大小的流在短時間內重疊。以 10 秒為統計粒度時，平均最大的流不一定是造成 incast 的那一條。TE 搬錯了流，等下一輪統計，再試一次。紓解壅塞要約 95 秒（圖 1）。對持續數週的訓練任務這可以接受，對推論的延遲目標則是違規。

Figure 1. Top: the previous cluster's controller in the cloud. The thin vertical lines mark the 10-s statistics rounds, and the three dots stand for the repeated tries, while the 95 s is the paper's production measurement. Bottom: Pegasus needs no statistics; consecutive packets of one message take different paths.圖 1。上：舊叢集的控制器在雲端的情形。細的直線標出 10 秒一輪的統計，三個點代表一再重試，95 秒則是論文的生產環境量測。下：Pegasus 不需要統計；同一則訊息的相鄰封包走不同的路徑。Drawn for this primer from Zou et al., SIGCOMM 2026, §5.1 and §7.2.

3. The design3. 設計

3.1 One DPU manages eight RNICs3.1 一台 DPU 管理八張 RNIC

Pegasus separates the DPU from the RNICs (Figure 4(b) of the paper, below). One self-built P-DPU (Arm cores plus an FPGA) sits in its own PCIe slot and PCIe domain. It is also the server's front-end NIC, with Ethernet, a storage hypervisor and the front-end vSwitch. Eight self-built FPGA P-RNICs, one per GPU, carry all tenant RDMA. The DPU controls them but is never in their data path.

Pegasus 把 DPU 與 RNIC 分開（下方論文 Figure 4(b)）。一台自研的 P-DPU（Arm 核心加 FPGA）插在自己的 PCIe 插槽、位於自己的 PCIe domain。它同時是伺服器的前端 NIC，帶有 Ethernet、儲存 hypervisor 與前端 vSwitch。八張自研的 FPGA P-RNIC（每顆 GPU 一張）承載所有租戶的 RDMA。DPU 控制它們，但從不在它們的資料路徑上。

Block diagram: P-DPU with FPGA and Arm parts and a DDR flow table; a bare-metal GPU server with the DPU in one PCIe domain and GPU-RNIC pairs under PCIe switches in another; a P-RNIC with DMA engine, flow table, RDMA block, packet spraying, credit-based CC and TX/RX. — Figure 4(b) of the paper (design). Left: the P-DPU; its Arm side runs the RNIC Manager. Middle: the DPU in its own PCIe domain, GPU–RNIC pairs under PCIe switches in another. Right: the P-RNIC, whose flow table, packet spraying and credit-based CC (orange) are Pegasus's additions.論文 Figure 4(b)（設計圖）。左：P-DPU，Arm 那一側執行 RNIC Manager。中：DPU 位於自己的 PCIe domain，GPU 與 RNIC 成對掛在另一個 domain 的 PCIe 交換器下。右：P-RNIC，其中流表、封包噴灑與 credit 壅塞控制（橘色）是 Pegasus 加上的部分。Zou et al., SIGCOMM 2026, Figure 4(b) (accepted PDF, ACM eReader).

Two-level flow tables. The cloud's SDN controller talks only to the DPU. The DPU keeps the authoritative software table and pushes incremental changes to a hardware table in each RNIC (Figure 2). A table only in DPU memory was rejected: every lookup would cross PCIe, and eight RNICs would contend. The RNIC table fits on chip because the AI cloud is small in this dimension. A GPU serves one model, the underlay-to-overlay address map is 1:1, and a VPC needs tens of thousands of entries, not the tens of millions of a traditional cloud. The table is a 4-way cuckoo hash (four candidate slots per key, checked in parallel) of 32K entries; lookup and translation take at most 20 ns. A hit adds a VXLAN header that the tenant never sees; a miss drops the packet before it reaches the switch. After a hash collision interrupted a tenant's training job, each table received a 32-entry stash, an overflow list checked after the four slots.

兩層流表。雲端的 SDN 控制器只與 DPU 溝通。DPU 保存權威的軟體表，並把增量變更推送到每張 RNIC 的硬體表（圖 2）。只把表放在 DPU 記憶體的做法被否決：每次查表都要跨越 PCIe，八張 RNIC 還會互相爭用。RNIC 的表放得進晶片，是因為 AI 雲在這個維度上很小。一顆 GPU 只服務一個模型，underlay 到 overlay 的位址對應是 1:1，一個 VPC 只需數萬個表項，而不是傳統雲的數千萬個。這張表是 32K 個表項的 4 路 cuckoo hash（每個鍵有四個候選位置，平行檢查）；查表加位址轉換最多 20 ns。命中時加上租戶看不到的 VXLAN 標頭；未命中時在封包到達交換器之前就丟棄。在一次雜湊碰撞中斷了某租戶的訓練任務之後，每張表都加了 32 個表項的 stash，也就是在四個位置之後檢查的溢位清單。

Figure 2. The control path (top) and the data path (bottom) of the two-level flow table. Updates are slow and rare; lookups happen for every packet, in the RNIC. Values as §4.1 and §4.2 of the paper state them.圖 2。兩層流表的控制路徑（上）與資料路徑（下）。更新慢而少；查表則在 RNIC 裡對每個封包進行。數值依論文 §4.1 與 §4.2。Drawn for this primer from Zou et al., SIGCOMM 2026, §4.

Updates over a narrow channel. A DPU change and the matching RNIC change commit as one transaction, and a write-ahead log records it. The only channel from the DPU to a remote RNIC is the RNIC's CSR (control and status register) interface: serial accesses of 4–8 bytes at about 4 MB/s. The DPU is in a different PCIe domain, so it must also check each RNIC's PCIe address, which changes after a restart. A per-RNIC state machine (Initial, Channel Established, Waiting, Down) checks the address only around restarts, not on every command. In production this cut a VPC flow update from 3.54 s to 1.5 s.

經由狹窄通道更新。DPU 的變更與對應的 RNIC 變更以同一筆交易提交，並記入預寫日誌。DPU 到遠端 RNIC 的唯一通道是 RNIC 的 CSR（控制與狀態暫存器）介面：每次 4–8 位元組的序列存取，約 4 MB/s。DPU 位於不同的 PCIe domain，所以它還必須檢查每張 RNIC 的 PCIe 位址，而這個位址在重新啟動後會改變。每張 RNIC 一個狀態機（Initial、Channel Established、Waiting、Down），只在重新啟動前後檢查位址，而不是每個指令都檢查。在生產環境中，這把一次 VPC 流表更新從 3.54 秒縮短到 1.5 秒。

What the split saves. The paper states 10.5% less power per server than eight DPUs. Our arithmetic with the paper's device powers (analysis, not measurement) shows that this figure matches the future ASIC RNIC; the deployed FPGA RNIC saves less:

拆開省下多少。論文說每台伺服器比八台 DPU 省 10.5% 功耗。用論文給的裝置功耗做我們的算術（分析，不是量測），可以看出這個數字對應的是未來的 ASIC RNIC；目前部署的 FPGA RNIC 省得較少：

Question問題	Calculation算式	Result結果
Extra power of 8 DPUs (≈150 W) instead of 8 CX7 RNICs (≈45 W, Figure 1 of the paper)以 8 台 DPU（約 150 W）取代 8 張 CX7 RNIC（約 45 W，論文 Figure 1）多出的功耗	$8 \times (150 - 45)$	$\approx 840$ W
Server power for which 840 W is the paper's 6.5–13.7%讓 840 W 等於論文 6.5–13.7% 的伺服器功耗	$840 \div 0.137,\ 840 \div 0.065$	$\approx 6.1$ – $12.9$ kW
Saving of 1 DPU + 8 FPGA P-RNICs (≤75 W) against 8 DPUs1 台 DPU + 8 張 FPGA P-RNIC（≤75 W）相對 8 台 DPU 的節省	$8 \times 150 - (150 + 8 \times 75)$	$450$ W≈ 3.5–7.4% of the server約為伺服器的 3.5–7.4%
Same with the future 30 W ASIC P-RNIC換成未來 30 W 的 ASIC P-RNIC	$8 \times 150 - (150 + 8 \times 30)$	$810$ W≈ 6.3–13.3%: contains 10.5%約 6.3–13.3%：涵蓋 10.5%

So the cost of a DPU per NIC is real, about 840 W per server. Part of Pegasus's power advantage, however, waits for its ASIC RNIC.

所以每張 NIC 配一台 DPU 的代價是真的，每台伺服器約 840 W。不過 Pegasus 的功耗優勢有一部分要等它的 ASIC RNIC 才會兌現。

3.2 Spray every packet, order at the receiver3.2 逐封包噴灑，在接收端排序

Each P-RNIC writes a new UDP source port into every packet, so standard switch ECMP sends consecutive packets of one flow over different paths. The switches need no adaptive routing. Chapter 8 forbids exactly this for a go-back-N NIC, because a new source port per packet reorders the flow. A P-RNIC accepts out-of-order packets, so Pegasus pays for the reordering at the receiver.

每張 P-RNIC 為每個封包寫入新的 UDP 來源埠，所以一般交換器的 ECMP 會把同一條流的相鄰封包送上不同路徑。交換器不需要 adaptive routing。第 8 章對 go-back-N 的 NIC 明確禁止這種做法，因為每個封包換來源埠會讓流亂序。P-RNIC 接受亂序封包，所以 Pegasus 在接收端為亂序付出代價。

The write-with-immediate problem. NCCL sends a chunk as RDMA writes and then a write-with-immediate: a write that also delivers 4 bytes and raises a completion entry (CQE) at the receiver. Under spraying, it can overtake the earlier writes. Commodity RNICs with spraying therefore hold it until every earlier write is acknowledged, which costs one extra round trip per chunk (Figure 3, left).

write-with-immediate 的問題。NCCL 以多個 RDMA write 送出一個 chunk，最後再送一個 write-with-immediate：這種 write 還會帶 4 位元組資料，並在接收端產生一筆完成項目（CQE）。噴灑之下，它可能超車先前的 write。因此開啟噴灑的市售 RNIC 會扣住它，直到先前每個 write 都被確認，每個 chunk 多花一個往返時間（圖 3 左）。

Figure 3. One sprayed NCCL chunk: writes W1–W3 (W3 overtakes W2) and the write-with-immediate WI. Left: the sender waits for all ACKs before WI. Right: WI follows W3 at once, and the receiver raises the CQE when its PSN bitmap is full up to the position that isCQE marks, about one round trip earlier.圖 3。一個噴灑的 NCCL chunk：write W1–W3（W3 超車 W2）與 write-with-immediate WI。左：發送端等全部 ACK 回來才送 WI。右：WI 緊接著 W3 送出，接收端在 PSN 位元圖補齊到 isCQE 標記的位置時產生 CQE，大約早一個往返時間。Drawn for this primer from Zou et al., SIGCOMM 2026, §5.3 and Figure 7.

Pegasus moves the ordering to the receiver. Each QP keeps two bitmaps over its window: a PSN bitmap (which packets have arrived) and an isCQE bitmap (which positions must raise a completion once all earlier packets are in). The receiver stores the 4-byte immediate and raises the CQE only when the window head passes that position. The sender sends the write-with-immediate right after the last write, and the extra round trip disappears (Figure 3, right). Each packet also carries its virtual address, rkey and length, so the RNIC writes it straight into GPU memory on arrival. This is the direct placement that IRN proposed (Section 7.6). The bitmap state costs at most 80 bytes per QP.

Pegasus 把排序搬到接收端。每個 QP 在它的視窗上維護兩張位元圖：PSN 位元圖（哪些封包已到）與 isCQE 位元圖（哪些位置在先前封包都到齊後必須產生完成項目）。接收端先存下 4 位元組的 immediate，等視窗前端越過該位置時才產生 CQE。發送端在最後一個 write 之後立刻送出 write-with-immediate，多出的往返時間就消失了（圖 3 右）。每個封包也帶著自己的虛擬位址、rkey 與長度，所以 RNIC 在封包到達時就直接寫進 GPU 記憶體。這就是 IRN 提出的直接放置（第 7.6 節）。位元圖的狀態每個 QP 最多 80 位元組。

3.3 Credits in the NIC, with one batch of slack3.3 NIC 中的 credit，外加一個 batch 的餘裕

Spraying spreads load, but it cannot stop incast: in all-to-all and tree all-reduce, many senders converge on one receiver. DCQCN reacts only after a queue forms, and as links move from 400 to 800 Gb/s the bandwidth-delay product grows faster than switch buffers. Pegasus therefore puts the core of credit-based congestion control (as in ExpressPass, NDP, Aeolus and EQDS) in the RNIC. Each QP sends its first BDP of data without credit. The sender then requests credits at line rate, one request per 16 KB batch of 4 packets. The receiver polls the requesting QPs round-robin and grants one credit per batch; credits travel in high-priority queues. A sender that collects more credit than it can use sends OFF, and ON once it has used up the surplus.

噴灑能分散負載，卻擋不住 incast：在 all-to-all 與樹狀 all-reduce 中，許多發送端會同時湧向同一個接收端。DCQCN 要等佇列形成後才反應，而當鏈路從 400 Gb/s 升到 800 Gb/s，頻寬延遲乘積長得比交換器緩衝還快。因此 Pegasus 把 credit 壅塞控制的核心（與 ExpressPass、NDP、Aeolus、EQDS 同一路線）放進 RNIC。每個 QP 的第一個 BDP 資料不需要 credit 就送出。之後發送端以線速請求 credit，每個請求對應一個 16 KB、4 個封包的 batch。接收端以輪詢方式服務提出請求的 QP，每個 batch 授予一個 credit；credit 走高優先佇列。若發送端收到的 credit 多到用不完，就送出 OFF，待多出的 credit 用掉後再送 ON。

The first hardware test failed in an unexpected way. Four FPGA RNICs ran NCCL-tests all-to-all, and throughput fell short although PFC and ECN counters stayed at zero and no OFF signal appeared. At times every active QP on the sender had zero credit. At 400 Gb/s a 16 KB batch lasts about 320 ns, so a credit must arrive every 320 ns. The receiver's transmit threads are often busy with its own batches and cannot send a credit in time, and switches add nanoseconds of jitter. The default rule sends

第一次硬體測試以出乎意料的方式失敗。四張 FPGA RNIC 執行 NCCL-tests all-to-all，PFC 與 ECN 計數器都是零，也沒有出現 OFF 訊號，吞吐卻不足。有些時候，發送端所有活躍 QP 的 credit 都是零。在 400 Gb/s 下一個 16 KB 的 batch 約持續 320 ns，所以每 320 ns 就必須有一個 credit 到達。接收端的傳送執行緒常常忙著送自己的 batch，來不及送出 credit，交換器也會加上數奈秒的抖動。預設規則送出的量是

\text{sendsize} = \min(\text{batch\_size},\ \text{re\_data},\ \text{re\_credit}) \tag{1}

where re_data is the data left to send and re_credit the credit left. Pegasus sends a full batch whenever the credit left is not negative:

其中 re_data 是尚未送出的資料，re_credit 是剩下的 credit。Pegasus 則只要剩下的 credit 不是負的，就送出一整個 batch：

\text{sendsize} = \min(\text{batch\_size},\ \text{re\_data}) \ \text{ if } \text{re\_credit} \ge 0, \ \text{ else } 0 \tag{2}

\text{sendsize} = \min(\text{batch\_size},\ \text{re\_data}) \ \text{ 若 } \text{re\_credit} \ge 0, \ \text{ 否則 } 0 \tag{2}

Credit can then fall below zero by at most one batch, and later grants repay it. Figure 8 of the paper shows the effect on the same testbed.

於是 credit 最多可以低於零一個 batch，之後的授予再把它補回。論文 Figure 8 顯示同一個測試床上的效果。

Grouped bar chart of bandwidth in Gbps from 300 to 400 against QP number 2 to 512; original credit CC bars vary between about 323 and 383, jitter-tolerant bars stay near 385 to 387. — Figure 8 of the paper (measured: four FPGA RNICs, NCCL-tests all-to-all, throughput from the RNIC's TX-byte register). Without the overdraft (purple), throughput ranges from about 323 Gb/s (512 QPs) to 383 Gb/s (2 QPs); with it (yellow), it stays at about 385–387 Gb/s. The axis starts at 300 Gb/s.論文 Figure 8（實測：四張 FPGA RNIC、NCCL-tests all-to-all，吞吐取自 RNIC 的 TX 位元組暫存器）。不允許透支時（紫），吞吐在約 323 Gb/s（512 個 QP）到 383 Gb/s（2 個 QP）之間；允許透支後（黃）維持在約 385–387 Gb/s。縱軸從 300 Gb/s 起算。Zou et al., SIGCOMM 2026, Figure 8 (accepted PDF, ACM eReader); values read from the figure.

Why does one batch of slack suffice? Panel 1 is our model of the credit loop. A late credit stalls a sender that has no spare credit; with the overdraft, the sender sends the next batch on time and the late credit repays the debt. The model shows that the overdraft absorbs jitter up to one batch time.

為什麼一個 batch 的餘裕就夠？面板 1 是我們對 credit 迴路的模型。遲到的 credit 會讓沒有備用 credit 的發送端停下來；允許透支時，發送端準時送出下一個 batch，遲到的 credit 再償還欠額。模型顯示，透支能吸收最多一個 batch 時間的抖動。

Panel 1 · model面板 1 · 模型

A credit loop with and without the one-batch overdraft有無「透支一個 batch」的 credit 迴路

Line rate線速

Batch per credit每個 credit 的 batch

Credit jitter, up tocredit 抖動上限

Credit loop time (our assumption)credit 迴路時間（我們的假設）

Batch time = credit interval

batch 時間 = credit 間隔

Free window (first BDP)

免費視窗（第一個 BDP）

Equation (1): wait for credit

式 (1)：等 credit

Equation (2): one batch ahead

式 (2)：先送一個 batch

Our model, not a measurement. The receiver returns one credit per batch it receives. The credit reaches the sender one loop time later, plus a random delay of 0 to the jitter limit (the same fixed random sequence for both rules). The sender starts with a free window of the first BDP, so without jitter it runs at line rate with no spare credit. Equation (1) waits for a whole credit; Equation (2) may run one batch ahead. Defaults are the paper's 400 Gb/s and 16 KB batch. The paper gives no loop time and no jitter size. The 2.6 µs loop and 330 ns jitter (about one batch, the time a credit can wait for a busy transmit thread) are our choices. The model leaves out QP count, protocol overhead and ON/OFF, so it does not reproduce Figure 8 bar by bar. Try 800 Gb/s: the same jitter now costs more, because one batch lasts half as long. Then try 64 KB batches.

這是我們的模型，不是量測。接收端每收到一個 batch 就回一個 credit。credit 在一個迴路時間之後到達發送端，再加上 0 到抖動上限之間的隨機延遲（兩種規則用同一組固定的隨機序列）。發送端一開始有第一個 BDP 的免費視窗，所以沒有抖動時剛好以線速運作、沒有任何備用 credit。式 (1) 要等到一整個 credit；式 (2) 可以先送一個 batch。預設值是論文的 400 Gb/s 與 16 KB batch。論文沒有給迴路時間，也沒有給抖動大小。2.6 µs 的迴路與 330 ns 的抖動（約一個 batch，也就是 credit 等待忙碌傳送執行緒的時間）是我們的選擇。模型沒有包含 QP 數、協定開銷與 ON/OFF，所以不會逐根重現 Figure 8。試試 800 Gb/s：同樣的抖動代價變大，因為一個 batch 只持續一半的時間。再試試 64 KB 的 batch。

Stop and think停下來想一想

In Panel 1, set the jitter to 600 ns: Equation (2) falls to about 379 Gb/s. Now move the batch from 16 KB to 64 KB. Equation (2) returns to line rate, while Equation (1) gains much less (342 to 352 Gb/s). Why? Compare the jitter with the batch time, then with the loop time. What does a 64 KB batch cost in exchange?

在面板 1 把抖動設成 600 ns：式 (2) 降到約 379 Gb/s。接著把 batch 從 16 KB 改成 64 KB。式 (2) 回到線速，式 (1) 卻只稍微改善（342 到 352 Gb/s）。為什麼？先把抖動和 batch 時間比較，再和迴路時間比較。換成 64 KB 的 batch，又要付出什麼代價？

3.4 Loss, PCIe incast and older NICs3.4 丟包、PCIe incast 與舊型 NIC

Loss versus reordering. Spraying keeps reordering limited, and loss is rare: over 20 months of production the network was loss-free more than 99.98% of the time. The receiver asks for a retransmission only when both the out-of-order degree and the share of missing packets in its bitmap pass thresholds. In a 127-thread stress test this cut unnecessary retransmissions from 1% (out-of-order threshold alone) to 0.006%.丟包與亂序。噴灑讓亂序有限，而丟包很少：20 個月的生產資料中，網路超過 99.98% 的時間沒有丟包。只有當亂序程度與位元圖中遺失封包的比例都超過門檻時，接收端才要求重傳。在 127 個執行緒的壓力測試中，這把不必要的重傳從 1%（只用亂序門檻）降到 0.006%。
PCIe incast inside the server. KV-cache and multimodal offload copy host memory to the GPU over the PCIe link that the RNIC also writes into. When an RNIC's internal queue passes a threshold, its credit generator slows down. TX PFC is on by default, but it is turned off per NIC when that NIC's TX PFC counters run high, so local PCIe congestion does not turn into pause frames in the fabric.伺服器內的 PCIe incast。KV-cache 與多模態卸載會經由 PCIe 把主機記憶體複製到 GPU，而 RNIC 也寫入同一條 PCIe 鏈路。當 RNIC 內部佇列超過門檻時，它的 credit 產生器就放慢。TX PFC 預設開啟，但某張 NIC 的 TX PFC 計數過高時就單獨關掉，讓本地的 PCIe 壅塞不會變成網路中的暫停幀。
Older NICs. P-RNICs mark their packets with 0x03 in a reserved field of the RoCE header (CX7 sends 0x00), credit and CNP signals are translated, and switches keep DCQCN and credit traffic in separate queues. The paper reports, without numbers, that mixed jobs perform on par with CX7-only clusters.舊型 NIC。P-RNIC 在 RoCE 標頭的保留欄位標記 0x03（CX7 送 0x00），credit 與 CNP 訊號互相轉譯，交換器則把 DCQCN 與 credit 流量放在不同佇列。論文表示混用的任務與純 CX7 叢集表現相當，但沒有給數字。

4. Implementation and deployment4. 實作與部署

Tencent built the hardware itself. The P-DPU is Arm plus FPGA; the P-RNIC is an FPGA card with 2 × 200 Gb/s ports, and an ASIC version is planned. AI traffic freed the FPGA budget for the new features. Mainstream libraries (NCCL, NIXL, UCCL, DeepEP) use RDMA write, not send/recv, so dropping hardware send/recv saved up to 7K ALMs (FPGA logic blocks). QP counts are small: at most 2,000 concurrent QPs per RNIC in AI training, against 10,000 in Tencent's storage clusters. Services that need send/recv use the front-end NIC. The paper gives no lines of code and does not describe the fabric: no tiers, oversubscription, switch model or buffer sizes.

騰訊自己打造了硬體。P-DPU 是 Arm 加 FPGA；P-RNIC 是具有 2 × 200 Gb/s 埠的 FPGA 卡，ASIC 版本在規劃中。AI 流量替新功能騰出了 FPGA 預算。主流函式庫（NCCL、NIXL、UCCL、DeepEP）使用 RDMA write 而非 send/recv，所以拿掉硬體 send/recv 省下最多 7K 個 ALM（FPGA 邏輯單元）。QP 數很少：AI 訓練中每張 RNIC 最多 2,000 個並行 QP，騰訊的儲存叢集則要 10,000 個。需要 send/recv 的服務改用前端 NIC。論文沒有給程式碼行數，也沒有描述網路本身：沒有層數、超額認購、交換器型號或緩衝大小。

By February 2026 Pegasus had run for two years and covered 8K GPUs in 1K servers, each with one DPU and eight RNICs. About 100 tenant tasks run per day, each on 1 to 2,400 GPUs: training, inference, recommendation, and tenants' own tests. Company-scale foundation-model training stays in dedicated clusters. Figure 9 of the paper shows the mix.

到 2026 年 2 月，Pegasus 已運行兩年，涵蓋 1K 台伺服器上的 8K 顆 GPU，每台一台 DPU 與八張 RNIC。每天約有 100 個租戶任務，每個使用 1 到 2,400 顆 GPU：訓練、推論、推薦，以及租戶自己的測試。公司級的基礎模型訓練則留在專用叢集。論文 Figure 9 顯示了任務組成。

Left: step CDFs of number of GPUs (8 to 2K) for recommendation, inference and training; inference starts near 0.98 at 8 GPUs, recommendation near 0.75, training near 0.26. Right: line plot of active tasks from 2026-01-11 to 2026-02-05, rising from 100 to about 230, peaking near 330 and settling near 260. — Figure 9(a) and 9(d) of the paper (production). Left: GPUs per tenant task over six months; more than 98.2% of inference tasks use one 8-GPU server, while training spreads from 8 to 2,400 GPUs. Right: active tasks from January 11, 2026, at least 100 and peaking above 300.論文 Figure 9(a) 與 9(d)（生產環境）。左：六個月內每個租戶任務使用的 GPU 數；超過 98.2% 的推論任務只用一台 8 GPU 伺服器，訓練則從 8 顆分布到 2,400 顆。右：自 2026 年 1 月 11 日起的活躍任務數，至少 100 個、峰值超過 300 個。Zou et al., SIGCOMM 2026, Figure 9(a) and (d) (accepted PDF, ACM eReader); (b) and (c) omitted.

The mix explains the management design. Tasks are small and short (about 27 hours on average), and hundreds run at once, so VPCs are built and torn down all the time. That is why seconds of flow-update time count as a production metric.

任務組成解釋了管理面的設計。任務小而短（平均約 27 小時），同時有數百個在跑，所以 VPC 不斷被建立與拆除。這就是為什麼數秒的流表更新時間會被當成生產指標。

5. What was measured5. 量到了什麼

The paper reports no simulation. Every result below is a measurement; the power figures of section 3.1 are analysis. The testbed is four servers, each with two Xeon CPUs, eight H20 GPUs and eight 2 × 200 Gb/s RNICs (CentOS 8, OFED 23.10), plus a five-RNIC incast setup under one ToR. Rows marked "appendix" come from Appendix A, and the main text cites both results.

論文沒有任何模擬結果。下面每個結果都是量測；第 3.1 節的功耗數字則是分析。測試床是四台伺服器，每台有兩顆 Xeon CPU、八顆 H20 GPU 與八張 2 × 200 Gb/s 的 RNIC（CentOS 8、OFED 23.10），另有一個在同一台 ToR 下五張 RNIC 的 incast 設定。標記「附錄」的列來自附錄 A；正文引用了這兩個結果。

Test測試	Setup設定	Measured實測結果
Credit jitter (Fig. 8)credit 抖動（Fig. 8）	4 FPGA RNICs, all-to-all, 2–512 QPs4 張 FPGA RNIC、all-to-all、2–512 個 QP	323–383 → about 385–387 Gb/s with the overdraft323–383 → 允許透支後約 385–387 Gb/s
Spurious retransmission (Fig. 12)不必要的重傳（Fig. 12）	127-thread stress test127 個執行緒的壓力測試	1% → 0.006%1% → 0.006%
Incast, credits vs DCQCN (Fig. 14, appendix)incast，credit 對 DCQCN（Fig. 14，附錄）	4→1, 5 RNICs under one ToR, perftest, 4–4,000 QPs4→1、同一 ToR 下 5 張 RNIC、perftest、4–4,000 個 QP	Up to 64 QPs: 44.6–82.7% less queueing delay. Above: DCQCN loses 20.2–32.5% of bandwidth, credits stay within 3%64 個 QP 以內：排隊延遲少 44.6–82.7%。以上：DCQCN 損失 20.2–32.5% 頻寬，credit 維持在 3% 以內
AllReduce with spraying (Fig. 13, appendix)噴灑下的 AllReduce（Fig. 13，附錄）	4 servers, CX7 with adaptive routing vs P-RNIC4 台伺服器，開啟 adaptive routing 的 CX7 對 P-RNIC	About 2× bus bandwidth at 8–16 MB (read from figure); about equal at 64 MB and above8–16 MB 時 bus bandwidth 約 2 倍（讀自圖）；64 MB 以上大致相同

The production results about one system are the 95 s TE relief time, the 99.98% loss-free time and the faster flow updates. The end-to-end case study is different. It runs the same models, parallelism and batch size on Pegasus and on Tencent's prior-generation AI cluster (CX7 RNICs, DCQCN, controller TE). These are two different clusters, and the NIC, the load balancing and the congestion control all changed between them, so no gain below belongs to one mechanism. The paper also prints the gains in its own convention, gain $= 1 - \text{old}/\text{new}$ , with Pegasus normalized to 1 in its Figure 10. The table turns them into ratios:

只涉及單一系統的生產結果是 TE 的 95 秒紓解時間、99.98% 無丟包時間與較快的流表更新。端到端的案例研究則不同。它在 Pegasus 與騰訊上一代 AI 叢集（CX7 RNIC、DCQCN、控制器 TE）上執行相同的模型、平行策略與批次大小。這是兩個不同的叢集，NIC、負載平衡與壅塞控制同時都不同，所以下面的任何增益都不能歸給單一機制。論文還用自己的換算方式印出增益：增益 $= 1 - \text{舊}/\text{新}$ ，並在 Figure 10 把 Pegasus 正規化為 1。下表把它們換成倍數：

Question問題	Calculation算式	Result結果
LLaMA 7B, 96 GPUs: old cluster's rate relative to Pegasus, from the printed 53.2%LLaMA 7B、96 顆 GPU：由印出的 53.2% 推回舊叢集相對 Pegasus 的速率	$1 - 0.532$	$0.468$ Figure 10(c) shows the same barFigure 10(c) 的長條相同
LLaMA 7B: Pegasus over the old clusterLLaMA 7B：Pegasus 除以舊叢集	$1 \div 0.468$	$\approx 2.14\times$ +114% in the usual convention一般換算為 +114%
70B dense, 32 GPUs, printed 7.5%70B 稠密、32 顆 GPU，印出 7.5%	$1 \div (1 - 0.075)$	$\approx 1.08\times$
7B MoE, 32 GPUs, printed 4.8%7B MoE、32 顆 GPU，印出 4.8%	$1 \div (1 - 0.048)$	$\approx 1.05\times$

So, on two different clusters in which the NIC, the load balancing and the congestion control all changed, Pegasus trained LLaMA 7B about 2.14× as fast, the 70B dense model about 1.08× and the 7B MoE model about 1.05×. For small gains the two conventions nearly agree; for the 53.2% they do not. The paper blames the large LLaMA gap on TE: one parameter set did not fit every task mix, and the old cluster needed manual tuning. In the same two-cluster comparison, with the same NIC, load-balancing and congestion-control changes, inference on 16 GPUs under different ToRs over one hour gained 3.7% in tokens per second and cut mean request completion time by 7.2%.

所以，在 NIC、負載平衡與壅塞控制同時都不同的兩個叢集上，Pegasus 訓練 LLaMA 7B 的速度約為 2.14 倍，70B 稠密模型約 1.08 倍，7B MoE 模型約 1.05 倍。增益小時兩種換算幾乎一致；53.2% 則差很多。論文把 LLaMA 的大落差歸因於 TE：一組參數無法適用所有任務組合，舊叢集需要手動調校。在同一個兩叢集比較中（同樣是 NIC、負載平衡與壅塞控制都改變），16 顆分屬不同 ToR 的 GPU 執行一小時推論，每秒 token 數增加 3.7%，平均請求完成時間減少 7.2%。

6. Limits and open questions6. 限制與未解的問題

The paper's own limits: the CSR channel is a bottleneck, and the authors ask vendors for better management interfaces. The small RNIC table suffers hash collisions, hence the stash and the advice to re-split VPCs. PCIe QoS is not available across vendors, so the PCIe remedies are workarounds. Native send/recv and the ASIC RNIC are future work. A careful reader adds:

論文自己承認的限制：CSR 通道是瓶頸，作者呼籲廠商提供更好的管理介面。RNIC 的表很小、雜湊碰撞嚴重，所以才有 stash 與重新拆分 VPC 的建議。跨廠商的 PCIe QoS 尚不可得，所以 PCIe 的對策只是權宜之計。原生 send/recv 與 ASIC RNIC 都是未來的工作。細心的讀者還會補上：

Attribution. The production comparison changes three things at once on different clusters. Only the testbed separates spraying (Figure 13) from credits (Figures 8 and 14), and only on four servers or five RNICs.歸因。生產比較在不同的叢集上同時改了三件事。只有測試床把噴灑（Figure 13）與 credit（Figure 8 與 14）分開，而且只用了四台伺服器或五張 RNIC。
A network paper without the network. No topology is given, so claims about load in the core cannot be tied to a design. The chapter overview's map leaves Pegasus out for this reason.沒有網路的網路論文。論文沒有給拓樸，所以關於核心層負載的說法無法連到某個設計。本章總覽的地圖也因此沒有擺放 Pegasus。
Scale. The target is more than 100,000 GPUs and VPCs; the deployment is 8K GPUs, and the end-to-end tests use 16–96 GPUs. The 32K-entry table and the CSR path are untested at the target.規模。目標是超過 100,000 顆 GPU 與 VPC；部署規模是 8K 顆，端到端測試只用 16–96 顆。32K 表項的表與 CSR 路徑都沒有在目標規模下測試。
Credits beside PFC and DCQCN. PFC stays on, and DCQCN and credit traffic share switches. Fairness between the two classes is not measured.credit 與 PFC、DCQCN 並存。PFC 仍開著，DCQCN 與 credit 流量共用交換器。兩類流量之間的公平性沒有量測。
One DPU, two roles. The DPU is the front-end NIC and the RNICs' control plane. A DPU failure reboots the whole server, and no failure rates are given.一台 DPU，兩種角色。DPU 既是前端 NIC，也是 RNIC 的控制平面。DPU 故障會讓整台伺服器重新開機，論文沒有給故障率。

7. How it connects7. 與其他內容的關聯

Tencent's two papers in this chapter answer one question under two tenancy models. Astral keeps the owner's model and buys bandwidth; Pegasus has no owner of the software and moves the fixes into the NIC. The table places both beside HPN and MegaScale as Chapter 13's matrix describes them.

騰訊在本第 13 章的兩篇論文，在兩種租戶模式下回答同一個問題。Astral 維持擁有者模式並買足頻寬；Pegasus 沒有軟體的擁有者，於是把修正搬進 NIC。下表依第 13 章的矩陣，把兩者與 HPN、MegaScale 並列。

	Pegasus	Astral	HPN	MegaScale
Who owns NCCL誰掌控 NCCL	The tenant租戶	Tencent (stock, PXN)騰訊（原版，PXN）	Alibaba阿里巴巴	ByteDance字節跳動
Load balance負載平衡	Per-packet spraying in the RNIC在 RNIC 逐封包噴灑	Source ports via switch hash; 5-s ECN controller以交換器雜湊挑來源埠；5 秒 ECN 控制器	Dual-plane wiring; host path choicedual-plane 接線；主機選路	2× uplinks; placement2 倍上行；配置
Congestion control壅塞控制	Receiver credits in the RNIC; PFC keptRNIC 中的接收端 credit；保留 PFC	DCQCN + PFCDCQCN + PFC	RoCEv2 over PFC (not detailed)PFC 上的 RoCEv2（未詳述）	Swift RTT + DCQCN ECNSwift RTT + DCQCN ECN
Topology拓樸	Not described未描述	1:1 fat tree, 128K GPUs1:1 fat tree，128K 顆 GPU	15K pod, 15:1 core15K pod，核心 15:1	3-tier Tomahawk 4 Clos三層 Tomahawk 4 Clos

Read the rows against Chapter 13. On load balance, Meta adds flows, HPN removes choices by wiring, MegaScale buys 2× uplinks, Astral steers source ports with a slow controller, and Pegasus gives up per-flow paths. On congestion control, MegaScale and Pegasus both found DCQCN poor under all-to-all; MegaScale added an RTT signal, and Pegasus replaced DCQCN with credits, Meta's clear-to-send moved from the library into the NIC. HPN's objection to rail-only, that one tenant cannot relay traffic through another tenant's GPU, is the constraint Pegasus is built around.

把這幾列對照第 13 章來讀。負載平衡方面，Meta 增加流數，HPN 用接線拿掉選擇，MegaScale 買 2 倍上行，Astral 以慢速控制器操控來源埠，Pegasus 則乾脆放棄逐流路徑。壅塞控制方面，MegaScale 與 Pegasus 都發現 DCQCN 在 all-to-all 下表現差；MegaScale 加上 RTT 訊號，Pegasus 則以 credit 取代 DCQCN，也就是把 Meta 的 clear-to-send 從函式庫搬進 NIC。HPN 反對 rail-only 的理由（一個租戶不能透過另一個租戶的 GPU 中繼流量）正是 Pegasus 據以設計的限制條件。

7.1 What an operator can take from it7.1 營運者可以學到什麼

Ask who owns the collective library. If you do, co-design as Meta and Astral do. If tenants do, the fixes must sit below the library, in the NIC.先問誰掌控集體通訊函式庫。若是你，就像 Meta 與 Astral 那樣共同設計。若是租戶，修正就必須放在函式庫之下，也就是 NIC 裡。
Compare the controller's reaction time with how fast the traffic changes. 10-s statistics fit periodic training, not overlapping medium flows.拿控制器的反應時間和流量變化的速度比較。10 秒的統計適合週期性的訓練，不適合互相重疊的中型流。
Budget the receiver work that spraying creates. Completion ordering, direct placement and loss detection all move to the receiver; small QP counts make the state affordable.為噴灑帶來的接收端工作編列預算。完成順序、直接放置與丟包偵測都移到接收端；QP 數少才負擔得起這些狀態。
Give credits slack. At 400 Gb/s a 16 KB credit covers about 320 ns, and the paper reports hardware jitter at the nanosecond level. One batch of overdraft absorbs it.給 credit 留餘裕。400 Gb/s 下一個 16 KB 的 credit 只涵蓋約 320 ns，而論文指出硬體有奈秒級的抖動。透支一個 batch 就能吸收。
Count NIC power per GPU. A DPU per NIC costs GPUs under a fixed power budget, and the saving of a split design depends on your own RNIC's power.以每顆 GPU 計算 NIC 功耗。每張 NIC 配一台 DPU 會在固定電力預算下吃掉 GPU，而拆分設計能省多少，取決於你自己 RNIC 的功耗。

Chapter 13: section 4 (a fourth answer on axis (c)), section 6 item 3 (bounded bytes in flight) and section 7, steps 3 and 5 and Panel 7, where tenancy and library ownership are inputs.第 13 章：第 4 節（軸 (c) 的第四種答案）、第 6 節第 3 項（限制在途位元組）與第 7 節第 3、5 步及面板 7，那裡租戶模式與函式庫擁有權都是輸入。
Chapter 8, section 5 and its spraying panel; Chapter 7, sections 3–4 (PFC, DCQCN) and 6 (direct placement, receiver credits); Chapter 9 for the DPU as front-end NIC.第 8.5 節與它的噴灑面板；第 7 章第 3–4 節（PFC、DCQCN）與第 6 節（直接放置、接收端 credit）；DPU 作為前端 NIC 的部分見第 9 章。
Guided readings: Meta RoCE (clear-to-send, QP scaling), IRN (selective retransmission, direct placement), DCQCN (the baseline beaten under incast).論文導讀：Meta RoCE（clear-to-send、QP scaling）、IRN（選擇性重傳、直接放置）、DCQCN（在 incast 下被超越的基線）。

Key numbers關鍵數字

Number	數字	Meaning (setup)
8K GPUs / 1K servers	Deployed by February 2026, two years in service (production)	2026 年 2 月的部署規模，已服務兩年（生產）
6.5–13.7%	Server power added by one DPU per RNIC (analysis, the paper's arithmetic)	每張 RNIC 配一台 DPU 增加的伺服器功耗（分析，論文的算術）
≈ 95 s	Congestion relief by the old controller TE in the cloud (production)	舊的控制器 TE 在雲端紓解壅塞的時間（生產）
32K; ≤ 20 ns	RNIC table entries (FPGA); lookup and translation time (hardware)	RNIC 表項數（FPGA）；查表與轉換時間（硬體）
3.54 s → 1.5 s	VPC flow update with the RNIC state machine (production)	加上 RNIC 狀態機後的 VPC 流表更新（生產）
≤ 80 B	Receiver bitmap state per QP for write-with-immediate ordering (hardware)	每個 QP 為 write-with-immediate 排序所需的位元圖狀態（硬體）
323–383 → ≈ 385–387 Gb/s	Credit CC without / with overdraft, 2–512 QPs (testbed, 4 FPGA RNICs)	credit 壅塞控制不透支 / 透支，2–512 個 QP（測試床，4 張 FPGA RNIC）
1% → 0.006%; > 99.98%	Unnecessary retransmissions (stress test, setting not stated); loss-free time over 20 months (production)	不必要的重傳（壓力測試，未說明環境）；20 個月的無丟包時間（生產）
≈ 2.14× / 1.08× / 1.05×	Training rate, LLaMA 7B (96 GPUs) / 70B dense / 7B MoE (32 GPUs each), against a different cluster whose NIC, load balancing and CC all differ (production; printed as +53.2% / +7.5% / +4.8%)	訓練速率，LLaMA 7B（96 顆 GPU）/ 70B 稠密 / 7B MoE（各 32 顆），對比 NIC、負載平衡與壅塞控制都不同的另一個叢集（生產；論文印成 +53.2% / +7.5% / +4.8%）
+3.7% / −7.2%	Inference tokens/s / mean request completion time, 16 GPUs for 1 hour, in the same two-cluster comparison with NIC, load balancing and CC all changed (production)	推論每秒 token 數 / 平均請求完成時間，16 顆 GPU、1 小時，同一個 NIC、負載平衡與壅塞控制都改變的兩叢集比較（生產）

Terms used on this page本頁用到的術語

Bare-metal AI cloud: Renting whole GPU servers to tenants with no hypervisor, while the provider still isolates each tenant's network.; 把整台 GPU 伺服器租給租戶、不用 hypervisor，同時業者仍隔離每個租戶的網路。
VPC, overlay, underlay: A tenant's virtual network; the addresses the tenant sees; the fabric's own addresses. Pegasus maps them 1:1 with VXLAN.; 租戶的虛擬網路；租戶看到的位址；網路本身的位址。Pegasus 以 VXLAN 將兩者 1:1 對應。
P-DPU, P-RNIC: Pegasus's own DPU (Arm + FPGA; front-end NIC and RNIC manager) and RNIC (FPGA; flow table, spraying, credits).; Pegasus 自研的 DPU（Arm + FPGA；前端 NIC 兼 RNIC 管理者）與 RNIC（FPGA；流表、噴灑、credit）。
Cuckoo hash, stash: A table where each key has four candidate slots checked in parallel; the stash is a small overflow list for collisions.; 每個鍵有四個平行檢查的候選位置的表；stash 是放置碰撞項目的小型溢位清單。
CSR channel: An RNIC's control and status registers, used as the DPU-to-RNIC management path (about 4 MB/s).; RNIC 的控制與狀態暫存器，被用作 DPU 到 RNIC 的管理通道（約 4 MB/s）。
Write-with-immediate, CQE: An RDMA write that also delivers 4 bytes and raises a completion queue entry at the receiver; NCCL uses it to signal a chunk.; 一種還會帶 4 位元組、並在接收端產生完成佇列項目的 RDMA write；NCCL 用它通知一個 chunk 已到。
PSN and isCQE bitmaps: Per-QP receiver bitmaps: which packets arrived, and which positions must raise a completion.; 每個 QP 在接收端的位元圖：哪些封包已到，以及哪些位置必須產生完成項目。
Credit-based congestion control: The receiver grants credits (here one per 16 KB batch); senders transmit only against credit, after a free first BDP.; 由接收端授予 credit（這裡每個 16 KB batch 一個）；發送端在免費的第一個 BDP 之後，只能憑 credit 傳送。
Overdraft: Equation 2: the sender may let its credit go negative by one batch, which absorbs nanosecond credit jitter.; 式 2：發送端可讓 credit 變成負的一個 batch，藉此吸收奈秒級的 credit 抖動。
TX PFC: A NIC sending pause frames into the fabric; Pegasus turns it off per NIC when internal PCIe congestion would spread.; NIC 向網路送出暫停幀；當伺服器內的 PCIe 壅塞可能擴散時，Pegasus 會單獨關掉該 NIC 的 TX PFC。