Measuring inside the server: FabricPerf, InfiniteHBD, PingPoint量測伺服器內部：FabricPerf、InfiniteHBD、PingPoint
How to observe an NVLink domain that has no NIC counters, how large a scale-up domain should be, and a GPU package treated as a network.如何觀察沒有 NIC 計數器的 NVLink 域、scale-up 域應該多大，以及把 GPU 封裝當成網路來看。
This page assumes Chapters 1–4. Every term it uses is defined there; the glossary on the contents page links to each definition. The papers here are recent and most of their results come from small testbeds or simulation, so read each number with its label.
本頁假設你已讀過第 1–4 章。這裡用到的每個名詞都在那些章節中定義過；目錄頁的詞彙表連到每個定義。這些論文都很新，多數結果來自小型測試平台或模擬，所以請連同標籤一起讀每個數字。
Terms from later chapters used here (4)本頁用到的後面章節名詞（4 個）
- Telemetry遙測（telemetry） Ch. 9 Counters and logs collected from hosts and switches to watch the network.從主機與交換器收集的計數器與日誌，用來監看網路。
- Jitter, order statistics抖動、順序統計 Ch. 10 Random step-to-step variation of each rank; the step waits for the maximum, which grows with the number of ranks.每個 rank 每步的隨機變動；步驟等待最大值，而最大值隨 rank 數成長。
- Standby pool, spare備用池、備援 Ch. 10 Spare machines kept ready so a failed one can be replaced without waiting for a repair.預備的備用機器，故障時可立即替換，不必等待修理。
- Simulation, packet-level simulation模擬、封包層級模擬 Ch. 12 Running a model of the workload and network in software; a packet-level simulator follows every packet through every queue.以軟體執行工作負載與網路的模型；封包層級模擬器追蹤每個封包經過每個佇列。
This page follows a byte from HBM down to the NIC, and its tools are the NIC's: counters, timestamps and PTP clocks. Three recent papers show that the network does not start at the NIC. It continues inside the server, across the NVLink domain, and inside the GPU package, between chiplets. There the usual tools see nothing. No NIC is on the path, the switches are closed, and the traffic is loads and stores that GPU threads issue. Two of the papers measure these hidden networks from inside GPU kernels. The third asks how large such a domain should be and how it should survive failures.
本頁跟著一個位元組從 HBM 往下走到 NIC，用的也都是 NIC 的工具：計數器、時間戳與 PTP 時鐘。三篇近期論文指出，網路並不是從 NIC 才開始。它延伸到伺服器內部、橫跨 NVLink 網域，甚至延伸到 GPU 封裝內部的 chiplet 之間。在那裡，平常的工具什麼都看不到。路徑上沒有 NIC，交換器是封閉的，流量則是 GPU 執行緒發出的 load 與 store。其中兩篇論文從 GPU kernel 內部量測這些隱藏的網路。第三篇問的是這種網域應該做多大，以及如何撐過故障。
FabricPerf: inside the NVLink domain. FabricPerf (SIGCOMM 2026) turns the measurement of a scale-up network into the profiling of NCCL's communication kernels. Probes on three instructions give five timestamps per transfer, a GPU-native PTP aligns the clocks, and a memory model lets L2 counters explain the fabric. The testbeds were an H100 NVL8 and a GB200 NVL16; on the GB200, the clock offset jittered by 101.8 ns on average inside a kernel. On the H100 with 24 channels, P99 channel latency was about 14 µs against a P50 near 7 µs; work stealing raised AllGather from 316.9 to 334.4 GB/s. On the GB200, L2 hit rates fell to 15% or less at 16 or more channels, and to about 0% with 32 channels at 4 MB buffers; eviction hints raised the ReduceScatter read hit rate from 0.31% to 35.17% on one GB200 node.
FabricPerf：NVLink 網域內部。FabricPerf（SIGCOMM 2026）把 scale-up 網路的量測變成對 NCCL 通訊 kernel 的剖析。掛在三個指令上的探針為每次傳輸給出五個時間戳，GPU 原生的 PTP 對齊時鐘，記憶體模型則讓 L2 計數器能解釋互連的行為。測試平台是 H100 NVL8 與 GB200 NVL16；在 GB200 上，kernel 內時鐘偏移的平均抖動是 101.8 ns。在 24 個 channel 的 H100 上，channel 延遲的 P99 約 14 µs，P50 約 7 µs；work stealing 讓 AllGather 從 316.9 提高到 334.4 GB/s。在 GB200 上，16 個以上 channel 時 L2 命中率掉到 15% 以下，32 個 channel 配 4 MB 緩衝時掉到約 0%；在單一 GB200 節點上，eviction hint 把 ReduceScatter 的讀取命中率從 0.31% 提高到 35.17%。
InfiniteHBD: how large the domain should be. Switch-based scale-up domains such as NVL72 are costly, and they waste GPUs when the tensor-parallel group does not divide the domain or a failure breaks it. InfiniteHBD (SIGCOMM 2025) moves the switch into the optical transceiver. A silicon-photonic switch sends the module's light to one of two fibers or loops it back, so nodes form a K-hop ring that can bypass a failed node. The hardware prototype measured 60–80 µs to reconfigure, 3.3 dB of average insertion loss, and less than 3.2 W for the switch core. Rings of 16 and 32 GPUs, built with fixed optics, used 77.1–77.3% of ring bandwidth for AllReduce, against 81.8% for an H100 8-GPU server without SHARP. A 348-day trace of about 3,000 GPUs had 2.33% of 8-GPU nodes faulty on average and 7.22% at p99. Its headline GPU-waste and MFU results are simulated, so they are not given here.
InfiniteHBD：網域應該做多大。以交換器為中心的 scale-up 網域（例如 NVL72）很昂貴，而且當張量平行群組無法整除網域、或故障把網域切斷時，就會浪費 GPU。InfiniteHBD（SIGCOMM 2025）把交換功能搬進光模組。一個矽光子交換器把光模組的光送到兩條光纖之一，或在模組內迴送，所以節點能組成可繞過故障節點的 K-hop 環。硬體原型量到 60–80 µs 的重新配置時間、平均 3.3 dB 的插入損耗，以及交換核心低於 3.2 W 的功耗。用固定光模組搭建的 16 與 32 顆 GPU 環，AllReduce 用到環頻寬的 77.1–77.3%，而沒有 SHARP 的 H100 8-GPU 伺服器是 81.8%。一份約 3,000 顆 GPU、348 天的故障紀錄中，8-GPU 節點平均有 2.33% 故障，p99 是 7.22%。論文最主要的 GPU 浪費與 MFU 結果是模擬值，所以這裡不列出。
PingPoint: inside the package. PingPoint (SIGCOMM 2026) treats an AMD MI300X as a network: 8 compute chiplets, 4 IO dies and 8 HBM stacks, joined by links with their own buffers, routes and backpressure. Only 28 of its 259 profiler counters concern the interconnect, so the authors inject Pingmesh-style pings into the running kernel on spare compute units and compare them with a calibrated baseline. Under load, the chiplet network took 436 of 949 ns of memory latency. The links between IO dies measured 2.4 TB/s, less than the 5.9 TB/s of HBM. Two MoE kernels on disjoint compute units still slowed each other: P99 latency rose up to 10.6× through shared IO-die links.
PingPoint：封裝內部。PingPoint（SIGCOMM 2026）把一顆 AMD MI300X 當成一個網路：8 個運算 chiplet、4 個 IO die 與 8 個 HBM 堆疊，以各有緩衝、路由與反壓的鏈路相連。它的 259 個 profiler 計數器中只有 28 個與互連有關，所以作者把 Pingmesh 式的 ping 注入執行中的 kernel、放在閒置的運算單元上，再與校準過的基準比較。在負載下，chiplet 網路占了 949 ns 記憶體延遲中的 436 ns。IO die 之間的鏈路量到 2.4 TB/s，比 HBM 的 5.9 TB/s 還少。兩個 MoE kernel 即使跑在不相交的運算單元上，仍經由共用的 IO die 鏈路互相拖慢：P99 延遲最多升高 10.6 倍。
| FabricPerf · SIGCOMM 2026 | |
|---|---|
| Problem問題 | No NIC in the NVLink domain, so no view below kernel timeNVLink 網域沒有 NIC，看不到比 kernel 時間更細的東西 |
| Mechanism機制 | Probes in NCCL kernels, GPU clock sync, memory-traffic modelNCCL kernel 內的探針、GPU 時鐘同步、記憶體流量模型 |
| Measured實測 | 101.8 ns clock jitter in a kernel; P99 ≈ 14 vs P50 ≈ 7 µs (H100, 24 channels); L2 read hit 0.31% → 35.17% with eviction hints (GB200 node)kernel 內時鐘抖動 101.8 ns；P99 ≈ 14 對 P50 ≈ 7 µs（H100、24 個 channel）；加 eviction hint 後 L2 讀取命中 0.31% → 35.17%（GB200 節點） |
| InfiniteHBD · SIGCOMM 2025 | |
| Problem問題 | Switch-based domains cost much and waste GPUs on faults交換器式網域昂貴，故障時浪費 GPU |
| Mechanism機制 | Optical switch in each transceiver; K-hop ring that bypasses failed nodes每個光模組內的光交換；能繞過故障節點的 K-hop 環 |
| Measured實測 | 60–80 µs switching, 3.3 dB loss, under 3.2 W (prototype); 77.1–77.3% vs 81.8% of ring bandwidth (16- and 32-GPU rings, fixed optics)切換 60–80 µs、損耗 3.3 dB、低於 3.2 W（原型）；環頻寬 77.1–77.3% 對 81.8%（16 與 32 顆 GPU 的環，固定光模組） |
| PingPoint · SIGCOMM 2026 | |
| Problem問題 | Chiplet links are invisible to GPU profilersGPU profiler 看不到 chiplet 之間的鏈路 |
| Mechanism機制 | Pings fused into the running kernel; per-link attribution against a baseline融合進執行中 kernel 的 ping；對照基準逐鏈路歸因 |
| Measured實測 | 436 of 949 ns of loaded memory latency; IO-die links 2.4 vs HBM 5.9 TB/s; P99 up to 10.6× for kernels on disjoint units (MI300X)負載記憶體延遲 949 ns 中占 436 ns；IO die 鏈路 2.4 對 HBM 5.9 TB/s；不相交運算單元上的 kernel，P99 最多 10.6 倍（MI300X） |
Where they agree, and where they differ. First, the slowest element sets the pace: the last channel of a collective, the tail of a chiplet link, the node that breaks a ring. Second, measurement has to move into the GPU. Lacking NIC and switch telemetry, FabricPerf and PingPoint reach the same method: probes inside kernels, memory counters and a calibrated baseline. Third, at terabytes per second, memory and network are one system: NCCL's buffers fall out of L2, and congestion at one HBM link backs up into the IO dies. They differ in the lever. FabricPerf rebalances work between channels. PingPoint argues that fixed routes make the placement of threads and data the tool. InfiniteHBD switches its optics around a failed node, in 60–80 µs for the hardware.
相同之處與不同之處。第一，最慢的元件決定節奏：集體操作的最後一個 channel、chiplet 鏈路的尾端、切斷環的那個節點。第二，量測必須搬進 GPU。在沒有 NIC 與交換器遙測的情況下，FabricPerf 與 PingPoint 得出同一套方法：kernel 內的探針、記憶體計數器與校準過的基準。第三，在每秒數 TB 的速度下，記憶體與網路是同一個系統：NCCL 的緩衝被擠出 L2，一條 HBM 鏈路的壅塞也會反壓回 IO die。它們的差別在手段。FabricPerf 在 channel 之間重新分配工作。PingPoint 主張，固定的路由讓執行緒與資料的擺放成為工具。InfiniteHBD 則切換光路繞過故障節點，硬體本身只需 60–80 µs。
What is still open. Nobody observes NVSwitch directly: FabricPerf infers it, and InfiniteHBD removes it. The scales are small: 16 GPUs, one package, a 32-GPU ring. The vendors differ, NVIDIA for FabricPerf and AMD for PingPoint, and open standards such as UALink may add a network layer that changes the tools. Read FabricPerf first, because it continues the NVLink rung of Section 2.2. Then read InfiniteHBD, which asks how large that rung should grow. Read PingPoint last: it shows that even the top rung, HBM, is reached through a network.
仍未解決的問題。沒有人直接觀察 NVSwitch：FabricPerf 是推論它，InfiniteHBD 則拿掉它。實驗規模都很小：16 顆 GPU、一個封裝、一個 32 顆 GPU 的環。廠商也不同，FabricPerf 是 NVIDIA，PingPoint 是 AMD；而 UALink 之類的開放標準可能加入一層網路層，改變這些工具。先讀 FabricPerf，因為它延續第 2.2 節的 NVLink 那一階。接著讀 InfiniteHBD，它問的是那一階該長到多大。最後讀 PingPoint：它顯示就連最上面那一階 HBM，也要經過一個網路才能到達。