Failure research: is it the network, and where?故障研究：是不是網路，問題在哪裡？

Section 10.1 to 4 treat a synchronous job as a series system: one bad part stops or slows every rank, and each failure costs $D + R$ plus the lost work. This page fills the detection term $D$ with a watchdog timeout or MegaScale's heartbeat. Four recent papers attack that term and the blast radius behind it. Read together, they split "what broke?" into three questions: is it the network, where in the network, and which machine. A fourth answer asks whether the system can avoid needing the answer at all. The figure places each paper on the recovery loop.

第 1 到第 10.4 節把同步任務當成串聯系統：一個壞零件就讓每個 rank 停下或變慢，每次故障的代價是 $D + R$ 加上損失的工作。本頁用 watchdog 逾時或 MegaScale 的心跳來填偵測項 $D$。四篇近期論文攻擊的正是這一項，以及它背後的影響範圍。合起來讀，它們把「什麼壞了？」拆成三個問題：是網路嗎、在網路的哪裡、是哪一台機器。第四種答案則問：系統能不能根本不需要這個答案。下圖標出每篇論文在恢復迴圈中的位置。

The papers give competing answers on four points.

這些論文在四個問題上給出不同的答案。

• Probes or metrics. R-Pingmesh and SkeletonHunter send probes between RNICs and infer faults from latency, loss and shared paths. Minder reads per-second host metrics, such as the PFC transmit rate, CPU, GPU and NVLink, and looks for the machine that stops behaving like the others. Probes see inside the fabric but not a GPU fault, as both probe papers say. Metrics see host faults, but a fabric fault that slows every machine equally makes no outlier. ByteDance runs Minder and R-Pingmesh side by side.探測還是指標。R-Pingmesh 與 SkeletonHunter 在 RNIC 之間發送探測，從延遲、丟包與共用路徑推斷故障。Minder 讀取每秒的主機指標，例如 PFC 送出速率、CPU、GPU 與 NVLink，尋找行為不再與其他機器相似的那一台。探測看得到網路內部，卻看不到 GPU 故障，兩篇探測論文都這麼說。指標看得到主機故障，但讓每台機器同樣變慢的網路故障不會產生離群者。字節跳動同時執行 Minder 與 R-Pingmesh。
• Trace or infer. R-Pingmesh copies each job's 5-tuples with eBPF from the RDMA stack of the host its Agent runs on, including hosts in public-cloud clusters. SkeletonHunter, in a multi-tenant container cloud, cannot hook the tenant's RDMA stack, so it infers the job's pairs from rails and from the rhythm of each RNIC's traffic.追蹤還是推斷。R-Pingmesh 在其 Agent 所在主機的 RDMA 協定堆疊上用 eBPF 複製每個任務的 5-tuple，這也包括公有雲叢集中的主機。SkeletonHunter 身處多租戶容器雲，無法掛鉤租戶的 RDMA 協定堆疊，所以從 rail 與每張 RNIC 流量的節奏推斷任務的配對。
• Thresholds or comparison. TPUv4's health daemon and preflight checks compare physical metrics with fixed in-spec thresholds. Minder argues that fixed thresholds fail on GPU fleets, because normal values depend on the task, so it compares machines with each other. Both can be right: link quality and firmware state do not depend on the task, while GPU temperature and use do.門檻還是比較。TPUv4 的健康 daemon 與 preflight 檢查，把實體指標與固定的規格門檻比較。Minder 主張固定門檻在 GPU 機群上行不通，因為正常值取決於任務，所以改為讓機器彼此比較。兩者都可能是對的：鏈路品質與韌體狀態與任務無關，GPU 溫度與使用率則有關。
• Evict or survive. Three papers end in eviction and a restart from a checkpoint. TPUv4 survives one fault class, the loss of an optical circuit switch, without a restart: it routes around the missing links at a measured cost of 0.5–8.6% in step time.逐出還是撐過。三篇論文以逐出與從檢查點重啟收尾。TPUv4 不必重啟就撐過一類故障，也就是失去一台光路交換器：它繞過缺少的鏈路，量到的代價是步時間增加 0.5–8.6%。

They agree on more than they dispute. All four exploit the regularity of training: recurring congestion, burst cycles, machines that behave alike, static routes per job. Most faults are host-side or link-level. In Minder's data, 55.8% of faults are hardware inside a host and ECC errors alone are 38.9%. R-Pingmesh finds flapping links the most common hardware failure, and TPUv4 reports daily failure rates of 0.08% for machines against 0.005% for cables. Faults outside the network look like network faults: a PCIe downgrade shows up as a surge of PFC pauses in both R-Pingmesh and Minder. And monitoring raises its own false alarms: a busy agent CPU explains 30 of R-Pingmesh's 50 RNIC reports.

它們的共識多於分歧。四篇都利用訓練的規律性：反覆出現的壅塞、突發週期、行為相似的機器、每個任務固定的路由。多數故障出在主機端或鏈路層。在 Minder 的資料中，55.8% 的故障是主機內的硬體，光是 ECC 錯誤就占 38.9%。R-Pingmesh 發現鏈路抖動是最常見的硬體故障，TPUv4 則回報機器每日故障率 0.08%、線纜只有 0.005%。網路以外的故障看起來像網路故障：PCIe 降級在 R-Pingmesh 與 Minder 裡都表現為大量湧出的 PFC 暫停。監控本身也會製造假警報：R-Pingmesh 的 50 個 RNIC 報告中，有 30 個出自忙碌的代理 CPU。

Several problems stay open. None of the four finds root causes; each names a location and leaves the rest to engineers. None addresses silent data corruption, the hardest row of Section 10.1. Detection time is weakly measured: R-Pingmesh's 20 s is its analysis period, SkeletonHunter gives an 8 s average without a distribution, and Minder's 3.6 s is compute time per call behind a 4-minute continuity rule, so its alerts arrive minutes after a fault. Recall rests on operator checks or user reports. TPUv4's 99.98% availability is stated without a definition, so we do not treat it as a measured number.

仍有幾個問題沒有解決。四篇都找不出根本原因；每篇只指出位置，其餘留給工程師。沒有一篇處理靜默資料損毀，也就是第 10.1 節最難的那一項。偵測時間量得不夠好：R-Pingmesh 的 20 秒是它的分析週期，SkeletonHunter 只給平均 8 秒而沒有分布，Minder 的 3.6 秒是每次呼叫的計算時間，背後還有 4 分鐘的持續性規則，所以它的警報在故障後數分鐘才到。召回率仰賴營運人員核對或使用者回報。TPUv4 的 99.98% 可用性沒有定義，所以我們不把它當成量測數字。

A note on sources: "Rerouting the torus" in Section 10.5 draws on the TPU v4 architecture paper (ISCA 2023, the guided reading); the NSDI 2024 paper is its operations side. A suggested order: R-Pingmesh first, for the vocabulary of probing and voting; SkeletonHunter next, for what changes under multi-tenancy; Minder third, which moves from the network to the machine; TPUv4 last, the opposite philosophy. Short on time? Read R-Pingmesh and TPUv4, the two ends.

關於來源：第 10.5 節的「重新接線的環面」取材自 TPU v4 架構論文（ISCA 2023，即論文導讀那一篇）；NSDI 2024 這篇是它的營運面。建議的閱讀順序：先讀 R-Pingmesh，學會探測與投票的詞彙；接著 SkeletonHunter，看多租戶下改變了什麼；第三讀 Minder，它從網路轉向機器；最後讀 TPUv4，相反的設計哲學。時間不夠？就讀 R-Pingmesh 與 TPUv4 這兩端。