Research digest 4 of 16 · Chapter 4: Collective communication

研究導讀 4 / 16 · 第 4 章：集體通訊

MCCS: A Service-based Approach to Collective Communication for Multi-Tenant CloudMCCS：以服務化方式為多租戶雲提供集體通訊

SIGCOMM 2024Yongji Wu, Yechen Xu, Jingrong Chen et al. (Duke University, Meta) · paper論文

After reading, you can explain why a cloud tenant's NCCL cannot choose a good ring, and how a provider-run collective service reorders rings, pins paths and prioritizes tenants while jobs run.讀完後，你能說明雲端租戶的 NCCL 為何選不出好的環，以及由供應商執行的集體通訊服務如何在任務執行中重排環、固定路徑並為租戶設定優先順序。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 4 ↗第 4 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

In a public cloud, NCCL runs inside each tenant's application and sees one big virtual switch: no racks, no link loads, no other tenants. It builds its rings in rank order and never changes them; in Meta's production trace, this put up to 2× the necessary flows across racks. MCCS moves collectives into a trusted service that the cloud provider runs on every host. Tenants keep the NCCL API through a thin shim. The provider chooses the ring order, the NIC and the path of every flow, and changes them between collectives. On a small testbed of 8 RTX 3090 GPUs whose fabric is emulated on one switch, MCCS averaged 2.4× NCCL's bandwidth from 8 to 512 MB on a 2:1 spine-leaf and made a priority tenant's training 34% faster than with ECMP. On an emulated ring of switches, reversing the ring around a background flow raised bandwidth from 1.7 back to 5.9 GB/s. The service adds 50–80 µs per collective, which hurts below 8 MB.

在公有雲中，NCCL 在每個租戶的應用程式內執行，只看到一台巨大的虛擬交換器：看不到機櫃、鏈路負載，也看不到其他租戶。它依 rank 順序建環，之後永不改變；在 Meta 的生產環境紀錄中，這讓跨機櫃的流最多達到必要數量的 2 倍。MCCS 把集體通訊移進由雲端供應商在每台主機上執行的可信服務。租戶透過一層薄的 shim 繼續使用 NCCL API。供應商決定環的順序、每個流的 NIC 與路徑，並在集體操作之間更改它們。在一個 8 顆 RTX 3090 GPU、網路由一台交換器模擬出來的小型測試平台上，MCCS 在 2:1 spine-leaf 上於 8 到 512 MB 平均達到 NCCL 頻寬的 2.4 倍，並讓高優先租戶的訓練比用 ECMP 快 34%。在模擬的交換器環上，繞開背景流把環反轉，讓頻寬從 1.7 GB/s 回到 5.9 GB/s。這個服務讓每次集體操作多花 50–80 µs，在 8 MB 以下會吃虧。

1. Where this paper sits1. 這篇論文的位置

Section 4.5 shows how NCCL builds its rings: once, when the communicator is created, with the order between hosts taken from the user's ranks. Section 3 shows that a ring runs at the pace of its slowest hop, and section 5 that each channel's connection is a flow that ECMP hashes onto one path. Section 11.1 lists what breaks when tenants share a fabric, and four isolation mechanisms. MCCS (Duke and Meta, SIGCOMM 2024) adds one at the collective layer, for a public cloud whose tenants cannot see the topology at all.

第 4.5 節說明 NCCL 如何建環：只在建立 communicator 時建一次，主機之間的順序取自使用者的 rank。第 3 節說明環的速度由最慢的一跳決定，第 5 節說明每個 channel 的連線是一個流，由 ECMP 雜湊到一條路徑上。第 11.1 節列出租戶共用網路時會壞掉的東西，以及四種隔離機制。MCCS（Duke 與 Meta，SIGCOMM 2024）在集體通訊層再加一種，對象是租戶完全看不到拓樸的公有雲。

2. The problem, measured2. 問題，用量測說話

The paper names three gaps of a library inside the tenant. Topology blindness: a good ring order needs the physical topology and the link loads, which the cloud hides; virtualization can even hide the PCIe layout that NCCL reads. A static strategy: NCCL fixes it at start, but the best choice depends on other tenants' traffic, which changes. Configuration-agnostic tricks: NCCL opens several connections to use parallel paths, but they may hash onto one physical path.

論文指出函式庫放在租戶內的三個缺口。看不見拓樸：好的環順序需要實體拓樸與鏈路負載，而雲端把它們藏起來；虛擬化甚至可能藏起 NCCL 要讀的 PCIe 配置。策略固定：NCCL 在開始時就固定策略，但最佳選擇取決於其他租戶不斷變化的流量。不看配置的技巧：NCCL 開多條連線想利用平行路徑，但它們可能被雜湊到同一條實體路徑。

Figure 1 of the MCCS paper: tenant-owned collective libraries compared with a provider-run MCCS service, with the three problems it addresses below — Figure 1. Figure 1 of the paper. Top: today each tenant (T1–T3) links its own collective library (CC); with MCCS the provider runs it as a service, and the application keeps a thin shim. Bottom: the three problems above. Green parts are controlled by the tenant, blue parts by the provider. A design figure.圖 1。論文的 Figure 1。上：現在每個租戶（T1–T3）各自連結集體通訊函式庫（CC）；有了 MCCS，由供應商把它當作服務執行，應用程式只留一層薄的 shim。下：上文的三個問題。綠色部分由租戶控制，藍色部分由供應商控制。這是設計圖。Screenshot of Wu et al., SIGCOMM 2024, Figure 1 (p. 1).

The measurement concerns ring order. NCCL chains hosts in rank order, so with random placement the ring crosses racks back and forth (Figure 2). The paper divides the cross-rack flows of a job's ring by those of an optimal ring. In Meta's production trace (spine-leaf, 2 hosts per rack, 8 GPUs and 8 NICs per host), this cross-rack ratio reached 2× in the worst case. A simulation shows that it grows with hosts per rack and job size (we omit its simulated number). Publishing the topology is no answer: providers keep topology and utilization confidential, the obstacle for hint-based designs such as NetHint.

量測的對象是環的順序。NCCL 依 rank 順序串接主機，所以隨機配置時，環會在機櫃之間來回穿越（圖 2）。論文把任務的環的跨機櫃流數，除以最佳環的跨機櫃流數。在 Meta 的生產環境紀錄中（spine-leaf，每機櫃 2 台主機，每台 8 顆 GPU 與 8 張 NIC），這個跨機櫃比（cross-rack ratio）最壞達到 2 倍。一項模擬顯示它隨每機櫃主機數與任務大小成長（模擬數字不列）。公開拓樸也不是解答：供應商要對拓樸與使用率保密，這正是 NetHint 這類提示式設計的障礙。

Figure 2. Two rings over the same 8 GPUs: hosts H1–H2 in rack A, H3–H4 in rack B, 2 GPUs each. Red arrows cross racks. Rank order with hosts placed A, B, A, B crosses 4 times; rack by rack crosses twice. With 2 hosts per rack, 2× is the worst case, the value that Meta's trace reached.圖 2。同樣 8 顆 GPU 上的兩個環：主機 H1–H2 在機櫃 A、H3–H4 在機櫃 B，各 2 顆 GPU。紅色箭頭跨越機櫃。依 rank 順序、主機配置為 A、B、A、B 時跨 4 次；逐機櫃排列只跨 2 次。每機櫃 2 台主機時，2 倍就是最壞情況，也就是 Meta 紀錄所達到的值。Drawn for this primer from Wu et al., SIGCOMM 2024, §2.2 and Figure 3.

3. The design3. 設計

3.1 Collectives as a provider service3.1 集體通訊作為供應商的服務

The MCCS service is a trusted user-space process on every host, run by the provider, with access to all GPUs and NICs (Figure 3). The tenant's application links a shim library with NCCL-compatible calls, so it needs no code change. The shim talks to the service through shared memory. The service allocates the tenant's GPU buffers and returns CUDA IPC handles; a collective names a buffer by allocation ID and offset, which the service checks. Shared CUDA events order the two processes' streams. Inside the service, a proxy engine per GPU holds the communicators, the strategy and the ring order, and transport engines carry all inter-host traffic, choose each flow's path and decide when it may send. A controller sees every tenant's communicators and traces.

MCCS 服務是每台主機上由供應商執行的可信使用者空間行程，能存取所有 GPU 與 NIC（圖 3）。租戶的應用程式連結一個提供 NCCL 相容呼叫的 shim 函式庫，所以不必改程式碼。shim 透過共享記憶體與服務溝通。服務替租戶配置 GPU 緩衝區並回傳 CUDA IPC handle；集體操作以配置編號加位移指定緩衝區，由服務檢查。共享的 CUDA event 排定兩個行程串流的先後。服務內部，每顆 GPU 有一個 proxy engine，保存 communicator、策略與環的順序；transport engine 承載所有主機之間的流量，決定每個流的路徑與何時可以送出。控制器看得到每個租戶的 communicator 與流量紀錄。

Figure 3. Left: each tenant's NCCL builds rings in rank order, blind to the fabric. Right: tenants link a shim; one MCCS service per host runs every collective, and a controller that sees the topology and all tenants sets ring order, paths and timing.圖 3。左：每個租戶的 NCCL 看不到網路，依 rank 順序建環。右：租戶連結 shim；每台主機一個 MCCS 服務執行所有集體操作，看得到拓樸與所有租戶的控制器決定環的順序、路徑與時機。Drawn for this primer from Wu et al., SIGCOMM 2024, §3–§4 and Figure 1.

3.2 Changing the strategy between collectives3.2 在集體操作之間更換策略

The provider may change a communicator's ring order, paths or NICs while the job runs. The change must happen between collectives, and every rank must switch at the same one. MCCS uses the fact that all ranks run the same collectives in the same order:

供應商可以在任務執行中更改 communicator 的環順序、路徑或 NIC。更改必須發生在集體操作之間，而且每個 rank 必須在同一個集體操作切換。MCCS 利用一個事實：所有 rank 以相同順序執行相同的集體操作：

Each proxy engine numbers its collectives; the numbers match across ranks.每個 proxy engine 為自己的集體操作編號；各 rank 的編號一致。
On a request, each proxy holds back new collectives and runs an AllGather of its last launched sequence number over the communicator's control ring.收到請求時，每個 proxy 暫緩新的集體操作，並在 communicator 的控制環上對自己最後啟動的序號做一次 AllGather。
Every rank now knows the maximum $m$ . Collectives up to $m$ use the old configuration and later ones the new; the maximum covers a rank that had already launched one more.每個 rank 都得知最大值 $m$ 。編號到 $m$ 為止的集體操作用舊配置，之後的用新配置；取最大值，是為了涵蓋已多啟動一個的 rank。
The proxies tear down the old connections and build the new ones, for example a reversed ring.各 proxy 拆除舊連線並建立新連線，例如反向的環。

When no change is requested, the protocol costs nothing. The paper calls the cost of a change "rather small" and gives no number.

沒有更改請求時，這個協定不花任何成本。論文說一次更改的成本「相當小」，但沒有給數字。

3.3 Policies from a controller3.3 由控制器套用的策略

The controller decides, and the engines enforce. The paper shows four policies. A locality-aware ring groups hosts by rack, then by pod, and enters each group once. Best-fit fair flow assignment (FFA), after Hedera, gives every flow the path with the least excess demand and takes the jobs' flows in turn. Priority flow assignment (PFA) reserves some paths for a high-priority tenant. Traffic scheduling (TS), after CASSINI, finds the idle gaps of a prioritized job and lets other jobs send only in them. To pin a flow to a path, the service writes a route ID into the RoCEv2 UDP source port, and the switch maps each source port to a path.

控制器做決定，各 engine 負責執行。論文展示四種策略。感知位置的環先依機櫃、再依 pod 把主機分組，每組只進入一次。最適公平流分配（FFA）取自 Hedera，給每個流超額需求最小的路徑，並輪流處理各任務的流。優先流分配（PFA）為高優先租戶保留部分路徑。流量排程（TS）取自 CASSINI，找出優先任務的閒置空檔，只讓其他任務在空檔中傳送。要把流固定在某條路徑，服務把路由編號寫進 RoCEv2 的 UDP 來源埠，交換器再把每個來源埠對應到一條路徑。

The paper's Figure 7 shows a change in action (Figure 4). Four switches form a ring, one server on each, and an 8-GPU AllReduce uses a clockwise ring. At 7.5 s a 75 Gb/s background flow starts on one clockwise link and leaves the job 25 Gb/s: bandwidth falls from 5.9 to 1.7 GB/s. At 12 s the controller reverses the ring. Every ring flow now uses a counterclockwise link, and bandwidth returns to 5.9 GB/s.

論文的 Figure 7 展示一次實際的更改（圖 4）。四台交換器連成一個環，每台接一台伺服器，一個 8 GPU 的 AllReduce 使用順時針的環。7.5 s 時，一條順時針鏈路上出現 75 Gb/s 的背景流，只留給任務 25 Gb/s：頻寬從 5.9 GB/s 掉到 1.7 GB/s。12 s 時控制器把環反轉。此後每個環上的流都走逆時針鏈路，頻寬回到 5.9 GB/s。

Figure 4. The paper's Figure 7 experiment. Left: the clockwise ring shares S0 → S1 with a 75 Gb/s background flow. Right: reversed, every flow runs counterclockwise and shares no link with it. Bandwidths are measured.圖 4。論文 Figure 7 的實驗。左：順時針的環與 75 Gb/s 的背景流共用 S0 → S1。右：反轉後所有流都逆時針傳送，不與它共用任何鏈路。頻寬為實測值。Drawn for this primer from Wu et al., SIGCOMM 2024, §6.2 and Figure 7.

In Panel 1 you build the ring and add the background flow.

在面板 1 中，你可以排出環並加入背景流。

Panel 1面板 1

Order the ring, then add a background flow排出環的順序，再加入背景流

Fabric網路

Ring order環的順序

Or click the GPUs in ring order或依環的順序點選 GPU

Background flow (75% of a link)背景流（鏈路的 75%）

Direction方向

Spine for cross-rack flows跨機櫃流的 spine

Network crossings

跨越網路的次數

Busiest link

最忙的鏈路

Ring rate (slowest hop)

環的速率（最慢的一跳）

AllReduce algbw in this sketch

此簡化模型的 AllReduce algbw

A simplified model of the paper's testbed. One ring; two GPUs on the same server talk through host memory, which we treat as fast. A link's free capacity is split equally among the ring flows on it, and a flow runs at its smallest share. Two racks: the 8-GPU tests use both 50 Gb/s virtual NICs of a server, so we send each hop between servers as two flows, one per NIC; each leaf has a 50 Gb/s link to each spine (2:1), and the numbers on a leaf–spine link count the flows going up (↑) and down (↓). ECMP picks a spine with a fixed hash of the GPU pair and NIC, one possible outcome of a random hash; "Assigned" gives each cross-rack flow the spine with the most capacity left, as MCCS's controller would. The background flow takes 37.5 of 50 Gb/s from rack A to rack B through spine 1. Ring of switches: links and server NICs are 100 Gb/s, as the paper's "75 Gb/s leaves 25 Gb/s" implies; flows take the shorter way, clockwise on a tie (↻ clockwise and ↺ counterclockwise flows per link), and the background flow takes 75 of 100 Gb/s on S0 → S1. The algbw is rate ÷ 8 ÷ 1.75 (Section 4.8), with no protocol overhead. Defaults: two racks, interleaved ranks, ECMP. Try "Locality-aware", then "Assigned", then the background flow and "Reversed": in two racks a ring crosses equally often in both directions, so reversing only reshuffles the hash; in the ring of switches it moves every flow off the busy link.

這是論文測試平台的簡化模型。只有一個環；同一台伺服器上的兩顆 GPU 經主機記憶體溝通，我們視為很快。一條鏈路的剩餘容量由其上的環流平分，每個流以它最小的那份速率前進。兩個機櫃：8 GPU 的測試會用到伺服器的兩張 50 Gb/s 虛擬 NIC，所以我們把伺服器之間的每一跳拆成兩個流，每張 NIC 一個；每台 leaf 到每台 spine 各有 50 Gb/s 鏈路（2:1），leaf–spine 鏈路上的數字是往上（↑）與往下（↓）的流數。ECMP 以 GPU 對與 NIC 的固定雜湊選 spine，是隨機雜湊的一種可能結果；「指派」則像 MCCS 的控制器一樣，給每個跨機櫃流剩餘容量最多的 spine。背景流經 spine 1 從機櫃 A 到機櫃 B，佔用 50 Gb/s 中的 37.5 Gb/s。交換器環：鏈路與伺服器 NIC 都是 100 Gb/s，這是論文「75 Gb/s 背景流只留下 25 Gb/s」所隱含的；流走較短的方向，平手時順時針（每條鏈路上 ↻ 為順時針、↺ 為逆時針的流數）；背景流佔用 S0 → S1 的 100 Gb/s 中的 75 Gb/s。algbw 為速率 ÷ 8 ÷ 1.75（第 4.8 節），不計協定開銷。預設：兩個機櫃、rank 交錯、ECMP。試試「感知位置」，再按「指派」，再開背景流並按「反轉」：在兩個機櫃中，環在兩個方向跨越的次數相同，所以反轉只是重新洗牌雜湊；在交換器環上，反轉則把每個流移出忙碌的鏈路。

4. Implementation4. 實作

MCCS is 13.5K lines of Rust: 1.5K for the shim and IPC, 6K for the control and management planes, and 6K for the transport engine and protocols. Its kernels are NCCL 2.17.1's ring AllReduce and AllGather, changed so that the proxies set up the ring buffers; trees, point-to-point and other collectives are not implemented. Transports are host shared memory and RDMA; NVLink is not supported. The engines use two more threads per GPU than NCCL. Traffic scheduling is hard-coded, with applications profiled offline.

MCCS 是 13.5K 行的 Rust：shim 與 IPC 1.5K 行，控制與管理平面 6K 行，transport engine 與協定 6K 行。它的 kernel 是 NCCL 2.17.1 的環形 AllReduce 與 AllGather，修改成由 proxy 建立環的緩衝區；樹、點對點與其他集體操作都未實作。傳輸方式是主機共享記憶體與 RDMA；不支援 NVLink。各 engine 每顆 GPU 比 NCCL 多用兩個執行緒。流量排程是寫死的，應用程式事先離線量測。

5. What was measured5. 量到了什麼

The testbed is small, and its fabric is emulated. Four servers each have 2 NVIDIA RTX 3090 GPUs and one 100 Gb/s ConnectX-5 NIC, split by traffic-class rate limits into two 50 Gb/s virtual NICs, one per GPU. One 100 Gb/s SN2100 switch emulates 2 leaves and 2 spines for 2 racks of 2 servers: switch-to-switch links are capped at 50 Gb/s and host links run at 100 Gb/s, so oversubscription is 2:1. Single-application tests use 4 GPUs (one per server) or all 8. Baselines: NCCL 2.17.1, NCCL(OR) with a hand-set optimal ring, and MCCS(-FA) with ECMP instead of flow assignment. Training traces (VGG-19, a 2.7B GPT) are replayed on MCCS. The 768-GPU results are flow-level simulation and are left out.

測試平台很小，網路是模擬出來的。四台伺服器各有 2 顆 NVIDIA RTX 3090 GPU 與一張 100 Gb/s 的 ConnectX-5 NIC，以 traffic class 限速分成兩張 50 Gb/s 的虛擬 NIC，每顆 GPU 一張。一台 100 Gb/s 的 SN2100 交換器模擬出 2 台 leaf 與 2 台 spine，組成 2 個機櫃、每櫃 2 台伺服器：交換器之間的鏈路限速 50 Gb/s，主機鏈路為 100 Gb/s，所以超額訂閱比為 2:1。單一應用的測試使用 4 顆 GPU（每台伺服器一顆）或全部 8 顆。基準：NCCL 2.17.1、手動設定最佳環的 NCCL(OR)，以及用 ECMP 取代流分配的 MCCS(-FA)。訓練流量紀錄（VGG-19、2.7B 的 GPT）在 MCCS 上重播。768 顆 GPU 的結果是流層級模擬，本頁略去。

Test測試	Measured on the testbed測試平台實測
Ring order alone, 512 MB AllReduce只改環順序，512 MB AllReduce	NCCL(OR) 56% (4 GPUs) and 78% (8 GPUs) faster than NCCLNCCL(OR) 比 NCCL 快 56%（4 GPU）與 78%（8 GPU）
Flow assignment, 8 GPUs, 512 MB AllReduce流分配，8 GPU，512 MB AllReduce	MCCS 46% faster than MCCS(-FA) and NCCL(OR)MCCS 比 MCCS(-FA) 與 NCCL(OR) 快 46%
Average over 8–512 MB, AllReduce and AllGather8–512 MB 平均，AllReduce 與 AllGather	MCCS 1.6× (4 GPUs) and 2.4× (8 GPUs) NCCL's algorithm bandwidthMCCS 為 NCCL 演算法頻寬的 1.6 倍（4 GPU）與 2.4 倍（8 GPU）
Service overhead, 4 GPUs, MCCS(-FA) vs NCCL(OR)服務開銷，4 GPU，MCCS(-FA) 對 NCCL(OR)	63% lower at 512 KB AllGather; 9.7% (AllGather) and 0.75% (AllReduce) lower at 8 MB; cause: 50–80 µs added latency512 KB AllGather 低 63%；8 MB 時低 9.7%（AllGather）與 0.75%（AllReduce）；原因是多出 50–80 µs 延遲
Background flow, then ring reversal背景流，然後反轉環	5.9 → 1.7 → 5.9 GB/s5.9 → 1.7 → 5.9 GB/s
Four multi-tenant setups, 128 MB AllReduce四種多租戶設定，128 MB AllReduce	FFA 75% faster than NCCL on average; shares near the expected 2:1:1, against 1.7:1 for A:B under ECMPFFA 平均比 NCCL 快 75%；頻寬分配接近預期的 2:1:1，ECMP 下 A:B 為 1.7:1
Training: VGG (A, 4 GPUs), GPT (B and C, 2 GPUs each)訓練：VGG（A，4 GPU）、GPT（B 與 C，各 2 GPU）	Under ECMP, A, B, C 18%, 22%, 14% slower than with FFA; PFA makes A 13% faster than FFA and 34% than ECMP; PFA plus TS makes B 16% faster than PFAECMP 下 A、B、C 比 FFA 慢 18%、22%、14%；PFA 讓 A 比 FFA 快 13%、比 ECMP 快 34%；PFA 加 TS 讓 B 比只用 PFA 快 16%

Figure 5 shows the single-application curves behind the first four rows.

圖 5 是前四列背後的單一應用曲線。

Figure 6 of the MCCS paper: algorithm bandwidth against data size for NCCL, NCCL(OR), MCCS(-FA) and MCCS, AllGather and AllReduce on 4 and 8 GPUs — Figure 5. Figure 6 of the paper: algorithm bandwidth against message size, AllGather and AllReduce, on 4 GPUs (top) and 8 GPUs (bottom); measured, with 95% intervals shaded. On 4 GPUs the optimal ring, NCCL(OR), matches MCCS for large messages; on 8 GPUs flow assignment adds more. Below a few MB, the service path costs bandwidth.圖 5。論文的 Figure 6：AllGather 與 AllReduce 的演算法頻寬對訊息大小，上為 4 顆 GPU、下為 8 顆 GPU；為實測值，陰影是 95% 區間。4 顆 GPU 時，大訊息下最佳環 NCCL(OR) 與 MCCS 相當；8 顆 GPU 時，流分配再多貢獻一截。小於幾 MB 時，走服務路徑反而損失頻寬。Screenshot of Wu et al., SIGCOMM 2024, Figure 6 (p. 8).

The reversal numbers agree with the bus-bandwidth rule of Section 4.8. A ring AllReduce on 8 GPUs moves $2 \times 7/8 = 1.75$ times its buffer through every hop:

反轉實驗的數字與第 4.8 節的匯流排頻寬規則一致。8 顆 GPU 的環形 AllReduce，每一跳要搬緩衝區大小的 $2 \times 7/8 = 1.75$ 倍：

Question問題	Calculation算式	Result結果
Bus bandwidth per hop at the measured 5.9 GB/s實測 5.9 GB/s 時每一跳的匯流排頻寬	$5.9 \times 1.75$	$\approx 10.3$ GB/sabout 83 Gb/s約 83 Gb/s
Highest algbw with 25 Gb/s left on one hop某一跳只剩 25 Gb/s 時的最高 algbw	$25 \div 8 \div 1.75$	$\approx 1.8$ GB/smeasured: 1.7 GB/s實測：1.7 GB/s

So 1.7 GB/s is close to what the free 25 Gb/s allows. The first row also shows that a hop carried about 83 Gb/s, more than one 50 Gb/s virtual NIC.

所以 1.7 GB/s 接近剩下 25 Gb/s 所允許的上限。第一列也顯示一跳承載約 83 Gb/s，超過一張 50 Gb/s 的虛擬 NIC。

6. Limits and open questions6. 限制與未解的問題

The paper has no limitations section, but it states several gaps: 50–80 µs per collective, which hurts below 8 MB; only ring AllReduce and AllGather; no NVLink; hard-coded traffic scheduling; two extra CPU threads per GPU; and no number for the cost of a change. A careful reader adds:

論文沒有限制一節，但提到幾個缺口：每次集體操作多 50–80 µs，在 8 MB 以下會吃虧；只有環形 AllReduce 與 AllGather；不支援 NVLink；流量排程寫死；每顆 GPU 多兩個 CPU 執行緒；更改的成本沒有數字。細心的讀者還應補上：

Scale. 8 consumer GPUs, 50 Gb/s virtual NICs, a fabric emulated on one switch; no production fabric was tested, and the larger-scale evidence is simulation.規模。8 顆消費級 GPU、50 Gb/s 虛擬 NIC、在一台交換器上模擬的網路；沒有測試任何生產網路，更大規模的證據是模擬。
The ring of switches. The paper does not say how it was emulated. Its numbers imply links faster than the 50 Gb/s of the spine-leaf setup: 75 Gb/s of background leaves 25 Gb/s, and a hop carried about 83 Gb/s.交換器環。論文沒有說明它如何模擬。它的數字隱含比 spine-leaf 設定的 50 Gb/s 更快的鏈路：75 Gb/s 的背景流留下 25 Gb/s，而且一跳承載約 83 Gb/s。
A new shared component. The service sits in every tenant's data path; the paper does not discuss what happens when it fails. Tenants must trust provider code, and the provider sees their communication patterns.新的共用元件。服務位於每個租戶的資料路徑上；論文沒有討論它故障時會怎樣。租戶必須信任供應商的程式，供應商也看得到它們的通訊模式。
Transport coupling. MCCS separates tenants by path and time, not by priority class, so PFC pauses can still couple them (Section 11.1).傳輸層的耦合。MCCS 以路徑與時間區隔租戶，而非優先等級，所以 PFC 暫停仍可能讓它們互相牽連（第 11.1 節）。

7. How it connects7. 與其他內容的關聯

Chapter 11, sections 1–2: MCCS is an isolation mechanism at the collective layer, and its locality-aware ring is the "order the ring pod by pod" rule of section 2.第 11 章第 1–2 節：MCCS 是集體通訊層的隔離機制，其感知位置的環正是第 2 節「逐 pod 排列環」的規則。
Chapter 4, section 5: the route ID in the UDP source port is the lever behind Meta's QP scaling and HPN's choice of connection; MCCS uses it across tenants.第 4.5 節：寫在 UDP 來源埠的路由編號，正是 Meta 的 QP scaling 與 HPN 選擇連線背後的槓桿；MCCS 把它用在租戶之間。
OptCCL and other solvers assume a known topology, which a provider service has.OptCCL 與其他求解器假設拓樸已知，而供應商服務正好知道拓樸。
Collectives Section 4.3: the 50–80 µs acts as extra $\alpha$ , so it costs most where the $\alpha$ term dominates, in small messages.第 4.3 節：這 50–80 µs 就像多出的 $\alpha$ ，所以在 $\alpha$ 項主導的小訊息上代價最高。
Theseus also switches schedules mid-job, but inside the library.Theseus 也在任務中途切換排程，但是在函式庫之內。

Key numbers關鍵數字

Number	數字	Meaning (testbed unless noted)
2×	Worst cross-rack ratio, Meta production trace, 2 hosts per rack	最壞的跨機櫃比，Meta 生產紀錄，每機櫃 2 台主機
+56% / +78%	Ring order alone, 512 MB AllReduce, 4 / 8 GPUs	只改環順序，512 MB AllReduce，4 / 8 GPU
1.6× / 2.4×	MCCS vs NCCL, average over 8–512 MB, 4 / 8 GPUs	MCCS 對 NCCL，8–512 MB 平均，4 / 8 GPU
50–80 µs	Added latency per collective	每次集體操作多出的延遲
5.9 → 1.7 → 5.9 GB/s	Background flow, then ring reversal	背景流出現，然後反轉環
34%	Priority tenant's training, PFA vs ECMP	高優先租戶的訓練，PFA 對 ECMP

Terms used on this page本頁用到的術語

MCCS service: A provider-run process on every host that executes all tenants' collectives.; 供應商在每台主機上執行的行程，負責執行所有租戶的集體操作。
Cross-rack ratio: Cross-rack flows of a job's ring divided by those of an optimal ring.; 任務的環的跨機櫃流數，除以最佳環的跨機櫃流數。
NCCL(OR): NCCL with a hand-set optimal ring; the baseline that isolates ring order.; 手動設定最佳環的 NCCL；用來單獨衡量環順序的基準。
FFA / PFA / TS: Fair flow assignment, priority flow assignment, and traffic scheduling in a prioritized job's idle windows.; 公平流分配、優先流分配，以及在優先任務閒置空檔中的流量排程。