Research digest 2 of 5 · Chapter 11: Sharing the cluster

研究導讀 2 / 5 · 第 11 章：共用叢集

LEVELLER: Fair Communication Scheduling via Progress-Rate Awareness in Multi-Tenant Training ClustersLEVELLER：以進度率感知實現多租戶訓練叢集的公平通訊排程

SIGCOMM 2026Geng Li, Yang Li, Mingyuan Zang, Jie Wu (China Telecom Cloud Computing Research Institute, BUPT, Temple University) · paper論文

After reading, you can explain how LEVELLER scores each training job by its progress rate and turns the switch priorities that Crux uses toward fairness, and what that costs.讀完後，你能說明 LEVELLER 如何以進度率為每個訓練任務評分，把 Crux 所用的交換器優先權轉向公平，以及代價為何。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 11 ↗第 11 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

On a shared link, a scheduler that maximizes efficiency can starve a job; Crux's static priority by GPU intensity is the example. LEVELLER scores each job by its normalized progress rate $R$ : the iterations it completed, divided by the iterations it would have completed alone. It aims at max-min fairness of $R$ . Giving strict priority to the job with the lowest $R$ at each moment provably reaches at least half of the max-min fair rate, also with periodic updates and in general networks; a controller and a shim under NCCL apply it through traffic classes. On 12 A100 GPUs it raised the minimum rate by up to 57% over Crux, at up to 15% lower average rate, and by 17–21% over CASSINI with four tenants.

在共享鏈路上，追求效率的排程器可能讓某個任務飢餓；Crux 依 GPU 強度設定的靜態優先權就是例子。LEVELLER 以正規化進度率 $R$ 為每個任務評分：它完成的迭代次數，除以它單獨執行時會完成的次數。目標是 $R$ 的 max-min 公平。每一刻把嚴格優先權給 $R$ 最低的任務，可證明至少達到 max-min 公平速率的一半，即使週期性更新、在一般網路上也成立；控制器與 NCCL 底下的 shim 透過 traffic class 執行它。在 12 顆 A100 GPU 上，它讓最低速率比 Crux 最多高 57%，平均速率最多低 15%；在四個租戶時比 CASSINI 高 17–21%。

1. Where this paper sits1. 這篇論文的位置

The Crux digest showed one answer to contention: rank jobs by GPU intensity and give the link to the top job. Section 11.1 shows the default answer: DCQCN gives each flow an equal share of the bottleneck. LEVELLER argues that neither is fair to jobs. Static priority can starve a low-ranked job. Equal bandwidth gives unequal training progress, because jobs differ in how much they communicate per unit of computation. What was missing is a job-level measure of what the network costs each job, and a scheduler that equalizes it with commodity priorities.

Crux 導讀展示了一種應對競爭的答案：依 GPU 強度為任務排名，把鏈路交給第一名。第 11.1 節展示了預設的答案：DCQCN 讓每條流平分瓶頸。LEVELLER 主張兩者對任務都不公平。靜態優先權可能讓排名低的任務飢餓。平分頻寬帶來不平等的訓練進度，因為各任務每單位計算所需的通訊量不同。缺少的是：一個衡量網路對每個任務代價的任務層級指標，以及用商用優先權把它拉平的排程器。

2. The problem, measured2. 問題，用量測說話

The paper reuses the Alibaba trace that Crux released: more than 30 concurrent jobs at peak, and over 36% of jobs exposed to inter-job contention. Its evidence of unfairness at scale is a flow-level simulation of that trace, so we give the mechanism through the paper's worked examples, not the simulated numbers.

論文沿用 Crux 公開的阿里追蹤資料：尖峰時超過 30 個任務同時執行，超過 36% 的任務曝露於任務間競爭。它在大規模下不公平的證據是對該追蹤的流級模擬，所以我們用論文的演算例說明機制，不列模擬數字。

A 2 s delay per iteration is catastrophic for a job with 1 s iterations (it runs at 1/3 of its ideal speed) and negligible for a job with 100 s iterations (100/102). On one link, three jobs with the same ideal iteration time of 3 units (compute 1 and communicate 2, twice; compute 2 and communicate 1) reach progress rates 0.5, 0.5 and 0.67 under flow-level fair share, so the third job runs 33% ahead. Figure 1 shows two jobs: Job 1 computes 1 and communicates 1; Job 2 computes 1 and communicates 2.

每次迭代延遲 2 s，對迭代 1 s 的任務是災難（只剩理想速度的 1/3），對迭代 100 s 的任務則微不足道（100/102）。在一條鏈路上，三個理想迭代時間同為 3 單位的任務（兩個計算 1、通訊 2；一個計算 2、通訊 1），在流級公平分享下的進度率是 0.5、0.5 與 0.67，第三個任務領先 33%。圖 1 是兩個任務：任務 1 計算 1、通訊 1；任務 2 計算 1、通訊 2。

Figure 1. The paper's two-job example (its Figures 3 and 4) over 15 time units. Fair share and static priority settle at cycles of 8; lowest-

R

-first keeps the link busy after

t = 1

and settles at the optimal cycle of 7.圖 1。論文的兩任務例子（其 Figures 3 與 4）的前 15 個時間單位。公平分享與靜態優先權都穩定在長度 8 的週期；最低

R

優先讓鏈路在

t = 1

之後一直忙碌，並穩定在長度 7 的最佳週期。Drawn for this primer from Li et al., SIGCOMM 2026, §2.4, §3.2 and Figures 3–4 (worked example); timelines from our fluid model of one link, which reproduces the paper's rates.

Fair share gives each job 0.75; static priority keeps Job 1 at 1 and Job 2 at 0.75. A schedule that switches priorities over time gives both 6/7 ≈ 0.86, so priorities must change during the job.

公平分享讓每個任務得到 0.75；靜態優先權讓任務 1 維持 1、任務 2 停在 0.75。隨時間切換優先權的排程讓兩者都得到 6/7 ≈ 0.86，所以優先權必須在任務執行中改變。

3. The design3. 設計

3.1 The metric: normalized progress rate3.1 指標：正規化進度率

Job $j$ starts at $t_j^0$ and completes $N_j(t)$ iterations by time $t$ . Its ideal iteration time $T_j^{\text{ideal}}$ is its time per iteration when it runs alone with full bandwidth. The normalized progress rate is

任務 $j$ 在 $t_j^0$ 開始，到時間 $t$ 完成 $N_j(t)$ 次迭代。它的理想迭代時間 $T_j^{\text{ideal}}$ 是它單獨以全頻寬執行時每次迭代的時間。正規化進度率（normalized progress rate）是

R_j(t) = \frac{N_j(t)}{(t - t_j^0) / T_j^{\text{ideal}}} = \frac{N_j(t)\, T_j^{\text{ideal}}}{t - t_j^0} \in [0, 1]

It is the inverse of the slowdown and compares jobs of any size or model. The objective is lexicographic max-min fairness of $R$ over a long window: no job's $R$ may rise by lowering a job whose $R$ is no larger. The model assumes dedicated GPUs and a constant computation time per iteration (Figure 2).

它是減速倍數的倒數，可以比較任何大小或模型的任務。目標是在長時間窗內 $R$ 的字典序 max-min 公平：任何任務的 $R$ 都不能靠降低一個 $R$ 不比它大的任務來提高。模型假設 GPU 專屬、每次迭代的計算時間固定（圖 2）。

Figure 2. On the left,

R

compares the iterations a job completed with the iterations it would have completed alone (the grey diagonal line), while on the right, every

\tau

the agents report

R

, the controller gives the lowest

R

the highest level, and the queues change which job advances.圖 2。左側以

R

比較任務完成的迭代次數與它單獨執行時會完成的次數（灰色斜線），右側則顯示每

\tau

，代理回報

R

，控制器把最高等級給

R

最低者，佇列隨之改變哪個任務前進。Drawn for this primer from Li et al., SIGCOMM 2026, §3.1, §4.1, Equation 1 and Figure 5.

3.2 Lowest rate first, with a proven bound3.2 最低速率優先，附可證明的界

On one link, let LCM be the least common multiple of the ideal iteration times and $\bar N_j = \mathrm{LCM} / T_j^{\text{ideal}}$ . Lemma 1 turns max-min fairness into a makespan problem, where the makespan is the time to finish $\bar N_j$ iterations of every job. All jobs reach the same optimal rate

在一條鏈路上，令 LCM 為各理想迭代時間的最小公倍數， $\bar N_j = \mathrm{LCM} / T_j^{\text{ideal}}$ 。Lemma 1 把 max-min 公平轉成 makespan 問題，makespan 是完成每個任務 $\bar N_j$ 次迭代所需的時間。所有任務達到相同的最佳速率

R^{*} = \frac{\mathrm{LCM}}{\min \text{makespan}(\bar N_j)}

Minimizing that makespan is NP-hard. Theorem 1 gives a simple rule: rank jobs by increasing $R_j(t)$ , and let a flow wait only for flows of lower- $R$ jobs. Any work-conserving, preemptive schedule that keeps this order is within a factor 2 of the optimal makespan, so $R_j \ge R_j^{*}/2$ for every job:

最小化這個 makespan 是 NP-hard。Theorem 1 給一條簡單規則：依 $R_j(t)$ 由小到大為任務排序，流只為 $R$ 更低的任務的流等待。任何維持這個順序、work-conserving 且可搶占的排程，都在最佳 makespan 的 2 倍以內，所以每個任務 $R_j \ge R_j^{*}/2$ ：

\begin{gathered} ms^{*} \ge \max\big(C_e^{\text{total}}, W_e^{\max}\big) \\ ms^{\sigma} \le C_e^{\text{total}} + W_e^{\min} \\ \Rightarrow\; ms^{\sigma} / ms^{*} \le 2 \end{gathered}

$C_e^{\text{total}}$ is all communication time on the link; $W_e^{\max}$ and $W_e^{\min}$ are the largest and smallest total computation of one job, because the link idles only while every job computes. For Figure 1:

$C_e^{\text{total}}$ 是鏈路上所有通訊時間； $W_e^{\max}$ 與 $W_e^{\min}$ 是單一任務總計算時間的最大與最小值，因為只有所有任務都在計算時鏈路才會閒置。以圖 1 為例：

Question問題	Calculation算式	Result結果
LCM of ideal iteration times 2 and 3; iterations per cycle理想迭代時間 2 與 3 的 LCM；每週期迭代次數	$6 \div 2;\ 6 \div 3$	$\bar N_1 = 3,\ \bar N_2 = 2$
Optimal rate, minimum makespan 7最佳速率，最小 makespan 為 7	$6 \div 7$	$\approx 0.857$
Flow-fair share, makespan 8流級公平分享，makespan 為 8	$6 \div 8$	$0.75$
Gain of the optimum over fair share最佳解相對公平分享的增益	$0.857 \div 0.75$	$\approx 1.14\times$ for both jobs兩個任務都是

The rule finds the optimal 7-unit cycle after a short transient (Figure 1, bottom; $R$ is 0.83 at $t = 36$ ). Re-ranking at every event is costly, so LEVELLER updates the order every $\tau$ . A misranked job loses at most $\tau$ of progress, and its lower $R$ then raises its priority (Corollary 1):

這條規則經過短暫的暫態後找到最佳的 7 單位週期（圖 1 下方； $t = 36$ 時 $R$ 為 0.83）。每個事件都重新排序代價太高，所以 LEVELLER 每 $\tau$ 更新一次順序。排錯的任務最多損失 $\tau$ 的進度，而它較低的 $R$ 隨後會提高它的優先權（Corollary 1）：

\min_j R_j^{\sigma,\tau}(t) \ge \tfrac{1}{2} R^{*} - \frac{\tau}{t - \max_i t_i^0}

The loss vanishes as $\tau / t \to 0$ ; with $\tau = 4$ the example enters a 28-unit cycle at the same 0.86. For general networks the paper reuses Crux's bottleneck-link reduction. The local converged rate of a link is the $R$ its jobs reach if every other link were infinite. Jobs on the link with the lowest such rate also have the lowest $R$ , so they hold the top priority everywhere and meet no second bottleneck; induction keeps the factor $\tfrac12$ network-wide (Corollary 2).

當 $\tau / t \to 0$ 時損失消失； $\tau = 4$ 時例子進入長度 28 的週期，速率同樣是 0.86。對一般網路，論文沿用 Crux 的瓶頸鏈路化約。一條鏈路的局部收斂速率（local converged rate）是：假設其他鏈路頻寬無限時，該鏈路上任務達到的 $R$ 。這個速率最低的鏈路上的任務， $R$ 也最低，所以它們在所有地方都握有最高優先權，不會遇到第二個瓶頸。對瓶頸做歸納，使 $\tfrac12$ 的因子在整個網路成立（Corollary 2）。

Panel 1面板 1

Efficiency first or fairness first?效率優先還是公平優先？

Jobs任務組

Update interval

\tau

(0 = every iteration)更新間隔

\tau

（0 = 每次迭代）

Job 1 compute任務 1 計算

Job 1 communication任務 1 通訊

Job 2 compute任務 2 計算

Job 2 communication任務 2 通訊

Job 3 compute任務 3 計算

Job 3 communication任務 3 通訊

Efficiency first: min R

效率優先：最低 R

Efficiency first: average R

效率優先：平均 R

Fairness first: min R

公平優先：最低 R

Fairness first: average R

公平優先：平均 R

Flow-fair share: min / average R

流級公平分享：最低 / 平均 R

Fairness vs efficiency first

公平優先相對效率優先

A deterministic model of one link, as in the paper's Figures 1, 3 and 4. Each job alternates compute and communication; communication is its link time at full rate. Efficiency first: static strict priority by GPU intensity $G_j w_j / c_j$ , Crux's rule with $k_j = 1$ . Fairness first: strict priority to the lowest $R$ , re-ranked at every iteration end ( $\tau = 0$ ) or every $\tau$ . Flow-fair share: sending jobs split the link equally, like DCQCN. $R$ is measured over 1,500 times the longest ideal iteration time from a common start, transient included. Defaults are the paper's Figure 3 pair, which reproduces its rates: (1, 0.75) for static priority, (0.75, 0.75) for fair share, 0.857 for both under lowest $R$ first. "Fig. 1 trio" gives 0.5, 0.5, 0.67 under fair share and 0.6 each under lowest $R$ first; static priority starves Job 2 there. "GPT + BERT" uses the model jobs of the Crux panel (Job 1 = GPT on 64 GPUs, Job 2 = BERT on 16, times in seconds) without its jitter. GPU utilization weights each job's compute by its GPUs. Raise $\tau$ : the first cycles in the timeline change, while the long-run rates stay close, as Corollary 1 predicts.

這是一條鏈路的確定性模型，與論文的 Figures 1、3、4 相同。每個任務在計算與通訊之間交替；通訊是它以全速使用鏈路的時間。效率優先：依 GPU 強度 $G_j w_j / c_j$ 設定的靜態嚴格優先權，即 $k_j = 1$ 的 Crux 規則。公平優先：把嚴格優先權給 $R$ 最低者，在每次迭代結束（ $\tau = 0$ ）或每 $\tau$ 重新排序。流級公平分享：傳送中的任務平分鏈路，類似 DCQCN。 $R$ 從共同的起點量到最長理想迭代時間的 1,500 倍，包含暫態。預設值是論文 Figure 3 的兩任務，重現其速率：靜態優先權 (1, 0.75)、公平分享 (0.75, 0.75)、最低 $R$ 優先時兩者都是 0.857。「Fig. 1 三任務」在公平分享下是 0.5、0.5、0.67，在最低 $R$ 優先下各為 0.6；靜態優先權在那裡讓任務 2 飢餓。「GPT + BERT」使用 Crux 面板的模型任務（任務 1 = 64 顆 GPU 的 GPT，任務 2 = 16 顆的 BERT，時間以秒計），但不加隨機變動。GPU 使用率以各任務的 GPU 數加權其計算。提高 $\tau$ ：時間軸上最初幾個週期會改變，而長期速率幾乎不變，正如 Corollary 1 所預測。

Stop and think停下來想一想

Switch to "GPT + BERT". Efficiency first gives GPT full speed and higher utilization. What does fairness first cost, and what does it buy BERT? Which goal suits a cloud that bills per GPU-hour?

切到「GPT + BERT」。效率優先讓 GPT 全速執行、使用率較高。公平優先付出多少，又為 BERT 換來什麼？按 GPU 小時計費的雲端適合哪個目標？

3.3 Few priority levels, and overlap3.3 少數優先權等級，以及重疊

Switches offer about 8 levels, some reserved, for 15–20 concurrent jobs on average. LEVELLER assigns levels per flow, and flows that never contend can be set independently. It builds a contention graph (one node per flow, an edge when flows of different jobs share a link) and prunes it: in each connected part, the flow with the lowest $R$ takes the highest free level; those nodes leave the graph; this repeats until one level is left for all remaining flows (Figure 3).

交換器約有 8 個等級，其中一些保留，而平均有 15–20 個任務同時執行。LEVELLER 以流為單位指派等級，從不競爭的流可以各自設定。它建立競爭圖（每條流一個節點，不同任務的流共用鏈路時連一條邊）並逐步修剪：在每個連通部分中， $R$ 最低的流取得最高的可用等級；這些節點離開圖；重複直到只剩一個等級，留給所有剩下的流（圖 3）。

Figure 3. The paper's two-level example (its Figure 6), solved by the pruning of its Figure 7. The lowest-

R

flow of each part (f1, f22) gets level 1, so Job 2's flows end on different levels and every link carries one level-1 and one level-0 flow.圖 3。論文的兩等級例子（其 Figure 6），以其 Figure 7 的修剪求解。每個部分中

R

最低的流（f1、f22）得到等級 1，所以任務 2 的兩條流落在不同等級，而每條鏈路上都是一條等級 1、一條等級 0 的流。Drawn for this primer from Li et al., SIGCOMM 2026, §4.2 and Figures 6–7.

A second correction uses overlap. If job $j$ 's communication that starts at $t_j^i$ can wait up to a slack without stalling the next operation (for example a DDP AllReduce during the backward pass), the local agent replaces the controller's priority $P_j(t)$ with the lowest level inside that window, with no coordination:

第二個修正利用重疊。如果任務 $j$ 在 $t_j^i$ 開始的通訊可以延後至多 slack 而不卡住下一個操作（例如反向傳遞期間的 DDP AllReduce），本地代理就在這個時間窗內以最低等級取代控制器給的優先權 $P_j(t)$ ，不需任何協調：

P'_j(t) = \begin{cases} \text{lowest}, & t_j^i \le t \lt t_j^i + \mathit{slack} \\ P_j(t), & \text{later} \end{cases}

3.4 Controller and agents3.4 控制器與代理

A logically central controller and one agent per tenant host form the system (Figure 4). The agent is a shim between the collective library and the transport and handles only inter-host flows. Every $\tau$ the controller collects each job's $R_j$ and {flow, path} map, computes levels and pushes them back. A new job first runs with the unique highest priority to measure $T^{\text{ideal}}$ ; re-profiling at semi-stochastic intervals keeps it current.

系統由一個邏輯上集中的控制器與每台租戶主機上的一個代理組成（圖 4）。代理是集體通訊函式庫與傳輸層之間的 shim，只處理主機之間的流。每 $\tau$ ，控制器收集每個任務的 $R_j$ 與 {流, 路徑} 對應，計算等級並推送回去。新任務先以唯一的最高優先權執行，以量測 $T^{\text{ideal}}$ ；在半隨機的間隔重新剖析讓它保持最新。

LEVELLER architecture: a controller above per-tenant agents between the collective library and the transport; one agent contains Priority Update, Rate Tracker, Priority Enforcer and Path Probe — Figure 4. The paper's Figure 5 (design). Left: the controller and per-tenant agents under the collective library (CCL). Right: one agent; ①–③ are the update loop: the controller asks, the agents report, the controller sends new levels.圖 4。論文 Figure 5（設計）。左：控制器與位於集體通訊函式庫（CCL）之下的各租戶代理。右：單一代理；①–③ 是更新迴圈：控制器發出請求、代理回報、控制器送回新等級。Li et al., SIGCOMM 2026, Figure 5 (design figure).

4. Implementation4. 實作

LEVELLER is about 2,000 lines of C++ and Python for PyTorch and NCCL, loaded with LD_PRELOAD, so training code does not change. The Rate Tracker parses the framework's log output for iteration counts. The Path Probe intercepts NCCL connection calls such as ncclIbRtrQp and sends INT probes with the same 5-tuple to learn each path. For TCP, iptables rules in the mangle table re-mark DSCP per 5-tuple. RDMA is harder, because Verbs forbids changing a queue pair's traffic class in the ready-to-send state. So LEVELLER performs a QP resurrection: it waits until every sent packet is acknowledged, resets the queue pair with ibv_modify_qp, and rebuilds it through INIT, RTR and RTS with the new traffic class.

LEVELLER 約 2,000 行 C++ 與 Python，支援 PyTorch 與 NCCL，以 LD_PRELOAD 載入，所以訓練程式碼不必修改。Rate Tracker 解析框架的日誌輸出以取得迭代次數。Path Probe 攔截 ncclIbRtrQp 等 NCCL 連線呼叫，並送出 5-tuple 相同的 INT 探測封包以得知每條路徑。對 TCP，mangle 表中的 iptables 規則依 5-tuple 重新標記 DSCP。RDMA 比較難，因為 Verbs 不允許在 ready-to-send 狀態改變 queue pair 的 traffic class。所以 LEVELLER 執行 QP 復活（QP resurrection）：等到每個已送出的封包都被確認，用 ibv_modify_qp 重設 queue pair，再以新的 traffic class 依序重建 INIT、RTR 與 RTS。

5. What was measured5. 量到了什麼

The testbed has three hosts, each with four A100 GPUs and four ConnectX-6 NICs. Five logical switches on two physical switches emulate three leaves and two spines; inter-switch links are rate-limited to 50 Gbps, with ECMP and DCQCN. Four models of 1.3B to 6B parameters train with DDP or FSDP on 2–4 GPUs per tenant, placed at random; each run is repeated at least ten times. CASSINI and Crux were re-implemented on the same testbed.

測試平台有三台主機，每台四顆 A100 GPU 與四張 ConnectX-6 NIC。兩台實體交換器上的五台邏輯交換器模擬三個 leaf、兩個 spine；交換器之間的鏈路限速 50 Gbps，使用 ECMP 與 DCQCN。四個 1.3B 到 6B 參數的模型以 DDP 或 FSDP 訓練，每個租戶 2–4 顆 GPU、隨機配置；每次實驗至少重複十次。CASSINI 與 Crux 在同一平台上重新實作。

Test測試	Measured實測結果
RDMA, three 2-GPU jobs on one bottleneck, 2 levels (Fig. 9)RDMA，三個 2 顆 GPU 的任務共用一個瓶頸，2 個等級（Fig. 9）	Iterations in one window: 14/27/14 (LEVELLER), 13/26/15 (CASSINI), 12/23/17 (Crux); minimum $R$ 0.69 vs 0.63 vs 0.61同一時間窗內的迭代次數：14/27/14（LEVELLER）、13/26/15（CASSINI）、12/23/17（Crux）；最低 $R$ 0.69 對 0.63 對 0.61
Four tenants vs CASSINI (Fig. 10)四個租戶，相對 CASSINI（Fig. 10）	RDMA: minimum +21%, average +6%; TCP: minimum +17%, average +4%RDMA：最低 +21%、平均 +6%；TCP：最低 +17%、平均 +4%
2–4 tenants vs Crux (Fig. 10)2–4 個租戶，相對 Crux（Fig. 10）	Minimum up to +57% (3 jobs, RDMA); average up to −15% (4 jobs, RDMA)最低最多 +57%（3 個任務，RDMA）；平均最多 −15%（4 個任務，RDMA）
Oversubscription 1:1 and 1:4 (App. F, Fig. 20)超額認購 1:1 與 1:4（附錄 F，Fig. 20）	Minimum vs CASSINI +18%, +30%; vs Crux +35%, +73%; average slightly below Crux最低相對 CASSINI +18%、+30%；相對 Crux +35%、+73%；平均略低於 Crux
Overhead, four tenants開銷，四個租戶	Controller–agent latency < 200 µs; a priority switch takes a few ms, at a minute-level cadence控制器到代理延遲 < 200 µs；一次優先權切換數 ms，頻率為分鐘級

Figure 5 shows the pattern of the worked examples on real hardware. Crux keeps the highest average but leaves one job far behind; LEVELLER's minimum and average almost coincide.

圖 5 在真實硬體上呈現了演算例的模式。Crux 保有最高的平均，卻讓一個任務遠遠落後；LEVELLER 的最低值與平均值幾乎重合。

Bar chart: minimum and average normalized progress rate for Crux, CASSINI and LEVELLER with 2, 3 and 4 jobs on TCP and RDMA — Figure 5. Figure 10 of the paper, measured on the 12-GPU testbed: minimum (solid) and average (pale, dashed) normalized progress rate with 2–4 tenants, TCP left and RDMA right. RDMA rates are lower because faster links raise contention.圖 5。論文 Figure 10，在 12 顆 GPU 的測試平台上實測：2–4 個租戶時的最低（實心）與平均（淺色、虛線）正規化進度率，左為 TCP、右為 RDMA。RDMA 的速率較低，因為較快的鏈路提高了競爭。Li et al., SIGCOMM 2026, Figure 10 (testbed, measured).

The 1,024-GPU results, including the largest gains and the claim that two levels suffice, come from a flow-level simulator fed by ASTRA-sim, so this page does not report them. The appendices, including Figure 20, are marked as not peer-reviewed.

1,024 顆 GPU 的結果，包括最大的增益與「兩個等級就夠」的說法，來自以 ASTRA-sim 為輸入的流級模擬器，所以本頁不列出。附錄（包括 Figure 20）標註為未經同儕審查。

6. Limits and open questions6. 限制與未解的問題

The paper names its efficiency cost against Crux, new jobs that look fast until enough iterations accumulate, and expert parallelism, whose iteration times vary (handled by re-profiling, not evaluated). A careful reader adds:

論文列出：相對 Crux 的效率代價、新任務在累積足夠迭代之前看起來很快，以及迭代時間會變動的專家平行（以重新剖析處理，未評估）。細心的讀者還應補上：

The bound is loose. $\tfrac12$ permits halving; the measured gains, not the bound, carry the case.界很寬鬆。 $\tfrac12$ 允許減半；支撐論點的是量到的增益，不是這個界。
$R$ is cumulative. A long-running job's $R$ moves slowly, so recovery after a disturbance takes time. $R$ 是累積的。長時間執行的任務 $R$ 變化很慢，所以干擾之後的恢復需要時間。
Job size is ignored. A 2-GPU and a 128-GPU job count the same, the opposite of Crux's weighting.忽略任務大小。2 顆 GPU 與 128 顆 GPU 的任務算一樣，正好與 Crux 的加權相反。
Priorities and PFC. End-to-end strict priority needs lossless classes with headroom, which the paper does not study.優先權與 PFC。端到端的嚴格優先權需要附 headroom 的無損類別，論文沒有研究這點。
Scale. 12 GPUs and models up to 6B.規模。12 顆 GPU、最大 6B 的模型。

7. How it connects7. 與其他內容的關聯

LEVELLER and Crux use the same knobs for opposite goals:

LEVELLER 與 Crux 用同樣的旋鈕追求相反的目標：

	Crux (SIGCOMM 2024)	LEVELLER (SIGCOMM 2026)
Goal目標	Cluster GPU utilization叢集 GPU 使用率	Max-min fairness of $R$ $R$ 的 max-min 公平
Score per job任務的分數	$k_j I_j$ , static $k_j I_j$ ，靜態	$R_j(t)$ , updated every $\tau$ $R_j(t)$ ，每 $\tau$ 更新
Shared knobs共用的旋鈕	Strict priority, about 8 levels, one bottleneck link per job, graph-based compression, path probes, the Alibaba trace嚴格優先權、約 8 個等級、每個任務一條瓶頸鏈路、以圖為基礎的壓縮、路徑探測、阿里追蹤資料
Who pays誰付出代價	Low-intensity jobs低強度任務	Average rate, up to −15% on the testbed平均速率，測試平台上最多 −15%

Deep dive §1: DCQCN's per-flow fairness is the "flow-fair share" row of the panel. §4: $R$ is the inverse of the per-job slowdown that exposed communication causes; its Panel 4 computes that slowdown.第 11.1 節：DCQCN 的逐流公平就是面板中的「流級公平分享」。第 4 節： $R$ 是暴露的通訊造成的任務減速倍數之倒數；其面板 4 計算的就是這個減速。
Guided readings: DCQCN gives the flow-level fairness that LEVELLER shows is unfair to jobs; Microsoft RoCE explains the 8 priority classes and per-class PFC that these levels ride on.論文導讀：DCQCN 提供了 LEVELLER 指出對任務不公平的流級公平；Microsoft RoCE 解釋了這些等級所依賴的 8 個優先權類別與逐類別 PFC。

Key numbers關鍵數字

Number	數字	Meaning and setup
0.75 → 0.86	Both jobs' $R$ : fair share vs max-min optimum (worked example)	兩個任務的 $R$ ：公平分享對 max-min 最佳（演算例）
1/2	Proven guarantee of lowest- $R$ -first, lazy updates and general networks included	最低 $R$ 優先的證明保證，含延遲更新與一般網路
+57% / −15%	Minimum / average $R$ vs Crux, 12-GPU testbed	相對 Crux 的最低 / 平均 $R$ ，12 顆 GPU 測試平台
+21% / +17%	Minimum $R$ vs CASSINI, four tenants, RDMA / TCP	相對 CASSINI 的最低 $R$ ，四個租戶，RDMA / TCP
< 200 µs; a few ms< 200 µs；數 ms	Controller–agent latency; one priority switch	控制器到代理延遲；一次優先權切換

Terms used on this page本頁用到的術語

Normalized progress rate $R$: Iterations completed times ideal iteration time, over elapsed time; 1 means no slowdown.; 完成的迭代次數乘理想迭代時間，除以經過時間；1 代表沒有減速。
Makespan: The time to finish a given set of work.; 完成一組給定工作所需的時間。
QP resurrection: Draining, resetting and rebuilding an RDMA queue pair to change its traffic class.; 排空、重設並重建 RDMA queue pair 以改變其 traffic class。