Add an on-demand Iris vs RCCL CCL benchmark job

[nirvedhmeshram]

Summary

Adds an on-demand job that benchmarks the iris.ccl collectives against RCCL and publishes a comparison table and graphs. Manual trigger or post-submit on main — deliberately not on pull requests.

This existed before and was lost

benchmark/ccl/comprehensive_sweep.py swept the collectives with a --benchmark_rccl flag, and the per-op scripts reported rccl_ms / rccl_bandwidth_gbps / rccl_ratio_percent. It was deleted in 15bba61 when the benchmarks migrated to iris.bench (#486), and the RCCL paths went with it.

What survived is benchmark/ccl/plot_sweep_results.py — still documented as "subplots comparing Iris vs RCCL bandwidth for each collective operation", still reading a CSV that nothing has produced since. This gives it a real input again.

Approach

Rather than restore the old subprocess driver, each CCL benchmark gets a backend axis:

@bench.axis("backend", ["iris", "rccl"])

Both series then come out of one sweep over identical shapes through the same timing harness, and the framework's existing CSV already carries the comparison. The process group is already nccl, so RCCL needs no extra setup. preamble_fn runs outside the timed region, so the copy_ needed to give dist.all_reduce out-of-place semantics does not pollute the measurement.

Two judgement calls worth reviewing

The RCCL baseline uses ordinary torch tensors, not Iris heap memory. The symmetric heap is allocated with the flags RMA requires, so timing RCCL against it would measure Iris's allocation choice rather than RCCL. Someone deciding between the two would call RCCL on normal tensors, so that is what is measured.

reduce_scatter is excluded. iris.ccl.reduce_scatter is (M, N) -> (M, N) — "each rank reduces its assigned tiles, stores locally" — whereas torch.distributed.reduce_scatter_tensor is (W*M, N) -> (M, N). Publishing those side by side would compare two different operations. Happy to add it if the intended correspondence is different from how I read it.

all_to_all is included with a documented caveat: Iris splits (M, N*W) along dim 1, dist.all_to_all_single splits along dim 0, so the RCCL path uses (M*W, N). Identical communication pattern and total bytes — W-1 peer messages of M*N elements — only the layout differs.

Cost

Not on pull_request by design. Three collectives across two backends is enough work to crowd out the test jobs on a shared 8-GPU pool. Ranks, sizes and repeat count are workflow_dispatch inputs; the default sweep is deliberately small (M=N ∈ {1024, 4096, 16384}, fp16, 8 ranks) and can be widened per run without editing anything.

Output

The job writes the table into the run summary and uploads CSVs and PNGs as artifacts (30-day retention). Three figures are produced, matching triton-shmem's plot_bench.py: bus bandwidth, latency, and the Iris/RCCL latency ratio.

Results on MI355X

Run on 8x MI355X (gfx950), ROCm 7.2.1, PyTorch 2.10 — 576 measurements: 3 collectives x {2,4,8} ranks x 10 message sizes (2 MiB - 1 GiB) x {fp16, bf16}. Ratios are Iris/RCCL bus bandwidth; above 1.00x means Iris is faster.

Collective	Ranks	2 MiB	64 MiB	1 GiB
all_gather	2 / 4 / 8	0.58 / 0.58 / 0.54	1.01 / 0.91 / 0.85	0.93 / 0.96 / 0.91
all_reduce	2 / 4 / 8	0.58 / 0.57 / 0.51	0.93 / 0.78 / 0.64	0.93 / 0.84 / 0.68
all_to_all	2 / 4 / 8	0.64 / 0.57 / 0.58	0.94 / 0.97 / 0.98	0.93 / 0.98 / 1.02

Two things worth a reviewer's attention:

The rank-scaling behaviour splits by collective. all_gather and all_to_all are close to rank-independent at large messages and sit near parity regardless of world size, with all_to_all crossing ahead at 8 ranks. all_reduce moves the other way as ranks double: 0.93x -> 0.84x -> 0.68x. Fixed per-launch overhead predicts the first pattern and not the second, which points at the two_shot algorithm rather than at Iris's launch path.

Small messages are overhead-dominated across the board (~0.5-0.6x at 2 MiB), converging as size grows. That is the expected shape and is not specific to any one collective.

Test plan

The sweep has been run on hardware: 576 measurements across the full M x N x dtype grid at 2, 4 and 8 ranks, with the figures and table above generated by the committed plotting path.

The plotting and table path is additionally verified against the real iris.bench CSV schema, including that skipped=True rows are dropped and a missing RCCL point renders as an em dash rather than a bogus ratio.

One operational note for anyone sweeping many shapes in a single process: the benchmarks do not free symmetric buffers between shape combinations, so they accumulate. all_to_all at 8 ranks over the whole grid exhausts a 64 GiB heap, since its output is world_size times the input. Splitting the sweep by M (one process per value) keeps each working set well inside the heap.

[drprajap]

can we include fp8 too?

[nirvedhmeshram]

there is no RCCL fp8 available so this would not have a baseline, happy to add if we just want to see them by themselves though. Let me know.

[nirvedhmeshram]

These are the graphs this CI is expected to generate (only on demand currently)

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 5 · 1

Open (6)

• This description contradicts the implementation in _grid() (and benchmark/ccl/README.md), which… · New
• The table merges all variants for a backend into a single size -> point map via dict.update(),… · New
• The Markdown table uses truthiness checks (if il, if rl, etc.). This will render valid 0.0… · New
• This allocates world_size CUDA tensors plus a fresh input tensor for every benchmark point. Even… · New
• gpu_time_ms now represents the slowest rank’s median, while all_times_ms appears to remain the… · New
• The PR description mentions 'Four collectives across two backends', but the workflow runner… · New

What changed in this PR

Adds an on-demand CI workflow to benchmark Iris CCL collectives against RCCL using a shared iris.bench timing harness, then publishes plots and a Markdown comparison table.

Changes:

• Extend iris.bench result aggregation to report cross-rank latency (slowest-rank median), plus min_time_ms and skew_pct in console/JSON/CSV.
• Add a backend sweep axis (iris vs rccl) to CCL benchmarks and a shared _common.py for axes and RCCL tensor allocation.
• Revive/modernize the CCL plotting pipeline to consume iris.bench CSV output and emit bandwidth/latency/speedup figures + Markdown summary; add a GitHub Actions workflow and runner script.

File	Description
iris/​bench/​_runner.py	Adds cross-rank aggregation and emits new CSV/JSON fields (min_time_ms, skew_pct).
iris/​bench/​_core.py	Documents new result semantics and fields on Result.
benchmark/​ccl/​plot_sweep_results.py	Reworks plotting/summary to consume iris.bench CSV and compare Iris vs RCCL.
benchmark/​ccl/​bench_all_reduce.py	Adds backend axis and RCCL baseline path; aligns bytes accounting.
benchmark/​ccl/​bench_all_gather.py	Adds backend axis and RCCL baseline path; aligns bytes accounting.
benchmark/​ccl/​bench_all_to_all.py	Adds backend axis and RCCL baseline path; aligns bytes accounting.
benchmark/​ccl/​_common.py	Centralizes sweep axes and RCCL “plain tensor” helper.
benchmark/​ccl/​README.md	Documents benchmark intent, axes, aggregation, and plotting outputs.
.github/​workflows/​iris-ccl-benchmark.yml	Adds on-demand/post-submit workflow to run sweep and publish artifacts/summary.
.github/​scripts/​run_ccl_benchmark.sh	Implements the sweep + plotting inside the container and writes artifacts.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.