Compact duplicate LoRA adapter tensors

[hershg]

Problem

Non-colocated merge_lora=false publication wrote every public LoRA tensor to safetensors independently. GLM-5.3's default share_expert_adapters=true export has 116,448 public keys but only 1,545 unique FP32 storages, so the old path serialized and reread about 61.5 GB per update.

What changed

• Detect exact duplicates by dtype, shape, and byte fingerprint.
• Serialize a shared-storage PyTorch artifact only when unique bytes are less than half of logical bytes.
• Preserve every public key and exact FP32 value.
• Keep safetensors for nonduplicated layouts.
• Publish through a temporary file plus atomic rename and remove the stale opposite format during representation changes.
• Rely on the existing vLLM loader to materialize independent BF16 destination tensors.

This PR contains only the compact-disk optimization. Native NCCL/NIXL transport work is separate.

Matched result

The profiler-identical native GLM-5.3 comparison used the same two-B300 topology, immutable image/runtime pins, 32K workload, rank-32 adapter, target modules, instrumentation, one excluded warmup, and three measured updates.

Publication + first sample	Update 1	Update 2	Update 3	Mean
Baseline	481.39s	495.87s	457.81s	478.36s
Compact	177.86s	164.75s	174.00s	172.20s

That is a measured 2.78x speedup.

Rank-0 compact means were:

• Bridge export: 22.89s, including 17.64s in nested CUDA-to-CPU waits.
• FP32 clone: 20.98s.
• duplicate detection: 49.67s.
• serialization and atomic write: 1.66s.
• vLLM load/materialization/registration/activation: 74.75s.
• first sample: 0.52s.

The retained production artifact was 1,232,385,444 bytes versus 61,521,624,984 bytes for baseline. An independent inspector verified 61,504,339,968 logical FP32 bytes, 1,221,967,872 unique tensor bytes, exact value/layout/alias fingerprints, and independent BF16 conversion storage.

A same-node-pair shared-filesystem control measured:

Exact-size payload	Durable write	Cross-node read
Compact, mean of 3	0.91s / 1.27 GiB/s	5.05s / 0.227 GiB/s
Baseline, 1 run	52.38s / 1.09 GiB/s	254.47s / 0.225 GiB/s

This explains most of baseline receiver latency as shared-filesystem reading. At compact size, raw reading is only about 5s of the 74.75s receiver RPC, leaving parsing, tensor reconstruction, BF16 materialization, registration, and activation as the main remaining receiver work.

Observed publication GPU peaks did not increase: trainer 225,705 MiB/GPU in both arms; inference 206,291 MiB/GPU baseline and 206,207 MiB/GPU compact. Peak cgroup current fell from 1,404.95 to 1,202.51 GiB on the trainer node and from 1,005.87 to 808.86 GiB on the inference node.

The GLM run completed repeated replacement and sampling, checkpoint, and unload. Its protocol validator accepted six publications and 54 phase records; the trace audit checked 64 inference traces across eight TP ranks and a 3.83-million-event trainer trace.

Scope boundary

This optimization depends on duplicated adapter values. The matched Qwen3-8B updated adapter had no comparable duplication and used the same 349,243,720-byte safetensors representation in both arms:

Qwen publication + first sample	Baseline	Compact
Mean of 3 measured updates	3.93s	4.09s

The single-pair 0.16s difference does not establish a regression. It shows that this optimization provides no benefit when the adapter is not duplicated. The same limitation applies when trainer.policy.model.lora.share_expert_adapters=false.

Correctness and validation

• Exact byte equality covers signed zero and NaN payloads.
• Compact serialization preserves public keys and shared storage.
• BF16 conversions of aliases use independent destination storage.
• Format changes remove the stale opposite artifact.
• Matched Qwen arms passed the pinned [GLM 5.3 Tests] Check trainer/inference LoRA logprob agreement #2182 numerical gates, produced readable checkpoints, sampled after updates, and unloaded.
• CPU CI passed skyrl_tests, check_code_quality, skyrl_train_tests, and skyrl_gym_tests on the code head.
• Focused Ruff checks and diff checks pass.
• The GPU workflow currently exits before tests because its Anyscale token is empty; it does not report a test failure in this change.