Skip to content

Fix deterministic checkpoint file sharding - #103

Merged
weixiao-huang merged 1 commit into
MoonshotAI:mainfrom
koriyoshi2041:fix-deterministic-checkpoint-shards
Sep 4, 2026
Merged

Fix deterministic checkpoint file sharding#103
weixiao-huang merged 1 commit into
MoonshotAI:mainfrom
koriyoshi2041:fix-deterministic-checkpoint-shards

Conversation

@koriyoshi2041

Copy link
Copy Markdown
Contributor

Problem

split_checkpoint_files() partitions the raw result of os.listdir(). Directory enumeration order is not guaranteed, so ranks on different nodes can derive different file-to-rank assignments for the same checkpoint. That can make checkpoint registration overlap some shards and omit others.

Fix

Sort the filtered .safetensors paths before slicing them across ranks.

The regression test presents the same directory contents in forward and reverse enumeration order and verifies that both produce the same complete, non-overlapping assignment.

Tests

uv run pytest -q tests/test_example_update.py
1 passed

uv run ruff check examples/update.py tests/test_example_update.py
All checks passed!

uv run ruff format --check examples/update.py tests/test_example_update.py
2 files already formatted

@weixiao-huang
weixiao-huang merged commit fee5a53 into MoonshotAI:main Sep 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants