feat: add Slurm retry and deterministic collection

[nabinchha]

📋 Summary

Implements the #869 retry-and-collection slice for persisted Slurm runs. Failed shards can be retried from fresh reconciliation state, and exactly one validated winner per planned shard is collected by a zero-GPU CPU Slurm job into a deterministic, atomically published dataset.

This PR is stacked on #913. PR #913 must merge first. The branch is exactly six retry/collection commits above the current #913 head (ce1fe44b). It consumes #913's public observation/reconciliation APIs, composes retry binding into #929's shell-phase one-node runtime, and adds capability APIs only; CLI wiring remains owned by #874#3.

🔗 Related Issue

Closes #869

🔄 Changes

✨ Added

• Persisted retry plans/status with exact sparse shard ownership, resume-and-shard locking, scheduler-identity validation, ambiguous-submission recovery, and winner-aware idempotency.
• Persisted collection plans/status/results with normalized profile-authorized destinations and exact winner/result/attempt/candidate validation.
• A zero-GPU CPU collection launcher and worker that performs bounded-memory Parquet input reads and deterministic Parquet, CSV, or JSONL partitioning.
• Same-parent staging and atomic no-overwrite publication with descriptor/path rebinding, exact-stage cleanup, and interruption/collision recovery.
• Adversarial regression coverage for stale state, conflicting winners, journal interruption, concurrent submissions, path replacement, output mutation, publication races, tampered retry bindings, concurrent collection observation, and bounded preemption/requeue liveness.

🔧 Changed

• Bind retry allocations to the persisted retry ID, retry-plan digest, and resolved resume mode from rendered submission through every shell control phase and client generation.
• Resolve collection destinations through the pinned workspace root plus unique normalized writable mounts, and reauthorize persisted destinations before collection.
• Separate per-collection worker ownership from short metadata transitions so reconciliation can observe and refresh state during bulk collection I/O.
• Use the image-build partition for collection jobs and canonical sacct JobID values for recoverable array shape.
• Expose the retry and collection capabilities through public Slurm state APIs without importing sibling-private names or changing CLI behavior.

🐛 Fixed

• Isolate scheduler observation timestamps per attempt so a clock-ahead sibling cannot prematurely expire another attempt's preemption-requeue window.
• Preserve profile.scheduler.bin_path in retry allocation scripts through the same bounded PATH policy used by initial generation scripts.
• Require candidate provenance to match the stable resolved-plan digest across producer, finalizer, and collector boundaries.
• Reconcile ambiguous retry and collection submissions from fresh processes by an immutable plan-derived job name, explicit current-UID ownership, and exact ordinary/sparse-array shape before binding the scheduler ID.
• Persist a bounded visibility deadline so accepted submissions recover without resubmission, while absent submissions can be retired and safely replaced.
• Fence delayed ambiguous allocations at the persisted retry-attempt or collection scheduler identity so only the recorded job can write, including when an earlier accepted allocation becomes visible after replacement.
• Bound partial sparse-array visibility: strict subsets wait through the reconciliation deadline, then retire as bounded absence so a retry cannot remain permanently PREPARED.
• Let a fast collection allocation wait outside the state lock for its exact scheduler binding instead of failing while the durable status is still PREPARED.

🧪 Testing

• Full Slurm suite after the current feat: reconcile persisted Slurm state #913 restack — 1,427 passed
• Current-head retry, reconciliation, service, renderer, golden, and shell runtime regressions — 110 passed
• make check-slurm — 194 files formatted and linted
• git diff --check
• Isolated wheel build/install/import and lazy CLI extension smoke test passes
• Unit tests added/updated
• E2E tests added/updated (worker/launcher integration through Slurm fakes)

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; this implements the approved Stage 2 plan)

---

Description updated with AI

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness, security, or repository-rule findings.

Summary

• Recovers ambiguous retry and collection submissions using immutable job names and validated scheduler shapes.
• Binds retry IDs, plan digests, scheduler identities, and effective resume modes through the allocation runtime.
• Collects validated shard winners into deterministic Parquet, CSV, or JSONL partitions.
• Uses private same-parent staging and atomic publication while rejecting destination collisions and input mutation.
• The three previous findings are resolved in the current code.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Persisted run state] --> B[Reconcile scheduler observations]
    B --> C{Failed shards?}
    C -->|Yes| D[Persist retry plan]
    D --> E[Submit sparse Slurm retry array]
    E --> F[Bind scheduler and retry identity]
    F --> G[Generate replacement candidates]
    G --> H[Finalize one winner per shard]
    C -->|No| H
    H --> I[Persist collection plan]
    I --> J[Submit zero-GPU collection job]
    J --> K[Validate winner and candidate chain]
    K --> L[Stream deterministic partitions]
    L --> M[Private same-parent stage]
    M --> N[Atomic no-overwrite publication]

Loading

_{Reviews (18) · Last reviewed commit: "fix(slurm): isolate attempt observation ..."}

[nabinchha]

Ownership note on the latest Greptile summary: I verified that both flagged execute-path behaviors—the production factory leaving artifact_publisher unset and ambiguous sbatch outcomes after run initialization—are inherited entirely from merged #926 at d7b24521. #915's exact range is 8139aa07..a79acf5f; it does not modify services/wiring.py, services/run.py, cli.py, or client/dependencies.py. No #915 code change is appropriate: this retry/collection slice is clean and its earlier ambiguous-recovery findings are addressed. The two M2 execute-path defects require a separate feat/slurm-execution follow-up.

[nabinchha]

Addressed the remaining Greptile summary finding in da8793b. The issue was valid: service status used the maximum timestamp from every active attempt as the observation time for all attempts, so a clock-ahead sibling could immediately expire another attempt’s five-minute preemption/requeue window. Scheduler evidence now uses per-identity timestamp floors, and service persistence uses each attempt’s own reconciled time. The new regression failed before the fix and now proves the preempted attempt remains pending and can reappear in the queue. Validation: 110 focused retry/reconciliation/service/renderer/shell tests and all 1,427 Slurm tests pass; make check-slurm and git diff --check also pass. Summary reviewed: #915 (comment)

[andreatnvidia]

Thanks for addressing the feedback! Everything looks good to me.