feat: reconcile persisted Slurm state

[nabinchha]

📋 Summary

Reconcile a persisted Slurm run from normalized active-queue and accounting observations in a fresh process. The status composition preserves bounded accounting lag and immutable terminal evidence, validates readiness and generation records, and refuses to report a winner when scheduler evidence contradicts it.

🔗 Related Issue

Part of #869 (869#3).

🔀 Merge Order

This PR is the direct observation/reconciliation follow-up on feat/slurm-execution, restacked onto ff766b14 after #929 merged. The restack drops the resolved-plan provenance, one-node dataset lease/finalization, and launcher-submission work now owned by #929. Child PR #915 remains based on this branch and must stay open.

🔄 Changes

• Add a public scheduler-observation collector over the normalized squeue/sacct client contract, including array-task and ordinary collection-job identities.
• Persist canonical per-attempt scheduler snapshots with restrictive permissions and atomic replacement.
• Add fresh-process run, shard, attempt, readiness, generation, and winner status composition through SlurmStateReconciler.
• Keep accounting lag nonterminal with a fixed reconciliation deadline and make reviewed terminal evidence immutable.
• Keep active-queue PREEMPTED and REQUEUED observations transitionable. Queue-absent accounting PREEMPTED evidence uses a fixed persisted requeue-visibility deadline, allowing a legitimate requeue to reappear before unrequeued work becomes terminal failure.
• Route public status() and cancel() lifecycle projection through the same persisted reconciler so unrequeued preemption cannot remain active after that fixed deadline.
• Query active-queue and accounting sources independently, treating only exact Slurm unknown-job queue failures as an absent row.
• Require a validated winner before a persisted candidate can make its shard and run succeed.
• Reject concurrent persisted-state changes and winner/scheduler contradictions instead of guessing status.
• Keep the reconciliation workflow composed across observation, storage, reading, derivation, and orchestration modules; no sibling imports of underscore-prefixed implementation names.
• Add package-install coverage for the observation and status APIs.

🧪 Testing

• Full Slurm suite after restack and public-status fix: 1,360 passed
• Focused observer, state-validation, launcher-client, and public-service suites: 93 passed
• make check-slurm — 176 files formatted and linted
• make test-slurm-wheel-install — CLI overhead 0.006s
• git diff --check

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; no architecture contract changed)

---

Description updated with AI

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable new defects remain, and all previous findings were manually resolved.

Summary

• Normalizes queue and accounting observations while preserving terminal evidence and bounded lag/preemption windows.
• Reconstructs run, shard, attempt, readiness, generation, and winner status in fresh processes.
• Adds immutable retry plans, ambiguous-submission recovery, and retry-specific resume-mode binding.
• Adds scheduler-backed collection jobs that verify winner inputs and atomically publish merged outputs.
• Routes public status and cancellation through the persisted reconciler.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    API[Public Slurm service] --> Reconciler[SlurmStateReconciler]
    Reconciler --> Queue[squeue observations]
    Reconciler --> Accounting[sacct observations]
    Queue --> Snapshot[Persisted scheduler snapshots]
    Accounting --> Snapshot
    Snapshot --> Status[Run / shard / attempt status]
    Status --> Retry[Retry coordinator]
    Retry --> RetryPlan[Immutable retry plan]
    RetryPlan --> Allocation[Retry array allocation]
    Allocation --> Candidate[Validated candidate output]
    Candidate --> Winner[Persisted shard winner]
    Winner --> Collection[Collection coordinator]
    Collection --> Merge[Verified deterministic merge]
    Merge --> Output[Atomically published collection]

Loading

_{Reviews (17) · Last reviewed commit: "feat: add Slurm retry and deterministic ..."}

[andreatnvidia]

One more cross-layer gap: SlurmStateWriter.acquire_dataset_workspace() exists, but the runtime path does not hold it while Data Designer is writing. If an UNKNOWN attempt is still alive but temporarily missing from scheduler views, a retry can start in the same resumable workspace. Wrapping the full client generation attempt in that lease would keep the original and retry writers from overlapping.

[andreatnvidia]

I found a provenance contract mismatch in the current base. The client produces a digest from the full per-attempt payload, but the finalizer accepts only ResolvedSlurmRunPlan.compute_sha256(). Normal worker output is therefore rejected before winner.json can be published, which also makes the collection path unreachable. Can we align the producer, finalizer, and collector around one contract and cover the complete worker-to-collection path?

[nabinchha]

Addressed the workspace-lease feedback from #913 (comment) in eec0b47. The allocation controller now acquires the existing shard dataset-workspace lease before client generation starts and holds it through process completion, result validation, and durable attempt-result binding. The regression asserts that both generation and publication occur while the lease is active and that the lease is released afterward. Validation: runtime controller 17 passed; full Slurm suite 1,274 passed; make check-slurm and git diff --check passed.

[nabinchha]

Addressed the provenance feedback from #913 (comment) in eec0b47. ClientWorker now emits the stable resolved-plan SHA-256 required by winner finalization instead of an attempt-specific digest. Added a real ClientWorker → SlurmStateWriter publication → winner-finalization regression, plus the direct producer assertion. Validation: client worker 18 passed; full Slurm suite 1,274 passed; make check-slurm and git diff --check passed. The downstream worker-to-collection continuation will be covered in child PR #915, where collection is owned.

[nabinchha]

Addressed the unrequeued-PREEMPTED liveness finding from #913 (comment) in final commit b1924e07. The stable invariant is that the fixed five-minute deadline measures continuous absence from squeue, not total PREEMPTED age. Queue-present PREEMPTED remains nonterminal with no absence deadline. Queue-absent accounting PREEMPTED starts or preserves one fixed deadline, and only continued absence beyond it becomes terminal failure so the attempt, shard, and run become retryable. If the job reappears in the queue, the absence deadline clears and PREEMPTED/PENDING/RUNNING can advance normally. Regressions cover active PREEMPTED beyond five minutes, accounting-only expiry, temporary loss of accounting evidence, and accounting-PREEMPTED → queue-PREEMPTED → PENDING recovery. Validation: 68 focused observation/model/transition tests; full Slurm suite 1,277 passed; make check-slurm and git diff --check passed.

[andreatnvidia]

Nice follow-through on the preemption edge cases. The latest change brings the public status and cancel paths onto the same bounded reconciliation flow. This looks good to merge.