fix(slurm): restore plugin activation boundary

[nabinchha]

📋 Summary

Restore the client plugin activation boundary for Slurm generation. The runtime wrapper now starts the plugin-aware client worker in a fresh Python interpreter, so the verified dependency overlay is activated before plugin and Data Designer configuration modules are imported.

This is needed because the runtime wrapper has already imported Data Designer configuration modules before it reaches the client phase. Activating a dynamic plugin overlay later in that same interpreter cannot retroactively rebuild the already-imported configuration type registry. As a result, a valid plugin-provided column can pass scheduling and endpoint readiness but still be rejected when generation starts. A fresh worker interpreter makes overlay activation happen before the plugin and configuration imports that depend on it.

Post-#929 one-node acceptance established that:

• The base path completes successfully.
• The plugin path reaches full endpoint readiness, then fails during configuration validation without this change.
• An allocation-local diagnostic reproduced premature configuration import as the cause.

Post-restack real one-node acceptance completed successfully: the plugin was installed and discovered, the model backend became ready, the fresh client worker generated all four requested records with the plugin-provided marker, and the winner was published automatically. The later review-response commit changes tests only, so the accepted runtime source tree is unchanged at the current head.

🔗 Related Issue

Post-#929 plugin activation regression; no separate issue.

🔄 Changes

• Add a bootstrap-safe ClientWorkerProcess component that owns the fresh-interpreter contract without importing configuration, interface, plugin, or worker modules.
• Compose that public component into the runtime client phase instead of hiding subprocess behavior in the runtime entrypoint.
• Refresh Python's public import caches before inventorying an attempt-local overlay, covering filesystems that preserve a directory timestamp across installation.
• Keep workspace leasing, candidate validation, and result publication in the runtime wrapper.
• Inject the component directly where narrow runtime tests need a controlled process result.
• Replace the synthetic plugin probe with an end-to-end regression that installs a locked fake-plugin wheel, runs preflight and _client through the default process component, verifies the generated plugin column, and verifies result/candidate publication.

🧪 Testing

• make test passes (not run; Slurm-only change)
• Unit tests added/updated
• E2E tests added/updated (not applicable)
• Default client-process plugin handoff regression
• make test-slurm: 1,430 passed after restacking on the latest shared base
• make check-slurm
• make test-slurm-wheel-install
• Pre-restack real one-node plugin acceptance: 4/4 records generated and plugin marker verified
• Post-restack real one-node plugin acceptance on the unchanged runtime source tree: 4/4 records generated and plugin marker verified
• git diff --check

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; existing activation boundary is preserved)

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness or repository-rule violations identified.

Summary

• Introduces a bootstrap-safe ClientWorkerProcess abstraction with a fixed module invocation.
• Delegates the runtime client phase to the fresh worker while retaining lifecycle and publication responsibilities in the runtime wrapper.
• Invalidates import caches before inspecting an attempt-local dependency overlay.
• Adds realistic coverage for installing, discovering, validating, and executing a plugin-provided column.

Diagram

sequenceDiagram
    participant Runtime as Slurm runtime wrapper
    participant Process as ClientWorkerProcess
    participant Child as Fresh Python interpreter
    participant Env as Dependency overlay
    participant Plugin as Plugin/config registry
    participant Engine as Generation engine

    Runtime->>Process: run(worker arguments)
    Process->>Child: python -m data_designer.slurm.client.worker
    Child->>Env: prepare and activate verified overlay
    Env-->>Child: overlay available on import path
    Child->>Plugin: import and discover plugin/config types
    Plugin-->>Child: plugin column registered
    Child->>Engine: validate configuration and generate records
    Engine-->>Runtime: candidate manifests and dataset
    Runtime->>Runtime: validate and publish candidate

Loading

_{Reviews (6) · Last reviewed commit: "test(slurm): exercise plugin worker hand..."}

[nabinchha]

Exact-head real one-node/one-GPU acceptance completed successfully.

• Plugin dependency installed and was discovered during preflight.
• The model backend reached readiness.
• Generation ran through the fresh client-worker process.
• All 4 requested records were produced with the plugin-provided marker verified.
• The winning output was published automatically.

No internal environment identifiers are included here; detailed evidence is retained privately.

[nabinchha]

Post-restack exact-head real one-node/one-GPU acceptance also completed successfully. Plugin preflight, backend readiness, fresh-worker generation, 4/4 records with the plugin marker, and automatic winner publication all passed. Detailed environment evidence remains private.

[andreatnvidia]

The updated regression now exercises the full plugin handoff through the default worker process, including installation, generation, and result publication. The activation boundary looks correct, and the original concern is covered. Looks good to me.