Skip to content

Fix macOS LLVM loading and cross-platform QIS recovery with CI coverage - #783

Open
qciaran wants to merge 8 commits into
devfrom
codex/macos-build-regression
Open

qciaran wants to merge 8 commits into
devfrom
codex/macos-build-regression

Conversation

@qciaran

@qciaran qciaran commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Fix macOS dynamic-library resolution failures and a Windows QIS recovery ABI mismatch uncovered by cross-platform verification.

  • PECOS's DYLD_LIBRARY_PATH export made rustc load Homebrew LLVM 21 instead of its bundled LLVM 22. Use DYLD_FALLBACK_LIBRARY_PATH and bypass cached compiler-probe failures in the sanitized macOS Python build. Remove the Rust 1.97.1 workaround after verifying stable builds.
  • Generated QIS programs bound panic(i32, ptr) to macOS libSystem's unrelated panic function, causing process exit/abort. Explicitly link the selected PECOS QIS FFI dylib on macOS so Mach-O's two-level namespace selects the correct runtime. Include the link-policy revision in compiled-program cache keys so Python extension upgrades cannot reuse old incorrectly linked programs.
  • Windows longjmp performs stack unwinding, but transfer-capable Rust FFI entry points and the recovery callback were declared extern "C". Use C-unwind on those boundaries to permit recovery without Rust's abort-on-unwind barriers. Keep releasing owned values and locks before transfer because Unix longjmp does not run destructors.
  • Replace the warning-only compiler canary with a blocking stable Cargo probe using the real PECOS build environment. Test installed LLVM wheel imports and bitcode generation on macOS, and run the QIS executor tests on every applicable PR. Raise that job's timeout to 30 minutes to accommodate the additional compilation.

Evidence and validation

  • Compiler regression-only commit 00b51575 failed as expected: https://github.com/PECOS-packages/PECOS/actions/runs/35016292928/job/104540441757 (rustc SIGSEGV in LLVM initialization).

  • After the LLVM fix, final Python CI at 10fe72da passed, including stable macOS builds and installed-wheel LLVM checks on Python 3.12 and 3.14: https://github.com/PECOS-packages/PECOS/actions/runs/35019112455.

  • The full Rust run exposed the additional QIS abort: https://github.com/PECOS-packages/PECOS/actions/runs/35021706111/job/104560321159. Reproduced the termination-test failure on unchanged local dev without DYLD variables. dyld_info -imports showed _panic (from libSystem); after the fix it shows _panic (from libpecos_qis_ffi).

  • New regression test checks the generated program's actual panic function address against the selected FFI library. Verified it fails with the old link policy and passes with the fix.

  • All 57 QIS unit tests pass locally, including exit/panic recovery, heap cleanup, and same-named program isolation. QIS Clippy with warnings denied, formatting, and whitespace checks pass.

  • Earlier LLVM validation: complete local Python build with no manual overrides, 16 LLVM IR/binding tests, module-version checks, and CLI Clippy all pass.

  • Final regular PR CI at 714a1a38 is green, including the new macOS QIS executor regression step and LLVM wheel runtime checks: https://github.com/PECOS-packages/PECOS/actions/runs/35026623706.

  • Full Rust verification at 714a1a38 passed on macOS and Linux: https://github.com/PECOS-packages/PECOS/actions/runs/35026678187. Windows failed in wrappers_free_program_heap_on_return_exit_panic_and_invalid_input: invalid heap_alloc transfers via the C shim's longjmp, Windows invokes RtlUnwindEx, and Rust aborts with panic in a function that cannot unwind at the extern "C" heap_alloc boundary.

  • Windows follow-up 6904c224: a new host-independent ABI regression aborts with the old heap_alloc ABI and passes with C-unwind. All 57 QIS and 129 FFI unit tests pass locally; Clippy with warnings denied, formatting, and whitespace checks pass. Regular macOS and Windows PR jobs now run both FFI and executor regressions, including actual C longjmp recovery, heap cleanup, and subsequent execution. Fresh PR CI and a full cross-platform Rust run are pending; keep this PR in draft until actual Windows verification passes.

  • Windows CI at 6904c224 now passes all 129 FFI tests and 12 executor tests, including real longjmp recovery, program heap cleanup, and successful execution after termination: https://github.com/PECOS-packages/PECOS/actions/runs/35034496393/job/104600374131. The corresponding macOS regression job also passed. Full cross-platform Rust verification remains pending: https://github.com/PECOS-packages/PECOS/actions/runs/35034531567.

PR-visible full-suite opt-in

Commit a7502509 adds the ci:full-rust label: adding it launches full Linux/macOS/Windows Rust validation as normal PR checks, and subsequent pushes rerun it while the label remains. Removing it restores the lightweight PR path. Unrelated label events do not cancel validation; post-merge and manual full runs remain unchanged. Usage is documented in .github/CI.md. Actionlint passes; zizmor reports only the two existing unpinned setup-just tool warnings.

The label is applied to this PR. GitHub confirmed a pull_request run on a7502509, titled Rust validation (full): https://github.com/PECOS-packages/PECOS/actions/runs/35042473499. This latest full run and regular PR checks are pending. The prior merged head e93ab326 had green regular checks and a code review with no actionable findings, but those results do not replace validation of this latest workflow change.

Latest verification on 1e132847

The label-triggered full run is visible in PR checks and passed Linux and macOS: https://github.com/PECOS-packages/PECOS/actions/runs/35043360816. Windows now fails in pecos-simulators test fast_corpus_matches_oracle_and_reference_bits: crossover_8t_256_terms amplitude[0].re is 3fa0000000000004 versus reference 3fa0000000000009 (five ULPs). This PR has no changes to crates/pecos-simulators relative to dev; the reference test originated in #716 and was later touched by #717. This is a separate numerical-reference issue, not the prior QIS unwind abort. Do not treat the latest full run as green; diagnosis/fix scope needs a separate decision rather than silently relaxing exact-reference assertions.

Supersedes #640. Related: #566.

@qciaran qciaran changed the title ci: catch stable Rust crashes in the macOS build environment Fix macOS LLVM library overrides and enforce stable Rust CI coverage Sep 15, 2026
@qciaran qciaran changed the title Fix macOS LLVM library overrides and enforce stable Rust CI coverage Fix macOS LLVM loading and QIS runtime symbol collisions with CI coverage Sep 15, 2026
@qciaran qciaran changed the title Fix macOS LLVM loading and QIS runtime symbol collisions with CI coverage Fix macOS LLVM loading and cross-platform QIS recovery with CI coverage Sep 15, 2026
@ciaranra
ciaranra marked this pull request as ready for review September 16, 2026 00:56
@ciaranra
ciaranra self-requested a review as a code owner September 16, 2026 00:56
@qciaran qciaran added the ci:full-rust Run full Rust tests on Linux, macOS, and Windows before merge label Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:full-rust Run full Rust tests on Linux, macOS, and Windows before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants