fix: GPU device-loss/hang from captures by disabling Opacity Micromaps - #12
fix: GPU device-loss/hang from captures by disabling Opacity Micromaps#12skurtyyskirts wants to merge 1 commit into
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 34 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughStageCraft adds a configurable capture safe mode that disables Opacity Micromaps before stage realization and periodically reapplies renderer overrides across capture switches. The implementation includes optional HdRemix integration, watchdog teardown, unit coverage, and versioned changelog entries. ChangesCapture GPU stability safe mode
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant StageCraftSetup
participant HdRemixBridge
participant DxvkRemix
participant Stage
StageCraftSetup->>HdRemixBridge: Push stability configvars
HdRemixBridge->>DxvkRemix: Apply custom preset with OMM disabled
StageCraftSetup->>Stage: Call open_stage after safe mode
StageCraftSetup->>HdRemixBridge: Reassert overrides on watchdog cadence
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR adds a StageCraft-side “capture safe mode” to prevent Vulkan device loss / hangs when opening or switching game-capture projects by forcing specific HdRemix renderer configvars (notably disabling Opacity Micromaps) before stage realization, then periodically re-asserting them via a watchdog to prevent capture presets from silently re-enabling the risky settings.
Changes:
- Apply HdRemix renderer overrides (Custom preset + ReSTIR GI + OMM disabled) before
open_stageand before capture import/switch realization, and add a periodic re-assert watchdog. - Add unit tests covering pre-open ordering, arming/disarming behavior, watchdog loop behavior, and watchdog task lifecycle.
- Bump
lightspeed.trex.control.stagecraftto 1.8.1 and document the fix in changelogs and new extension settings.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| source/extensions/lightspeed.trex.control.stagecraft/lightspeed/trex/control/stagecraft/setup.py | Implements the capture safe-mode override push + watchdog re-assert logic and hooks it into project open and capture switching. |
| source/extensions/lightspeed.trex.control.stagecraft/lightspeed/trex/control/stagecraft/tests/unit/test_setup.py | Adds unit tests to ensure the override happens before realization and that watchdog behavior is correct. |
| source/extensions/lightspeed.trex.control.stagecraft/config/extension.toml | Bumps extension version, adds optional HdRemix bridge dependency, and introduces new safe-mode settings with defaults. |
| source/extensions/lightspeed.trex.control.stagecraft/docs/CHANGELOG.md | Documents the 1.8.1 fix and the new settings/behavior at the extension level. |
| CHANGELOG.md | Adds a top-level release note for the capture safe-mode mitigation. |
| async def _reassert_watch_loop(self, interval: float): | ||
| """Re-assert the overrides every ``interval`` seconds while a capture stays loaded.""" | ||
| try: | ||
| while self._safe_mode_capture_path is not None: | ||
| await asyncio.sleep(interval) | ||
| # The project may have been closed/swapped out while we slept. | ||
| if self._safe_mode_capture_path is None: | ||
| break | ||
| try: | ||
| self._push_stability_configvars() | ||
| except Exception as error: # noqa: BLE001 - a transient hiccup must not kill the loop | ||
| carb.log_warn(f"Periodic capture stability re-assert failed: {error}") | ||
| except asyncio.CancelledError: | ||
| pass |
There was a problem hiding this comment.
Fixed in 765af20. An event-driven teardown on stage CLOSING would race open_stage: opening project B over project A fires A's CLOSING after safe mode was armed for B, stripping protection exactly while B realizes. Instead the watchdog loop is now stage-aware — it skips the push while no stage is present (transition in flight) and disarms + exits when the stage root layer is anonymous (project closed). Unit tests added for both paths.
| carb.log_info( | ||
| f"Disabled Opacity Micromaps for capture project '{source_path}' (graphicsPreset=Custom, " | ||
| "integrateIndirectMode=ReSTIR GI, opacityMicromap.enable=0) before stage realization to " | ||
| "avoid a GPU device-loss/hang; re-asserting periodically." | ||
| ) | ||
| self._start_reassert_watchdog() |
There was a problem hiding this comment.
Fixed in 765af20. The arm log no longer claims periodic re-assert; that is only logged by _start_reassert_watchdog when the watchdog actually starts (interval > 0, task created).
| def _set_hdremix_configvar(key: str, value: str): | ||
| """Set an HdRemix (dxvk-remix RtxOptions User-layer) config variable through the runtime bridge.""" | ||
| from lightspeed.hydra.remix.core import hdremix_set_configvar as _hdremix_set_configvar # noqa: PLC0415 | ||
|
|
||
| _hdremix_set_configvar(key, value) | ||
|
|
There was a problem hiding this comment.
Fixed in 765af20 with a module-level optional-import sentinel (try/except ImportError) — same bridge import style as lightspeed.hdremix.renderer_settings/settings_bridge.py, kept guarded because lightspeed.hydra.remix.core is optional here (absent in headless/CLI apps). _set_hdremix_configvar raises when the bridge is unavailable, which the existing arm/re-assert handlers already catch and log.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@source/extensions/lightspeed.trex.control.stagecraft/lightspeed/trex/control/stagecraft/setup.py`:
- Around line 95-99: Move the hdremix_set_configvar import out of
_set_hdremix_configvar and replace it with a module-level optional-import
sentinel using try/except ImportError. Update _set_hdremix_configvar to use that
sentinel while preserving behavior when the optional module is unavailable, and
remove the function-local import.
- Around line 557-565: The _apply_capture_safe_mode_on_switch method must disarm
an already-armed safe mode when _SETTING_AUTO_SAFE_MODE is disabled, rather than
returning with the watchdog and _safe_mode_capture_path still active. Mirror the
project-open teardown behavior before the early return, and add coverage for
switching captures with autoDisableOpacityMicromaps off after safe mode was
already armed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro
Run ID: 0e8a0669-9bef-449c-ac00-60c1bc7829c2
📒 Files selected for processing (5)
CHANGELOG.mdsource/extensions/lightspeed.trex.control.stagecraft/config/extension.tomlsource/extensions/lightspeed.trex.control.stagecraft/docs/CHANGELOG.mdsource/extensions/lightspeed.trex.control.stagecraft/lightspeed/trex/control/stagecraft/setup.pysource/extensions/lightspeed.trex.control.stagecraft/lightspeed/trex/control/stagecraft/tests/unit/test_setup.py
765af20 to
657bc21
Compare
Opening a capture project (or switching captures) in StageCraft could fault the Vulkan device (VK_ERROR_DEVICE_LOST) or hang the main thread seconds after open_stage. A game capture is thousands of small alpha-tested meshes, and building Opacity Micromaps (OMM) for all of them during stage realization can overrun the path tracer's GPU working set -- there is no per-capture GPU budget ceiling to fall back on, so the device faults instead of degrading. Prim count does not reliably predict which captures do this (a 599 KB capture hung while larger ones did not), and OMM is only a render-time optimization for alpha-tested geometry, so disabling it never changes visuals and never affects editing or asset replacement. lightspeed.trex.control.stagecraft now disables OMM (graphicsPreset=Custom, integrateIndirectMode=ReSTIR GI, opacityMicromap.enable=0, pushed through the HdRemix bridge) for any capture project before open_stage realizes the stage, and re-asserts it every opacityMicromapReassertIntervalSeconds (default 5s) via a watchdog -- dxvk-remix re-applies the capture's own graphics preset a few seconds after realization (re-enabling OMM) without touching the carb /rtx/* nodes, so a one-shot override silently loses. The same override is applied before a switched-to capture is realized. Gated by autoDisableOpacityMicromaps (default on). The watchdog is stage-aware (skips pushes during stage transitions, disarms itself when the project is closed) and is torn down on destroy and whenever the setting is off. Adds the optional lightspeed.hydra.remix.core dependency for the bridge and bumps the extension to 1.8.1. Verified on an RTX 5090: a Tomb Raider: Legend capture project that previously device-lost the GPU now logs the pre-open override plus the watchdog start and opens cleanly with no Aftermath dump.
657bc21 to
2c2d21c
Compare
Problem
Opening a capture project (or switching captures) in StageCraft can fault the Vulkan device (
VK_ERROR_DEVICE_LOST) or hang the main thread seconds afteropen_stage, dropping an Aftermath dump. A game capture is thousands of small alpha-tested meshes; building Opacity Micromaps (OMM) for all of them during stage realization can overrun the path tracer's GPU working set, and with no per-capture GPU budget ceiling to fall back on, the device faults instead of degrading.Prim count does not reliably predict which captures do this — in testing, a 599 KB capture hung while much larger ones did not — so a "heavy capture" heuristic is not a safe gate.
Fix
lightspeed.trex.control.stagecraftnow disables OMM for any capture project beforeopen_stagerealizes the stage, by pushingrtx.graphicsPreset=Custom,rtx.integrateIndirectMode=ReSTIR GI, andrtx.opacityMicromap.enable=0through the HdRemix bridge (lightspeed.hydra.remix.core, added as an optional dependency). The same override is applied before a switched-to capture is realized.A watchdog re-asserts the override every
opacityMicromapReassertIntervalSeconds(default 5s): dxvk-remix re-applies the capture's own graphics preset a few seconds after realization — re-enabling OMM without touching the carb/rtx/*nodes, so no settings callback fires — which means a one-shot override silently loses and the GPU faults ~1 minute later anyway.hdremix_set_configvarwrites only to the dxvk-remix RtxOptions User layer, so the periodic re-push cannot trigger a render-settings reload and is a runtime no-op when the value is already current. The watchdog is stage-aware: it skips pushes during stage transitions and disarms itself when the project is closed (stage-close events can't drive teardown — they fire mid-open_stagewhen loading one project over another, which would strip protection exactly while the new capture realizes).OMM is a render-time optimization for alpha-tested geometry only, so disabling it never changes visuals and never affects editing or asset replacement — it trades a little alpha-tested ray-tracing performance for stability. The behavior is gated by a new
autoDisableOpacityMicromapssetting (default on); the watchdog is torn down ondestroyand whenever the setting is off. The extension is bumped to 1.8.1 with changelog entries.Verification
Disabled Opacity Micromaps for capture project ... before stage realization) plus the watchdog start, dxvk-remix confirms[RTX] Opacity Micromap: disabled/Integrate Indirect Mode: ReSTIR GI - activated, and the project opens cleanly with no device loss and no Aftermath dump (including after sitting past the point where the capture's preset used to re-enable OMM).tests/unit/test_setup.py, including a regression test pinning that the override runs beforeopen_stage(an earlier attempt hooked the stage-OPENED event, which is after realization and too late), plus coverage for the toggle-off teardown paths and the stage-aware watchdog.format_code.bat --verify,lint_code.bat all,check_forbidden_words,check_test_file_location, andcheck_changelogall pass locally.Note for reviewers
This is deliberately a toolkit-side mitigation. The root-cause alternative would be a per-capture OMM-build VRAM budget/throttle in dxvk-remix so OMM could stay enabled — happy to rework in that direction if you'd prefer the fix live at the renderer level.