Skip to content

Add batched visualization markers and asset attachment - #654

Draft
yuecideng wants to merge 4 commits into
mainfrom
codex/visualization-markers
Draft

yuecideng wants to merge 4 commits into
mainfrom
codex/visualization-markers

Conversation

@yuecideng

@yuecideng yuecideng commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Description

Add render-only mesh markers shared by native DexSim and Viser, with RGBA transparency and reusable box, sphere, cylinder, capsule, cone, arrow, frame, and custom-mesh prototypes.

Marker groups default to all SimulationManager.num_envs; update, visibility, and clear accept selected env_ids. scope="world" provides a single global batch. There is no new public arena_index. Groups can attach to registered object/robot/articulation roots or links with local offsets, follow host simulation updates, and detach while preserving world poses. Existing legacy drawing APIs remain available.

Automatic attachment refresh preserves browser capture cadence and isolates visual failures from simulation accounting. Manual mutations validate and publish synchronously with rollback. Native markers are excluded from camera outputs by default; explicit overlay inclusion retains the engine's documented Hybrid NRD limitation.

Docs CI also fetches the architecture snapshot’s pinned evidence commit when it is outside current branch history after a squash merge. Exact-source assertions and generated data remain unchanged.

Refs #129.

Dependencies: native rendering uses DexSim's generic spawn.create_render_actor, RenderDesc/MaterialDesc, and RenderBody routing/material capabilities (engine commit 3224e21f0) in MR !1423. The specialized DebugMesh factory and object subtype have been removed from that engine change. Keep this PR in draft until the engine release/integration is qualified; no unpublished dependency version is pinned.

Type of change

  • New feature
  • Documentation update

Validation

Current consumer commit: b5ecae69.

  • Black 26.3.1 (black .) and git diff --check: passed; 1069 Python files unchanged by formatting.
  • Visualization/manager/context checks: 329 passed, 3 failed. The three failures are the existing removed-com_quaternion descriptor mismatch below; the changed marker consumer suite passes.
  • Private generic native integration: 2 passed, 1 explicit COM dependency skip. DexSim's three-renderer overlay/image/material lifetime and independent-segment tests: 9 passed in isolated renderer processes on engine 3224e21f0; six additional native failure/isolation regressions passed. Formal CI is tracked in its MR.
  • Public API documentation: 2089/2089 exports covered. Updated simulation/visualization context and API guidance; independent consumer review found no actionable defects.
  • CPU benchmark at 64 environments × 128 boxes: full state update 1.217 ms, selected 16-environment update 0.317 ms, detached snapshot 203.634 ms (five repetitions). These measure unchanged CPU state/snapshot paths, not renderer throughput or native instancing; no speedup claim.
  • Current-head remote CI at b5ecae69: lint and documentation integration tests passed; broad CPU reported 4639 passed, 92 failed, 1 skipped (176.98 s). The exact 92 failed test IDs match the preceding revision: 88 constructor com_quaternion errors, 2 dynamic com_quaternion errors, 1 inertia-shape error and 1 interrupted-warning cascade. Documentation build also passed (502 s); the current-head run is terminal failure because of those CPU failures, and later test lanes did not run. This PR remains draft and is not CI-green.

Remaining integration work

  • The existing EmbodiChain Spawn adapter still passes com_quaternion to a DexSim descriptor which now expects body-frame inertia. This pre-existing incompatibility causes the three excluded manager cases and blocks the real registered-articulation attachment smoke before attachment starts. Root/link following, replacement, safe removal, and no implicit preparation are covered at manager level. Migrate COM/inertia together and qualify real assets before removing draft status; do not silently discard the orientation.
  • True native batch/GPU instancing remains separate engine work. The current native adapter publishes individual mesh handles; Python batch semantics do not imply one native bulk call.
  • Arbitrary USD marker import, text, and new streaming point-cloud machinery remain follow-up scope.

Screenshots

No screenshots attached; behavior is covered by focused CPU/native tests and the runnable example.

Checklist

  • I have run the black . command to format the code base.
  • I reviewed affected documentation and agent context and updated the simulation/visualization guidance and API inventory.
  • Public API changes are reflected in the API docs (python docs/scripts/check_api_docs.py).
  • I have added tests that prove the feature works.
  • Dependencies have been updated, if applicable. Pending an engine release containing the required capability.

@yuecideng yuecideng added enhancement New feature or request rendering Things related to rendering (eg, performace, efficiency, bug) visualization Features or functions for visualization with the simulation dexsim Things related to dexsim labels Sep 18, 2026
@yuecideng
yuecideng requested a review from acrlw September 19, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dexsim Things related to dexsim enhancement New feature or request rendering Things related to rendering (eg, performace, efficiency, bug) visualization Features or functions for visualization with the simulation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant