Add browser skill-sequence authoring for atomic skills

[Yuan-Xinyi]

Description

Closes #15.

Adds interactive skill-sequence authoring to the browser visualization. Instead of hand-writing motion code, a user picks a scene entity, chooses an atomic skill, appends it as a card, compiles the sequence, replays it on a translucent preview robot, and executes it:

pick entity → choose skill → Add card → set params → Compile → Preview → Execute

The authored artifact is a plain ActionInvocation sequence, so AtomicActionEngine remains the single source of truth for skill semantics — nothing about skills is reimplemented in the UI. M1 covers move_end_effector, pick_up, place on the UR5 + gripper tutorial scene.

Design rationale, including a survey of how MoveIt/RViz, Isaac Sim, CoppeliaSim, viser/pyroki, SAPIEN, Franka Desk, UR PolyScope, MoveIt Task Constructor and RobotStudio approach this, is summarised in the branch handover notes. Three ideas were taken directly: RViz's Plan/Execute split (preview never touches physics), PolyScope's per-node state colouring, and MTC's per-stage segment replay.

Layering (bottom-up)

All new authoring code lives under embodichain/lab/visualization/authoring/:

Module	Role
protocol.py	Immutable card model with five states (unconfigured / ready / running / succeeded / failed), sequence snapshots, command objects; card params frozen recursively
session.py	Sequence editing, compilation into grounded invocations with per-card waypoint segments back-filled from the compiled trajectory, preview joint positions, execution
preview.py	Playback cursor and per-link anchored FK driving the translucent preview robot
panel.py / bridge.py	Viser sidebar panel; simulation-thread bridge turning browser commands into session calls
execution.py	StepwiseExecution, a generator advancing one simulation step per call so the browser keeps rendering during execution

Generic capabilities added to the visualization stack

Each is inert until used, so existing behaviour is unchanged:

• panels.py — a backend-neutral PanelSpec(panel_id, build, apply_state, title) contract. No authoring concept enters the Viser backend; with no panel registered the sidebar folder set is byte-for-byte unchanged (guarded by a test).
• SceneExporter — preview node groups (preview nodes reuse the source robot's geometry IDs and carry their own poses).
• SceneNode.opacity — new field defaulting to 1.0; the visibility mask is multiplied by it, which is exactly equivalent at the default (x * 1.0 is exact in IEEE-754).
• VisualizationRuntime — forwards preview updates and carries a panel-command channel alongside the existing gizmo/pick channels.

Notable implementation detail

Preview poses use per-link anchoring (anchor_i = real_pose_i @ inv(fk_current_i)) rather than anchoring on the chain root. Root-only anchoring left the preview robot offset by exactly 1 cm from ee_link downward: the URDF chain and the DexSim model disagree by a fixed offset on this robot. That is the same pre-existing discrepancy identified in #620 (URSolver reporting success with a 1 cm FK residual), tracked separately; once it is fixed the cheaper root anchoring can be restored.

Type of change

• New feature (non-breaking change which adds functionality)

Screenshots

A recording of the panel driving one sequence end to end. Every state change in it comes from a real browser input event dispatched at the page — clicks on the canvas and the controls, and typed target coordinates — not from calling the session or the bridge directly.

What it shows, in order:

1. Pick an entity. Click-to-pick is enabled and the cube is clicked in the 3-D view; the panel reports Picked entity: cube.
2. Build the sequence. A move_end_effector card is added and stays yellow unconfigured until a target is typed and applied, then turns ready. A pick_up card is added and bound to the picked cube. A place card is added with its own target.
3. Compile. The panel reports 320 waypoints and gives each card its own range — 0–80, 80–200, 200–320 — so it is visible which part of the trajectory belongs to which skill.
4. Preview. The translucent preview robot replays the trajectory next to the real one, with the end-effector path drawn as a polyline. Play/Pause, Step forward and the frame slider all work, and a marker walks down the card list to show which skill the current frame belongs to. The physical scene does not move during any of this.
5. Execute. The translucent robot hides for the duration — the plan and its execution never animate side by side on separate cursors — while the orange polyline stays, leaving the planned path visible for the real robot to follow. The real robot runs the trajectory while the status line counts waypoints and the cards turn running then succeeded one at a time; the browser keeps rendering throughout, which is what the stepwise execution driver in this PR is for. The preview returns when the run ends.

The same sequence runs headlessly via --headless_smoke, which is what CI can assert on.

Checklist

• I have run the black . command to format the code base.
• I have made corresponding changes to the documentation
• Public API changes are reflected in the API docs
• I have added tests that prove my feature works
• Dependencies have been updated, if applicable (none)

Validation

pytest tests/visualization/ -o addopts="" --run-gpu   -> 183 passed
  (the 121 pre-existing visualization tests are unmodified and still pass)
python scripts/tutorials/visualization/skill_sequencer.py --headless_smoke
  -> compiles 3 cards to 320 waypoints, segments [0,80) [80,200) [200,320)
  -> preview leaves physics untouched: max |dqpos| = max |dpose| = 0
  -> stepwise execution runs 350 host ticks without blocking the browser
  -> object moves 0.43 m; all cards end succeeded
python scripts/tutorials/visualization/skill_sequencer.py --viser
  -> server and panel come up; interactive click-through not yet signed off
black --fast --check .                 -> 952 files unchanged
python docs/scripts/check_api_docs.py  -> 2001/2001 exports documented
context.py check                       -> agent context map: ok

Preview being side-effect free is asserted directly: joint positions, joint targets and body poses are bit-identical before and after producing preview frames.

Known limitations

• The Sphinx build could not be verified locally (sphinx and pypandoc are not installed in this environment). The new page was checked mechanically: it is in the overview/sim toctree, every {doc} target resolves, and fences are balanced.
• Compilation still runs synchronously on the simulation thread, so a long grasp-sampling pass freezes the browser. Execution no longer does; compilation is the remaining half of that problem.
• Rotation for move_end_effector / place is not exposed in the panel (cards fall back to the top-down default), and pick_up targets can only be bound by clicking.
• SequencePreview.capture_inputs() assumes it is the only producer of overlays and preview groups; a second producer would need to merge the tuples.
• Simulation-backed tests carry the gpu marker and need --run-gpu with the not slow filter removed.

Unrelated but worth flagging for a separate fix: whenever allow_commands=True (which visualization/cli.py sets for any --viser run), Viser registers a scene pointer callback, and its frontend disables camera controls on pointer-down and only re-enables them on pointer-up over the canvas. Releasing the mouse outside the canvas leaves the camera permanently disabled until the page is reloaded. This predates this PR and affects every Viser session; making entity picking a togglable mode would fix it.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the latest preview-suppression changes have coherent lifecycle handling and focused regression coverage.

Summary

The latest changes hide and pause the translucent preview robot while stepwise execution controls the real robot, restore it after completion or cancellation, retain the planned path overlay, and add documentation and regression coverage for this lifecycle.

• Synchronizes preview visibility with stepwise execution state.
• Preserves the preview cursor while hidden and pauses invisible playback.
• Restores preview visibility immediately after cancellation or completion.
• Updates the tutorial recording and user-facing lifecycle documentation.

Diagram

sequenceDiagram
    participant Host
    participant Bridge as AuthoringBridge
    participant Exec as StepwiseExecution
    participant Preview as SequencePreview
    participant Runtime as VisualizationRuntime

    Host->>Bridge: update()
    Bridge->>Exec: advance()
    alt execution remains active
        Bridge->>Preview: set_suppressed(true)
        Bridge->>Preview: pause()
    else execution finishes or is cancelled
        Bridge->>Preview: set_suppressed(false)
    end
    Bridge->>Runtime: publish panel state
    Host->>Preview: capture_inputs()
    Preview-->>Runtime: preview visibility + planned path

Loading

_{Reviews (8) · Last reviewed commit: "Re-record the sequencer GIF and show it ..."}

[Yuan-Xinyi]

三条 P1 已全部修复并推送(12cf3f5),逐条回复见上方 inline。它们有个共同点:都在浏览器命令到达仿真线程的路由与生命周期上。

其中「点选被 sim.update() 抢先消费」这条特别值得记一笔——自动化测试全绿却没能拦住它,因为 --headless_smoke 是用编程方式绑定实体的,绕开了真实的 UI 路径。已补上显式断言反向顺序会丢 pick 的回归测试。

验收(本机,py3.11 + dexsim 0.5.0):

pytest tests/visualization/ -o addopts="" --run-gpu   -> 188 passed  (183 基线 + 5 新增)
skill_sequencer.py --headless_smoke                    -> PASS,物体位移 0.407 m,三卡 succeeded
black --fast --check .                                 -> 952 files unchanged
check_api_docs.py                                      -> 2001/2001
context.py check                                       -> ok

另外提一下:本 PR 目前只触发了 Greptile,lint / build / test 三个 workflow 没有起来(分支上查不到 workflow run),对比 #620 是三项都跑的。可能需要 maintainer 批准首次运行。

[Yuan-Xinyi]

已合并 main(7f8bd58),冲突解决,分支恢复 MERGEABLE。

冲突只有一处:docs/source/overview/sim/index.rst——main 把 "Documentation quality notes" 标题改成了小写,而本分支在同一区块加了 skill_sequencer 条目。两边意图都保留。

适配 main 的 API 变更(都在测试侧,产品代码无需改动):

• RigidBodyAttributesCfg 已被有意从公开 cfg 门面移除(main 里有测试专门断言这一点),仿真测试改用 create_tutorial_rigid_body_physics() 建 cube,与 main 更新原子技能教程的写法一致,并透传 sim.is_newton_backend
• exporter 的 soft-object / cloth-object 两套查询在 main 里合并成了 deformable 一套,场景导出测试替身相应改为 get_deformable_object_uid_list / get_deformable_object

本机验证到什么程度(如实说明):

pytest tests/visualization/ -o addopts="" --run-gpu   -> 186 passed, 5 errors

186 个非仿真测试全部通过。剩下 5 个依赖仿真的测试在本机无法运行,原因是合并后的 main 需要比本机所装 dexsim 0.5.0 更新的引擎,与本 PR 无关——已验证 main 自带的 tests/sim/test_sim_manager.py 在本机同样失败,报同一个错:

AttributeError: 'dexsim.cuda.pybind.WorldConfig' object has no attribute 'log_startup_info'

逐层加本地守卫绕过后还会撞上 articulation.py:1580 的下一处不兼容,因此没有继续。这 5 个测试在合并前(99e7eec)是通过的(190 passed),需要 CI 用正确版本的 dexsim 复验。--headless_smoke 同理,本机跑不了。

顺带:CI 目前仍只触发了 Greptile,lint / build / test 三个 workflow 始终没有起来。