Benchmark seven more atomic skills

[Yuan-Xinyi]

Description

Third of three stacked changes. Base is #641, which is itself based on #620; review those first.

Extends the suite from five skills to twelve. Each new benchmark reports the full success ladder from #641 and a task metric defined by what the skill is supposed to achieve, not by whether a planner returned:

Skill	task_success is measured as	Result
open_door	hinge reaches the commanded angle	✅ 0.0001 rad error
press	button reaches its commanded depression	✅ 6.00 mm peak
slide	drawer reaches the commanded travel	✅ 0.1794 m, 0.0006 m error
pour	container reaches the commanded tilt	✅ 0.781 rad peak
axis_align	residual angle to the target axis	✅ 1.571 rad → 0.01 rad
hand_over	delivery distance, no drop	✅ 0.057 m
twist	knob reaches the commanded rotation	❌ reported as failing

Two skills did not run at all, and neither failure was in the benchmarks

Press and Slide plan Cartesian-linear segments. #640 now requires those to use strategy='ik_interp', but their tutorials still request 'motion_gen', so the guard rejects them. This is not benchmark-only: the press tutorial fails on main today with exactly the error the benchmark hit —

ValueError: cartesian_linear skill segments require strategy='ik_interp';
motion_gen backends do not guarantee exact Cartesian samples.

Both tutorials and both benchmarks now request the strategy the primitives require. Verified by running the press tutorial itself, which completes again.

OpenDoor: the replay rate, never the task tolerance

At the shared default the arm trails the plan by 0.66 rad and the tracking gate rejects the run, even though the door reaches its commanded angle. Raising only the replay rate — four times the shared default for this skill — brings tracking inside the gate and the hinge to within 0.001 rad. The task tolerance is unchanged.

Twist is a real failure and is reported as one

The planned end-effector rotation about the knob axis is exactly the commanded 0.7854 rad, with correct sign, at 0.026 rad of arm tracking error — so the skill plans correctly. The gripper wedges against the knob rather than holding it, and the knob joint has near-zero stiffness, so the knob over-rotates and keeps being dragged during retract. Neither the success gate nor the scene stiffness was adjusted to turn this green; BENCHMARK_REPORT.md proposes three ways to gate it and recommends one.

Type of change

• New feature (non-breaking change which adds functionality)
• Bug fix (the press / slide strategy regression)

Screenshots

N/A — reports are Markdown tables written to outputs/benchmarks/.

Checklist

• I have run the black . command to format the code base.
• I have made corresponding changes to the documentation
• Public API changes are reflected in the API docs (none; benchmark scripts)
• I have added tests that prove my fix works (the benchmarks are the executable artifact; every skill below was run)
• Dependencies have been updated, if applicable (none)

Validation

Every new benchmark run on this branch, current engine:

open_door   plan=True valid=True exec=True task=True   hinge_err=0.0001
press       plan=True valid=True exec=True task=True   peak=6.00 mm
slide       plan=True valid=True exec=True task=True   travel=0.1794 m, err=0.0006
pour        plan=True valid=True exec=True task=True   peak=0.7811 rad
axis_align  plan=True valid=True exec=True task=True   1.5709 -> 0.01 rad
hand_over   plan=True valid=True exec=True task=True   dist=0.0568 m, no drop
twist       plan=True valid=True exec=True task=False  rot=2.195 vs 0.785 commanded
scripts/tutorials/atomic_action/press.py --headless --auto_play -> completes

BENCHMARK_REPORT.md records the measured ladder, timings and task values per skill, the bugs found and fixed, and the threshold questions that need a decision rather than a default — including one that matters: press's 80 %-of-stroke rule passes regardless of commanded depth on this asset, so it currently proves only that the button was pressed.

Not built: push_object has no tutorial to source a scene from, and the two coordinated dual-arm skills are left for a follow-up.

[greptile-apps]

Confidence Score: 1/5

Fix in-process dispatch and the Pour/HandOver false-positive task gates before merging, and satisfy the public-API annotation requirement.

Findings

1.  In-process actions lack required arguments ▶
2.  Off-axis tumbles count as pours ▶
3.  Reversed pours pass the gate ▶
4.  Tabletop drops escape detection ▶
5.  Videos replay a different execution ▶
6.  Failed recordings consume video quota ▶
7.  Recording changes the height diagnostic ▶
8.  Exported helpers lack parameter annotations ▶
9.  Report describes outdated replay settings ▶

Fix with agent prompt

### Issue 1
scripts/benchmark/atomic_action/run_benchmark.py:46-52
The new actions cannot run through `--in_process`: `_make_child_args()` omits their case-selection fields, but each new `run_all_benchmarks()` accesses its field directly before applying the smoke profile. For example, `--action open_door --in_process --profile smoke` raises `AttributeError` on `args.door_cases` instead of producing a report. Populate the new action-specific defaults when constructing the child namespace, preferably using each module's argument parser.

### Issue 2
scripts/benchmark/atomic_action/pour_benchmark.py:146-150
This helper returns the total relative rotation angle; the requested axis only determines its sign. A 45-degree tumble perpendicular to the pour axis therefore returns positive 45 degrees because the zero dot product is forced to a positive sign. The 45-degree benchmark can consequently credit an off-axis slip as a successful pour. Measure rotation about the requested axis and add a perpendicular-axis counterexample test.

### Issue 3
scripts/benchmark/atomic_action/pour_benchmark.py:294-296
Taking the absolute value of both rotations removes the direction check from the signed task metric. If a slipping object rotates −45 degrees around the requested axis for the +45-degree case, this reports zero error and `task_success=True`. The measurement and primitive use the same axis convention, so compare the signed achieved rotation with the signed command rather than their magnitudes.

```suggestion
        rotation_error = abs(
            trace.peak_signed_displacement - case.rotate_angle_rad
        )
```

### Issue 4
scripts/benchmark/atomic_action/hand_over_benchmark.py:238-247
The drop threshold is below the support table, so it cannot detect an object dropped onto that table during transfer. The target lies inside the table footprint, only 0.10 m above its surface. An object resting near the target at z≈0.53 m passes both the 0.12 m delivery tolerance and the z≥0.35 m check despite losing its grasp prematurely. Track grasp retention or an unexpected return to the support surface during transfer, while allowing the primitive's intentional final release.

### Issue 5
scripts/benchmark/atomic_action/open_door_benchmark.py:286-291
The recorder uses four physics steps per waypoint, whereas OpenDoor is scored at 64 and the other new benchmarks at 16. For these contact tasks, that changes tracking and achieved motion-not just playback speed-so a video attached to a successful case can depict a different outcome. Record during the measured replay or give the recording path the same replay cadence and terminal hold settings.

### Issue 6
scripts/benchmark/atomic_action/slide_benchmark.py:318-319
A failed or unavailable recording is appended to `video_paths` as an empty string. With `--record_video --record_failed_video` and the default case limit, a planning failure with no trajectory can consume the quota, suppress a later recordable case, and inflate the reported video count. Append only nonempty recording paths, as the other new benchmarks do. Twist has the same issue.

### Issue 7
scripts/benchmark/atomic_action/pour_benchmark.py:329-330
`final_object_z_m` is read after the optional reset and recording replay, while the rotation trace and starting height describe the scored run. Enabling recording can therefore replace this diagnostic with the height from a different execution-or the reset pose if recording cannot start. Capture the final height immediately after physical validation, before resetting the scene for video.

### Issue 8
scripts/benchmark/atomic_action/axis_align_benchmark.py:115
The exported `object_axis_angle_rad()` leaves all three parameters unannotated, violating the repository directive to fully annotate public APIs. Add `RigidObject` and tensor parameter annotations, using type-only imports where appropriate. The exported `signed_rotation_about_axis_rad()` in Pour has the same omission. This repository requirement must be satisfied before merging.

### Issue 9
scripts/benchmark/atomic_action/BENCHMARK_REPORT.md:73-86
The report presents 16 steps per waypoint as OpenDoor's chosen replay rate, but the committed benchmark uses 64 because 16 fails its tracking gate. Together with the opening claim that the changes are uncommitted, this leaves readers unable to tie the reported results to the reviewed implementation. Identify the measured revision and settings, and separate historical experiments from the current benchmark configuration.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds seven atomic-skill benchmarks, extending the registered suite to twelve, with planning, motion-validity, execution, and physical task-success reporting. Press and Slide tutorials now select the required Cartesian interpolation strategy.

• Uses segment-specific measurements for articulated tasks and peak rotation for Pour.
• Retains Twist's physical task failure rather than loosening its gate.
• Requires fixes for in-process dispatch and false-positive Pour/HandOver measurements before merging.
• Recording consistency, report provenance, and public-helper annotations also need attention.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Select benchmark case] --> B[Compile skill sequence]
  B --> C{Planning succeeded?}
  C -- Yes --> D{Motion valid?}
  D -- Yes --> E[Replay and sample physical state]
  E --> F{Tracking within tolerance?}
  F -- Yes --> G[Evaluate skill-specific task metric]
  C -- No --> H[Record failure stage]
  D -- No --> H
  F -- No --> H
  G --> I[Markdown report]
  H --> I

Loading

_{Reviews (1) · Last reviewed commit: "feat(bench): benchmark seven more atomic..."}

[greptile-apps]

In-process actions lack required arguments

The new actions cannot run through --in_process: _make_child_args() omits their case-selection fields, but each new run_all_benchmarks() accesses its field directly before applying the smoke profile. For example, --action open_door --in_process --profile smoke raises AttributeError on args.door_cases instead of producing a report. Populate the new action-specific defaults when constructing the child namespace, preferably using each module's argument parser.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/run_benchmark.py
Line: 46-52

Comment:
**In-process actions lack required arguments**

The new actions cannot run through `--in_process`: `_make_child_args()` omits their case-selection fields, but each new `run_all_benchmarks()` accesses its field directly before applying the smoke profile. For example, `--action open_door --in_process --profile smoke` raises `AttributeError` on `args.door_cases` instead of producing a report. Populate the new action-specific defaults when constructing the child namespace, preferably using each module's argument parser.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Off-axis tumbles count as pours

This helper returns the total relative rotation angle; the requested axis only determines its sign. A 45-degree tumble perpendicular to the pour axis therefore returns positive 45 degrees because the zero dot product is forced to a positive sign. The 45-degree benchmark can consequently credit an off-axis slip as a successful pour. Measure rotation about the requested axis and add a perpendicular-axis counterexample test.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/pour_benchmark.py
Line: 146-150

Comment:
**Off-axis tumbles count as pours**

This helper returns the total relative rotation angle; the requested axis only determines its sign. A 45-degree tumble perpendicular to the pour axis therefore returns positive 45 degrees because the zero dot product is forced to a positive sign. The 45-degree benchmark can consequently credit an off-axis slip as a successful pour. Measure rotation about the requested axis and add a perpendicular-axis counterexample test.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Reversed pours pass the gate

Taking the absolute value of both rotations removes the direction check from the signed task metric. If a slipping object rotates −45 degrees around the requested axis for the +45-degree case, this reports zero error and task_success=True. The measurement and primitive use the same axis convention, so compare the signed achieved rotation with the signed command rather than their magnitudes.

Suggested change

	rotation_error = abs(
	abs(trace.peak_signed_displacement) - abs(case.rotate_angle_rad)
	)
	rotation_error = abs(
	trace.peak_signed_displacement - case.rotate_angle_rad
	)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/pour_benchmark.py
Line: 294-296

Comment:
**Reversed pours pass the gate**

Taking the absolute value of both rotations removes the direction check from the signed task metric. If a slipping object rotates −45 degrees around the requested axis for the +45-degree case, this reports zero error and `task_success=True`. The measurement and primitive use the same axis convention, so compare the signed achieved rotation with the signed command rather than their magnitudes.

```suggestion
        rotation_error = abs(
            trace.peak_signed_displacement - case.rotate_angle_rad
        )
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Tabletop drops escape detection

The drop threshold is below the support table, so it cannot detect an object dropped onto that table during transfer. The target lies inside the table footprint, only 0.10 m above its surface. An object resting near the target at z≈0.53 m passes both the 0.12 m delivery tolerance and the z≥0.35 m check despite losing its grasp prematurely. Track grasp retention or an unexpected return to the support surface during transfer, while allowing the primitive's intentional final release.

Knowledge Base Used: Simulation lab

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/hand_over_benchmark.py
Line: 238-247

Comment:
**Tabletop drops escape detection**

The drop threshold is below the support table, so it cannot detect an object dropped onto that table during transfer. The target lies inside the table footprint, only 0.10 m above its surface. An object resting near the target at z≈0.53 m passes both the 0.12 m delivery tolerance and the z≥0.35 m check despite losing its grasp prematurely. Track grasp retention or an unexpected return to the support surface during transfer, while allowing the primitive's intentional final release.

**Knowledge Base Used:** [Simulation lab](https://app.greptile.com/dexforce/-/custom-context/knowledge-base/dexforce/embodichain/-/docs/simulation-lab.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Videos replay a different execution

The recorder uses four physics steps per waypoint, whereas OpenDoor is scored at 64 and the other new benchmarks at 16. For these contact tasks, that changes tracking and achieved motion—not just playback speed—so a video attached to a successful case can depict a different outcome. Record during the measured replay or give the recording path the same replay cadence and terminal hold settings.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/open_door_benchmark.py
Line: 286-291

Comment:
**Videos replay a different execution**

The recorder uses four physics steps per waypoint, whereas OpenDoor is scored at 64 and the other new benchmarks at 16. For these contact tasks, that changes tracking and achieved motion—not just playback speed—so a video attached to a successful case can depict a different outcome. Record during the measured replay or give the recording path the same replay cadence and terminal hold settings.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Failed recordings consume video quota

A failed or unavailable recording is appended to video_paths as an empty string. With --record_video --record_failed_video and the default case limit, a planning failure with no trajectory can consume the quota, suppress a later recordable case, and inflate the reported video count. Append only nonempty recording paths, as the other new benchmarks do. Twist has the same issue.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/slide_benchmark.py
Line: 318-319

Comment:
**Failed recordings consume video quota**

A failed or unavailable recording is appended to `video_paths` as an empty string. With `--record_video --record_failed_video` and the default case limit, a planning failure with no trajectory can consume the quota, suppress a later recordable case, and inflate the reported video count. Append only nonempty recording paths, as the other new benchmarks do. Twist has the same issue.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Recording changes the height diagnostic

final_object_z_m is read after the optional reset and recording replay, while the rotation trace and starting height describe the scored run. Enabling recording can therefore replace this diagnostic with the height from a different execution—or the reset pose if recording cannot start. Capture the final height immediately after physical validation, before resetting the scene for video.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/pour_benchmark.py
Line: 329-330

Comment:
**Recording changes the height diagnostic**

`final_object_z_m` is read after the optional reset and recording replay, while the rotation trace and starting height describe the scored run. Enabling recording can therefore replace this diagnostic with the height from a different execution—or the reset pose if recording cannot start. Capture the final height immediately after physical validation, before resetting the scene for video.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Exported helpers lack parameter annotations

The exported object_axis_angle_rad() leaves all three parameters unannotated, violating the repository directive to fully annotate public APIs. Add RigidObject and tensor parameter annotations, using type-only imports where appropriate. The exported signed_rotation_about_axis_rad() in Pour has the same omission. This repository requirement must be satisfied before merging.

Context Used: CLAUDE.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/axis_align_benchmark.py
Line: 115

Comment:
**Exported helpers lack parameter annotations**

The exported `object_axis_angle_rad()` leaves all three parameters unannotated, violating the repository directive to fully annotate public APIs. Add `RigidObject` and tensor parameter annotations, using type-only imports where appropriate. The exported `signed_rotation_about_axis_rad()` in Pour has the same omission. This repository requirement must be satisfied before merging.

**Context Used:** CLAUDE.md ([source](https://github.com/dexforce/embodichain/blob/main/CLAUDE.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Report describes outdated replay settings

The report presents 16 steps per waypoint as OpenDoor's chosen replay rate, but the committed benchmark uses 64 because 16 fails its tracking gate. Together with the opening claim that the changes are uncommitted, this leaves readers unable to tie the reported results to the reviewed implementation. Identify the measured revision and settings, and separate historical experiments from the current benchmark configuration.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/BENCHMARK_REPORT.md
Line: 73-86

Comment:
**Report describes outdated replay settings**

The report presents 16 steps per waypoint as OpenDoor's chosen replay rate, but the committed benchmark uses 64 because 16 fails its tracking gate. Together with the opening claim that the changes are uncommitted, this leaves readers unable to tie the reported results to the reviewed implementation. Identify the measured revision and settings, and separate historical experiments from the current benchmark configuration.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Something wrong

[matafela]

[P2] Initialize action-specific defaults for in-process dispatch
The newly registered actions are also exposed through --in_process, but _make_child_args() does not populate their required case-selection attributes, such as door_cases, press_cases, and align_cases. Each new benchmark reads its attribute directly, causing an AttributeError; all seven new actions are affected, including the smoke profile. Please initialize the child namespace through the selected module’s add_benchmark_args() defaults before applying shared overrides, and cover this dispatch path with a focused regression check.