Add task catalog and independent physical objective evaluation

[yuecideng]

Description

Task discovery currently exposes runnable environments without a shared task-level description or qualification record, and expert completion alone does not establish a measured physical outcome. This PR adds a shared catalog/gallery and an optional independent physical objective, connected by a repeated_pick_place expert/replay evaluation example.

Refs #106 and #655. This implements the first incremental slice; it does not close the broader evaluation/composition roadmap.

• Add task-local catalog.yaml, show-task, category filtering and static HTML export from one catalog model. Preserve existing task IDs and uncataloged tasks; associate RL support with the deployment referenced by the trainer configuration. Package task READMEs and validation records alongside configs.
• Add an opt-in ordered stable-region objective with per-environment progress, sustained-duration requirements, revocable final success, isolated snapshots and partial reset. Physical results remain separate from program completion, segment acceptance, termination and dataset persistence. Stable pose/velocity does not establish gripper release.
• Add an expert/dynamic-replay runner with bounded initial XY variation, effective seeds, actual initial poses, configuration/component snapshots and separate execution, physical, acceptance and persistence outcomes. Preserve legacy raw-policy replay; explicit expert-controller trajectories validate joint layout and cadence. Record settling frames only in the diagnostic trajectory, without extending accepted demonstration segments.

Review by commit:

1. a772c1cf: objective lifecycle, controller replay, evaluation runner and focused tests.
2. 87da14af: catalog/gallery, packaged examples, documentation and context.
3. 66c555d3: user-facing task catalog/physical evaluation guide, CLI reference and Supported Tasks navigation.

Dependencies: no new Python dependencies. Real physical qualification requires a DexSim build compatible with the repository's existing spawn descriptor API.

Validation and remaining limitation

• Final focused suite: 123 passed:

  PYTHONPATH=. python -m pytest tests/test_task_catalog.py tests/test_main.py tests/gym/envs/test_objectives.py tests/gym/envs/test_replay_actions.py tests/lab/scripts/test_evaluate_task_objective.py tests/test_task_program_package_data.py -q --tb=short

• Expanded affected suite: 952 passed, 6 skipped, 18 deselected, 2 failed and 2 setup errors. All four unsuccessful cases reach the installed DexSim API's unsupported com_quaternion argument. The descriptor failure was also reproduced on unchanged base c101a805; this is not a green full-suite claim. Affected cases: test_modular_env_tutorial::test_tutorial_rigid_objects_compile_to_spawn_descriptors, test_profiler_integration::TestProfilerIntegration::test_records_step_reset_sections, test_base_env::TestBaseEnvCPU::test_env_rollout, and test_embodied_env::TestCPU::test_env_rollout.
• Independent catalog, integration and runner reviews have no unresolved findings.
• Full-repository black . with Black 26.3.1; API documentation gate 2091/2091; context validation and routing checks passed.
• Wheel build and packaged-resource verification passed. An unpacked-wheel gallery smoke imported no simulator modules; the final wheel's relevant source and resource contents match the checkout.
• Sphinx dummy and full HTML builds exited 0 with 739 warnings each; documentation is not warning-free. The API-doc checker tests passed (8 tests). Browser checks verified the new guide, Supported Tasks navigation, CLI/API pages, downloadable startup evidence and desktop/mobile layout; all four checked pages had no local HTTP failures.

The native expert/replay attempt stops during simulator construction, before any physical rollout. The checked-in startup record therefore reports an error and unavailable physical outcome, rather than success or measured task failure. End-to-end physical qualification remains pending a compatible DexSim build. Learned-policy adapters, named multi-run experiments and broader predicates remain follow-up work.

Documentation and context

The new Task Catalog and Physical Objectives guide is linked from How-to Guides, CLI Reference and Supported Tasks. It explains discovery, catalog authoring, gallery export, objective semantics and evaluation outcomes, using literal includes of the packaged YAML examples. The repeated-pick-place README documents baseline and perturbed evaluation commands and the objective's limits. Public API entries and the env-framework context owner cover discovery, objective lifecycle and recording semantics. Other affected topic owners were reviewed; their existing contracts remain accurate.

Type of change

• New feature (non-breaking change which adds functionality)
• Documentation update

Screenshots

No screenshot attached. Generate the static gallery directly from packaged task metadata:

embodichain list-task --config-root embodichain_tasks=embodichain_tasks/configs/tasks --category manipulation --export-html task-gallery.html
embodichain show-task embodichain_tasks:repeated_pick_place

Checklist

• I have run the black . command to format the code base.
• I reviewed affected documentation and agent context, updated it where needed, or explained why no update was needed.
• Public API changes are reflected in the API docs (python docs/scripts/check_api_docs.py).
• I have added tests that prove the feature works at the covered boundaries; native qualification limitations are documented above.
• Dependencies have been reviewed; no dependency changes are required by this PR.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable new issue was found in the changes since the previous review.

Summary

This PR adds a shared task catalog and static gallery, introduces an optional ordered physical-objective evaluator, and provides an expert-versus-dynamic-replay evaluation workflow with independent execution, physical, acceptance, and persistence outcomes.

• Adds catalog-backed list-task, show-task, category filtering, and HTML export while retaining uncataloged tasks.
• Integrates per-environment ordered-placement objectives with stable-duration checks, snapshot isolation, and partial resets.
• Adds schema-aware controller-action replay and a repeated-pick-place evaluation runner.
• Documents catalog authoring, physical-objective semantics, deployment discovery, and the current simulator compatibility limitation.
• The changes since the previous review are documentation and navigation updates whose commands, references, and included resources match the implementation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    C[catalog.yaml] --> M[Shared task catalog model]
    G[Runnable deployment configs] --> M
    M --> L[list-task]
    M --> S[show-task]
    M --> H[Static HTML gallery]

    G --> E[EmbodiedEnv]
    O[objective.yaml] --> E
    E --> P[Ordered physical objective]
    E --> X[Expert execution]
    X --> T[Controller trajectory]
    T --> R[Dynamic replay]
    P --> Q[Independent physical outcome]
    X --> A[Execution and acceptance outcomes]
    R --> Q

Loading

_{Reviews (2) · Last reviewed commit: "docs: add task catalog and physical eval..."}

[greptile-apps]

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

•  Controller trajectories replay incorrectly embodichain/lab/gym/envs/embodied_env.py:2031 ▶

  When an explicit ControllerAction is recorded, these lines store the final controller targets. However, save_trajectory() does not mark the trajectory as expert_controller, so dynamic replay treats it as a raw policy action and applies ActionManager preprocessing again. For deployments with transforms such as delta-position actions, replay can issue different commands from those originally recorded. The evaluation runner patches its own artifact, but trajectories saved through the public API remain affected.