feat(agents): TypeSafe world state with way to the target, open sides and doorways

[spomichter]

Contribution path

Draft, stacked on #4216 (feat/typesafe-nav-eval), which is stacked on the TypeSafe agent. The diff is the world state the agent sends to the model, the question wording that refers to it, and the scorer used to find it. The TypeSafe v2 rewrite is #4212; this needs porting to whichever agent lands.

Problem

In the 84-case habitat_nav run the TypeSafe arm reached 34 of 84. 45 of the 50 failures share one cause: the JSON state gives the model the straight-line bearing to the target and eight scan sectors, and the questions only ever turn toward the target, so anything on that line ends in a give-up, a forward/backward dither or a false finished. Measured on that run's requests: side and rear sectors read "5.0 m clear" in 816 of 840 readings (the scan is forward-facing and an empty sector is reported as max range), 22% of the 20 listed objects are above 1.5 m, and the target is last in the list in 121 of 168 requests.

Solution

dimos/agents/typesafe/world_state.py builds one document per tick, everything relative to the robot:

{"goal": "go to the couch",
 "robot": {"motion": "driving", "last_drive": {"x": "forward", "y": "none", "yaw": "turn_right"},
           "recent": {"moved_m": 1.2, "turned_deg": 188, "target_closer_m": 0.9, "pattern": "advancing"}},
 "objects": [{"target": true, "label": "couch", "bearing": "ahead_right", "bearing_deg": -16,
              "distance": "far", "distance_m": 10.19, "width_m": 1.5},
             {"label": "potted_plant", "bearing": "left", "distance_m": 0.39, "width_m": 0.8}],
 "way_to_target": {"state": "blocked", "blocked_by": "wall", "blocked_at_m": 6.4,
   "open_sides": [{"side": "left", "kind": "corner", "bearing_deg": -1, "detour_deg": 15, "clear_m": 3.6},
                  {"side": "right", "kind": "doorway", "bearing_deg": -20, "detour_deg": 4,
                   "range_m": 4.9, "width_m": 1.0, "target_beyond": true}],
   "going_around": {"side": "left", "for_s": 22}},
 "free_space": {"ahead": {"clear_m": 3.4, "state": "clear", "by": "wall"}, "...": "8 directions"}}

• _objects_3d: the target first, then up to 5 obstacles within 4 m that overlap the 0.1-0.9 m body band above the robot's floor with a footprint of at least 0.2 m. No world coordinates. bearing_word: ahead is within 15 degrees; behind always says behind_left or behind_right.
• way_to_target: one body-wide line from the robot to the target's box against the floor-level footprints of objects and walls and against the depth scan gives clear (+ room, narrowed_on) or blocked (+ blocked_by, blocked_at_m; obstacle when only the scan sees it). ray_cast sends single rays every 5 degrees; on each side of the line the nearest direction that runs free past the blocker is open, a range jump of 1 m between neighbouring rays is corner. doorways finds 0.7-3.0 m gaps in straight runs of wall boxes that the robot has a free line to and reports bearing, range, width and whether the target lies beyond that wall. One wall as seen from the robot; nothing is searched or chained, and no side is ranked or recommended.
• Memory (kept by the agent between ticks): recent over 8 s with a pattern word (starting, advancing, still, stuck, turning_on_the_spot, moving_without_getting_closer), going_around (the side the model's own picks began steering to, kept until it closes), been_there (a way whose far point lies on the robot's own trail older than 10 s), and 30 s of scan returns within 2.5 m so a direction seen blocked stays unlisted when the robot looks away.
• free_space: 8 directions from the same footprints plus the scan, each naming what is there; unseen when nothing is known.
• demo_objects.py publishes walls (no longer excludes ^wall).
• drive.py / TASK: same six answers. Forward, strafe and turn steer by a "steering bearing": the target's bearing when the line is clear, else the bearing of one listed open side, keeping the side already chosen. finished needs the target near, the robot stopped and no wall on the line. decode, _steer, the config and the module's streams are unchanged.
• misc/evals/ts_state_bench.py: evo benchmark over habitat_nav cases. Per case the suite's grade when reached, not counting a reach with a ground-truth wall between the final pose and the target box, else 0.4 x the furthest fraction reached of the planner arm's driven route; one simulator at a time by file lock; a run with no valid model ticks is a harness error, not a score. ts_state_frozen.py: the gate that limits a search to the state builder and question text (whitelist, AST pins on decoding/steering/config/streams and on the _tick data flow, ground-truth hash, greps for planner or reference-route references and scene ids).

Found by an evo loop of 13 experiments (exp_0013). Same tasks, 84-case run (one attempt, 300 s) vs this branch (120 s):

set	84-case run	this branch
7 tuning tasks	2 of 7	11 of 14 runs (two each), 80% of route covered on average
9 held-out tasks, 5 scenes never used for tuning	0 of 9	7 of 9 (one run each, on the earlier exp_0008 state)

An oracle run with the bearing to a look-ahead point on the true route reached 4 of 6, so the controller and model were not the limit. Input is ~3,500 tokens per tick (was ~3,980).

Known problems:

• 102344403_cabinet_2 reached in the 84-case run and fails 0 of 2 here: the robot shuttles through the one doorway beside its start.
• Outcomes vary run to run (toilette 1 of 2); single runs cannot rank states.
• Only 5 of the 34 tasks that passed in the 84-case run have been re-run; the held-out set has not been re-run on this exact state.
• Walls come from the ground-truth boxes of feat(habitat): HSSD scene ground truth as 3D and top-down 2D detections #4214; the HM3D scene has none. About 30 geometric constants were sized while looking at the tuning tasks.
• nav_metrics reached has no line-of-sight test (a robot within 1 m of a target behind a wall counts as arrived, for every arm); the scorer here rejects that, the suite does not.

How to Test

uv run pytest dimos/agents/typesafe -q          # 33 tests
# one live case (TYPESAFE_API_KEY, Habitat built, ground truth from #4214 under misc/habitat/ground_truth)
python misc/evals/ts_state_bench.py --cases 106366410_174226806_couch --repeats 2

ruff, mypy (strict) and pre-commit pass on the changed files. The three type fixes after the scored experiment do not change behaviour.

Checklist

• I have read and approved the CLA.

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
5998	1	5997	103

View the full list of 1 ❄️ flaky test(s)

dimos.e2e_tests.test_voice_browser::test_hold_to_talk_ships_a_decodable_recording[firefox]

Flake rate in main: 68.75% (Passed 5 times, Failed 11 times)

Stack Traces | 22.6s run time

voice_bridge = ('http://127.0.0.1:64782/', [])
fake_mic_page = <Page url='http://127.0.0.1:64782/'>

    def test_hold_to_talk_ships_a_decodable_recording(
        voice_bridge: tuple[str, list[AudioChunk]], fake_mic_page: Page
    ) -> None:
        url, chunks = voice_bridge
        chunks.clear()  # the module fixture is shared across both engines
        fake_mic_page.goto(url)
        mic = fake_mic_page.get_by_test_id("chat-audio_in-mic")
        # Enabled == transport connected; the manifest already placed the panel.
        expect(mic).to_be_enabled(timeout=120_000)
        expect(mic).to_have_attribute("data-state", "idle")
    
        mic.hover()
        fake_mic_page.mouse.down()
>       expect(mic).to_have_attribute("data-state", "recording", timeout=15_000)
E       AssertionError: Locator expected to have attribute 'recording'
E       Actual value: arming 
E       Call log:
E         - Expect "to_have_attribute" with timeout 15000ms
E         - waiting for get_by_test_id("chat-audio_in-mic")
E       
E       Aria snapshot:
E       - button "hold to talk": talk

chunks     = []
fake_mic_page = <Page url='http://127.0.0.1:64782/'>
mic        = <Locator frame=<Frame name= url='http://127.0.0.1:64782/'> selector='internal:testid=[data-testid="chat-audio_in-mic"s]'>
url        = 'http://127.0.0.1:64782/'
voice_bridge = ('http://127.0.0.1:64782/', [])

dimos/e2e_tests/test_voice_browser.py:104: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}