feat(judges): one message_history for every judge path

[donei003]

Stacked on #89 — base is feature/ld-judges-tool-trajectory, so the diff is just this change. Retarget to main when #89 merges.

Problem

#89 gave the trajectory to the offline judge only. The two online paths built their own message_history and neither included it — so a trajectory rubric silently degraded to grading prose when run online, and a judge grading the same response saw a different conversation depending on which path reached it.

They had already drifted before the trajectory made it visible:

Path	message_history was
Offline evals	row input + trajectory + output + format block
Online inline (run_judges)	user input + output + format block
Online deferred (run_judge)	output + format block — no input at all

A deferred judge was grading a response with no request beside it. That is a pre-existing bug this PR also fixes.

Change

judge_scoring.build_message_history is now the only place a history is built — in the module that already owns the {score, reasoning} contract, for exactly the same reason. All three paths call it, and a test asserts the inline and deferred paths produce byte-identical output for one row.

Online capture happens in execute_and_track / execute_and_stream, which now return the rendered trajectory alongside response and track_data. client.py and the two per-node graph.py judge runs thread it through.

JudgeTask gains user_input and trajectory — plain strings, since every field on it has to survive pickling to a worker thread; a test pins that.

trajectory.py moves from evaluations/ to the package root, since it is no longer evaluations-specific.

The NativeTool decision you asked about

Recording is composed inside wrap_tool_handlers, on the original tool map, so the recorder still sees a NativeTool as a NativeTool and skips it — identically to offline.

Wrapping the tracked map instead was the tempting option, because native calls are locally observable online: wrap_tool_handlers substitutes a callable tracking stub. But that stub returns nothing, so recording it would show a judge a tool call with an empty result while the provider's real result stayed invisible — worse than not showing it. Tests assert the native tool is absent from the online trajectory and that $ld:ai:tool_call still fires underneath the recorder.

Tell me if you'd rather natives appear online with an explicit "result not observable" marker; it's a small change now that one function owns the rendering.

Graph-level judges get no trajectory, deliberately

graph_judge grades a final answer produced across several nodes. Splicing their trajectories together would describe a conversation that never happened, so it gets "". Per-node judges inside a graph do get their own node's.

Validation

• uv run pytest -q — 1282 passed, 11 skipped
• uv run mypy packages/client/src/launchdarkly_ai_server — clean; ruff check / format --check — clean
• 13 new tests in test_judge_message_history.py: the builder's ordering and skipping, the trajectory reaching both online paths, inline-vs-deferred agreement, JudgeTask picklability, online capture through the real execute_and_track, native-tool exclusion, and $ld:ai:tool_call surviving the composition

Spec follow-up for ai-sdks-monorepo §3.13/§3.14 to come once this shape is agreed.

🤖 Generated with Claude Code

---

Note

Overview
Unifies what judges see across inline online, deferred (JudgeTask), and offline evaluation paths by introducing judge_scoring.build_message_history as the only place {{message_history}} is assembled: user input, rendered tool trajectory, model output, then JSON formatting instructions (empty parts skipped).

Online invocations now capture tool trajectories in execute_and_track / execute_and_stream (per-invocation TrajectoryRecorder, composed inside wrap_tool_handlers on the original tool map so NativeTool stays unrecorded). The rendered trajectory string is returned with the generation result and threaded through client.py, graph.py, run_judges, and build_judge_tasks.

Deferred judges gain JudgeTask.user_input and JudgeTask.trajectory (picklable strings), fixing the prior bug where deferred grading omitted the request entirely. Offline scoring in evaluations/runner.py uses the same builder instead of inline joins.

trajectory moves from evaluations/ to the package root for shared online/offline use. Docs and test_judge_message_history.py assert inline vs deferred byte-identical history and online capture behavior.

^{Reviewed by Cursor Bugbot for commit 398dbad. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 398dbad. Configure here.}

[cursor]

Handoff tools pollute judge trajectories

Medium Severity

execute_and_track now records every callable into the trajectory, including synthetic __handoff_* tools that route() injects. Per-node judges therefore see those names under tools available and as real calls, and a node with no customer tools still gets a trajectory block. wrap_tool_handlers already omits the same tools from $ld:ai:tool_call as non-work.

Additional Locations (2)

• packages/client/src/launchdarkly_ai_server/graph.py#L372-L383
• packages/client/src/launchdarkly_ai_server/trajectory.py#L130-L140

 

^{Reviewed by Cursor Bugbot for commit 398dbad. Configure here.}