Skip to content

🐛 Fix(evaluation): run trials in runtime service - #3954

Open
cj2026-bit wants to merge 4 commits into
hotfix/v2.6.1from
fix-evaluation-runtime-dispatch-v261
Open

cj2026-bit wants to merge 4 commits into
hotfix/v2.6.1from
fix-evaluation-runtime-dispatch-v261

Conversation

@cj2026-bit

@cj2026-bit cj2026-bit commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Bug

Evaluation APIs failed in the v2.6.0 / v2.6.1 environment with thread-manager errors:

  • Creating an evaluation task failed with ThreadManagerNotRunning: Thread manager for runtime is created.
  • The trial-run API returned 990105 Failed to run trial evaluation.

Root cause

The issue was introduced by PR #3917, which unified thread lifecycle management and thread pools.

Config and Runtime are separate processes, and each process owns its own in-process thread-manager instances. The evaluation code used runtime_thread_manager directly from the Config process. The Runtime manager is started only in the Runtime process, so the Config process's instance remained in the CREATED state and rejected task submission with ThreadManagerNotRunning.

The trial-run path had the same architectural problem: its public endpoint lived in Config but directly executed the Agent and evaluators in the Config process, relying on the unstarted local Runtime manager.

Solution

  1. Use config_thread_manager for Config-side preparation and dispatch work when creating an evaluation task.
  2. Keep the public POST /agent-evaluations/trial-run API unchanged; Config continues to handle user authentication and request validation.
  3. Add an internal Runtime trial-run endpoint, with Config forwarding the request using an internal JWT.
  4. Run Agent execution and evaluator scoring in the Runtime process using runtime_thread_manager, avoiding direct cross-process thread-pool references.
  5. Do not change the database schema or the public request and success-response contracts.

Validation

  • Targeted tests: 291 passed, 4 skipped.
  • Full CI test job: passed; all repository tests passed.
  • Total coverage: 89.7% (reported as 90% by the coverage summary).
  • Codecov patch coverage: 95.23% (target 90%).
  • Codecov upload: passed (coverage report).
  • SonarCloud analysis and all six image-build checks: passed.
  • Updated pure-logic and endpoint test stubs to cover the Config/Runtime manager split.

Test screenshots

Screenshots will be added by the test owner.

image image

Route trial evaluations through the authenticated Config-to-Runtime proxy and use Config's manager only for creation-stage preparation. Keep Agent execution and evaluator scoring in Runtime.

Co-authored-by: Codex <noreply@openai.com>

Generated-by: gpt-5
@cj2026-bit cj2026-bit added the ai-assisted PR authored or co-authored with AI assistance label Sep 18, 2026
@cj2026-bit cj2026-bit self-assigned this Sep 18, 2026
@cj2026-bit cj2026-bit removed the ai-assisted PR authored or co-authored with AI assistance label Sep 18, 2026
Keep pure-logic service import tests aligned with the Config and Runtime thread-manager split.

Co-authored-by: Codex <noreply@openai.com>

Generated-by: gpt-5
@cj2026-bit cj2026-bit changed the title fix(evaluation): run trials in runtime service 🐛 Fix(evaluation): run trials in runtime service Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants