Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,7 @@ Most use an LLM as a judge. All of them pick between an **efficient** model and
|---|---|---|---|
| **[Capability](docs/routing_algorithms/llm_classifier_routing.md)** | The first request is judged by an LLM. | `llm_classifier` | 71.2% at $79.32 |
| **[Stage](docs/routing_algorithms/stage_router_routing.md)** | Tool responses are judged by pattern matching or an LLM. | `stage_router` | 72.7% at $68.19 |
| **[Plan/Execute](docs/routing_algorithms/plan_execute_routing.md)** | A capable model plans, then an efficient model executes after the first edit. | `plan_execute` | not yet benchmarked |
| **[Capability + Stage](docs/routing_algorithms/composite_routing.md)** | Combines the two above. | `composite` | not yet benchmarked |
| **[Escalation](docs/routing_algorithms/escalation_router_routing.md)** | Starts efficient. Responses are judged by an LLM for issues, then escalated. | `llm_classifier` + `mode = "escalation"` | 75.7% at $85.00 |
| **[Advisor Gate](docs/routing_algorithms/advisor_gate_routing.md)** | One model serves every turn; a stronger advisor approves its plans and "done" claims, or sends it back. | `advisor` | lifts a weak executor 43.8% → 54.7% |
Expand Down
16 changes: 16 additions & 0 deletions benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,22 @@ benchmark/server-configs/tb-lite-single-gpt-5-5.toml
benchmark/server-configs/tb-lite-single-opus-4-7.toml
```

To benchmark the plan/execute boundary, use
`benchmark/server-configs/tb-lite-plan-execute-opus-kimi.toml`. It starts each task on Opus and
hands the preserved trajectory to Kimi after the first edit or write tool call:

```bash
bash benchmark/run-baseline.sh \
--harbor-path benchmark/datasets/openthoughts-tblite-closed-book \
--server-config benchmark/server-configs/tb-lite-plan-execute-opus-kimi.toml \
--model switchyard \
--agent codex \
--reasoning-effort xhigh \
--n-tasks 1 \
--n-concurrent 1 \
--max-retries 0
```

By default, the runner starts in the background and prints the PID, log path, and kill command.

## Book Modes
Expand Down
26 changes: 26 additions & 0 deletions benchmark/server-configs/tb-lite-plan-execute-opus-kimi.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Plan/execute configuration for Harbor Terminal-Bench Lite.
# Harbor selects this route with --model switchyard.

schema_version = 1

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

[targets.capable]
id = "anthropic/claude-opus-4.7"
llm_client = "openrouter"

[targets.efficient]
id = "moonshotai/kimi-k2.7-code"
llm_client = "openrouter"

[routes.switchyard]
id = "switchyard"
type = "plan_execute"
capable_target = "capable"
efficient_target = "efficient"
1 change: 1 addition & 0 deletions crates/libsy/src/algorithms.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ pub mod fall_through;
pub mod llm_class;
pub mod noop;
pub mod passthrough;
pub mod plan_execute;
pub mod rand;
pub mod stage;
pub mod subagent;
Expand Down
Loading
Loading