feat(libsy): add an up-front capability gate to the escalation route - #681
feat(libsy): add an up-front capability gate to the escalation route#681linj-glitch wants to merge 5 commits into
Conversation
The escalation route judges trouble after it has happened, and a hand-off
mid-task makes the strong tier redo the work, so for tasks that were never
going to fit the weak tier the cheapest moment to escalate is the first
request. A task-level rule in the trajectory-judge prompt cannot set that
split reliably: worded as "spans several modules" or "the hardest
minority" it latched almost every multi-file task in practice, because the
judge has no reference distribution to calibrate against.
`escalation.gate = { base_threshold }` runs the packaged capability
forecaster once per session, on the first request, through the escalation
judge's target, and applies capability mode's threshold policy to its
p_solve. Below the threshold the session latches to the capable tier before
the efficient tier is called; otherwise the request proceeds and the
trajectory judge takes over. Unusable verdicts and failed gate calls fall
open to the efficient tier. `threshold_step` and `prompt` mirror
capability mode.
Adds EscalationGateConfig with validation, the gate construction next to
the capability classifier so both share one policy, four route tests and a
validation test, schema and guide documentation, and a changelog entry.
Signed-off-by: Lin Jia <linj@nvidia.com>
|
Signed-off-by: Lin Jia <linj@nvidia.com>
…hreshold in its evidence The gate's usefulness depends on where the threshold sits relative to the forecaster's p_solve distribution, and nothing surfaced that distribution: the capability evidence was overwritten by the gate's own marker and no log line carried the score. The gate now logs every verdict with the forecast evidence at info level and copies score and threshold into its latch evidence. Adds a read-back accessor on Driver for wrapping components. Signed-off-by: Lin Jia <linj@nvidia.com>
4b9e3ec to
2a9869f
Compare
The gate forecast runs once per session and rewards a strong model; the trajectory judge runs on every weak turn and is where a cheap model belongs. With one classifier_target for both, a deployment either pays a frontier model per turn (1,500 verdicts per 30 sessions measured) or gates with a weak forecaster. gate.classifier_target names a separate target for the forecast, resolved and validated by the runner like classifier_target and included in the route's callable targets; unset, the gate keeps using the route's judge. Signed-off-by: Lin Jia <linj@nvidia.com>
|
First benchmark results for the gate, DeepSWE-v1.1 DEV30 subset, Codex 0.149.1, run total = agent tokens of every member plus judge and gate calls in relative units (uncached input 1, cached 0.1, output 8 per million tokens, all at GPT-5.6 rates). Sol max alone scores 26 to 28 of 30 for 54 units. Kimi-K3 cheap tier, Sol max strong tier, gate at base_threshold 0.50 through a Sol-medium target, per-turn judge (stuck bar only) on DeepSeek flash: 26 of 30 for about 46 units. The gate latched 10 sessions on turn one with forecasts between 0.18 and 0.48 and kept the sessions forecast at 0.52 to 0.72; Kimi solved 11 of the 12 it kept to the end; the stuck bar latched 8 late. Without the gate, every prompt-level task rule latched 19 to 30 of 30 sessions on turn one and the route cost 52 to 66 units. Luna cheap tier at medium effort, gate at 0.60: the gate half worked (14 of the 15 tasks it sent to Sol solved, forecasts spread 0.22 to 0.78 with no cluster at the threshold) but Luna solved 0 to 1 of the tasks it kept, so that pair lands at 12 to 16 of 30 for 41 to 47 units. A 64-task confirmation of the Kimi configuration is running; I will post it here before asking for review. |
|
64-task confirmation of the Kimi configuration (gate 0.50 through Sol-medium, DeepSeek stuck-only per-turn judge, generic route): 38 solved of 58 completed members, 65.5 percent; six members failed before the harness ran and left no artifacts. Same 64 tasks, other arms: Kimi-K3 alone 40 and 35 (58.6 percent), Sol max alone 41 to 46 (68 percent). Run total 1.30 units per member, about 84 for the set, against about 116 for Sol max alone and 56 for Kimi alone. The gate latched 16 of 58 sessions on turn one, the stuck bar latched 14 more late, and Kimi kept 28 and solved 16 of them. So on both the 30-task and the 64-task sets the gate puts the route at the lower edge of Sol max's score band, four to five tasks above the cheap tier alone, at 16 to 28 percent below Sol's total. Without the gate, every prompt-level task rule latched 19 to 30 of 30 sessions on turn one and cost as much as or more than Sol alone. |
…dence ladder Asked for a probability, judges cluster on a few round values and, given a stated cutoff, write a number just under it (0.35 against 0.40 in more than half of one run's verdicts). Asked for one rung of a fixed ladder they rank more reliably. verdict_scale = "ordinal" on capability routes and escalation.gate.verdict_scale switch the packaged prompt and schema to eight named rungs from surely to almost_surely_not; each rung maps to the midpoint of the frequency band it names, so base_threshold keeps selecting the split and a threshold between two rungs latches exactly the rungs below it. The rung is recorded in the evidence next to the score. Signed-off-by: Lin Jia <linj@nvidia.com>
1 participant
After merge, an escalation route can carry
escalation.gate = { base_threshold = 0.4 }. On the first request of a session the route calls the judge with the packaged capability-classifier prompt, reads its p_solve forecast for the task, and latches the session to the strong tier when the forecast is below the threshold, before the weak tier is called; otherwise the request proceeds and the trajectory judge takes over as today.threshold_stepandpromptbehave as in capability mode, unusable verdicts and failed gate calls fall open to the weak tier, and the runner needs no change becauseescalationalready deserializes intoEscalationJudgeConfig.The reason is cost. A hand-off mid-task makes the strong tier redo the work (90 to 120 strong calls per escalated session on DeepSWE regardless of when the latch fired), so the cheapest moment to escalate a task that was never going to fit the weak tier is turn one, and a task-level rule in the judge prompt cannot set that split: an absolute wording latched 30 of 30 DEV sessions and a relative wording 28 to 29 of 30, while capability mode's numeric threshold sent 14 of 30 to the strong tier in the same setting. The gate reuses capability mode's threshold policy, prompt and schema unchanged.
Tests: 354 pass across libsy and the runner including five new ones (latch before the efficient call, pass then trajectory judge, once per session, fall-open on an unusable verdict, threshold validation); clippy is clean; the server dry-run accepts the new table. Kept as a draft until the DEV benchmark on the Luna and Kimi pairs shows a selective split and a run-total saving against the strong tier alone.