Skip to content

fix(query): stop tool-failure-loop guard from conflating distinct Bash failures - #928

Merged
agentforce314 merged 2 commits into
agentforce314:mainfrom
yibocai-airbnb:fix/loop-guard-fallback-tail
Sep 14, 2026
Merged

fix(query): stop tool-failure-loop guard from conflating distinct Bash failures#928
agentforce314 merged 2 commits into
agentforce314:mainfrom
yibocai-airbnb:fix/loop-guard-fallback-tail

Conversation

@yibocai-airbnb

Copy link
Copy Markdown
Contributor

Summary

_normalize_error_category's generic fallback (no named pattern matched — InputValidationError, PermissionError, etc.) keyed off the head of the tool result: text.lower()[:MAX_FALLBACK_CATEGORY_LENGTH] (120 chars).

Bash tool results are assembled as stdout + stderr + exit-code sentence, in that order. Any script that prints even a short startup banner or progress output before crashing has its actual differentiator — a traceback's exception line, a compiler's final error — pushed past that 120-char window. So a script that prints an identical banner on every run collapses every distinct bug behind it into one fallback category.

Observed live, not hypothetical: benchmarking a local model on an agentic JAX/MNIST task, three genuinely different bugs behind an identical stdout banner were categorized as one recurring signature and tripped the loop guard after attempt 3 — even though each attempt had actually fixed the prior bug and hit a new one. The guard punished the shape of the script's own output, not whether the model was actually looping — the same category of defect #777 fixed elsewhere in this same guard.

Fix

Prefer the tail of the (exit-code-stripped) text, and specifically the tail of a matched Traceback (most recent call last): block when present, since that isolates the exception line from any stderr preamble.

This only changes behavior when the text exceeds the 120-char window — below that, [:120] and [-120:] are identical, so short results are untouched. A genuinely-recurring failure (identical traceback) still produces an identical tail, so real-loop detection is not weakened; see the fuzz test below for that property proven at scale rather than asserted.

Tests

  • 3 unit tests: the core regression case (distinct tracebacks behind a shared banner must not collapse), a generic tail-vs-head check, and same-traceback-still-matches.
  • 2 differential fuzz tests (2000 seeded trials each), in the same spirit as fix(harness): stop the loop guard and plan mode from killing headless runs #777's own fuzz methodology:
    • Reproduces the bug at scale: the old head-slice collides on >90% of trials whenever the shared banner is >=120 chars; the new scheme collides on 0%.
    • No-regression property: an identical traceback tail is still recognized as the same category regardless of how much random, unrelated stdout noise varies in the head.

Full existing suite (test_tool_failure_loop_guard.py, test_ch06_tools_round4_A.py) still green — 42 + 13 tests, 0 failures.

Test plan

  • python3 -m unittest tests.test_tool_failure_loop_guard tests.test_ch06_tools_round4_A -v — 55/55 pass
  • Reproduced the original bug against the pre-fix code path (differential fuzz, TestFallbackCategoryFuzz)

🤖 Generated with Claude Code

…h failures

The generic fallback error-category keyed off the first 120 chars of a tool
result. Bash results are stdout+stderr+exit-code sentence in that order, so
a script that prints a startup banner before crashing produced an identical
120-char prefix for genuinely different bugs, tripping the loop guard after
3 attempts even when each attempt fixed the prior bug and hit a new one.

Now prefers the tail (and a matched traceback's tail specifically), where
the actual exception line lives, so distinct failures behind a shared
stdout prefix are no longer collapsed into one signature.
2000 seeded trials each, in the spirit of this guard's own per-batch
counting fix (differential fuzz against the prior behavior rather than
one hand-picked example):

1. Reproduces the bug at scale: a >=120-char shared banner collides
   almost every trial under the old head-slice, and zero times under the
   new tail-preferring slice, for otherwise-distinct tracebacks.
2. No-regression property: an identical traceback tail is still
   recognized as the same category no matter how much random, unrelated
   stdout noise varies in the head -- the real-loop case the fix must
   not weaken.
@agentforce314
agentforce314 merged commit da47bff into agentforce314:main Sep 14, 2026
@agentforce314

Copy link
Copy Markdown
Owner

LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants