fix(query): stop tool-failure-loop guard from conflating distinct Bash failures - #928
Merged
agentforce314 merged 2 commits intoSep 14, 2026
Conversation
…h failures The generic fallback error-category keyed off the first 120 chars of a tool result. Bash results are stdout+stderr+exit-code sentence in that order, so a script that prints a startup banner before crashing produced an identical 120-char prefix for genuinely different bugs, tripping the loop guard after 3 attempts even when each attempt fixed the prior bug and hit a new one. Now prefers the tail (and a matched traceback's tail specifically), where the actual exception line lives, so distinct failures behind a shared stdout prefix are no longer collapsed into one signature.
2000 seeded trials each, in the spirit of this guard's own per-batch counting fix (differential fuzz against the prior behavior rather than one hand-picked example): 1. Reproduces the bug at scale: a >=120-char shared banner collides almost every trial under the old head-slice, and zero times under the new tail-preferring slice, for otherwise-distinct tracebacks. 2. No-regression property: an identical traceback tail is still recognized as the same category no matter how much random, unrelated stdout noise varies in the head -- the real-loop case the fix must not weaken.
Owner
|
LGTM. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_normalize_error_category's generic fallback (no named pattern matched —InputValidationError,PermissionError, etc.) keyed off the head of the tool result:text.lower()[:MAX_FALLBACK_CATEGORY_LENGTH](120 chars).Bash tool results are assembled as
stdout + stderr + exit-code sentence, in that order. Any script that prints even a short startup banner or progress output before crashing has its actual differentiator — a traceback's exception line, a compiler's final error — pushed past that 120-char window. So a script that prints an identical banner on every run collapses every distinct bug behind it into one fallback category.Observed live, not hypothetical: benchmarking a local model on an agentic JAX/MNIST task, three genuinely different bugs behind an identical stdout banner were categorized as one recurring signature and tripped the loop guard after attempt 3 — even though each attempt had actually fixed the prior bug and hit a new one. The guard punished the shape of the script's own output, not whether the model was actually looping — the same category of defect #777 fixed elsewhere in this same guard.
Fix
Prefer the tail of the (exit-code-stripped) text, and specifically the tail of a matched
Traceback (most recent call last):block when present, since that isolates the exception line from any stderr preamble.This only changes behavior when the text exceeds the 120-char window — below that,
[:120]and[-120:]are identical, so short results are untouched. A genuinely-recurring failure (identical traceback) still produces an identical tail, so real-loop detection is not weakened; see the fuzz test below for that property proven at scale rather than asserted.Tests
Full existing suite (
test_tool_failure_loop_guard.py,test_ch06_tools_round4_A.py) still green — 42 + 13 tests, 0 failures.Test plan
python3 -m unittest tests.test_tool_failure_loop_guard tests.test_ch06_tools_round4_A -v— 55/55 passTestFallbackCategoryFuzz)🤖 Generated with Claude Code