Conversation
paul-v-snet
requested changes
Sep 23, 2026
Comment on lines
+103
to
+106
| def make_base(create, name="ASICloud"): | ||
| provider = llm.AIProvider(name, "ASI_API_KEY", "minimax/minimax-m3", "https://example.invalid/v1") | ||
| provider._client = NS(chat=NS(completions=NS(create=create))) | ||
| return provider |
Collaborator
There was a problem hiding this comment.
make_base is defined but never used.
Comment on lines
+134
to
+156
| def test_empty_reply_out_of_budget_is_explained(): | ||
| create = FakeCreate(chat_response("", "length")) | ||
| assert sent_text(make_openrouter(create).chat(PROMPT)) == llm.LLM_EMPTY_RESPONSE_MESSAGE | ||
|
|
||
|
|
||
| def test_empty_reply_with_stop_is_not_blamed_on_the_budget(): | ||
| create = FakeCreate(chat_response("", "stop", completion_tokens=0, reasoning_tokens=0)) | ||
| assert make_openrouter(create).chat(PROMPT) == "" | ||
|
|
||
|
|
||
| def test_openai_empty_reply_out_of_budget_is_explained(): | ||
| create = FakeCreate(responses_response("", "incomplete", "max_output_tokens")) | ||
| assert sent_text(make_openai(create).chat(PROMPT, max_tokens=120)) == llm.LLM_EMPTY_RESPONSE_MESSAGE | ||
|
|
||
|
|
||
| def test_openai_empty_reply_without_incomplete_reason_returns_empty(): | ||
| create = FakeCreate(responses_response("", "completed", output_tokens=0, reasoning_tokens=0)) | ||
| assert make_openai(create).chat(PROMPT) == "" | ||
|
|
||
|
|
||
| def test_asione_empty_reply_out_of_budget_is_explained(): | ||
| create = FakeCreate(chat_response("", "length")) | ||
| assert sent_text(make_asione(create).chat(PROMPT)) == llm.LLM_EMPTY_RESPONSE_MESSAGE |
Collaborator
There was a problem hiding this comment.
All test_*_empty_reply_* tests also pass when chat() crashes, since each provider's chat() returns "" when it catches an exception. Is this the expected behavior?
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #337, based on its branch. Unit tests only, no behaviour change.
Autotests/unit/test_llm_budget.py(15 tests), registered inrun_mandatory, covers what #337 added: thefinish_reasonandincomplete_reasonchecks before the notice, the notice itself on an empty reply that ran out of budget for OpenRouter, OpenAI and ASI:One, the ASI:One reasoning budget mapping, the OpenRouter reasoning body,[LLM_USAGE]at INFO on both APIs, and an API error returning an empty string. The provider modules are loaded by file path withopenaiandconfigstubbed, so the tests need no container, network or token.Earlier versions of this PR also changed provider behaviour: a retry without reasoning, one notice per streak, and keeping a truncated reply out of the loop. All three are dropped after the discussion here and in #337.
How Has This Been Tested?
CI:
tests/pytest.sh65 passed, Phase 1 144 passed (129 + 15 new), Phase 2 6 passed, MeTTa tests green.Checklist