Universal tool calling: tool name is not constrained by grammar, so long-context / creative-sampler runs produce near-miss names

[Rw5xJ7y]

koboldcpp 1.120 (Windows, koboldcpp.exe, CUDA / kobolcpp_cublas.dll)
Windows 11 Pro 10.0.22631, i7-13700K, 128 GB DDR4, 4x RTX 3090 (96 GB VRAM)
Model: unsloth/DeepSeek-V4-Flash-0731-GGUF, UD-Q8_K_XL (150.75 GiB, arch = deepseek4)
Client: Cherry Studio [v2.0.12 - v2.0.14] via the OpenAI-compatible /v1/chat/completions endpoint
Launch config (relevant keys): contextsize 292144, quantkv bf16, noswa true, autofit true,
batchsize 1024, threads 8, usemmap false, jinja true, debugmode 1

Universal tool calling: tool name is not constrained by grammar, so long-context / creative-sampler runs produce near-miss names

Summary

Cherry Studio exposes MCP tools with hashed names such as mcp__CherryMemory__createEntities_603184f9d0bef38c8450. In a 165k-token conversation with creative sampler settings (temperature 1.0, top-p 0.95, DRY and min_p enabled), stage 2 repeatedly produced names that were almost right:

mcp__CherryMemory__call_57130
mcp__CherryMemory__createEntits_603184f9d0bf38c8450
mcp__CherrryMemory__createEntities_60_3184f9d0be_f38c8450
mcp__CherryMemory__crerateEntities_603184f90bef38c8450
mcp__CherryMemory__getEntities_603184f9

and in one round an empty array:

Generating (5 / 16384 tokens)
Output: [ ]

Stage 1 had chosen the correct name in its reasoning JSON. The client rejected each call ("Model tried to call unavailable tool ..."), the model retried, and this looped for 1 h 41 min until the client's tool-call limit stopped it.

Lowering temperature to 0.5 / top-p 0.9 and disabling DRY and min_p fixed it immediately: the same request then completed successfully. So the root cause is that the tool name is free-form text in stage 2 and repetition-suppressing samplers corrupt it.

Request

Constrain tool_name (and ideally the argument keys) to the provided tool list via the grammar used in stage 2, and/or run the tool-call stages with neutral sampler settings independent of the user's creative settings.

Related: #2464 #2465 #2466

[LostRuins]

This one is a little bit harder to solve, so I think for now I will do this: If jinja tools is enabled and tools are provided, the default temperature will be 0.5 instead of 0.7

[Rw5xJ7y]

Update: with the DSML fix on the rolling build, corrupted tool-call output now passes through silently as content. Under temperature 1.0 + DRY + min_p the model emitted invoke NAME="..." (uppercase) and, in another run, dropped the hash suffix from the tool name; temperature 0.5 with DRY/min_p off works every time.

On the default-temperature change: clients like Cherry Studio, SillyTavern and Open WebUI send temperature/top_p/DRY explicitly on every request, so a changed default won't reach those users. If feasible, suspending repetition-suppressing samplers (DRY, rep pen) while a tool-call block is being generated would be the robust fix.

[LostRuins]

In Cherry Studio what temperature is it sending? You should lower the temperature there.
If that's not possible, you can hardcode a lower temperature from the backend side by setting --gendefaults though im not sure if that's what you want.

Applying constrained sampling on the tool name itself isn't that easy since no grammar is actually applied during the normal jinja tool calling (tokens are generated normally), basically the backend doesn't know whether tool calling is currently in progress or not

[Rw5xJ7y]

Understood, thanks for explaining — I'll keep switching samplers client-side for tool turns (--gendefaults would have to apply globally, which I can't do since the same assistant/thread is used for creative writing at temperature 1.0).

One small note in case it's useful: for DSV4 the whole DSML opening marker is a single special token (128825, <｜DSML｜), so "a tool call is in progress" is knowable from the moment that token is sampled. Suppressing DRY/rep-pen from that point until the closing marker would be enough — the failures I saw were single-character corruptions (invoke NAME= instead of name=, a dropped character in the hashed tool name), not structural ones, so full grammar constraints probably aren't needed. Either way, feel free to close.