Conversation
… serves; a model with no vocabulary refuses by name (PMAT-3609) BPETokenizer::new required `<unk>` in every vocabulary. Qwen3.5 has none: it declares tokenizer.ggml.eos_token_id and no unknown_token_id key at all. So `apr serve` refused a real, complete vocabulary, while a GGUF/APR with NO vocabulary loaded, because GH-226's placeholder synthesised `token{i}` with `<unk>` in slot 0. The less-specified input was the one accepted. This follows the operator's three constraints (#3609, 2026-09-20) and the cop's rulings (2026-09-21): - The unknown token is Option: `new(vocab, merges, impl Into<Option<&str>>)`. A named token must exist. With no unknown token, CONSTRUCTION refuses, naming the first byte that has neither a `<0xNN>` token nor a byte-level glyph token (with_merges also requires every glyph), so encode never meets an unencodable byte: nothing is dropped and nothing is synthesised. With an unknown token present, encode is unchanged. - No vocabulary refuses by name at all 13 placeholder sites (apr-cli serve x7, aprender-serve cli x6). - apr serve's tokenizer.json path uses the DECLARED model.unk_token. The vocabulary-only AppState constructors use `vocabulary_unk_token` (the vocabulary's own `<unk>`, else none), the by-name interim #3675 replaces. - Prose: the GH-226 comment, the book's api-server example and examples/model_cache.rs no longer teach `<unk>` as a requirement. - Fixtures: REAL Qwen3.5-0.8B and TinyLlama GGUF headers, every key and scalar verbatim, only the per-token arrays sliced (generate.py, MANIFEST.json with each source header's sha256). They use the `.gguf-header` extension because .gitignore ignores `*.gguf`. Verified: aprender-serve lib 15892 passed (tokenizer 215, 9 new); apr-cli lib 7283 passed; clippy -D warnings clean on both; cargo check with cuda; contracts lib 1666; deny ok. MUTANT: restoring the unconditional `get("<unk>").ok_or_else(..)` turns the real-Qwen3.5 row RED with the original error "Unknown token '<unk>' not in vocabulary". Found and filed separately: #3677 (greedy encode reads Latin-1 characters as byte-level glyph tokens, so `é` decodes to U+FFFD). Closes #3609 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false) {
"ticket": "PMAT-3609",
"head": "e5b2e75410f4f3af6da07baeb5388dba47ae8572",
"width": 3,
"executor": "agy",
"agreed": false,
"auto_merge": {
"checked": true,
"was_armed": false,
"disarmed": false,
"note": "auto-merge not armed"
},
"lanes": [
{
"lane": 1,
"verdict": "NO-VERDICT",
"findings": 0
},
{
"lane": 2,
"verdict": "PASS",
"findings": 0
},
{
"lane": 3,
"verdict": "PASS",
"findings": 4
}
]
} |
Contributor
Author
|
quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false) {
"ticket": "PMAT-3609",
"head": "e5b2e75410f4f3af6da07baeb5388dba47ae8572",
"width": 3,
"executor": "agy",
"agreed": false,
"auto_merge": {
"checked": true,
"was_armed": false,
"disarmed": false,
"note": "auto-merge not armed"
},
"lanes": [
{
"lane": 1,
"verdict": "FAIL",
"findings": 4
},
{
"lane": 2,
"verdict": "PASS",
"findings": 0
},
{
"lane": 3,
"verdict": "PASS",
"findings": 4
}
]
} |
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
Contributor
Author
|
Seat-fill request for aprender-b3 (cop ruling 2026-09-21): all three quorum seats on PMAT-3609 need a non-author measured fill. Round 3 recorded AGREED 3/3 PASS. None of the three lanes judged this tree:
So none of the three PASSes counts as a measurement.
Both commits are local and not pushed. The PR head on GitHub is still git worktree add /mnt/nvme-raid0/agent-wt/seat-3609 9cc89e3a9 && cd /mnt/nvme-raid0/agent-wt/seat-3609Acceptance commands. Please run each one at
Layout facts the lanes misread, all checked at
Disk note: |
Contributor
Author
Non-author measured seat-fill for PMAT-3609 (aprender-b3, 10:19Z): judged sha
|
| # | Constraint / check | Command | Measured | vs claim |
|---|---|---|---|---|
| 1 | C1 dir. 1: a declared unknown token resolves; the cop's encode guard | cargo test -p aprender-serve --lib tokenizer |
215 passed, 0 failed. All 9 tokenizer_tests_unk_3609 rows ok, incl. no_unknown_token_and_a_byte_with_no_token_is_refused_by_name and tinyllama_real_header_declares_unk_and_it_still_resolves |
✅ matches (215, 9 rows) |
| 2a | C1 dir. 2: placeholders gone | git grep -nE 'token\{i\}|format!\("token\{' -- crates/apr-cli/src/commands/serve crates/aprender-serve/src/cli |
1 hit (routes.rs:222) at 9cc89e3a9, 14 on origin/main |
✅ matches |
| 2b | C1 dir. 2: refuses by name | git grep -n 'no_vocabulary(' … |
2 definitions (handler_gpu_completion.rs:482, mod_server_commands.rs:13), 13 call sites (apr-cli 7, aprender-serve 6) |
✅ matches, but see finding A |
| 3 | C3: mutation turns the real Qwen3.5 fixture RED | In BPETokenizer::new, replace the match unk_token.into() with the pre-#3609 unconditional token_to_id.get("<unk>").ok_or_else(..)?, then cargo test -p aprender-serve --lib qwen35_real_vocabulary_builds_a_tokenizer_with_no_unknown_token |
RED, rc=101: a real vocabulary without <unk> is not refused (#3609): UnsupportedOperation { … "vocabulary must include <unk> token" } (panic at tokenizer_tests_unk_3609.rs:71). File restored with git checkout --, tree clean |
✅ the row discriminates |
| 4 | The fixture is a real header | generate.py ~/models/Qwen3.5-0.8B-IQ4_XS.gguf … --keep 1024; sha256sum |
output 0855f961…cb082, equal to MANIFEST output_sha256 and to the tracked qwen3.5-0.8b.gguf-header. Source header (first 10,943,082 bytes) 2eb5e86e…5d36, equal to MANIFEST. Parsed independently: GGUF v3, 46 keys, eos_token_id = 248046, no tokenizer.ggml.unknown_token_id |
✅ matches |
| 5 | C2: prose and the query audit | pmat query --literal '<unk>' --files-with-matches (81 files); … 'unknown_token' (17) |
Union 87 files. Diffed against every path the body's audit lists: 2 unlisted, docs/roadmaps/entries/PMAT-3609.yaml and docs/roadmaps/roadmap.yaml. Both are this ticket's own text, added by 9cc89e3a9 after the audit was written. api-server.md and examples/model_cache.rs: 0 "must include <unk>" sentences |
✅ holds (the counts drifted 82→81 and 13→17 because of those 2 files and the index) |
| 6 | Builds | cargo check -p aprender-serve [--features cuda], cargo check -p apr-cli [--features cuda] |
all four rc=0 | ✅ matches |
| 7 | Complexity | bash scripts/check_complexity_ratchet.sh at 9cc89e3a9 |
PASS (D2): 79a3af79d vs 9cc89e3a9 … none new, none grown |
✅ on the ratchet, but see finding B |
Verdict: PASS on all three constraints, with two findings.
- A — C1 direction 2 is compiled, not measured. "No vocabulary refuses by name" is evidenced by 13 call sites and four clean
cargo checks. No test sends a request to a model with no vocabulary and observes the named refusal. The code is there, but its behaviour is unmeasured, and a later refactor that drops one call site stays green. That's worth a behavioural row: one serve handler with an empty vocab returns the namedno_vocabularyerror. I'd not block on it, because the constraint asked for the refusal, which exists. - B — claim 7 as worded ("no function above cognitive 25") is false.
decode(tokenizer.rs:385) is cyclomatic 15, cognitive 41. It is the same 41 onorigin/main(tokenizer.rs:344), and the diff's hunks don't touch it, so the ratchet is right that nothing is new or grown. The accurate wording is "no function new or grown over the threshold;encodekept under it by extractingbyte_fallback_id". routes.rs:222, the fabricatedtoken0/token1SSE stream, is out of The BPE tokenizer requires a '<unk>' token that Qwen3.5 does not have — a universal unknown-token is an assumption, not a fact #3609's scope as the PR says. R13:apr servehas three different HTTP surfaces and which one you get depends on the format of the file you passed #2507 namesroutes.rsand "the two meanings of/generate" but not this stream, so it belongs as a note on R13:apr servehas three different HTTP surfaces and which one you get depends on the format of the file you passed #2507 rather than a new issue.
This was referenced Sep 21, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3609
BPETokenizerrequired a<unk>token. Qwen3.5 has none: it declarestokenizer.ggml.eos_token_idand nounknown_token_idkey at all. Soapr serverefused a model with a real, complete vocabulary, while a model with no vocabulary loaded, because a placeholder synthesised one. The less-specified input was the one accepted. This PR applies the operator's three constraints from the 2026-09-20 ruling, plus the cop's 2026-09-21 rulings.What changes
Option, and absent means absent.BPETokenizer::new(vocab, merges, unk_token: impl Into<Option<&str>>).Some(t)/"t":tmust be in the vocabulary, or the constructor errors. Existing"<unk>"callers compile unchanged (From<&str> for Option<&str>).None: no unknown token. Construction refuses, by name, if any byte has neither a<0xNN>token nor a byte-level glyph token.with_mergesalso requires every glyph, because the merge path maps bytes to glyphs only. Soencodenever meets an unencodable byte. Nothing is dropped and nothing is emitted as a stand-in. This is the cop's "refuse, never drop" guard; it is enforced at construction becauseencodeis infallible and has dozens of callers.encodewith an unknown token is unchanged (<0xNN>, then the unknown token). Without one, it falls back to the byte-level glyph.apr-cliserve, 6 inaprender-servecli. It isModelLoadFailedinapr-cli(exit 6) andUnsupportedOperationinaprender-serve: "… has no vocabulary to tokenize with, so there is nothing to serve; refusing to substitute placeholder tokens (The BPE tokenizer requires a '<unk>' token that Qwen3.5 does not have — a universal unknown-token is an assumption, not a fact #3609)". This removes QA: SmolLM2 systematic 2-test failure across all sizes (92.6% pass rate) #226'stoken{i}+vocab[0] = "<unk>"fallback.apr serve's tokenizer.json path uses the declaredmodel.unk_token(null for Qwen). The vocabulary-only constructors (theAppState::*_and_vocabfamily) usevocabulary_unk_token(&vocab): the vocabulary's own<unk>entry, or none. That is a by-name interim the cop approved. Threading the declaredtokenizer.ggml.unknown_token_idthrough every loader is Thread the DECLARED unknown token (tokenizer.ggml.unknown_token_id / tokenizer.json unk_token) through every tokenizer loader — #3609 uses a by-name interim #3675, filed before this PR.api-server.md) andexamples/model_cache.rsnow teach the unknown token as optional and the model's own.Fixtures: real headers, truncation named
crates/aprender-serve/tests/fixtures/gguf-header-slices/:generate.pycuts a real GGUF down to its header. Every key and every scalar/string value is kept verbatim; only the per-token arrays are cut to a leading slice.MANIFEST.jsonrecords each source file's header sha256.qwen3.5-0.8b.gguf-header(39 KB)Qwen3.5-0.8B-IQ4_XS.gguf, header sha2562eb5e86e…eos_token_id = 248046; nounknown_token_id; no<unk>in the vocabularytinyllama-1.1b-chat.gguf-header(19 KB)tinyllama-1.1b-chat-v1.0.Q4_K_M.ggufunknown_token_id = 0=<unk>The
.gguf-headerextension is deliberate:.gitignore:47ignores*.gguf, and a.gguffixture would silently never be committed.Evidence
Case rows (
crates/aprender-serve/src/tokenizer_tests_unk_3609.rs): the real Qwen3.5 header declares EOS and no unknown token · the real Qwen3.5 vocabulary builds a tokenizer with no unknown token and round-trips ASCII/CJK/newline (the mutation target) · the real TinyLlama header's declared<unk>still resolves to id 0 (direction 1) · a declared unknown token is still emitted for an unencodable byte · a named unknown token missing from the vocabulary is refused · no unknown token + byte-level glyphs encode every byte · no unknown token +<0xNN>tokens encode every byte · no unknown token + a byte with no token is refused, naming the byte ·vocabulary_unk_tokennames only a present<unk>.The
pmat queryaudit (constraint 2)pmat query --literal '<unk>'(82 files) and--literal 'unknown_token'(13 files), plusgit grepover docs (95 and 18 files). The union is 108 files. Every one is fixed or kept, with its reason:<unk>. Declared unknown tokens still resolve (direction 1), and these call sites compile unchanged viaFrom<&str> for Option<&str><unk>/ comments on token id 0apr/tokenizer_loading.rs:237builds a SentencePiece tokenizer with the literal"<unk>": legitimate for a unigram model, and the same by-name interim #3675 replaces with the declarationTokenizer(token.rs,tokenizer_vocabulary.rs,examples/tokenization.rs): a word vocabulary has no byte fallback, so OOV words need an unknown id. Different algorithmapr/special_tokens.rs,apr/mapped_apr_model.rs): recognises<unk>as a control token WHEN PRESENT; requires nothingchat_template_template.rs: tokenizer_configunk_token)tokenizer.ggml.unknown_token_id(llama_tokenizer/gguf.rs:150); its BPE/Unigram trainers emit<unk>as training output; GGUF export/builder tests write<unk>as data. None requires<unk>of an input vocabularyapr tokenize trainconfig);<unk>is an output it defines, not an input requirement<unk>as a token, theapr tokenizetrainer, and tests documenting<unk>for unseen characters<unk><unk>as dataparse_stage_list_rejects_unknown_token,an_unknown_tokenization_method_is_refused)pmat queryindex hit with no literal occurrence in the current file (stale index entry)108 files, 0 unclassified.
fixed: changed in this PR (9)
crates/apr-cli/src/commands/serve/chat.rscrates/apr-cli/src/commands/serve/handler_gpu_completion.rscrates/apr-cli/src/commands/serve/handlers.rscrates/apr-cli/src/commands/serve/handlers_include_01.rscrates/aprender-serve/book/src/examples/api-server.mdcrates/aprender-serve/examples/model_cache.rscrates/aprender-serve/src/api/mod_app_state_gpu.rscrates/aprender-serve/src/api/mod_app_state_new.rscrates/aprender-serve/src/tokenizer.rskept: aprender-serve tests, benches and test helpers whose vocabularies DECL (34)
crates/aprender-serve/benches/cache.rscrates/aprender-serve/benches/tokenizer.rscrates/aprender-serve/src/api/chat_completions_stream.rscrates/aprender-serve/src/api/openai_handlers.rscrates/aprender-serve/src/api/tests/apr_model_routes_2609.rscrates/aprender-serve/src/api/tests/batch_completions_tokenizer_2465.rscrates/aprender-serve/src/api/tests/chat_template_contract.rscrates/aprender-serve/src/api/tests/embed_and_envelope_2376.rscrates/aprender-serve/src/api/tests/openai_compat_2375.rscrates/aprender-serve/src/apr/tests_apr_header.rscrates/aprender-serve/src/apr/tests_apr_metadata.rscrates/aprender-serve/src/apr/tests_f16.rscrates/aprender-serve/src/apr/tokenizer_tests.rscrates/aprender-serve/src/cache_tests.rscrates/aprender-serve/src/gguf/inference/forward/single_tests.rscrates/aprender-serve/src/gguf/loader_gguf_model_02.rscrates/aprender-serve/src/gguf/loader_gguf_read.rscrates/aprender-serve/src/gguf/tests/decode_byte.rscrates/aprender-serve/src/gguf/tests/phase35_rope.rscrates/aprender-serve/src/gguf/tests/tests_35.rscrates/aprender-serve/src/gguf/tests/vocabulary_single_gguf.rscrates/aprender-serve/src/layers/tests/qa_012_latency.rscrates/aprender-serve/src/registry_create.rscrates/aprender-serve/src/registry_tests.rscrates/aprender-serve/src/tokenizer_contract_tests.rscrates/aprender-serve/src/tokenizer_tests_bpe_encode.rscrates/aprender-serve/tests/apr_coverage.rscrates/aprender-serve/tests/gguf_model_coverage.rscrates/aprender-serve/tests/gguf_vocab_tests.rscrates/aprender-serve/tests/integration_multi_model_api.rscrates/aprender-serve/tests/property_tests.rscrates/aprender-serve/tests/property_tokenizer.rscrates/aprender-serve/tests/qwen3_moe_serve_dispatch_v1.rscrates/aprender-serve/tests/tokenizer_stress.rskept: aprender-serve examples whose sample vocabulary declares `` / com (1)
crates/aprender-serve/examples/par_001_check_embeddings.rskept: SentencePiece/Unigram (5)
crates/aprender-core/src/text/llama_tokenizer/tests_gguf_sentencepiece.rscrates/aprender-core/src/text/tokenize/tests_sentence.rscrates/aprender-serve/src/tokenizer_sentence_piece.rscrates/aprender-serve/src/tokenizer_sentencepiece_encode.rscrates/aprender-serve/src/tokenizer_tests_sentencepiece_viterbi.rskept: `apr/tokenizer_loading.rs (1)
crates/aprender-serve/src/apr/tokenizer_loading.rskept: word-level `Tokenizer` (3)
crates/aprender-serve/examples/tokenization.rscrates/aprender-serve/src/token.rscrates/aprender-serve/src/tokenizer_vocabulary.rskept: special-token classification (2)
crates/aprender-serve/src/apr/mapped_apr_model.rscrates/aprender-serve/src/apr/special_tokens.rskept: parses a DECLARED unknown token (1)
crates/aprender-serve/src/chat_template_template.rskept: aprender-core tokenizers/converters/tests (27)
crates/aprender-core/examples/create_test_transformer_apr.rscrates/aprender-core/src/citl/neural/tests.rscrates/aprender-core/src/format/converter/export_include.rscrates/aprender-core/src/format/converter/gguf_export_config.rscrates/aprender-core/src/format/converter/tests/tokenizer_parse.rscrates/aprender-core/src/format/converter/tests/tokenizer_parse_vocab_padding.rscrates/aprender-core/src/format/converter/tokenizer_loader.rscrates/aprender-core/src/format/gguf/api_tests.rscrates/aprender-core/src/format/gguf/builder.rscrates/aprender-core/src/format/test_factory/harness_impl.rscrates/aprender-core/src/format/v2_dequant_tests/tests_layout_writer_flags.rscrates/aprender-core/src/text/bpe/mod.rscrates/aprender-core/src/text/bpe/tests.rscrates/aprender-core/src/text/bpe/tests_encode_decode.rscrates/aprender-core/src/text/chat_template/tests.rscrates/aprender-core/src/text/chat_template/tests_huggingface.rscrates/aprender-core/src/text/llama_tokenizer/construction.rscrates/aprender-core/src/text/llama_tokenizer/gguf.rscrates/aprender-core/src/text/llama_tokenizer/mod.rscrates/aprender-core/src/text/llama_tokenizer/tests.rscrates/aprender-core/src/text/llama_tokenizer/tests_decode.rscrates/aprender-core/src/text/llama_tokenizer/tests_gguf_parsing.rscrates/aprender-core/src/text/tokenize/bpe_training.rscrates/aprender-core/src/text/tokenize/mod.rscrates/aprender-core/src/text/tokenize/tests.rscrates/aprender-core/src/text/tokenize/tests_bpe_loading.rscrates/aprender-core/src/text/tokenize/unigram_training.rskept: aprender-train (2)
crates/aprender-train/src/tokenizer/bpe.rscrates/aprender-train/src/tokenizer/config.rskept: apr-cli test data and trainer config (6)
crates/apr-cli/src/commands/embed_viz_classifier.rscrates/apr-cli/src/commands/rosetta_fail_closed_tests.rscrates/apr-cli/src/commands/stamp.rscrates/apr-cli/src/commands/tokenize.rscrates/apr-cli/tests/falsification_crux_f_18.rscrates/apr-cli/tests/falsification_tokenizer_data.rskept: aprender-orchestrate (2)
crates/aprender-orchestrate/src/serve/banco/inference.rscrates/aprender-orchestrate/src/serve/banco/inference_tests.rskept: CodeBERT (2)
contracts/codebert-tokenizer-validation-v1.yamlcrates/aprender-contracts-staging/contracts/codebert-tokenizer-validation-v1.yamlkept: documentation showing a vocabulary JSON sample that contains `` a (1)
book/src/tools/apr-spec.mdkept: substring match, not an unknown TOKEN (2)
crates/apr-cli/src/commands/test_llm_band.rscrates/aprender-serve/src/inference_trace/save_tensor_stage.rskept: historical records (8)
crates/aprender-serve/coverage_report/html/coverage/home/noah/src/realizar/src/tokenizer.rs.htmldocs/audits/quorum-PMAT-3571.jsondocs/dogfood-templates/albor-370m-v1-dogfood-template.mddocs/qa/prompts/QA-MASTER-AUDIT-TRACE.mddocs/specifications/archive/APR-SPEC-v2-draft.mddocs/specifications/archive/qwen3-perf-parity.mdevidence/section-61-5g-1-re-encode-2026-05-10/README.mdevidence/serve/3571/LOADER.mdkept: `pmat query` index hit with no literal occurrence in the current file (2)
crates/aprender-orchestrate/examples/migrations/pytorch-inference/output.rscrates/aprender-serve/examples/trace_layer0_detailed.rsThe new files (
tokenizer_tests_unk_3609.rs, the fixtures) are this PR's additions.contracts/bpe-tokenization-v1.yamlalready states the invariant this makes true:vocabulary_lookupsays "unknown tokens handled by byte fallback", so no contract text changes.Found, filed, not in this PR
éencodes as byte 0xE9 and decodes to U+FFFD. It is independent of the unknown token and pre-existing for every byte-level GGUF served throughAppState. The Qwen round-trip rows therefore use ASCII/CJK/newline.apr showcase's pipeline substitutes token ids[1,2,3,4,5]when a model has no vocabulary (commands/showcase/pipeline.rs:165). It is a demo command and not a tokenizer.Overlap
The stale PMAT-3041 PRs #3344 and #3349 (opened 09-16, both DIRTY) touch
serve/handler_gpu_completion.rsandserve/handlers_include_01.rsin different functions. Whichever lands second resolves the text.🤖 Generated with Claude Code