Skip to content

chore: 0.69.1 merge-back — release/0.69.1-batch-2 + rc/0.69.1-x (X2) into main - #4046

Open
noahgift wants to merge 646 commits into
mainfrom
chore/0.69.1-merge-back
Open

noahgift wants to merge 646 commits into
mainfrom
chore/0.69.1-merge-back

Conversation

@noahgift

@noahgift noahgift commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Post-publish merge-back of the 0.69.1 release into main. Not armed: the release cop merges it after publish.

What it merges

Two --no-ff merges on a branch cut from main @ 49fe19c:

  1. release/0.69.1-batch-2 @ 4f81bde (freeze v2)

  2. rc/0.69.1-x @ d8a6df5 (X2, the 0.69.1 release binary) = freeze v2 plus two scoped hotfixes:

  3. fix/ladder-emergency-crux-smoke @ 4587520: the judge chain (8a9e0a74f9814503517c49d86dc34587520). certification: carry a cell's receipt forward when the diff since its apr_sha doesn't touch that cell's inference path #4037 and Release process for 0.70+: CRUX-smoke gate on the release binary; full ladder + full CRUX move to nightly; #4033 levers wired into the workflow #4045 build on it. Its emergency scope is version-pinned to 0.69.1, so it is inert on main.

  4. fix/prepub-skip-cfg-test-includes @ 47bb92b (f5, aprender-serve: #3985 fusion guard test include_str!s contracts/kernel-fusion-v1.yaml from OUTSIDE the crate — cargo test from the published package cannot compile #4048): check_package_includes skips escapes in #[cfg(test)]-only module files. It touches scripts/check_package_includes.sh and scripts/lib/resolve_includes.py only.

release/0.69.1-batch-2 is an ancestor of rc/0.69.1-x, so merge 2 only adds the 5 hotfix commits. All four merges were conflict-free against main @ 49fe19c.

Pre-push checklist (re-run after merge 4, HEAD 0d16799)

check result
cargo fmt --all -- --check rc 0
cargo test -p aprender-contracts --lib 1701 passed, 0 failed
cargo deny check advisories advisories ok
cargo check --workspace rc 0
pmat roadmap aggregate --check roadmap == aggregate(167 fragments), idempotent
cargo test -p aprender-mcp 206 passed, 0 failed
cargo test -p aprender-serve --lib qwen35_known_issue 4 passed
bash scripts/check_model_ladder.sh --self-test 147 cases, 0 bad, rc 0
bash scripts/check_package_includes.sh --self-test SELF-TEST PASSED, rc 0

CI fixes made on this branch

file:line (before the insert) rule reason
scripts/check_crux_ollama_in_lock.sh:152 SEC001 eval is a literal printf argument (ollama's "eval count"/"eval rate" fixture), not an eval call
scripts/check_ladder_serve_teardown.sh:119 SEC001 evals a function body awk-extracted from this repo's own scripts/model_ladder.sh; no external input
scripts/check_ladder_serve_teardown.sh:191 SEC001 evals function bodies awk-extracted from this repo's own scripts/model_ladder.sh; no external input
scripts/check_moe_routes_through_one_dispatch.sh:183 SEC010 copies git-tracked sources into $T, a mktemp -d dir this script creates and removes on EXIT
scripts/check_publish_preflight.sh:630 SEC010 self-test fixture under the script's own mktemp -d dir
scripts/check_publish_preflight.sh:643 SEC010 self-test fixture under the script's own mktemp -d dir
scripts/crux_inference_dogfood.sh:628 SEC010 $WORK is the script's own mktemp -d dir, followed by a sha12 and a thinking mode
scripts/crux_inference_dogfood.sh:673 SEC010 under $WORK (mktemp -d), with a whitelisted verb
scripts/crux_sweep_shards.sh:54 SEC010 $OUT is the operator's required --out argument
scripts/model_ladder.sh:1358 SEC010 $OUT_DIR is the operator's --out, and RECEIPT_BASE is the host id from host_id

Verified after the fix:

  • check_bashrs_gate.sh and its --self-test PASS (412 files, 0 SEC/DET/IDEM).
  • The shell-lint ratchet PASSES.
  • check_model_ladder --self-test gives 147/0 and check_publish_preflight --selftest gives 47/47.
  • The three edited checks pass when run.

Note for the merger

🤖 Generated with Claude Code

noahgift and others added 30 commits September 23, 2026 10:41
…3957)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…952-crux-fold2

# Conflicts:
#	docs/roadmaps/roadmap.yaml
…d after merging F6

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ll has its bf16 row (#3962, #3957 F6)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…clusion removed — RED on 2 real defects

Stacked on feat/3950-iq2xxs-gemv-admission@472144f04 (PR #3969), because the
named exclusion `is_q4k_q6k_strategy` exists only there.

Widened guard, first run: 24/26 strategy kernels assemble and name-check clean. Two do not:
- Dp4aSIMDQ4KGemv: kernel_name() "dp4a_simd_q4k_gemv", PTX entry "dp4a_q4k_gemv"
  (lookup fails at launch; no executor caller found, only a proptest).
- FusedKVHwDp4aQ4KGemv: generate_ptx() returns "" — generate_q4k_gemv_ptx
  (kernels_generate_gemm_cuda.rs) has no arm, so unwrap_or_default. Executor
  q4k_mwv_gemv.rs:602 (fused_qkv_hw_dp4a_q4k_gemv_into) launches it.
PARKED for #3975 (cop reprioritized). Not yet fixed; must-RED plant not yet run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…uld not fail on the case it exists for

judge_thinking_on_output already made "no <think> at all" red ("the thinking mode this
leg exists to judge was never entered"), but a block that opened and closed with nothing
inside it went straight to the answer check. "Skipped the reasoning, answered" was scored
the same as "reasoned, answered". The #3907 cells prompted this: Qwen3.5-0.8B Q4_K_M
already had a closed block and an answer ('$2 +') at budget 8.

Now a closed block with only whitespace inside is a named FAIL, "closed EMPTY within N
tokens", distinct from never-entered, unclosed and wrong-answer.

Oracle: two planted empty blocks (bare, and whitespace-only) are must-RED; the
closed-with-content case stays green as the positive control. Mutant (guard disabled
with `false &&`) -> the test panics at the planted case, rc=101; restored under a trap.
49 golden/thinking lib tests pass; clippy -D warnings and fmt are clean.

Not measured: whether Q4_K_M really emits an empty block on the ON leg. That is the
clean-room re-run this fix makes meaningful.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 5103e07)
…cords per-engine think closure (#3962)

The hf and vLLM drivers split the think block off before writing `text`. An
unclosed block becomes `reasoning` plus an EMPTY text, which the oracle read
as no_answer_tag, never as a loop. The certifier now rebuilds the raw reply
from `reasoning` + `text`. The receipt gains think_closure, {model|prompt:
{leg:engine: closed|unclosed}}, so the judge can join the cop's ruling:
- every oracle leg unclosed means the model's behaviour at greedy (RED-MODEL, F9);
- the oracle closed while apr looped means an apr defect.

The oracle also treats a lone </think> (a template-prefilled <think>) as the
end of reasoning, so a tag drafted in the reasoning never counts. Both fixes
were planted out and went RED.

Refs #3962

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hen the sweep re-proves the cause (#3957 F9, F10)

Operator rulings 2026-09-23 on #3957: "proceed with (a)" (F9) and "c" (F10). A held file
may stay in the capability report as an explicit RED cell that does not block, but only as a
claim about CAUSE that THIS sweep re-proves. Otherwise the row is plain FAIL, so neither
verdict can become a deferral in disguise (the #3880 amnesty rule).

RED-MODEL (inventory.red_model, scripts/lib/model_ladder_redmodel.py). The evidence is the
CRUX receipts bound to the cut (cop ruling: one mechanism, not a second llama.cpp runner):
- the thinking-ON greedy ids of apr and llama.cpp on the IDENTICAL GGUF are equal, compared
  as raw lists by the judge (no `equal` flag is read);
- llama.cpp's text reproduces the declared defect (think_never_closed | think_empty);
- a sibling positive control closes with content, and its ON control cells are GREEN;
- apr CPU ids == GPU ids;
- optional bf16_reproduces: every hf/vLLM bf16 leg shows the defect too (the 2B candidate);
- the defect is the whole failure: the row with golden_output neutralised must be green;
- the key covers thinking ON only: the file's thinking-OFF CRUX cells must still be GREEN.

RED-UNSUPPORTED (inventory.red_unsupported): the header architecture matches the key, and
`apr run --gpu` was OBSERVED refusing by name (capability::no_cuda_forward_reason):
rc != 0, no fallback, 0 stdout bytes. A green row of that architecture refuses the key.
The producer now records the header architecture, the run's stdout bytes (stdout and stderr
are split) and the refusal line verbatim.

Key hygiene: a #NNNN ticket; a key matching no held file is refused (judged only when every
required receipt was accepted -- the real stale 0.69.1 receipts showed the false "matches
nothing" otherwise); a key matching a green row is STALE; one key per file.

Case table: 30 new cases (4 green, 26 must-RED, each varying one fact, generated by
gen_redmodel_cases.py) and 25 new mutants, each killed by the case that isolates its rule.
self-test: 113 case(s), 0 bad.

Refs #3957, #3951, #3948, #3977

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-think, OS-picked port (#3978)

`apr code` hardcoded `apr serve run … --gpu`, so it had no CPU lane and every CRUX
code cell on the cpu backend was a refusal. It also capped every request at 1024
tokens with no flag to pin it, and asked for port 19384 + pid%1000, which collides
across concurrent sessions.

- `--no-gpu` / `--gpu` (conflicting) choose the child's device; the default stays --gpu.
- `--max-tokens N` makes every request ask for exactly N, overriding the manifest
  and APR_AGENT_MAX_TOKENS_CAP.
- `--think off|on`: `on` is REFUSED by name before anything launches, because
  apr serve has no thinking-ON path (#3723) and a silently ignored flag would be a
  false pin.
- The port is reserved by binding 127.0.0.1:0. A child that loses the reservation
  race (address in use) is relaunched on a fresh port, up to 3 times.
- cmd_code keeps its signature and delegates to cmd_code_with(…, CodeServeOptions).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ning, and reports what it did (#3961, #3948)

#3961: the hybrid ON leg did `let (on_text, _used_gpu) = golden_output_runtime(..)`:
the longest generation the gate runs, the one most exposed to a mid-run fallback,
was judged on text alone, so a CPU-served answer passed as a GPU cell (#3922).
Now thinking_on_leg_failure() runs runtime_golden_backend() on the ON leg exactly
as on the OFF cases, and a pass records the ON leg's backend, think_body_chars
and generated_tokens (golden_output_runtime now returns generated_token_count).

#3948 (cherry-picked 5103e07) + its fold quorum's required changes
(do-not-implement-as-written, 2/2):
  1. single-token bypass: `<think>.</think>` passed. A closed block now needs
     MIN_THINK_WORDS = 3 alphanumeric words; 0 words keeps "closed EMPTY".
  2. RED-MODEL routing for Qwen3.5-0.8B-Q4_K_M: owned by #3957 F9 (aprender-36).
  3. contracts/thinking-budgets-v1.yaml records Qwen3.5-0.8B-Q4_K_M with its own
     measured basis (empty at 16/64/2048; llama.cpp agrees, #3951).
  4. the hybrid ON-leg failure carries `[budget basis - ...]` like the dense leg.
  5. tests for space/tab whitespace and single-character/one-token bodies.

Must-RED, run on the unfixed logic first (behaviour-preserving helper, then fix):
  an_on_leg_that_fell_back_is_red                               FAILED -> ok
  an_on_leg_failure_names_its_budget_basis                      FAILED -> ok
  a_think_block_of_one_character_or_one_token_is_not_reasoning  FAILED -> ok
Positive controls ok before and after. apr-cli thinking/golden/loud_truncation
tests: 52 passed; clippy -D warnings rc 0; fmt clean. The #3907 source-read
guard still finds both ON-leg call strings.

Refs #3961 #3948 #3957

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…al prints 178 bytes of verbose preamble (#3957 F10)

Measured on lambda, apr 0.69.1 (7b8aa7e), Qwen3.5-35B-A3B-UD-IQ4_XS, run --gpu --verbose:
rc 12, the refusal by name on stderr, and 178 stdout bytes, all four of them apr's own
`verbose:` preamble lines. A stdout-byte rule would have judged the REAL refusal RED.
The producer now records generated_bytes (stdout minus the verbose preamble) beside
stdout_bytes, and the judge reads generated_bytes. The fixture refusal text is the measured
line, verbatim.

Refs #3957

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…Dp4aSIMDQ4KGemv names the entry its PTX declares

Turns the #3970 widened guard (every_emitted_kernel_assembles +
the_name_the_launcher_looks_up_exists_in_the_ptx_it_compiles) GREEN.

1. FusedKVHwDp4aQ4KGemv: generate_ptx returned "" (the live generator,
   kernels_generate_gemm_cuda.rs, has no arm; the only arm lived in
   cuda/generate.rs, which nothing include!s). Its sole launcher,
   q4k_mwv_gemv.rs:602 fused_kv_hw_dp4a_q4k_gemv_into, is UNREACHABLE:
   #[deny(dead_code)] on it => rustc "method is never used" under
   --features cuda. The crate-wide allow(dead_code) (lib.rs:63) hid it.
   fused_qkv_hw_dp4a_q4k_gemv_into — the live QKV path (layers/rope.rs:111)
   — runs three q4k_gemv_into launches; the fused K+V phase was measured
   -3.1% and never wired. Deleted: the launcher, the KernelType variant,
   its kernel_name arm, the orphan arm, the guard-list entry. No verb,
   model or flag could reach it, so there is no runtime behaviour change
   and no device A/B to run.

2. Dp4aSIMDQ4KGemv: emits Dp4aQ4KGemvKernel PTX (entry dp4a_q4k_gemv)
   but kernel_name() said "dp4a_simd_q4k_gemv". Now "dp4a_q4k_gemv".
   No executor constructs it (tests only). ptxas sm_70 (guard) and
   sm_89 (with_target) assemble clean.

Must-RED (guard at 65baa3e + 2ae6b0a cherry-pick): 20 passed / 2
failed, exactly these two. After: 24 passed / 0 failed.

Refs #3976 #3970

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…22 tensor

The device A/B trusts dequantize_iq2_s as its oracle. Its comment cited 50 random blocks
(PMAT-3477), not the model the oracle is used on -- a gap the IQ3_XXS bar (#3963) exposed.

All five real type-22 tensors (blk.{8,9,10,17,21}.ffn_down), decoded by gguf-py
(llama.cpp @ df03399) and by dequantize_iq2_s, compared element-wise with the bytes read
by gguf-py's own reader: 18,350,080 values, EVERY ONE BITWISE IDENTICAL, max ulp 0.

The comparison script is the one that caught a planted decoder bug on Q2_K (#3960), so it
is shown able to fail. The probe is #[ignore] and one-shot; it names itself when it does
not run rather than passing silently.

Refs #3953, #3963, #3960

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ne .err narration)

Lanes: gemini-3.1-pro-high, gemini-3.8-flash-high and gemini-3.1-pro-low, measured from agy (gpt-oss exhausted;
Claude lanes refused for a Claude-authored diff). 3 distinct conversation ids, structured_output verdict
PASS x3, 0 sibling references, receipt-lint ok. The harness exits NOT AGREED because partial=true: every
lane's .err holds agy's own 'root agent idle; waiting ... background task(s)' narration (paiml-implement#311).
There is no dissent and no finding.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hat do (#3978, aligned with #3723)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ing it in gpu-q deadlocked the box (#3964)

gpu-q ends in `exec flock $LOCK choom -- <cmd>`, and the ladder's apr_locked does the
same, so a caller routinely HOLDS the fleet lock when it runs this script. The script then
did `exec 9>"$LOCK"; flock 9`: a new open file description, and no bound. flock(2) locks
conflict across descriptions, so the inner call waited FOREVER while its own ancestor held
the lock. Because that ancestor held the FLEET lock, every GPU job on the box stalled
behind it. The mechanism was proven on a private lock before anything was changed: outer
`flock $T`, inner `exec 9>$T; flock -w 3 9` -> blocked.

It also mis-read a legitimate holder. wait_empty samples the card WITHOUT the lock, so the
current lock-holder's own GPU use (dd's CRUX leg with Ollama resident inside its hold)
read as contention.

Now:
- If an ANCESTOR holds $LOCK, use that hold: skip the lockless wait and the re-take.
  Detected from /proc/locks, not an environment marker. A marker can be set by anyone; an
  ancestor's held flock cannot be faked. It's ANCESTRY, not "someone holds it": an
  unrelated holder is exactly the contention this script waits for.
- Otherwise take the lock with `flock -w "$WAIT"`. If ancestry detection ever regresses,
  that turns into a named 75 instead of a hang.
- Release only a lock we took.
- The foreign-process checks are unchanged. Holding the lock excuses no foreigner.

Self-test: 4 rows added, each bounded by `timeout 20` so a regression FAILS the table:
   9  caller already holds the lock -> proceeds, EXCLUSIVE, no deadlock   (f5's row)
  10  caller holds it, a foreigner arrives mid-run -> still CONTENDED
  11  caller holds it, card already occupied -> refused, command never runs
  12  lock held by a NON-ancestor -> not inherited; bounded wait, then refused
12/12 on the shipped script. Mutants:
  detection disabled                        -> 9/12, row 9 = 75 "not free" (the bound works)
  detection disabled + unbounded flock 9    -> rows time out at 124: the pre-fix deadlock,
                                               now caught as a failure instead of a hang
Also tested through gpu-q's EXACT chain (flock -> choom -> script, private lock, stub
card): rc 0 in 2 s, "held by ancestor pid N", EXCLUSIVE.

The table's summary counted a 0/1 flag, so it printed "11/12 rows" with three rows failing.
It now counts failures.

bashrs: 0 errors on both files.

Refs #3964

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…at every m, `gemm_bt` is [n,k] (#3975)

Quorum verdict (2/2, "alternative 2: explicit layout at the API").

CudaExecutor::gemm documented B as [k,n] but switched to the Gemv kernel at
m == 1, which reads [n,k]. So a [k,n] caller was wrong at m=1 (GpuModel: the
91.9% gpu_cpu_trace_compare divergence) and an [n,k] caller was wrong at m>1
(the batched cached path).

- gemm(a, b[k,n], ..): GemmTiled at every m. The m=1 Gemv shortcut is gone.
- gemm_bt(a, b[n,k], ..): m=1 uses the coalesced warp-reduce Gemv, unchanged,
  so decode is not slowed. m>1 uses a tiled A @ B^T kernel, KernelType::GemmBtTiled,
  emitted by trueno's GemmBackwardAKernel, whose grad_c @ B^T is the same product
  bound as (M,N,K) = (m,k,n). No runtime transpose of weights.
- CudaScheduler::matmul stays [k,n]; new CudaScheduler::matmul_bt is [n,k].
- Callers of dequantized [out,in] weights, routed by layout:
    cached/thread-safe.rs + cached/scheduler.rs  CUDA -> matmul_bt
    same, wgpu/CPU fallback + forward/acceleration.rs  -> HybridScheduler::matmul_transpose_b
  (HybridScheduler::matmul is [k,n] too; those fallbacks were wrong at every m.)
- fused_matmul_cuda deleted, not ported: nothing ever set cuda_executor to Some.
- test_imp_109b's "reference" was HybridScheduler::matmul on the same [out,in]
  weight, which agreed with the bug by construction. It is now a CPU oracle.

Acceptance, gpu_exclusive_run.sh on lambda (RTX 4090), card own-only:
the 5 must-REDs from 9ce6080 plus test_gemm_m1_uses_gemv [9.5, 4.75] are GREEN,
plus a tile-boundary sweep (m,k,n) in {(1,70,45),(2,70,45),(17,33,16),(37,70,45),
(64,128,96)} through both entry points. 41/41 across gemm_layout, cuda::executor::gemm,
test_imp_109b and gpu::scheduler::core. Without cuda: test_imp_109{a,b,c} 3/3.

LANE (jidoka J6): the tests moved to gguf::cuda::gemm_layout_tests_3975, the prefix
ci.yml's cuda-unit job (yoga, real GPU) selects: `--list` with its exact filter
selects 5 of 76. No workflow edit. batch_matmul_gpu_prefer_cuda and
get_cuda_scheduler are now pub(crate) so the tests can reach them from there.

Refs #3975 (PMAT-3975)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…med protocol faults, hf/vLLM unclosed think, per-model admission (#3957 F6)

Four joins the #3962 producers (aprender-19 code/serve, aprender-dd certification)
depend on; each was a wrong verdict on the judge before this commit (5 rows BROKE):
  R1  serve ROUTE is an additive key part: two routes of one prompt collided,
      and a wrong /api/chat hid behind a right /v1 route (rc 0, one GREEN cell).
  R2  `protocol_fault` is named ("protocol fault: stream_truncated"), not
      "stdout JSON has no text field".
  dd  hf/vLLM write an UNCLOSED think as `reasoning` + "": rebuilt the way
      crux_prompt_certify.driver_raw does, so it reads "unclosed think", not
      no_answer_tag.
  J2  `admitted_by_sha`: a prompt the certification did not admit for THIS model
      is RED on that cell, however right the answer.
Also merges aprender-dd feat/3962 @ fec2da7 (driver_raw, v2 drift check).
Table: 116 ok, 0 broke; 11 mutants, each killed by its NAMED row (two mutant
specs were wrong on the first run -- a non-matching sed and a must-name that
matched the ok-text -- and were fixed, not the judge).

Refs #3957 #3962

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…n-inherited-lock' into release/0.69.1-batch-2
…09/010 call_sites to the live generator

crates/aprender-serve/src/cuda/generate.rs was compiled by nothing. Proof:
appending `THIS IS NOT RUST @@@ {{{` to it left `cargo check -p
aprender-serve --features cuda --lib` AND `cargo test ... --lib --no-run`
at rc=0. The live generator is kernels_generate_gemm_cuda.rs
(include!d from cuda.rs). After deletion: cuda lib, cuda lib-tests and
no-cuda lib all build (rc=0).

The orphan was a shadowed artifact (CLAUDE.md #8): #3976's
FusedKVHwDp4aQ4KGemv arm lived ONLY there, which made the kernel look
generated. contracts/kernel-fusion-v1.yaml cited four call_sites inside
it, and those line numbers had drifted even there (202/267/270/273 pointed
at FusedResidualRmsNorm, two `},` and a DeltaRule arm). They now name
the arms in the live file (:250, :362, :365, :368). pv validate: 0 errors.
fusion_gate_contract_falsify passes (5/5), but it only checks that the
string "call_site:" occurs, never that the path exists.

Refs #3976

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…son}; NotMeasured never prints PASSED (#3973)

WIP: the full aprender-serve --features cuda suite is still running; the receipt
follows as a separate commit. What is ALREADY proven, on RTX 4090:
  both fail-open branches -> NotMeasured with a named reason (device test)
  positive control, a real prompt -> Validated, min cosine 0.9993
  must-RED, each planted separately, each RED on its own assertion:
    probe branch returns Validated   -> red
    cpu-ref branch returns Validated -> red
    status line says PASSED for NotMeasured -> red
  fmt 0; clippy -p aprender-serve --lib --features cuda -D warnings 0;
  cargo check -p apr-cli --features cuda 0.

API: validate_gpu_first_token now returns F2Outcome; f2_status_line is the ONE line
the run path prints. Routing is unchanged: only Mismatch leaves the GPU.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… — GPU-free (#3975)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ow is producible from the CLI (#3723, #3957 F9)

Cop decision on #3957 F9, option (1): keep the RED-MODEL proof strict ("apr = llama.cpp on the
identical GGUF, same token ids") and give apr a thinking toggle, so the thinking-ON row exists.
Measured by aprender-83: `apr run` had no think flag, realizar routed every Qwen3/3.5 to the
no-think template, and a pre-rendered ON prompt had its specials zero-width-escaped.

- realizar `chat_template::apply_thinking_mode(rendered, Option<bool>)`: `on` removes the
  trailing EMPTY `<think></think>` prefill of the rendering. This is the derivation `apr qa`'s
  golden ON leg already uses (`without_thinking_prefill`, #3724). `off` and absent leave the
  rendering byte-identical. `on` on a template with no thinking mode is REFUSED by name,
  never answered in OFF mode.
- `InferenceConfig.thinking` + `with_thinking`, applied at all three prepare_tokens sites
  (GGUF, SafeTensors, APR) after the template renders. The prompt is never pre-rendered.
- ONE `ThinkingArg` declaration (the BackendArg pattern), flattened into `apr run` and `apr
  chat`. Name and values are shared with `apr code` (#3978, aprender-19). `--batch-jsonl`
  refuses the flag by name rather than ignoring it.
- crux_inference_judge report_greedy: the greedy key gains `thinking`, and each engine's RAW
  record (generated_ids, generated_text, greedy, special, max_tokens) or its refusal is
  carried into the receipt. The ladder's F9 join compares the lists itself.

Must-RED unit cases: ON renders the thinking prompt, not the no-think one (a mutant mapping
ON to OFF fails `thinking_on_removes_the_empty_prefill`); OFF/absent still render no-think
byte for byte; ON on ChatML is refused; a non-empty or non-trailing block is not a prefill.
The clap cases: on/off/absent on run and chat; other values are rejected.

Not in this commit (#3723 done_when items 2 and 4): the serve per-request toggle, and a separate
`reasoning` field in --json.

Refs #3723, #3957, #3978

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
noahgift and others added 10 commits September 23, 2026 18:09
…: CRUX smoke on both hosts from the release binary

OPERATOR, verbatim, relayed by the release cop aprender-cf, 2026-09-23: "for this
particular release we need emergency surgery.  after a binary is build we really only
care about CRUX on gx10 and lambda (but "smoke test") style.  anything huge, must be
nightly only".

`check_model_ladder.sh --scope crux-smoke` judges CRUX smoke receipts INSTEAD of the
model ladder, under the contract entry `ladder.emergency_scopes` (name crux-smoke,
release "0.69.1", date, verbatim quote, hosts [lambda, gx10], thinking [off, on]).
scripts/lib/crux_smoke_scope.py requires: every named host has CRUX receipts; every
receipt is from the RELEASE BINARY (apr sha == the cut exactly, no equivalence) and did
not DECLINE; for every host x certified model (read from the prompt certification) x
thinking mode, at least one cell, every cell GREEN, and a GREEN positive control. It
prints "OPERATOR EMERGENCY SCOPE: CRUX smoke only" first and last, never "every rung
green"; the ladder is not run. Refused for any release other than 0.69.1.

Must-RED (hermetic table in --self-test): a host missing; a certified model missing on
a host; the positive control RED; no positive control; a non-control cell RED beside a
GREEN control; a receipt at a sha other than the cut; the scope used for 0.70.0. Green
row is the control. 7 mutants, each killed (red-cells-ok first SURVIVED -- the
control-RED row was also caught by the control rule; a non-control-RED row was added).
End to end through the gate with the real certification (3 models) and cut X2
d8a6df5: rc 0, 12 host x model x mode combinations checked; --version 0.70.0 -> RED.
self-test 147/0 + tables; pv ok.

Refs #3710 #3957

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…'s ADMITTED modes

Cop correction, 2026-09-23: "3 certified models x thinking OFF/ON" was the cop's
wording, not the operator's (the operator said only "CRUX on gx10 and lambda (but
"smoke test") style"). The smoke matrix is certified model x the modes the prompt
certification ADMITS for it (admitted_by_sha_thinking), printed on every run:
  on the real 0.69.1 certification: 4B-Q4_K_M -> off,on; 2B-Q4_K_M -> off (0 ON prompts
  admitted: its ON leg loops at greedy); 4B-UD-Q4_K_XL -> off,on  = 5 cells per host.

Must-RED (table): an ADMITTED mode missing on a host; a certified model with ZERO
admitted modes; an UNADMITTED mode's green cells counted toward the pass (they never
are). Control: an OFF-only model with no ON cells at all is GREEN. Mutants
zero-modes-ok and all-modes-counted killed (all-modes-counted first SURVIVED: every
fixture carried cells in both modes; the OFF-only row was added). A certification
without admitted_by_sha_thinking is RED. End to end with the real certification and
cut X2: rc 0, the matrix printed. self-test 147/0 + tables.

Refs #3710 #3957

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…y module FILES (#4048)

A module declared out of line under a test-only cfg (#[cfg(test)] mod guard;) is test code
though its own text has no cfg -- aprender-serve's fusion_call_site_guard_3985.rs. Its escaping
include_str! is SKIPPED by name on stderr (same mechanism as the wasm skip), transitively for
the modules it declares. cfg(any(test, ...)) modules and files with a test cfg on only some
items are still judged. Case table rows 14-18.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…the judge read a field no producer writes

Reported by aprender-3a (2026-09-23) on a real receipt: lambda's X2 shard
evidence/crux/0.69.1/d8a6df53a/shards/00fe7986ff5f-off/lambda-gpu.json
(32 cells GREEN) was judged "from apr sha None". crux_inference_dogfood.sh
writes only apr.version_line ("apr 0.69.1 (d8a6df5)") and a SHORT
harness.sha. apr_sha_of read apr.sha / apr_sha, a shape the producer never
emits. The case table passed only on self-authored fixtures.

apr_sha_of now:
- still honours an explicit apr.sha / apr_sha;
- otherwise parses `apr --version`'s line (harness.sha is the harness
  checkout, not the binary) and resolves the short sha in this checkout with
  git rev-parse --verify <short>^{commit}, rejecting anything unambiguous-
  failing;
- fails closed on a dirty or unparseable line;
- refuses a receipt where any cell's engines.apr.version names a different
  binary.
Fixes load_crux, crux_smoke_scope and model_ladder_redmodel alike (one helper).

New fixture rows copy the REAL receipt's apr/harness/cell shape: real-shaped
binds to the cut (green); another binary, an unresolvable short sha, a
-dirty line and a mixed-binary cell are each RED. Must-RED: 4587520's
module fails the real-shaped green row. Four binding mutants are each killed
by their named row (version-line-unread and short-sha-unresolved by the
green row, dirty-accepted by the dirty row, mixed-cells-ok by the mixed
row). Self-test 147 cases, 0 bad.

Real receipt, fixed judge: apr_sha_of -> d8a6df5;
lambda 00fe7986ff5f thinking=off: 32 CRUX cells GREEN, control GREEN.

Refs #3957

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 23, 2026

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=4046 head=6db770d2ffbb14afd1158ebd610f37a443c5a60e verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

@noahgift

Copy link
Copy Markdown
Contributor Author

Armed by the cop (aprender-cf) on an operator SELECTION in the cop session, among options the cop worded: "Arm as is" (2026-09-23). That includes the release branch's ci.yml step "The crates.io cascade can FINISH" (#3892, from 2f9fe27). A MERGE commit (not squash), so d8a6df5 stays an ancestor of main (publish preflight R4).

@noahgift
noahgift enabled auto-merge September 23, 2026 16:53
noahgift and others added 3 commits September 23, 2026 18:55
…(--scope crux-smoke)

The preflight asks the judge's own --scope path (scripts/lib/crux_smoke_scope.py, not re-implemented)
with the CUT (--cut-commit, default HEAD): R7 accepts CRUX-smoke receipts from the release binary in
place of the model matrix, prints OPERATOR EMERGENCY SCOPE, and keeps the model-matrix rows as
EVIDENCE only. It adds the one binding the judge cannot see: HEAD may differ from the cut only under
evidence/. The reader's must-REDs (another release, another binary's receipts, a host missing) are
proven by the judge's smoke table; 10 new preflight rows prove the wiring.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…sh merge queue)

main's merge queue squashes (repo ruleset), so a release cut is never an ancestor of main and R4 as
written could not pass. R4 now passes on ancestry OR when git diff HEAD <main> is empty over
crates/ src/ Cargo.toml Cargo.lock ('R4 cut content in main via squash <sha>'). Refused, naming the
paths: a crate file differing between the cut and main; a cut whose crate files are not in main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…Cargo.toml Cargo.lock), as R4 does

'HEAD differs from the cut only under evidence/' would refuse the real release: the tag sits on
main's squash of #4046, whose scripts/ and contracts/ (the scope's own reader and entry) arrive after
the cut. What must equal the cut is what cargo publishes. Row: tooling after the cut passes; a source
change still refuses (mutant 'no binding' killed by it).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
noahgift added a commit that referenced this pull request Sep 23, 2026
…he scope from main (--scope-tree); --via PR head

Cop ruling (b): tag and publish exactly the smoked cut. R4's content arm becomes containment: the
cut's published changes since merge-base(cut, main) reverse-apply to main's tree (throwaway index,
-C0: main moving on beside a change is expected), Cargo.lock at hunk level or SEMANTICALLY via
lock_contained.py over ladder_equiv's canonical delta (dedupes are not changes). Refused, by file:
a change missing on main, reverted by main, or existing only on the cut. --via SHA proves containment
through history when main edited over a cut line afterwards: HEAD is an ancestor of SHA and SHA's
changes are contained. --scope-tree reads the ruling from main's checkout (a cut predates its ruling);
that tree must be ON main. Self-test 56/56; mutants 7/7 killed. Simulation (main + squash of #4046,
ROOT = the cut): R4 ok via 0dccce2, R7 red ONLY for 'no CRUX receipt', R3/R5 the only other reds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rge drifted

workspace-test-shard (2/3) on #4046 refused at "Decide the test tier":
scripts/tree_reader_tests.txt had drifted from the sources. Derived-only entries
were apr-cli --lib commands and aprender-orchestrate --lib agent::driver::chat_template.
Regenerated with `bash scripts/check_tree_reader_tests.sh --update`, which passes.

The update also ledgers one NEW unwired reader that arrived with the release:
aprender-serve --test tokenizer_nonascii_4005 (9684a7a, #4005). No workflow
names it yet. It is ledgered deliberately rather than wired here, because a
merge-back should not take a CI-wiring decision. Wiring it is a follow-up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…', not as a format

guard-cargo on #4046 failed at check_shell_lint_ratchet.sh with "bashrs errors grew
6 -> 10": four SC2059 errors (a variable used as the printf format string) in the
#4048 self-test fixtures. gate needs guard-cargo, so #4046 could not pass.

$guard is a literal Rust line with a trailing \n and no '%', so printf '%b' "$guard"
writes the same bytes. The self-test still passes, and the ratchet is back at its
baseline of 6.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
noahgift added a commit that referenced this pull request Sep 23, 2026
…rmat (bashrs SC2059 x4)

Same bytes ($guard carries no '%'); the shell-lint ratchet grew 6 -> 10 on #4046 because of them.
aprender-6c applied the identical fix there. Shell-lint ratchet: 6, PASS; --self-test passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
noahgift added a commit that referenced this pull request Sep 23, 2026
…tion; carry #4046's '%b' fixture fix

The shell-lint ratchet read 11 on this branch: 4 SC2059 inherited from the base (fixed identically
on #4046 by aprender-6c) and 1 SC2107 of mine. Now 6, PASS; checks and self-tests green.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
noahgift added a commit that referenced this pull request Sep 23, 2026
…re the if), SEC011 (guarded cleanup); '%b' fixture fix

The ratchet read 12: 4 inherited SC2059 (fixed identically on #4046) plus my SC2135 and SEC011. The
KEEP_T debug hook that hid the rm -rf guard from bashrs is gone. Now 6, PASS; checks green.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
noahgift added a commit that referenced this pull request Sep 23, 2026
…yte-identical, sha256 9dea3704...c23fdd6)

The cut's own check_publish_preflight.sh (called with no args by cascade-publish.sh) predates the
recorded operator emergency scope, so its R7 would judge the stale model matrix. Cop ruling (aprender-cf,
2026-09-23, under the operator's emergency-scope order): the cascade runs with the two documented
seams -- PUBLISH_PREFLIGHT_MAIN_REF=<#4046 final head> (R4 on true ancestry) and
PUBLISH_PREFLIGHT_LADDER_JUDGE=<this file> -- and the announcement + GH release notes say R7 ran through
this seam under the emergency scope. The file refuses unless ROOT HEAD is the cut, then execs main's
judge with --scope crux-smoke --cut-commit --crux, certification from the scope tree. Proven on the
simulation: the cut's own preflight -> R4 ok, R7 ok; only R3/R5 red pre-tag.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts bring to main's gate

guard-cargo on #4046 failed "bashrs-gate: FAIL 10 SEC/DET/IDEM error(s)". Neither side
was red alone: main's check_bashrs_gate.sh (--no-ignore) had never scanned the
release branch's scripts. Every finding was read in context and is a false positive
for its rule:
- SEC001 x1: 'eval' as a literal printf argument (ollama's "eval count" fixture).
- SEC001 x2: evals function bodies awk-extracted from this repo's own model_ladder.sh.
- SEC010 x7: paths under the script's own mktemp -d dir, or the operator's --out.

Each gets the per-line `# bashrs disable-next-line=<rule>` directive (honoured under
--no-ignore), with the reason on its own line above; a trailing reason voids the
directive. Comments only. check_bashrs_gate --self-test and the gate itself pass
(412 files, 0 SEC/DET/IDEM), and so do the shell-lint ratchet, the ladder self-test
147/0, the preflight self-test 47/47, and the three edited check scripts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant