Correct the keccak gate and prove bound necessity - #950
Draft
jotabulacios wants to merge 11 commits into
Draft
Conversation
Configuration D of the rho necessity analysis claims the lane's output is completely free: with neither range check present, the cyclic system in `right` solves for an arbitrary target. The check backing that claim only asked whether the shift identity holds for the constructed pair - and `L[j]` is derived FROM the identity, so it held by construction and could not fail. Nothing verified the target was reached. With the target now compared, it turns out it was not. The recurrence indexed `c[(j+1)%4]` while the closed form for `r0` directly above it solves `R[j-1] - 2**16*R[j] = c[j]`, so the two disagreed by one and the construction hit its intended pi halfwords on 0 of 25 lanes. Indexed `c[j]`, which is what `r0` was derived for, the lanes hit their target 25 of 25. The verdict for configuration D is unchanged - the system is solvable, the output is free - but the exhibited construction now demonstrates it.
The board's three "sound - implied" cells were backed by `check(2**16 > MASK16)`: a comparison of two constants, true whatever the chip does. And the argument it stood for does not cover the two configurations where the column left unchecked is `right`, because there the deviation is not a small integer at all - over the field, any `left` in [0, 2**16) admits `right = (in*2**rnc - left)*inv(2**16)`, a full-size field element. What closes those configurations is the ByteAlu operand that reads the unchecked column. The BITWISE table holds byte rows only, so `checked_byte + this` in [0, 255] confines `this` to [-255, 255]; every term of the identity then stays under 2**33, the field identity IS the integer identity, and Euclidean division is unique. That chain is now in the model: `operand_summand_window` derives the window, `difference_form_is_exact` checks the magnitude step the difference form had silently assumed, and `surviving_deviation` sweeps all 2**16 input halfwords against one interval per column. It replaces both the constant comparisons and the four hand-picked inputs that stood in for theta's configuration B, and it asserts the honest pair lies inside the modelled intervals - the failure that would make a "pinned" verdict meaningless. The windows rest on each column being read by exactly ONE operand byte, so `combinatorics.py` grows the theta analogue of that premise: the four carries are a permutation of the four rotated_C low bytes. It is exposed as `premises()` and imported by both necessity scripts, rather than being a file whose docstring asks you to run it first. Verdicts are unchanged, but each is now the output of a sweep that fails when its inputs do: widening the left interval to a halfword and a bit, or dropping rot_left's check, both produce survivors.
`witness_fullchip.py` asserted that at least one output lane differs from the reference - which is also what a bug in its own hand transcription of the round produces. Both failure modes were reachable: making Dxz read column (x+1) instead of (x+4), or dropping chi's NOT, left the script printing FULL-CHIP WITNESS VERIFIED while reporting 12 wrong lanes instead of 2. That is the fail-open direction discipline 1 of the README calls the only dangerous one. Honest and forged rows now come out of one `build_row(tamper=...)`, and the honest row must be EXACTLY the reference - cross-checked against keccak_ref and against model_dataflow's mirror, the one test_dataflow.py validates - before any claim about the forged row is made. Both mutations above now fail that control. The forgery is also no longer demonstrated on a lane picked by list order, which happened to be (0,0): the one lane with RHO = 0, no rotation at all. All 11 saturated lanes are forged in turn, 10 of them with a non-zero rotation, and the wrong output lanes must be a subset of the lanes that can move - the single pi lane reading the forged source, itself read by three chi lanes - which pins where the forgery leaks instead of only counting lanes.
Nothing in the repo ran anything under formal_verification/: no Makefile target, no workflow. The gate's only automated guard was the Rust digest test, and its documented remedy - re-run the directory, then update the digests - is satisfied by pasting the new constant without running anything. The daily LOC report meanwhile counts the directory as its own "formal verification" section, so those lines are reported under that heading with nothing executing them. `make verify-keccak` runs the half that needs no solver: the FIPS-202 reference anchors, the concrete mirror, the combinatorial premises, both necessity boards and the full-chip witness. About three seconds. A new workflow runs it on every pull request that touches the directory, on any base branch, so stacked PRs are covered too. The QF-BV gate itself stays manual - it needs z3's Python bindings and about three minutes - exactly as the directory README documents.
The README promoted the necessity table to the headline result without saying how a cell is decided, which left the two "implied" rows reading as bare assertions. It now names the mechanism - the operand window, the integrality step, the complete sweep - and says which single configuration has no per-column window and is therefore decided by a witness instead. It also records what the two implied halves are worth, because they are not the same kind of saving: rho's is 100 AreBytes sends, theta's is 20 degree-3 polynomial constraints, which are the reason this AIR declares max_degree 3. Neither is proposed as an optimization here. The file list and the run instructions follow the code: `combinatorics.py` carries the theta premise and is imported rather than run by hand, the necessity scripts sweep rather than sample, the witness has a positive control, and `make verify-keccak` is the one command CI runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The gate's line citations are its only audit trail into the Rust. The README states
that faithfulness of the model to the Rust is a human obligation, and the citations
are how someone discharges it. All of them are wrong.
rs:539-588("theta: Cxz XORchain") lands on the KeccakRc sender,
rs:796-870("chi: AND then XOR") lands onIota, and
rs:593-631/rs:723-766nameBusInteraction::sender(BusId::Hwsl, …)blocks that #889 deleted outright; the file now contains zero.
test_ref.pycitesexecution.rs:646-680forKECCAK_RC/KECCAK_RHO, which sit at 782 and 810. Theywere moved by #876, an unrelated hint-ecall PR that grew the file by 136 lines, so
the cited range is now
ArithOp::Divand the citation was stale three days beforethis branch opened.
The README also declares a scope gap as its first follow-up: QF-BV cannot test
whether the
AreBytes/IS_BITbounds are sufficient modp. It cannot be closedin the same model. The bounds are carried as the width of the bitvectors, and mod
2^nthe factor2^16is a zero divisor, so even a widened model would report thedecomposition pinned. Mod the Goldilocks prime
2^16is invertible.Separately, the AIR's count tests (1031 interactions, 1480 columns, 140 constraints)
catch anything that adds or removes one of those, but not a rewiring that keeps them.
Changing the first
BusValueof the ρARE_BYTESpair fromcols::rot_lefttocols::rot_rightleaves all three counts untouched, is satisfied by every honesttrace, and makes ρ forgeable.
Description
Cites each modeled equation by the construct it names — the
// --- <Group>: … ---banner title, or thecols::/KeccakRndConstraintssymbol — instead of a line number. Names survive the churn that broke these from
two unrelated directions. No executable change; the AST of the three Python files
is identical modulo docstrings, and the board is unchanged.
Drops the README section that catalogued where the stale numbers really pointed,
since with the citations fixed it no longer describes anything.
Adds
field_model.py, the companion integer-mod-pmodel of the inline θ/ρidentities, with a switch per range check.
in · 2^rnc = right · 2^16 + leftpinsnothing on its own over a field, since for any
leftthere is exactly oneright.What the range checks do is rule out the deviation
(L, R) → (L − 2^16·d, R + d),and they do not do it equally:
Cxz_left/Cxz_right)rot_left/rot_right)leftrange checkrightrange checkleftandrightenter with weights1and2^16. Boundingleftkills thedeviation outright; bounding
rightonly narrows it. In θ the residual window isclosed by the parity of
left, the shift being by one, and by the carry being asingle bit. In ρ,
rightis a halfword andd = ±1fits exactly at saturation,which
theta = 0xFFFFFFFFFFFFFFFFreaches on all 25 lanes.Adds
combinatorics.py, which checks the premises the ρ result rests on, none ofwhich had been checked: π is a bijection on the lanes, all 400
rot_left/rot_rightbyte columns are read exactly once by a pi operand, and thepi offsets are even so a pi halfword reads a single source halfword.
Adds
witness_fullchip.py, which builds the ρ forgery as a whole KECCAK_RND rowfrom a reachable message state instead of a lane in isolation: all 140 constraints
and every ByteAlu operand satisfied, two of twenty-five output lanes differing from
FIPS-202.
Replaces the README's "both are load-bearing" with the table above, and states that
the
24/24 UNSATverdict is conditional on the range checks existing.Adds
test_keccak_rnd_air_structure_is_pinned, which digests the bus wiring (ids,multiplicities, column indices, linear coefficients) and the captured constraint IR
(ops, dimensions, field constants, roots). FNV-1a is written out because
std'shasher is not stable across toolchains. When it fails the wiring changed, and the
directory has to be re-run before the digests are updated.
#889 replaced the HWSL lookups with these identities and dropped 120 sends per row.
Under the lookup,
leftandrightwere pinned individually and dropping eithercheck was harmless; under the identity, dropping
rot_left's is forgeable. The 100saved ρ sends were paid for with a range check that changed status, and nothing
outside this directory records it.
Tests
combinatorics.py,necessity_theta.py,necessity_rho.pyandwitness_fullchip.pypass. The sound results need no solver: they are exactinteger-bounding arguments over finite, complete enumerations, so they run in
seconds.
test_ref.py,test_dataflow.py,z3_parallel.py(24/24 UNSAT, 5/5 negative controls SAT, positive control PASS),tamper_test.py(clean UNSAT, 4/4 tampers SAT).test_keccak_rnd_air_structure_is_pinnedwas checked against the tamper it existsfor. With
cols::rot_left→cols::rot_rightapplied to the ρARE_BYTESpair,test_keccak_bus_interaction_counts,test_keccak_column_counts,test_keccak_constraint_counts,test_pi_virtual_matches_rotateandtest_keccak_core_round_state_consistencyall pass, and only the digest fails.cargo fmt --checkandcargo clippy --all-targets -D warningsclean.