Skip to content

Correct the keccak gate and prove bound necessity - #950

Draft
jotabulacios wants to merge 11 commits into
docs-keccak-formal-verification-baselinefrom
fix/keccak-gate-bound-necessity
Draft

Correct the keccak gate and prove bound necessity#950
jotabulacios wants to merge 11 commits into
docs-keccak-formal-verification-baselinefrom
fix/keccak-gate-bound-necessity

Conversation

@jotabulacios

@jotabulacios jotabulacios commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Motivation

The gate's line citations are its only audit trail into the Rust. The README states
that faithfulness of the model to the Rust is a human obligation, and the citations
are how someone discharges it. All of them are wrong. rs:539-588 ("theta: Cxz XOR
chain") lands on the KeccakRc sender, rs:796-870 ("chi: AND then XOR") lands on
Iota, and rs:593-631/rs:723-766 name BusInteraction::sender(BusId::Hwsl, …)
blocks that #889 deleted outright; the file now contains zero. test_ref.py cites
execution.rs:646-680 for KECCAK_RC/KECCAK_RHO, which sit at 782 and 810. They
were moved by #876, an unrelated hint-ecall PR that grew the file by 136 lines, so
the cited range is now ArithOp::Div and the citation was stale three days before
this branch opened.

The README also declares a scope gap as its first follow-up: QF-BV cannot test
whether the AreBytes/IS_BIT bounds are sufficient mod p. It cannot be closed
in the same model. The bounds are carried as the width of the bitvectors, and mod
2^n the factor 2^16 is a zero divisor, so even a widened model would report the
decomposition pinned. Mod the Goldilocks prime 2^16 is invertible.

Separately, the AIR's count tests (1031 interactions, 1480 columns, 140 constraints)
catch anything that adds or removes one of those, but not a rewiring that keeps them.
Changing the first BusValue of the ρ ARE_BYTES pair from cols::rot_left to
cols::rot_right leaves all three counts untouched, is satisfied by every honest
trace, and makes ρ forgeable.

Description

  • Cites each modeled equation by the construct it names — the
    // --- <Group>: … --- banner title, or the cols:: / KeccakRndConstraints
    symbol — instead of a line number. Names survive the churn that broke these from
    two unrelated directions. No executable change; the AST of the three Python files
    is identical modulo docstrings, and the board is unchanged.

  • Drops the README section that catalogued where the stale numbers really pointed,
    since with the citations fixed it no longer describes anything.

  • Adds field_model.py, the companion integer-mod-p model of the inline θ/ρ
    identities, with a switch per range check. in · 2^rnc = right · 2^16 + left pins
    nothing on its own over a field, since for any left there is exactly one right.
    What the range checks do is rule out the deviation (L, R) → (L − 2^16·d, R + d),
    and they do not do it equally:

    dropped θ (Cxz_left/Cxz_right) ρ (rot_left/rot_right)
    the left range check sound, implied forgeable
    the right range check sound, implied sound, implied
    both forgeable forgeable, output entirely free

    left and right enter with weights 1 and 2^16. Bounding left kills the
    deviation outright; bounding right only narrows it. In θ the residual window is
    closed by the parity of left, the shift being by one, and by the carry being a
    single bit. In ρ, right is a halfword and d = ±1 fits exactly at saturation,
    which theta = 0xFFFFFFFFFFFFFFFF reaches on all 25 lanes.

  • Adds combinatorics.py, which checks the premises the ρ result rests on, none of
    which had been checked: π is a bijection on the lanes, all 400
    rot_left/rot_right byte columns are read exactly once by a pi operand, and the
    pi offsets are even so a pi halfword reads a single source halfword.

  • Adds witness_fullchip.py, which builds the ρ forgery as a whole KECCAK_RND row
    from a reachable message state instead of a lane in isolation: all 140 constraints
    and every ByteAlu operand satisfied, two of twenty-five output lanes differing from
    FIPS-202.

  • Replaces the README's "both are load-bearing" with the table above, and states that
    the 24/24 UNSAT verdict is conditional on the range checks existing.

  • Adds test_keccak_rnd_air_structure_is_pinned, which digests the bus wiring (ids,
    multiplicities, column indices, linear coefficients) and the captured constraint IR
    (ops, dimensions, field constants, roots). FNV-1a is written out because std's
    hasher is not stable across toolchains. When it fails the wiring changed, and the
    directory has to be re-run before the digests are updated.

#889 replaced the HWSL lookups with these identities and dropped 120 sends per row.
Under the lookup, left and right were pinned individually and dropping either
check was harmless; under the identity, dropping rot_left's is forgeable. The 100
saved ρ sends were paid for with a range check that changed status, and nothing
outside this directory records it.

Tests

  • combinatorics.py, necessity_theta.py, necessity_rho.py and
    witness_fullchip.py pass. The sound results need no solver: they are exact
    integer-bounding arguments over finite, complete enumerations, so they run in
    seconds.
  • The existing board is unchanged: test_ref.py, test_dataflow.py,
    z3_parallel.py (24/24 UNSAT, 5/5 negative controls SAT, positive control PASS),
    tamper_test.py (clean UNSAT, 4/4 tampers SAT).
  • test_keccak_rnd_air_structure_is_pinned was checked against the tamper it exists
    for. With cols::rot_leftcols::rot_right applied to the ρ ARE_BYTES pair,
    test_keccak_bus_interaction_counts, test_keccak_column_counts,
    test_keccak_constraint_counts, test_pi_virtual_matches_rotate and
    test_keccak_core_round_state_consistency all pass, and only the digest fails.
  • cargo fmt --check and cargo clippy --all-targets -D warnings clean.

Configuration D of the rho necessity analysis claims the lane's output is
completely free: with neither range check present, the cyclic system in
`right` solves for an arbitrary target. The check backing that claim only
asked whether the shift identity holds for the constructed pair - and `L[j]`
is derived FROM the identity, so it held by construction and could not fail.
Nothing verified the target was reached.

With the target now compared, it turns out it was not. The recurrence indexed
`c[(j+1)%4]` while the closed form for `r0` directly above it solves
`R[j-1] - 2**16*R[j] = c[j]`, so the two disagreed by one and the
construction hit its intended pi halfwords on 0 of 25 lanes. Indexed `c[j]`,
which is what `r0` was derived for, the lanes hit their target 25 of 25.

The verdict for configuration D is unchanged - the system is solvable, the
output is free - but the exhibited construction now demonstrates it.
The board's three "sound - implied" cells were backed by
`check(2**16 > MASK16)`: a comparison of two constants, true whatever the
chip does. And the argument it stood for does not cover the two
configurations where the column left unchecked is `right`, because there the
deviation is not a small integer at all - over the field, any `left` in
[0, 2**16) admits `right = (in*2**rnc - left)*inv(2**16)`, a full-size field
element.

What closes those configurations is the ByteAlu operand that reads the
unchecked column. The BITWISE table holds byte rows only, so
`checked_byte + this` in [0, 255] confines `this` to [-255, 255]; every term
of the identity then stays under 2**33, the field identity IS the integer
identity, and Euclidean division is unique. That chain is now in the model:
`operand_summand_window` derives the window, `difference_form_is_exact`
checks the magnitude step the difference form had silently assumed, and
`surviving_deviation` sweeps all 2**16 input halfwords against one interval
per column. It replaces both the constant comparisons and the four
hand-picked inputs that stood in for theta's configuration B, and it asserts
the honest pair lies inside the modelled intervals - the failure that would
make a "pinned" verdict meaningless.

The windows rest on each column being read by exactly ONE operand byte, so
`combinatorics.py` grows the theta analogue of that premise: the four carries
are a permutation of the four rotated_C low bytes. It is exposed as
`premises()` and imported by both necessity scripts, rather than being a file
whose docstring asks you to run it first.

Verdicts are unchanged, but each is now the output of a sweep that fails when
its inputs do: widening the left interval to a halfword and a bit, or
dropping rot_left's check, both produce survivors.
`witness_fullchip.py` asserted that at least one output lane differs from the
reference - which is also what a bug in its own hand transcription of the
round produces. Both failure modes were reachable: making Dxz read column
(x+1) instead of (x+4), or dropping chi's NOT, left the script printing
FULL-CHIP WITNESS VERIFIED while reporting 12 wrong lanes instead of 2. That
is the fail-open direction discipline 1 of the README calls the only
dangerous one.

Honest and forged rows now come out of one `build_row(tamper=...)`, and the
honest row must be EXACTLY the reference - cross-checked against keccak_ref
and against model_dataflow's mirror, the one test_dataflow.py validates -
before any claim about the forged row is made. Both mutations above now fail
that control.

The forgery is also no longer demonstrated on a lane picked by list order,
which happened to be (0,0): the one lane with RHO = 0, no rotation at all.
All 11 saturated lanes are forged in turn, 10 of them with a non-zero
rotation, and the wrong output lanes must be a subset of the lanes that can
move - the single pi lane reading the forged source, itself read by three chi
lanes - which pins where the forgery leaks instead of only counting lanes.
Nothing in the repo ran anything under formal_verification/: no Makefile
target, no workflow. The gate's only automated guard was the Rust digest
test, and its documented remedy - re-run the directory, then update the
digests - is satisfied by pasting the new constant without running anything.
The daily LOC report meanwhile counts the directory as its own "formal
verification" section, so those lines are reported under that heading with
nothing executing them.

`make verify-keccak` runs the half that needs no solver: the FIPS-202
reference anchors, the concrete mirror, the combinatorial premises, both
necessity boards and the full-chip witness. About three seconds. A new
workflow runs it on every pull request that touches the directory, on any
base branch, so stacked PRs are covered too.

The QF-BV gate itself stays manual - it needs z3's Python bindings and about
three minutes - exactly as the directory README documents.
The README promoted the necessity table to the headline result without saying
how a cell is decided, which left the two "implied" rows reading as bare
assertions. It now names the mechanism - the operand window, the integrality
step, the complete sweep - and says which single configuration has no
per-column window and is therefore decided by a witness instead.

It also records what the two implied halves are worth, because they are not
the same kind of saving: rho's is 100 AreBytes sends, theta's is 20 degree-3
polynomial constraints, which are the reason this AIR declares max_degree 3.
Neither is proposed as an optimization here.

The file list and the run instructions follow the code: `combinatorics.py`
carries the theta premise and is imported rather than run by hand, the
necessity scripts sweep rather than sample, the witness has a positive
control, and `make verify-keccak` is the one command CI runs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant