feat(prover): chunk the accelerator tables and project them for storage - #919
feat(prover): chunk the accelerator tables and project them for storage#919Oppen wants to merge 2 commits into
Conversation
KECCAK, KECCAK_RND, ECSM, ECDAS, HINT and COMMIT had no max_rows entry: each one padded its whole op list into a single table, so a keccak- or ECSM-heavy epoch built one table whose height is proportional to guest data. At epoch_size_log2=23 that is a single multi-GB allocation no storage mode can stream. KECCAK, KECCAK_RND, ECSM and ECDAS were also absent from auto_storage's projection entirely, so the Ram/Disk decision under-projected exactly the programs most likely to need Disk. Every one of these chips evaluates row-locally (no `main(1, ..)` reference), so their rows split across tables the way the core chips' do, and the buses they drive are a multiset argument that does not care which table a row sits in. KECCAK_RND splits on whole permutations, since one call is 24 contiguous rows. The limits follow the existing effective-width model, and `accelerator_max_rows_track_effective_width` pins each width to the AIR so a new column or bus cannot leave a limit stale. Breaking: the six tables move out of FIXED_TABLE_COUNT (11 -> 5) into TableCounts, which changes the sub-proof layout and the statement absorbed into the transcript (tags bumped to V4/V3). Prover and verifier must be deployed together; earlier binaries cannot verify these proofs.
…fied - Drift tests only ran guests with zero keccak/ECSM calls, so the new projection formulas were exercised at count 0 only. `count_table_lengths_matches_keccak_trace` runs a three-permutation guest, making KECCAK and KECCAK_RND non-empty. - The chunking test stopped at trace shape; a split table's buses only matter once proved. It now proves and verifies the multi-chunk trace. - The ECDAS per-call bound backs a storage projection, so exceeding it must fail in release too: debug_assert -> assert with the offending count. - Revert CONTINUATION_GLOBAL_TAG to V2: the global statement absorbs no table counts, so this change does not alter it.
|
Automated review pass, with measurements. Run on The description undersells this: it's a prove-time win, not just a memory bound. Real ethrex block,
Recursion step (
The time win is 3× bigger in recursion than on a block, and the keccak count is why: the block
Chunking soundness checked and came back clean: none of the six |
What
The six data-proportional accelerator chips — KECCAK, KECCAK_RND, ECSM, ECDAS, HINT,
COMMIT — get
max_rowslimits and chunking, and all of them are projected inauto_storage.Why
Split out of the #874 review, where the DMA table was flagged for having no
max_rowsentry. The gap is the whole accelerator family, not DMA:
n.next_power_of_two().max(4)),so table height is proportional to guest data with no ceiling. At
epoch_size_log2 = 23a keccak- or ECSM-heavy epoch is a single multi-GB table — and a single table is the one
thing no storage mode can stream:
disk-spillbounds memory per chunk, and there wasonly ever one chunk.
auto_storage::table_specsatall, so the Ram/Disk estimate under-projected precisely the programs most likely to
need Disk. (COMMIT was already projected.)
Why chunking is sound here
Every one of these
ConstraintSets is row-local — none referencesmain(1, ..)— so noconstraint spans a chunk boundary, and the buses they drive are a LogUp multiset argument
that is indifferent to which table a row sits in. KECCAK_RND splits on whole permutations:
one call is 24 contiguous rows (
ROUNDS_PER_OP), and its limit divides by that.Limits
Same effective-width model as the core chips (
main_cols + 3 × buses, MEMW's 127 @ 2^19 asthe baseline):
accelerator_max_rows_track_effective_widthpins each width to the AIR, so adding a columnor a bus to one of these chips fails a test instead of silently leaving its limit stale.
MaxRowsConfig::small()deliberately keeps the production values for these six: shrinkingthem to 2^5 splits a committed output or one ECSM ladder into dozens of sub-proofs and
slows every test that uses it. The chunking test shrinks them itself.
Projection
TableLengthsgainskeccak/keccak_rnd/ecsm/ecdas/hintpadded-row counts;count_table_lengthscounts the ecalls andtable_specsincludes all of them plusKECCAK_RC. ECDAS is an upper bound (
ecdas::MAX_STEPS_PER_ECSM, ≤2 ladder steps per scalarbit), in the same spirit as the existing LT/MUL/DVRM/BRANCH bounds — the drift test asserts
>=for it and exact equality for the other four.Breaking
The six tables move out of
FIXED_TABLE_COUNT(11 → 5) intoTableCounts. That changesthe sub-proof layout and the statement absorbed into the transcript, so the domain tags are
bumped (
STATEMENT_V3→V4, continuation epoch/globalV2→V3). Prover and verifiermust ship together; earlier binaries cannot verify these proofs. #874 and #876 also touch
FIXED_TABLE_COUNT, so whichever lands second rebases.Validation
make test-prover: 560 passed, 0 failed, 24 ignored.make lint: clean (all four clippy passes, includingdebug-checks,disk-spill,cuda).accelerator_chunking_tests: effective widths pinned; a 3-call keccak guest at onecall per chunk produces one KECCAK and one KECCAK_RND chunk per call, each padded to 32
rows; default limits keep a small program at one chunk per accelerator.
make testadditionally fails on this machine in-p executor --libwithdyld: symbol not found in flat namespace '__end'— a link/env failure in a crate thisPR does not touch (running the prebuilt
executortest binary directly reproduces it,and the binary contains no prover code). Not investigated here.
Not in scope
Epoch sizing is still cycle-count-only (
resume_with_limit), so an epoch's row budget canstill be blown with no cycle-count change. Bounding that needs the executor to stop an
epoch on a weighted row budget, which breaks the "intermediate epoch = exactly 2^k cycles"
invariant (
continuation.rs:1327) — deliberately left out.