Reproducible 202609 chain, step 8: tools/verify_release.py — hash a release, re-verify any host or mirror against it - #356
Merged
Conversation
… or mirror against it (chain step 8) hash: every .parquet/.json under a dir → release_hashes.json (bytes + sha256). check: HEAD size fast-fail then streamed sha256 per file against --base URL or --dir; --only/--skip-prefix (skipped is reported, never counted as verified); JSON --report; exit 0 only when every listed file matches. Explicit User-Agent (Cloudflare 403s Python-urllib). Tested: 16 local files verified; live vocab_labels_202608 verified against data.isamples.org; tampered sha256 and wrong size both FAIL with exit 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy
…otherwise); refuse redirects; URL-encode + validate paths; contain --dir; strict GET framing; complete hash mode (Codex round 1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy
…ete walks, strict manifest typing, canonical paths (Codex round 2) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy
…lace), commonpath containment, non-empty release id, 'matched when read' contract (Codex round 3) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy
…links), report written before the verdict and its failure is FAILED (Codex round 4) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy
Contributor
Author
|
🤖 rbotyee — Codex round 6 (final) verdict, verbatim: No P1/P2 findings. Verdict: LGTM Round 5’s issue is resolved:
Offline checks passed: syntax compilation, local hash/check smoke tests, and |
rdhyee
added
the
CC+Codex+LGTM
Contributor
Author
|
🤖 rbotyee — staging check before merge: fork Pages run 33207335311 on this branch built green; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: keep going on Part B under the standing rule ("keep going as long as you get CC+Codex+LGTM, deployment to rdhyee looks good"). What I did: wrote the release verifier the plan's step 8 calls for, took it through six Codex rounds, and tested each verdict path against local files and one live production file. No published file changes.
What it is
tools/verify_release.pyhash --dir <release dir> --release-id <id> --out <manifest.json>— every.parquet/.json(except*.manifest.jsonbuild sidecars) with bytes + SHA-256. Complete for a quiescent directory: no filters, symlinked entries and unreadable subtrees are errors, not omissions.check --manifest <manifest.json> (--base https://host | --dir <mirror>) [--only glob] [--skip-prefix p] [--report r.json]— re-reads every listed file and compares size + SHA-256.--allow-partial) when a filter left files unchecked; FAILED (exit 1) on any mismatch, missing file, or operational error (including a report that can't be written).Accept-Encoding: identityand anyContent-Encodingfails the file; HEAD size fast-fail; GET status/framing checked; short bodies fail; paths percent-encoded per segment; manifest strictly validated (schema tag, counts, path grammar, types);--dircontainment; destinations (--out,--report) may not sit inside the tree they describe, be symlinks, or hard-link a listed file; atomic writes.The #334 release manifest (size/ETag) stays the Explorer's boot-time cross-check; this is the byte-level truth for a release, usable against R2, a mirror, or an unpacked Zenodo deposit.
Evidence
16 local 202609 derived files hashed and re-verified; live
vocab_labels_202608.parquetVERIFIED againstdata.isamples.org; tampered hash → FAILED/1; wrong size → fast FAILED/1 (no body fetched);--only→ PARTIAL/3;../x.json,sub/./x.json, boolfile_count, fragment-bearing--base, symlinked subdir, hard-linked--report/--out, unwritable report dir — each rejected as specified. Requires an explicit User-Agent (Cloudflare 403s Python-urllib).Review
Codex ×6, each asked to refute "VERIFIED means every listed byte matches": R1 changes-required (partial→VERIFIED; unencoded URLs; redirects; containment; framing;
hash --only) → R2 (report inside target; walk completeness; fragment base; typing; canonical paths) → R3 (hard links; contract wording) → R4 (destination symlinks; report-write ordering) → R5 (stdout manifest) → R6 LGTM. Verdict posted below.🤖 Generated with Claude Code
https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy