ci: resume snapshot from batch-archive and publish with a patch tag - #15
Merged
Conversation
Adds a manually-triggered Batch Sync workflow exposing every batch-export flag as a typed input: start/end block, max requests per second, block range limit, output path, compression and verbosity. The job runs on the self-hosted bee runners, whose egress addresses are inside the RPC provider's allowlist, so no credential is needed there. If it is ever moved to a hosted runner the endpoint requires HTTP Basic auth, so the Compose endpoint step builds an authenticated URL when a user and password are available and falls back to the bare URL when they are not. The composed value is masked and passed through the environment rather than echoed, and the endpoint itself comes from a secret so no infrastructure hostname is committed here. The final sync step is deliberately a failing TODO: the export half is runnable, but the publish destination has not been decided yet.
Replaces the skeleton's failing Sync TODO with the real flow: check out ethersphere/batch-archive, resume the export from the snapshot at a selectable tag (latest semver tag by default), and push the refreshed file back as a conventional chore commit titled with the last block number, tagged with the next patch version. The start_block, output and compress inputs are gone: resuming pins the start to the archive's cursor, the destination path is fixed by the archive layout, and a resumed .gzip stays gzip. --slim=true is passed explicitly so the snapshot shape does not silently follow a CLI default change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014WoaRY6SZi2gUPPmQEhJgr
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014WoaRY6SZi2gUPPmQEhJgr
- route dispatch inputs through env instead of interpolating them into the run script, closing a shell-injection path on the runner - push commit and tag with --atomic so a rejected main push cannot leave an orphaned tag that a later run would resume from - restrict tag resolution and the patch bump to strict vX.Y.Z tags via grep + sort -V, so a pre-release or non-semver tag can neither become the resume default nor corrupt the bump Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014WoaRY6SZi2gUPPmQEhJgr
The export now always runs to the latest finalized block: a hand-picked end past the finalized head could commit logs from a block that later reorgs, which an append-only archive can never shed, and an end below main's last block could tag a shorter snapshot than its predecessor. Removing the input makes both mistakes impossible instead of guarded. The snapshot commit body now names the resume tag and links the workflow run that produced it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014WoaRY6SZi2gUPPmQEhJgr
# Conflicts: # .github/workflows/batch-sync.yml
Deduplicate rationale stated in both the header and at steps, drop narration of plainly visible code, and hoist the input-injection rule to the header where it covers every step. Adds the two facts that were missing: the full secrets list an operator needs, and why cancel-in-progress must stay false. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014WoaRY6SZi2gUPPmQEhJgr
The workflow run link in the commit body expires with Actions log retention; naming the actor keeps attribution in the archive itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014WoaRY6SZi2gUPPmQEhJgr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on the workflow skeleton from #12 (merged): replaces its failing
SyncTODO with the real publish flow.What it does
archive_taginput, defaulting to the latest semver tag when left empty; a nonexistent tag fails with a clear error.export --resume … --output … --slim=true(--slimpassed explicitly so the snapshot shape does not silently follow a CLI default change). The export always runs to the latest finalized block — see input changes below.mainas a conventional commit titled with the last block number —chore: update snapshot to block number <N>(decimal, extracted from the last NDJSON entry, matching the archive's history) — with a body naming the resume tag and linking the workflow run. Tags the next patch version and pushes commit + tag atomically.Input changes
start_blockend_blockoutputarchive/export.ndjson.gzipcompress.gzipstays gzip; the CLI ignores the flag on resumemax_request,block_range_limit(default raised to 10000),verbositysurvive.Hardening (post-review)
env:instead of${{ inputs.* }}interpolation in run scripts, closing a shell-injection path on the self-hosted runner.git push --atomic— a rejected main push cannot leave an orphaned tag that a later run would resume from.vX.Y.Ztags, so a pre-release or non-semver tag can neither become the resume default nor corrupt the bump. The new tag is bumped from the highest existing semver tag, not the resume tag, so resuming an older tag can never collide.Notes
GHA_PAT_BASICmust havecontents: writeon batch-archive — in ethersphere/bee it is only used for repository-dispatch, so its scope should be verified; if it is read-only, the fix is a one-line secret-name swap.scheduletrigger stays commented; the workflow is manual-only for now. Before ever enabling it, the inputs need${{ inputs.x || 'default' }}fallbacks — theinputscontext is empty on schedule events.🤖 Generated with Claude Code
https://claude.ai/code/session_014WoaRY6SZi2gUPPmQEhJgr