Allow concurrent writers with Caterva2 - #703
Merged
Merged
Conversation
Pre-sized uninit arrays written once per chunk, with the offsets block carrying the completion record. Measured what the format allows first: a chunk whose old content was special appends rather than compacting, so a first write costs ~0.5 ms and moves no other chunk's offset. Records why the frame length cannot serve as a validator (a zeros write can leave it unchanged) and why the .b2lock generation counter can, and why chunk writes do not generalise to fsspec/S3 the way the reads did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A pre-sized array is filled a chunk at a time, by as many writers as there are chunks: `C2Array.update_chunk` posts one, and the subscriber refuses a slot that already holds anything. `written_chunks` reads what the fill recorded in the frame's own offsets, so progress costs one range read and no endpoint of its own. Reading the index turned out not to be the same question as reading blocks of a chunk: `serves_blocks` also weighs whether splitting a chunk would pay, which is nothing to do with whether the frame has offsets to read. Split the geometry half out, so a frame of small chunks can still say which of them were written. `invalidate_index` drops what was read of a frame that has since been written to -- the header as well as the offsets, since a write moves the frame's length and its payload extent, and the offsets are found through both. Nothing is read until the next lookup asks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phases 1, 2, 3 and 5. Phase 4 stays open with what was learned about it written down: the ETag is the freshness half of the stamp question and does not answer the identity half, which is what tells an array that was appended to from one that was replaced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stand-in accepts chunk writes now, so --write lays out an empty array of the dataset's geometry, fills it with the dataset's own chunks, and times the three things the design rests on: the fill serial and with several writers, what the server pays to store a chunk into an empty slot against over a live one, and what reading the progress costs from the offsets against walking the chunks. Two of those numbers came out wrong before they came out right, and both were the measurement's fault. Rewriting a live chunk with bytes of its own length is the case the frame skips the move for, so it has to carry a chunk of a different compressed size or it measures nothing; and the first FsspecNDSource of a process pays for fsspec's own import, which is 77 ms of nothing to do with reading offsets. The ratio a rewrite costs is the dataset's own -- it is whatever payload follows the chunk -- so the row says how many bytes were moved rather than leaving a bare multiple to be read as a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two questions wanted different answers, and mtime:cbytes was answering neither well. Which array this is now comes from the nonce a subscriber writes into vlmeta on the first chunk written: a size and an mtime can both be repeated by a different array at the same path, and a cache served against that one is wrong in every chunk without saying so. Whether it has changed since keeps the mtime and the size -- but only while the array is still being filled. A complete array is stamped by its nonce and size alone, since every slot is claimed and every write to it refused, so a cache of it stands where before an mtime that moved for reasons of its own threw the whole thing away. The plan wanted the stamp to hold still *during* a fill, on the grounds that a cache's chunks are all still where they were. They are not: a cache built while a chunk was unwritten holds the zeros an unwritten chunk reads as, and the run-length offset it had, and both are wrong the moment a writer fills that slot. A frozen stamp serves those zeros for good, which a test now pins by freezing one and watching it happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review found three ways the write path could serve wrong data, and all three are fixed here with a test apiece. A handle reads api/info once when it opens the array, and `stamp` is built from exactly the fields a write moves -- so a handle used to fill an array kept its pre-fill stamp for good, and a Proxy given that handle adopted a cache built while the chunks were still empty. A write marks the metadata stale now, and whatever asks reads it again. `serves_blocks` weighs whether splitting a chunk into blocks would pay, which is nothing to do with whether the frame has an index worth reading -- but the two shared one memoized source, so touching the block path first (max_ranges is a public property, and Proxy reads it) cached None and left written_chunks raising NotRanged ever after. The judgement belongs at the call, not in the slot. A complete array stamps `n<nonce>:<cbytes>` and a filling one whose subscriber reports no mtime stamped the same string, which is the one collision the whole design exists to prevent. Both carry a marker now. Also from the review: invalidating the index no longer empties what index_state hands a cache, nor drops layouts a written chunk can never invalidate; _forget_index takes the lock a source is built under; the run-length tag is read in the host's byte order rather than always little-endian; and the fill bench built a C2Array per chunk rather than per writer, which charged every timed write an untallied api/info. The numbers move little -- 7.6x rather than 7.0x for eight writers over a simulated WAN -- and the plan is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FrancescAlted
force-pushed
the
cat2-concurrent-writers
branch
from
August 21, 2026 08:11
cdd56d0 to
f4a4dc1
Compare
A review of the branch found the freshness only ever reached the handle that did the writing. What that left, and what it is now: * `meta` is read when a handle is opened and never again of itself, so a reader that outlived someone else's chunks handed a `Proxy` the stamp of the array as it was -- which its cache matched and the bytes no longer did. `refresh_stamp()` looks again, and a `Proxy` calls it before judging a cache; a complete array costs nothing there, since nothing can write to one. * `index_state` handed a cache the offsets a write had already moved, and the stamp guarding them said the array had not changed -- true of the array, false of the offsets. Nothing stale is handed over now. * Reading `api/info` again is a round trip, and a write of the handle's own could land inside it; that answer is dropped rather than stored as current. * Every property built on `meta` refreshes, not just `stamp` and `vlmeta`, so `cbytes` and `serves_blocks` no longer depend on what was read first. * `written_chunks()` reads the frame rather than an index a `Proxy` cache left behind, and no longer spends an `api/info` it has no use for. * A write that was refused still forgets what the handle believed: the refusal is the one answer that proves another writer moved the frame. * `invalidate_index` drops the block layouts too. Where a chunk is says nothing about the bytes at that position still being the ones its blocks were mapped from, and the method promises nothing against a rewrite in place. * An offset coding a run-length kind nothing can rebuild is refused when the index is read, rather than mid-fetch where there is no fallback for it. * `aupdate_chunk` invalidates off the event loop, instead of parking it on the lock an in-flight source open holds. * One request helper for both ways of posting a chunk, one URL builder, one vectorized reader of the run-length tag, and offsets out of a cache brought back into the host's byte order. * The `update_chunk` example built a chunk without the array's blocksize, which is one the subscriber refuses. Tests for each of the above, including the block path, which nothing reached before: `__getitem__` goes through `api/fetch` and never touches the index. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`index_state` and `adopt_index` are how a `Proxy` carries a frame's offsets into its cache and takes them up again on the next run. Nothing else calls them, and nothing outside `ByteRangeNDSource` implements them: a source over a Blosc2 frame inherits both, and a source over anything else has no index to keep. Published, they read as a caller's API, and their contract is not one -- what `index_state` hands back is only safe against a stamp that says the bytes have not moved, which is a precondition nothing in the signature carries. So: `_index_state` and `_adopt_index`, which autodoc leaves out on its own. `written_chunks` and `invalidate_index` stay as they are -- how far a fill has got is a question a caller does ask. Neither name has shipped in a tag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`is_fsspec_url` excluded `http(s)://` as reserved for Caterva2. Nothing was
holding the reservation: a subscriber names its datasets by root and path, not
by URL, so `C2Array` is entered through a `blosc2.URLPath`, which `open`
dispatches on by type long before a string is inspected. What the exclusion
bought was `blosc2.open("https://host/big.b2nd")` falling through to the local
path branch and reporting a URL as a missing file.
fsspec reads http(s) in ranges wherever the server answers them, which is what
`FsspecNDSource` wants and no more than what it wants of `s3://`. So a frame
behind nginx, a CDN or an S3 website endpoint now opens whole, through
`cache_storage=`, or a piece at a time with `lazy=True`, like every other URL.
The write guards gain by it too: saving to an `https://` URL says so now,
instead of quietly creating a local file with the URL for a name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading a remote array is spread across four reference pages and a tutorial, and the three things this cycle added -- block-granular reads, a stamp that says a cache has gone stale, and filling an array from several writers -- have no home outside the release notes. So: one page, task-first. Which of the three ways in to use, what the cache holds and when it is thrown away, what blocks buy (5-17x on S3, 0.14 s against 1.01 s on cat2.cloud), how a fill is coordinated by the array refusing a second write (7.6x with 8 writers), and the fifteen lines it takes to give a transport of your own the lot. The figures are the published ones, not re-measured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guide said "upload it with your Caterva2 client", which is the one step of the fill a reader cannot work out for themselves. It is `cat2-client upload`, which ships with Caterva2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements the necessary changes to allow concurrent writers with Caterva2. Still preliminary.