A viewer for contributed observations, and an explorer for the census

[zmaril]

An upload used to end with a link to the bytes. Following it got you a
zstd-compressed frame of parquet files — honest, and unreadable without the
client, a decompressor and a parquet reader.

Now the site reads it for you:

cratebank: your observation is public at https://cratebank.io/view/sessions/schema=…/….cbnk.zst
cratebank: readable there until the next compaction, typically 05:00 UTC daily, which merges it into the public tables

Four commits: the viewer, then somewhere to walk around in it, then the truth
about how long any of it lasts, then saying that truth at the moment it
matters.

1 — Show a contributor the observation they just uploaded

• crates/worker-ingest answers with view alongside url, from a new
  VIEW_BASE binding. Each field is absent when its domain is not bound, as
  url already was.
• crates/client prints view when the collector offers one and url
  otherwise, so cargo cratebank serve is unaffected.
• infra/worker/site.ts is a new Worker on the site. It reads R2 through
  its own BUCKET binding, bundle.ts decompresses and unframes, render.ts
  renders, view.css styles.
• build-site.ts also writes dist/site/viewer.css, which the pages link
  rather than inline.

2 — Make the viewer somewhere you can walk around

/view/ used to be a 404 explaining what a key looks like. It is now the top
of the space, and every partition between it and an object is a page: schema
fingerprints, then years, months and days, then the uploads — each level
newest first, with a trail back up from anywhere. The front page also walks
down to the newest day in the last few schemas so recent uploads are one click
from the top; that walk is capped, because a bucket is not. A prefix nothing is
filed under is a 404 rather than an empty page pretending to be a place.

An observation opens with what a person came to find out — an M2 Pro compiled
42 units in 37.4 s on dev — rather than the first of 47 fields, then lists
what is in the object with row counts, so tables can be jumped to instead of
scrolled past. Truncated tables link to the rest of themselves; ?rows=N
raises the 200-row cap as far as 2000.

Bots are refused, four ways: /robots.txt disallows /view/,
X-Robots-Tag is on every response, every page repeats it in a robots meta
tag, and a request whose user agent names a crawler, a scraper or a link
preview gets a 403 pointing at data.cratebank.io. The regex deliberately
avoids a bare bot match — CUBOT NOTE 20 is a phone, not a crawler — and is
tested against six bot agents and four browser ones. A user agent is a claim
rather than proof, so this stops the cooperative and the lazy; what it protects
is not secrecy (every byte is public) but the cost of serving pages nobody
reads.

3 — Say what a staged object's lifetime actually is

Opening the deployed preview against the production bucket showed sessions/
empty: worker-compact deletes every bundle it merges, so with a daily
cron a viewer link is good for about a day.

The viewer had been telling anyone who followed a dead link that uploads are
kept indefinitely and they must have mistyped the key. Both halves were wrong,
and sending that person off to check their typing is the worst answer
available. So the 404 now names compaction and points at the published tables
where the rows went, an observation page says the object is staged rather than
archived, and the listing says it shows the recent past rather than the census.

4 — Tell a contributor how long their link is good for

The lifetime is now said where somebody is actually looking at the link: a
second line under the URL the client prints.

The schedule comes from the collector, not the client. Ingest carries a
COMPACTION var and returns it beside url and view. A client that
hardcoded 05:00 UTC would be claiming it about cargo cratebank serve and
about anybody else's collector, neither of which compacts at all — so a
collector that says nothing has nothing said on its behalf and the line is
simply not printed. "Typically" because a deploy compacts too.

COMPACTION must be kept in step with the cron in
crates/worker-compact/wrangler.jsonc. Nothing enforces that; both are
commented to point at each other.

Decisions worth a look

• R2 binding, not data.cratebank.io. Same origin, so no CORS rule to keep
  in step, and the pages do not depend on the public data domain. It costs the
  site Worker a bucket binding.
• run_worker_first: ["/view/*"]. Every other path is still an asset
  served without waking any code.
• Rendered on the server, no script.
• Bounded work. Only sessions/ is ever read or listed, so the published
  tables and the client manifest sharing the bucket stay outside the viewer;
  200 rows per table by default, counts taken from parquet footers; objects
  over 16 MB compressed are linked rather than opened.

Testing

• npm test — 15 tests: framing, row counts, escaping, routing, key and prefix
  shapes, percent-encoding, an object that will not open, browsing, the bot
  refusal, and the header on every response.
• cargo test --workspace --all-features (56), clippy -D warnings,
  fmt --check, and cargo check --target wasm32-unknown-unknown for the
  ingest Worker.
• End to end against wrangler dev: a bundle from cratebank_core POSTed to
  the ingest Worker, the returned view URL opened on the site Worker, all
  nine tables rendered from R2; then browsing across three objects in two
  schemas, ?rows=, the empty-prefix 404. Checked light and dark. The real
  ingest Worker was re-run after commit 4 to confirm compaction comes back in
  the JSON.
• Against this PR's deployed preview, bound to the production bucket: assets
  serve without waking the Worker, the viewer answers, Googlebot gets a 403
  without the bucket ever being touched, robots.txt and X-Robots-Tag are
  live.

Still open, for the repo owner

Both are about the same gap and neither is mine to decide:

1. A viewer link expires. Making it permanent means either keeping merged
   bundles (unbounded storage, and compaction would need to remember what it
   has read) or rebuilding the page from the published tables.
2. README.md and site.md say sessions.parquet carries the key of the
   blob each row came from. It has no such column and nothing writes one, so
   there is no path from a published row back to its object — which is also
   what blocks option (2) above.

Leaving the link as a receipt and saying so plainly is what this PR does.

---

5 — The build, drawn

Cargo's --timings report answers three questions — when each unit ran, how busy the machine was, and where the time went. A contributor has seen that page already; it is the thing they ran to make this object. A viewer that showed only tables was asking them to rebuild it by eye.

All three are drawn now, from the stored tables, as server-rendered SVG with no script — an SVG <title> is a tooltip in every browser, and a chart that needs JavaScript is a chart that does not work in a curl.

And then two Cargo cannot draw, because it does not sample:

	Cargo	here
parquet v59.2.0	4.26s, 100% frontend	codegen 71%, borrowck 7%, type_check 7% — 584 MB peak

Worth staring at: Cargo says 100% frontend, the samples say 71% codegen. Not a contradiction — Cargo's "frontend" is wall time up to rmeta, and a unit with no rmeta split reports all of it that way. The samples are what actually ran. That gap is the argument for collecting phases at all.

A fully cached build says so rather than drawing 167 zero-width rows. That will be the common case now the organisation contributes from CI with warm caches.

6 — One table where there were five

units, phases, unit_flags, compiler_units and artifacts are all keyed to a unit, so reading them apart meant doing the join by eye, five times, for every unit you were curious about.

#  Unit            Target  Total  Frontend  Phase mix  Peak memory  Output     Unblocked  Flags          Features
80 parquet 59.2.0  lib     4.26s  4.26s     ▇▇▇▇▁▁     584 MB       2 · 35 MB             debuginfo 2 …  snap

Artifacts join on unit_index — exact, not by name; the only table that can. Sorted longest first and every column sorts, which is what replaced a separate "slowest units" list: a table you can sort already is one.

• Phase mix is a shape, not a number. Forty units read as forty bars at a glance and as forty percentages not at all.
• Features and flags are pills. clone-impls, default, derive, extra-traits, full is a sentence you have to read; five pills is a shape you count.
• run_id is gone from every table — identical on all 4,000-odd rows, and in the rail once.
• Seconds show two decimals, with the stored value in data-sort so sorting and the JSON download keep the precision the page does not need.

The charts were being built from the first 200 rows. phases has 1,957 and the display cap applied to the join too — 18 phase bars instead of 165. The tables a join needs are read in full now while the raw sections still show 200 with show them all. A gantt drawn from a tenth of the data is worse than one that admits it cannot draw.

7 — A page, rather than a list of tables

Two columns. Configuration in a sticky rail — the glance, sessions, build_config, the contents — because checking a chart against which rustc was this is the thing the single column made you scroll back up for. Measurements in the column that scrolls. The stored tables fold into a <details>: still the object, still what the JSON download reads, no longer in the way.

Full width. main was capped at 46rem by the site's stylesheet — a measure for prose. This is a page of tables and charts, where every rem is another column on screen.

Sorting is sortable-tablesort, 1.7 KB of vanilla JavaScript copied out of node_modules at build time, not a CDN link. A page that reaches off its own origin to sort a table stops sorting when someone else's host has a bad day.

Download as JSON reads the DOM:

for (const section of document.querySelectorAll("section[data-table]"))

Every row is already on the page, so embedding them again would double the page to save a walk over it — and it makes the download honest, containing exactly what is on screen, with ?rows= changing both together.

8 — A tree

/view/ was one level of partitions per page: four clicks to reach one of a dozen objects, which is a filing system standing in for a list. It is the whole staged keyspace now, as nested <details open> — no script, and every object one click from the top.

One list without a delimiter returns keys rather than one level of them. That is affordable because compaction deletes what it merges: this is a day or two of uploads, never the history of the census.

9 — /data/*, and somewhere to ask questions

The published tables had no CORS headers. Checked before assuming:

$ curl -I -H "Origin: https://cratebank.io" -H "Range: bytes=0-99" \
    https://data.cratebank.io/units.parquet
HTTP/2 206
accept-ranges: bytes
← no access-control-allow-origin

Range requests work, so partial parquet reads would be efficient — but a browser on this site could not fetch a byte. /data/* serves the same objects from this origin with the request's own headers passed to R2, which is what keeps a query to three or four ranged reads instead of a download. The allowlist is shapes rather than names, so a new table needs no edit here, and sessions/ is deliberately excluded — staged objects belong to the viewer.

/explore/ runs DuckDB against those tables and hands the Arrow to Perspective, which pivots, filters and plots it in a toolbar — 13 chart types, no second query written. Every table is a view over the parquet, not a copy, so opening the page costs one footer read each and a query costs only the columns it names.

opens on              15 rows in 64 ms
switch to units      167 rows in 63 ms
a join across tables  20 rows in 36 ms

Both engines are self-hosted; nothing is fetched from a CDN and no query leaves the tab. Perspective's 3.4 MB of wasm is a static asset. DuckDB's 34 MB is past what an asset may be, so it lives in the bucket keyed by its version — an immutable URL caches for a year, an overwritten one cannot cache at all — and the deploy publishes it once per DuckDB release, checking the public URL because that is the one the browser will use.

A loading screen covers the panel while ~40 MB arrives, with a real percentage from DuckDB's own progress handler, an indeterminate sweep when there is no share to report, and no animation under prefers-reduced-motion.

Four bugs, none findable without running it

• Missing perspective-client.wasm. The plain ESM builds want every wasm module initialised by hand with a URL; getting it wrong is a blank page. The .inline.js builds carry their own.
• worker.table is not a function. perspective.worker() resolves to the client; it is not one.
• Failed to open RecordBatchStreamReader. Two copies of apache-arrow — duckdb-wasm pins ^17, I had installed 21 — so tableToIPC did not recognise the table and returned nothing. Pinned to 17 and deduped. Also "file" → "stream"; Perspective's reader never looks for a footer.
• ~200 NaN SVG paths per table switch. Perspective keeps its configuration across a load, so a bar chart grouped by phase, handed a result with no phase, drew a hundred d="…vNaNh…" paths and reported nothing. A changed shape resets to the table before loading — resetting after still let one frame render against the old config, which is why the first fix did not take.

And one that had nothing to do with the browser: three template literals in chart.ts had a NUL byte where a space belonged. Invisible in an editor, type-checked fine, and it stayed hidden because both sides of the memory and phase joins had it — so those worked, and the flags join, written later with a real space, was the one that silently returned nothing.

Also corrected

site.md claimed sessions.parquet carries the key of the blob each row came from. It has no such column and has not since the bundle format landed, which is item (2) of Still open above — now stated where it is read, and the viewer links to the bytes it read instead. The viewer link on the front page is its own section and relative, and both the site and the viewer have a favicon.

Checked

tsc clean on both projects, 15 viewer tests, 79 Rust tests, clippy clean with all features. /explore/ driven in a browser end to end: 0 console errors, four benign warnings from Perspective noting it is in inline mode.

Not checked

Nothing tests the explorer or the /data/* guards. The two are worth a test each before this is relied on.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PWfUsPpGVt97QK8RYUF5U7
https://claude.ai/code/session_01361quprG7tUgaVFKAmKtJZ

[github-actions]

cratebank preview

Open the static preview

Run automatically for a PR authored by @zmaril.

OpenTofu plan

No infrastructure changes.

Plan output

cloudflare_workers_custom_domain.site["www.cratebank.io"]: Refreshing state... [id=2426388fdde2ff5a2083a1458e423f27566fb994]
cloudflare_workers_custom_domain.site["cratebank.io"]: Refreshing state... [id=41b6bb2dc82ba0dacaa13bb7056f0304a92e9065]
cloudflare_r2_bucket.cratebank: Refreshing state... [id=cratebank]
cloudflare_workers_route.data_root[0]: Refreshing state... [id=8392a042ec984865a4842bde55fa086c]
cloudflare_workers_custom_domain.ingest[0]: Refreshing state... [id=6edef664db9c85623282947938570ffb8054b932]
cloudflare_r2_custom_domain.data[0]: Refreshing state...

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.