Skip to content

Take the back end apart, and say how much of the build was watched - #36

Merged
zmaril merged 17 commits into
claude/payload-upload-viewer-300zosfrom
claude/codegen-visibility-gwe6cp
Aug 27, 2026
Merged

Take the back end apart, and say how much of the build was watched#36
zmaril merged 17 commits into
claude/payload-upload-viewer-300zosfrom
claude/codegen-visibility-gwe6cp

Conversation

@zmaril

@zmaril zmaril commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Stacked on #35 — base is claude/payload-upload-viewer-300zos, so the diff here is the commits on top of it.

codegen was one name over the whole back end. On the one macOS build in the census with phase data it is 95.5% of every sample on the parallel threads and 41% of the build's CPU, in a single bucket meaning "the compiler was compiling".

A phase is three names now

Segment (frontend, backend, link, plus wait and unknown), phase, and detail. The old flat list is the middle level; the back end splits into monomorphize, ir_build, optimize, machine_code and lto, and the third level says which pass family in the optimiser, whether machine code was instruction selection or register allocation or the assembler, which half of type checking. codegen survives as the answer for back-end work no finer marker claimed.

The back end has to be matched innermost-first: rustc runs work items on worker threads, so the outermost marker there is always the thread's own entry closure inside rustc_codegen_llvm (28,756 of ~30,000 codegen samples in one profile). The markers are rustc's own C++ shims — LLVMRustOptimize, LLVMRustWriteOutputFile, LLVMRustPrepareThinLTOImport, LLVMRustLinkerAdd — which bound exactly the operations worth naming and are as stable as the symbol table.

link leaves codegen entirely. It lives in rustc_codegen_ssa::back::link, so running cc was being published as compiling.

Two things the profile already carried and this threw away

The thread name. rustc names each worker after its work item and codegen unit. That was read for one boolean and discarded, which was wrong twice: lto cgu.NN matched none of the prefixes tested, so thin-LTO work (19,252 samples against opt's 10,903 in a release build) was counted as the serial compile, and a unit's sixteen codegen units were one number.

The clock. Consecutive samples that stay in one phase become a span with a start and an end, so "when did this unit stop type checking and start emitting code" is a lookup rather than a reconstruction. A thread unsampled for more than 10 ms starts a new span rather than stretching one over a gap nothing measured.

Linux had no phase data at all, and now does

Every Linux session in the census publishes 100% unattributed — 1.7M samples, four sessions. The profile gives rustc, librustc_driver.so and libLLVM.so a null codeId on Linux while the symbol sidecar keys them by debug id, and the lookup only ever asked by code id. 6.6% of frames resolved before; 100% after. Anything published before this fix has no phase data whatever its rows say.

Coverage is now a published fact

A phase mix is a statement about whatever the sampler saw, and nothing said how much that was. sessions.parquet gains sampled_span_s and sampler_lost_events, and --rate-hz exists to act on them. Same 200-unit build, two rates:

rate wall lost events compiles missed wall clock unaccounted
4999 Hz 275s 39,920 24 of 205, incl. one 12.7s 5.3%
999 Hz 203s 787 13 of 205, all < 1s 0.8%

The default rate dropped whole compilations and inflated the build by 35%.

--public

Measuring somebody else's open source project needed a key in their manifest. --public (or CRATEBANK_PUBLIC=1) is the same declaration made by whoever runs the build. The rule it feeds is now a named function, package_is_public, conjoined with "is this a workspace member" — so neither the key nor the flag can publish a path dependency from outside the tree, a private git checkout, or a private registry, and that is what its test asserts.

Charts

A timeline: one row per thread per unit, each band a run of samples drawn where it happened. And "Inside each phase", which is the only place the third level is drawn — each group a share of its own phase, bars in the phase's own colour with the label carrying identity.

chart.ts reached the 1,500-line limit on the way and split three ways along the boundary that was already there: chart.ts (the build as Cargo saw it), phases.ts (the compiler as the sampler saw it), chartkit.ts (what both draw with). Byte-for-byte identical output.

Tables

phases.parquet gains segment, phase, detail, thread, cgu. phase_spans.parquet is new: the same samples in time order, at the first two levels only (at 5 kHz the third changes almost every sample — 101,492 spans for 274,020 samples with it, 15,114 without). Both are written from one pass over one profile, so a phase mix and a timeline of it cannot disagree. Ten tables now, not nine; no migration, since nothing is consuming this yet.

Checked

  • cargo fmt --check, cargo clippy --workspace --all-features --all-targets clean, cargo test --workspace --all-features, both wasm worker crates check, npm run check in infra/worker.
  • End to end against the reference collector on Linux, and the viewer rendered from that payload.
  • Measured on bun's Rust workspace (99 crates, 219 units), which needed --public to see at all:
debug release (lto = "fat", codegen-units = 1)
wall 203s 652s
backend / frontend 58 / 37 79 / 17
optimize (share of backend) 3% 59%
machine_code 44% 23%
ir_build 24% 7%
lto 2.4%, of which 95% merge

Fat LTO's cost is not the merge — that is 17 seconds. It is that the merged module then gets optimised as one unit. With codegen-units = 1 the tail is literally one thread: bun_runtime opt cgu.0, 113 seconds.

Left alone

Two gaps are written up in docs/collection.md rather than papered over: the residual ~4% unattributed (mostly stacks that resolve to LLVM frames and nothing else, which want an unwinding fix rather than another marker), and the fact that on a hundred-crate workspace the LLVM-side level three does not resolve at all, though it resolves fine on a single-crate build of the same workspace with the same toolchain.

🤖 Generated with Claude Code

https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5

claude added 4 commits August 27, 2026 12:04
Every Linux build in the census publishes 100% `unattributed`: 1.7M
samples across four sessions, no phase data at all. That is not a
compiler doing nothing recognisable, it is a lookup with the wrong key.

samply writes a library three ways -- `codeId`, `debugId`, and the
`breakpadId` that is the debug id with an age digit stuck on the end --
and this only ever asked by code id. A code id is a build id, and an ELF
library need not carry one: on Linux `rustc`, `librustc_driver.so` and
`libLLVM.so`, which is to say every library a compile actually runs in,
arrive with `codeId: null` while the symbol sidecar knows them perfectly
well by debug id. Nothing errors. Every frame resolves to nothing, every
stack comes back empty, and `classify` answers `unattributed` for a
build it was never shown.

Measured on a release build of a two-dependency workspace, stable 1.94
on x86_64 Linux: 6.6% of frames resolved before, 100% after, and 35% of
samples had no symbol at all on the stack rather than 0%.

The load side already knew about all three spellings -- it indexes by
both ids -- so this is the lookup catching up with it. Sorting the
symbol table is the other half: `resolve` binary-searches it, so its
order is a correctness requirement rather than something the sidecar
promises.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
`codegen` was one name over the whole back end. On the one macOS build
in the census with phase data it is 95.5% of every sample on the
parallel threads and 41% of the build's CPU, in a single bucket whose
content is "the compiler was compiling". Four crates in that build are
more than half codegen and nothing says which half.

It could not have been otherwise. rustc runs back-end work items on
worker threads, so the outermost marker on such a stack is always the
thread's own entry closure inside `rustc_codegen_llvm` -- in one profile
28,756 of ~30,000 codegen samples resolve their outermost marker to
`std::thread::lifecycle::spawn_unchecked<LlvmCodegenBackend>`. The
outermost rule is right for the frontend, where name resolution driven
by macro expansion is expansion; inside the back end it can only ever
answer `codegen`.

So a sample now carries three names. A **segment** -- frontend, backend,
link -- which is the level two builds are worth comparing at. A
**phase**, the old vocabulary plus the five the back end turns out to
be: `ir_build`, `optimize`, `machine_code`, `lto`, and `monomorphize`,
matched innermost-first by rustc's own C++ shims (`LLVMRustOptimize`,
`LLVMRustWriteOutputFile`, `LLVMRustPrepareThinLTOImport`), which bound
exactly the operations worth naming and are as stable as the symbol
table. And a **detail**: which pass family in the optimiser, whether
machine code was instruction selection or register allocation or the
assembler, which half of type checking. `codegen` survives as the
answer for back-end work no finer marker claimed -- 2% of a real build.

`link` moves out of `codegen` entirely. Linking is inside
`rustc_codegen_ssa::back::link`, so running `cc` has been published as
compiling.

Two more things the profile already carried and this threw away.

rustc names each worker thread after the work item and the codegen unit
it was given. That was read for one boolean and discarded, which was
wrong twice: `lto cgu.NN` matched none of the three prefixes tested, so
thin-LTO work -- 19,252 samples against `opt`'s 10,903 in a release
build, the largest single class there is -- was counted as the serial
compile, and the sixteen codegen units of a unit were one
indistinguishable number. Both are kept now, as `thread` and `cgu`.

And every sample has a timestamp, which was counted and dropped, so
"when did this unit stop type checking and start emitting code" had to
be guessed at downstream from totals. Consecutive samples that stay in
one phase become one span with a start and an end. A thread that goes
unsampled for more than ten milliseconds starts a new span rather than
stretching the last one over a gap nothing measured -- without that a
two-sample run three seconds apart draws as a three-second band.

`phases.parquet` gains segment, phase, detail, thread and cgu;
`phase_spans.parquet` is new and carries the same samples in time order.
Both come from one pass over one profile, so a phase mix and a timeline
of it cannot disagree. Spans stop at the second level: at 5 kHz the
detail changes almost every sample, and encoding it produced 101,492
spans for 274,020 samples, 61% of them one sample long, against 15,114
without it.

Measured end to end on Linux, release, thin LTO: backend 64%, frontend
31%, link 1%, unattributed 4%. Inside the optimiser: memory 21%, loops
20%, instcombine 15%, analysis 12%, inlining 8%, and 24% that named no
family.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
Every phase chart on the page was a mix: a unit's samples added up and
spread across its bar in proportion. That can say a unit was 60% back
end and cannot say when it stopped checking and started emitting, which
is the question a compile actually raises. The answer was in the data
and nothing drew it.

So there is a chart of it. One row per thread of one rustc, each band a
run of consecutive samples, drawn where it happened. The first row of a
unit is the serial compile; the rows under it are its codegen units, one
thread each, running while the serial row does rather than after it. On
a dev build of this repository the shape is immediate: `parquet` checks
for seven seconds and then one `opt cgu.00` runs alone from 7.5s to
past 12s, which is the tail the whole build waits on and was previously
one orange band's worth of "codegen".

Units are measured from their own start rather than the build's, so two
can be compared by shape without hunting for them on the build clock.

The palette is re-cut for nineteen phases in four families: hue is the
segment, lightness is the position in it. The warm family is six long
because the back end is six phases; it used to be three, because the
back end used to be one phase with nothing inside it.

Three bounds, all of them stated on the page rather than left to look
like a build with no tail: sub-pixel runs merge into the phase that held
most of their samples, bands under four pixels lose the tooltip nobody
could land on them anyway, and the chart stops at 400 rows or 6,000
bands. On a 163-unit build that is 26 units and 425 KB of the page
rather than 1.3 MB.

The explorer gets the questions the new columns make askable: what the
back end was doing, which codegen unit everything waited for, and when
a crate stops checking and starts emitting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
The schema, the capture manifest, the collection notes and the client's
own README all described one flat list of phases. They describe the
hierarchy now, including which of the two `link`s is a wall-clock span
and which is sampled CPU inside rustc, and that summing across `thread`
gives a number comparable to nothing.

There is also a note on what is still unattributed, with the query to
watch it. It is 4.1% of a Linux release build, concentrated on the
thin-LTO threads, and two things are known about it and neither chased:
stacks that resolve to LLVM frames and nothing else, which want an
unwinding fix rather than another marker, and rustc's parked process
main thread, which would be better named than left in `unknown`.

Anything published before the id fix has no phase data at all, whatever
its rows say. That is worth writing down where a reader of the census
will find it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown

cratebank preview

Preview removed because this pull request is closed.

Six findings, all mine, all real.

`sample.rs` was 1528 lines against a 1500-line limit, and the split it
wanted was already there: what a stack was doing is a different job from
running the profiler and reading its output. The marker tables,
`classify`, the parked-thread leaves and the thread naming move to
`phase.rs` -- 753 lines of "what was this", against 813 of "get the
data" -- and rustc's thread names sit with the symbol markers, because a
stack says what the code was doing and the name says which work item it
was doing it for.

The stack walk was nine levels deep: a chain of nodes, each naming a
frame, each frame reaching its symbol through three more tables, all of
it inline in the middle of a function about something else. It is a
`Stacks` type now, and the four nested `if let`s are four `?`s.

Shaping the payload was three nested closures deep for the same reason
and is two functions instead. The session fixture in core's tests built
a whole build observation as one literal, which put its innermost span
nine levels in; the threads are a helper now, which also makes the
invariant it exists to test -- the same samples as a total and as a span
-- something you can read in one line rather than count braces to find.

No behaviour change: 91 tests pass, and two release builds measured
before and after differ by 1.2 points of back-end share, which is the
same run-to-run variance two builds of the identical binary show.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
The hierarchy went in and the page stopped at two levels of it. `optimize`
is 42% of this build and the chart said so; what the optimiser was
actually doing was in `phases.parquet` and nowhere on the page.

So each phase that has anything finer to say gets a group, and each group
sums to its own phase rather than to the build -- `regalloc` is a tenth
of making machine code and a fortieth of the compile, and the first is
the number this chart exists for. Bars are the phase's own colour,
because a detail is not a separate thing from its phase; the label tells
them apart, and every bar carries one. That also keeps the palette at
nineteen rather than inventing forty more hues nobody validated.

`(no finer detail)` is drawn rather than dropped: 40% of machine code and
23% of the optimiser reached the phase and no further, and that is worth
seeing next to the parts that did resolve. A phase that never says
anything finer gets no group at all, since one bar at 100% labelled
"(no finer detail)" is a row that says nothing.

On a release build of a two-dependency workspace: type checking is 65%
typeck, 20% well-formedness, 6% coherence. Borrow checking is two thirds
the checker and a third the MIR pipeline it drives. The optimiser is 21%
memory, 20% loops, 15% instcombine. Machine code is 29% instruction
selection against 9% register allocation. Thin LTO is 70% reading other
modules' bitcode in.

The share label is anchored after its bar, in a class rather than an
attribute: `.tick` sets `text-anchor: middle` and CSS beats a
presentation attribute, so the attribute version printed every
percentage on top of the bar end it was measuring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
Two failures, both from the last commit and both mine.

`chart.ts` reached 1,575 lines against a 1,500-line limit. I said in the
commit before it that the file was twenty-one lines from the limit and
that the next feature would have to make the call; the next feature was
the one after it, and it did not.

The boundary was there to be found: what Cargo reports about a build is
not what samply reports about the compiler that ran it. So `chart.ts`
keeps the build -- lanes, the dependency graph, concurrency, the verdict,
the units table -- `phases.ts` takes the three sampled-phase charts, and
`chartkit.ts` holds what both draw with, which is the part that is
genuinely shared: the phase order, the colour a phase gets, reading a
table by column name, the axis, a unit. 795, 493 and 356 lines.

The rendered page is byte-for-byte what it was before the split.

The other failure is a scratch screenshot script I left in
`infra/worker` and swept up with `git add -A`. The `workers` job asserts
the repository contains no `.js`, `.mjs` or `.cjs` at all -- everything
here is TypeScript and the built bundle is not committed -- so one stray
`.mjs` fails it, which is the check working. It belongs in a scratch
directory, and that is where it is now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
claude added 4 commits August 27, 2026 12:56
Measuring somebody else's open source project is a thing the census wants
to be able to do, and the only way to say "these crates are public" was a
key in their manifest. So `--public`, or `CRATEBANK_PUBLIC=1`: the same
declaration, made by whoever runs the build.

It is the same decision either way, and the manifest stays the better
place for it when the manifest is yours -- it is reviewable, and it
persists, where a flag is a decision made once in a shell. Both meet in
one variable in `project`, so nothing downstream can tell them apart or
needs to.

The rule they feed is now a function with a name, `package_is_public`,
because a flag that widens a privacy boundary should have the boundary
written down where it can be tested. It is conjoined with "is this a
workspace member", so neither the key nor the flag can publish a path
dependency from outside the tree, a private git checkout, or a private
registry -- and that is what the test asserts, along with the fact that a
crates.io package is public because of where it came from and asks
nobody.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
A phase mix is a statement about whatever the sampler saw, and until now
nothing said how much that was. On bun the answer is 2%: a 351-second
build whose last sampled unit finished at 8.6 seconds. Every number
drawn from that run describes the first nine seconds and reads as if it
described the build.

Two facts now travel with the observation. `sampled_span_s` is the time
from the first sampled process starting to the last sample taken, which
against `elapsed_s` is the coverage. `lost_events` is what samply
reports on stderr and nothing was reading -- perf events the kernel
dropped because the sampler could not keep up, which is the mechanism:
drop the records that say what a process is and its unit does not arrive
unattributed, it does not arrive at all. Both are columns on
`sessions.parquet`, so a reader can filter on coverage before believing
a phase number rather than recomputing it from unit start times.

Reading samply's stderr means draining a second pipe, so it gets its own
thread: whichever pipe fills first would otherwise stop the build.

And `--rate-hz`, because a long build is the case for sampling less
often. The default stays 4999 -- samply's fixed cost per invocation
dominates on anything worth measuring, and a low rate starves small
units -- but a build long and wide enough to overflow the perf buffer
has a lever now, and the two new columns are how you tell whether you
need it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
Two new columns want the sentence that tells a reader why they are
there: a run that covers 2% of its build has phase tables about 2% of
that build, and dropped events mean units are missing rather than
unattributed. Both belong next to the numbers they qualify.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
Measuring a hundred-crate workspace turned up a gap worth writing down
before it is forgotten: on that build every `machine_code` sample came
back with no detail, across three runs at two sampling rates, while the
phase itself resolved fine.

That asymmetry is the clue. `machine_code` is recognised by
`LLVMRustWriteOutputFile`, which is a rustc symbol in
`librustc_driver.so`; everything that would say which part of it --
instruction selection, register allocation, the assembler -- is an
`llvm::` symbol in `libLLVM.so`. A single-crate build of the same
workspace, same toolchain, same sampler, resolves 1.3 million libLLVM
frames and lands on exactly the leaves a debug build should have.

So the symbols exist, the sidecar carries them, and something about the
larger run loses that one library. Until that is understood, a missing
level three means "not resolved here" and never "the compiler did not do
that", and this says so where a reader of the census will find it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
The `lto` phase read 0.3 seconds on a build whose profile is
`lto = "fat"`, which is not a small number, it is a wrong one. Every
marker in that phase was a thin-LTO shape -- `PrepareThinLTOImport`,
`FunctionImporter`, the ThinLTO buffer shims -- and fat LTO does not go
through any of them. It merges every module into one and then optimises
that, so its merge step was reaching no marker at all and its optimise
step was landing in `optimize`.

Half of that was right. Optimising the merged module *is* optimising,
and it stays where it was. The merge was simply missing, and now has the
three names it happens under: `LLVMRustLinkerAdd`,
`rustc_codegen_llvm::back::lto` already covered `run_fat`, and
`llvm::Linker::linkModules`.

Measured on bun's release build, 652 seconds, 99 crates, one codegen
unit, fat LTO: `lto` goes from 0.3s to 17.1s, every sample of it in
`bun_runtime`, 16.3 of them the merge itself. The 426 seconds of
optimising after the merge stay in `optimize`, which is what they are.

The markers went in unvalidated -- a two-dependency test workspace with
fat LTO turned out to merge too little to sample even once at 999 Hz --
so this is the real build standing in as the test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
@zmaril zmaril changed the title Take the back end apart Take the back end apart, and say how much of the build was watched Aug 27, 2026
On a hundred-crate workspace every `machine_code` sample came back with
no detail, at two sampling rates, on three runs -- while the phase
itself resolved fine. A single-crate build of the same workspace, same
toolchain, same sampler, resolved 1.3 million `libLLVM` frames.

The asymmetry was the whole clue. `machine_code` is recognised by
`LLVMRustWriteOutputFile`, which is a rustc symbol in
`librustc_driver.so`; everything that says *which part* of making
machine code is an `llvm::` symbol in `libLLVM.so`. So a fault that
reached only the second library would look exactly like this.

It did. The sidecar for that build has 48 entries and exactly one
repeated debug id: `libLLVM.so` twice, with 3,032 symbols and with 108.
`load` keyed them into a map, so the 108-symbol stub replaced the
compiler's symbol table, and nothing errored -- the phase markers went
on resolving from the library that arrived once. Small builds were
unaffected because their sidecars carry one copy, which is why every
test and every probe passed.

Copies are merged now rather than replaced, deduplicated and sorted
because `resolve` binary-searches them. On the same 91,414 samples that
turns 94% unlabelled into 47% assembler, 15% instruction selection, 3%
register allocation; across the whole build, samples carrying a level
three go from 31.7% to 52.5%.

The test builds a sidecar with one library in it twice and asserts both
halves survive, including an address that lands between them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
claude added 3 commits August 27, 2026 17:14
`unattributed` was 5.1% of bun's workspace build, and a bucket that big is
not a rounding error -- it is a set of things nobody had looked at. Looking
at them, they were four kinds of work, each nameable:

  * the incremental cache being written -- more than half of it, on a build
    that is not even incremental;
  * the compiler process rather than the crate: the dynamic loader, the
    thread and allocator bring-up rustc pays once per unit, the arenas it
    drops afterwards;
  * questions any step can ask and that answer themselves recursively --
    trait solving, const evaluation, layout -- reached so deep through the
    query system that no caller survives on the stack;
  * proc macros, which run on their own stack with no rustc frame on it.

So there is a fourth segment, `process`, for the compiler's own time, and
five new phases. 5.1% -> 0.09%, and what is left is stacks of one frame the
unwinder could not get past, which is what the bucket is for.

The back end also says which part of itself now: `ir_build` was one
undivided bucket and is `body`, `layout` and `debug_info`; `machine_code`
gained `target_init`, `frame` and `debug_info` and its bare share went 8.8%
-> 3.6%; and the time the back end spends destroying the LLVM module after
the object file exists is `free_module`, which was 1.5% of one build.

Two things worth keeping from doing it:

`sample.rs` grows a probe -- ignored by default, pointed at a saved profile
-- that prints the whole breakdown and then the stacks behind one bucket.
Marker tables are only ever extended by looking at what no marker claimed,
and that looking should not be a script written from scratch each time.

And a test that no marker matches a frame a thread is rooted at. Four of
them briefly did, which put 28% of a build in `startup`: the outermost rule
claims every sample on a thread whose entry frame a marker names. The
mistake is invisible in the number a marker change is usually judged by,
because the unattributed share went down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
The phase tables answer "what did the compiler do" and cannot answer "did it
use the machine while doing it". Those come apart: bun's release build is
722 CPU-seconds spread over 519 seconds of wall clock on four cores -- 35%
-- and its last 430 seconds are one thread with three cores idle. Nothing
in a phase breakdown says that. Every phase in it looks healthy.

So `sessions.parquet` gains three columns, swept out of the spans that were
already there: `sampled_cpu_s`, `peak_threads`, and `serial_tail_s`, the
trailing seconds during which never more than one thread was working. That
last one is the number that says a build cannot be made faster by adding
cores, and for a fat-LTO one-codegen-unit release profile it is most of the
build.

The CPU figure has an independent check: the machine's own `cpu_busy_mean`,
read from /proc and never near this code, says 38% where the spans say 35%.

The probe prints all three now, so a saved profile can be asked the same
question without a payload.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
The paragraph at the top of a build page answers "did this use the machine"
with an average -- N cores, X% unused -- which a build can pass while
ending with one thread and three idle cores for most of its wall clock.
`serial_tail_s` is the sharper answer, so it goes in the same paragraph
when it is a real share of the build.

Also notes, in both places the vocabulary is written down, that the
`process` segment and the `process` thread are different columns saying
different things: one is which thread a sample was on, the other is what
kind of work it was.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
Twenty-six phases in five families only works if the ramp inside a family
is ordered, and two of them were not: `mir_build` and `borrowck` sat at
0.471 and 0.464 relative luminance, which is two names for one colour and
they are adjacent in the legend. The five blues sat inside 0.03 of each
other and separated only by hue.

Both families are respaced and the spacing is written down next to them,
because it is the kind of thing that drifts back.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015iY4ffQbZC8ix4g4fs1BR5
@zmaril
zmaril marked this pull request as ready for review August 27, 2026 17:48
@zmaril
zmaril merged commit a946c2f into claude/payload-upload-viewer-300zos Aug 27, 2026
12 checks passed
@zmaril
zmaril deleted the claude/codegen-visibility-gwe6cp branch August 27, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants