bench/render-cpu: where a render's time and CPU actually go (draft — scaffolding to strip)

[harper-joseph]

Draft — the harness and its findings are ready; the experiment scaffolding is not, see "Before merge".

Adds bench/render-cpu/: a benchmark for the render path, and a README that records what it measured, including the results that contradict what we believed.

Why this is worth merging rather than deleting

It was commissioned to answer "can we cut CDP calls, maybe combine some evaluate calls". The answer is no, and the numbers say why: Runtime.callFunctionOn is 15 of 435 CDP messages per render, and all in-page counting and gating together is 44ms of a 7,450ms render — 0.59%. Six separate implementations of that idea measured within noise; folding the DOM count into the scroll pass measured +31% wall because it buys an extra scroll pass. That entire line of work is now written down as a dead end with numbers, so nobody spends another day on it.

What it found instead: the settle budget closes almost entirely to deliberate Node-side sleeps, and scroll.topSettleMs — 600ms of a 4,050ms render, taken twice at a package default nobody overrides — was in no attribution at all.

What is in it

• bench.js — one render's latency and its phase split (scroll / network-idle / plateau / topSettle / gate), on a deterministic PDP-shaped fixture.
• load.js — renders/second and CPU-seconds/render at a given concurrency, because a change that only removes waiting looks enormous on an idle machine and a change that only removes CPU looks like nothing.
• fleet.js — process shapes (N workers × M slots) at equal total slots.
• aging.js — does an aged browser get slower. Unfinished (stopped at 25%); the README says so rather than letting a partial result read as a negative.
• fixture-server.js — the fixture in its own process, required above c≈8: in-process, one Node event loop was serving all CDP traffic and every HTTP response, worth −22% batch at c=24.
• instrument.js — CDP message counting, Performance.getMetrics read through puppeteer's own session, and process-tree CPU bucketed by Chrome --type=.

Four hazards it documents, each of which produced a wrong number first

1. page.metrics() silently drops TaskOtherDuration, V8CompileDuration, DevToolsCommandDuration and ProcessTime — the four most useful counters.
2. Process-tree CPU must be sampled while pages are still open, or a render whose context has exited takes its accounting with it.
3. V8CompileDuration is useless as an instrument: V8 compiles lazily on background threads.
4. resolveConfigForJob caches by (device + matching override names) and only invalidates on the overrides array identity, so a config that spreads another and adds a field resolves to the previous render's cached config for every URL matching an override. This is a latent bug in shipped code, not in the bench — production is safe only because a config reload happens to produce a new array, and nothing enforces that. It cost two wasted measurement runs before it was found.

Before merge

packages/browser/src/experiments.ts and the experiments.* branches in renderer.ts are measurement scaffolding — they exist so a candidate and the code it replaces can be measured in one process. They must be stripped before this merges; the runners and the README do not depend on them. Flagging rather than doing it here so the branch stays reproducible against its own results while #187 is in review.

Related: #185 (the parked persistent-profile candidate cites this harness).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive benchmarking suite under bench/render-cpu to measure render times, CPU usage, memory footprint, and concurrency characteristics of the renderer. It also adds temporary measurement scaffolding (experiments.ts), registers in-page helpers (inPage.ts), and integrates these experimental optimizations into the main renderer (renderer.ts). The feedback from the reviewer focuses on enhancing robustness by adding defensive checks (such as optional chaining and fallback mechanisms) when accessing window.__prerender inside browser-evaluated contexts to prevent potential runtime exceptions from failing the entire render job.

[gemini-code-assist]

To enforce defensive programming and ensure robustness, we should add appropriate null/undefined checks or optional chaining before accessing window.__prerender properties. If the page fails to load the bootstrap script, window.__prerender will be undefined, causing a hard TypeError crash during the element counting phase. Adding optional chaining with a fallback to the standard countDomElements / countMatchingElements walk ensures the render can degrade gracefully instead of failing entirely.

Suggested change

	const countElements = async (): Promise<number> => timed('countMs', () => countElementsNow());
	const countElementsNow = async (): Promise<number> => {
	if (experiments.monitor) return await page.evaluate(() => window.__prerender.read().elements);
	if (experiments.nativeCount) return await page.evaluate(() => window.__prerender.nativeElements());
	if (experiments.installHelpers) return await page.evaluate(() => window.__prerender.countDomElements());
	return await page.evaluate(countDomElements);
	};
	const countMatching = async (selector: string): Promise<number> => {
	if (experiments.nativeCount || experiments.monitor)
	return await page.evaluate((sel) => window.__prerender.nativeMatching(sel), selector);
	if (experiments.installHelpers)
	return await page.evaluate((sel) => window.__prerender.countMatchingElements(sel), selector);
	return await page.evaluate(countMatchingElements, selector);
	};
	const countElements = async (): Promise<number> => timed('countMs', () => countElementsNow());
	const countElementsNow = async (): Promise<number> => {
	if (experiments.monitor) return await page.evaluate(() => window.__prerender?.read?.().elements ?? countDomElements());
	if (experiments.nativeCount) return await page.evaluate(() => window.__prerender?.nativeElements?.() ?? countDomElements());
	if (experiments.installHelpers) return await page.evaluate(() => window.__prerender?.countDomElements?.() ?? countDomElements());
	return await page.evaluate(countDomElements);
	};
	const countMatching = async (selector: string): Promise<number> => {
	if (experiments.nativeCount || experiments.monitor)
	return await page.evaluate((sel) => window.__prerender?.nativeMatching?.(sel) ?? countMatchingElements(sel), selector);
	if (experiments.installHelpers)
	return await page.evaluate((sel) => window.__prerender?.countMatchingElements?.(sel) ?? countMatchingElements(sel), selector);
	return await page.evaluate(countMatchingElements, selector);
	};

References

1. In browser-evaluated contexts (such as Puppeteer's page.evaluate), implement defensive checks (e.g., Array.isArray guards, try/catch blocks) even if they are redundant with upstream validation. A runtime exception inside page.evaluate can fail the entire render job, so it is critical to handle errors gracefully and skip malformed rules rather than throwing.

[gemini-code-assist]

Add a defensive check to ensure window.__prerender is defined before attempting to read its properties or call its methods. If the helpers are not loaded, handle this gracefully by returning a fallback value (e.g., 0) instead of throwing an error, as a runtime exception inside page.evaluate can fail the entire render job.

Suggested change

	async (stepMs: number, frac: number, useMonitor: boolean, useNative: boolean) => {
	const helpers = window.__prerender;
	const count = useMonitor
	? helpers.read().elements
	: useNative
	? helpers.nativeElements()
	: helpers.countDomElements();
	await helpers.scrollPass(stepMs, frac);
	return count;
	},
	async (stepMs: number, frac: number, useMonitor: boolean, useNative: boolean) => {
	const helpers = window.__prerender;
	if (!helpers) {
	return 0;
	}
	const count = useMonitor
	? helpers.read().elements
	: useNative
	? helpers.nativeElements()
	: helpers.countDomElements();
	await helpers.scrollPass(stepMs, frac);
	return count;
	},

References

1. In browser-evaluated contexts (such as Puppeteer's page.evaluate), implement defensive checks (e.g., Array.isArray guards, try/catch blocks) even if they are redundant with upstream validation. A runtime exception inside page.evaluate can fail the entire render job, so it is critical to handle errors gracefully and skip malformed rules rather than throwing.

[gemini-code-assist]

Use optional chaining and fall back to the local waitForStep function if window.__prerender is undefined. This prevents the waitFor gate from failing completely if the bootstrapped helpers are missing or blocked.

				if (experiments.combineWaitFor || experiments.existsShortCircuit) {
					const existsOnly = experiments.existsShortCircuit && minCount === 1;
					const mode: 'walk' | 'native' = experiments.nativeCount || experiments.monitor ? 'native' : 'walk';
					let step: { scrolled: boolean; count: number; exists: boolean };
					try {
						step = experiments.installHelpers
							? await page.evaluate(
									(opts: Parameters<typeof waitForStep>[0]) => window.__prerender?.waitForStep?.(opts) ?? waitForStep(opts),
									{ anchor: rule.selector, content: contentSelector, scroll: doScroll && !hasScrolled, existsOnly, mode }
								)
							: await page.evaluate(waitForStep, {
									anchor: rule.selector,
									content: contentSelector,
									scroll: doScroll && !hasScrolled,
									existsOnly,
									mode,
								});
					}

References

1. In browser-evaluated contexts (such as Puppeteer's page.evaluate), implement defensive checks (e.g., Array.isArray guards, try/catch blocks) even if they are redundant with upstream validation. A runtime exception inside page.evaluate can fail the entire render job, so it is critical to handle errors gracefully and skip malformed rules rather than throwing.

[gemini-code-assist]

Ensure robust error handling in the combined tail experiment. If window.__prerender is undefined or any of its methods throw, wrapping both extractStructuredOffers and postProcess in try-catch blocks and returning null allows Node-side code to fall back to the standard non-combined path instead of failing the entire render.

				if (experiments.combineTail) {
					const ppStart = Date.now();
					const tail = await page.evaluate(
						(cap: number, opts: unknown, patterns: string[]) => {
							const helpers = window.__prerender;
							if (!helpers) return null;
							let offers: Array<string | null> | null = null;
							try {
								offers = helpers.extractStructuredOffers(cap);
							} catch {
								offers = null;
							}
							let content: string | null = null;
							try {
								content = helpers.postProcess(opts, patterns);
							} catch {
								content = null;
							}
							return { offers, content };
						},
						STRUCTURED_OFFER_CAP,
						config.postProcess,
						config.block.urlPatterns
					);
					if (tail && tail.content !== null) {
						timings.postProcess = Date.now() - ppStart;
						job.structuredOffers = tail.offers;
						return tail.content;
					}
				}

References

1. In browser-evaluated contexts (such as Puppeteer's page.evaluate), implement defensive checks (e.g., Array.isArray guards, try/catch blocks) even if they are redundant with upstream validation. A runtime exception inside page.evaluate can fail the entire render job, so it is critical to handle errors gracefully and skip malformed rules rather than throwing.