Readiness contracts: stop a render when the page says it is complete, and report it when it is not

packages/browser/README.md

@@ -291,6 +291,29 @@ include what you change:
 			"pathPattern": "^/product/", // only product pages have this widget
 		},
 	],
+	// optional: per-page-type statements of what a COMPLETE render contains. When one governs a
+	// render it REPLACES the timer-based settle — see "Readiness contracts" below. Absent → no-op.
+	"readiness": {
+		"onSatisfied": "quiet", // stop once the contract holds AND the DOM has been still for quietMs
+		"quietMs": 250,
+		"unmetGraceMs": 1000, // stop waiting on a clause that has never been true once everything else holds
+		"contracts": [
+			{
+				"name": "product",
+				"pathPattern": "^/product/",
+				"require": [
+					{ "name": "price", "selector": "[data-price]", "nonEmptyText": true, "textMatches": "\\$\\s?\\d" },
+					{ "name": "hydrated", "selector": "astro-island", "shed": "ssr" },
+					{ "name": "no-skeletons", "absent": ".skeleton" },
+					{ "name": "grid-or-empty", "anyOf": [{ "selector": ".tile", "minCount": 1 }, { "selector": ".no-results" }] },
+					{ "name": "rails-filled", "every": ".rail", "contains": ".slide" },
+					// only required when the page's own JSON-LD says the content should exist
+					{ "name": "reviews", "selector": ".review", "onlyIf": { "jsonLdNumber": "aggregateRating.ratingCount" } },
+				],
+				"observe": [{ "name": "product-links", "selector": "a[href^=/product/]" }], // reported, never waited on
+			},
+		],
+	},
 	"postProcess": {
 		"stripScripts": true, // remove executable <script> (keeps application/ld+json etc.)
 		"inlineEmptyStyleSheets": true,
@@ -372,6 +395,69 @@ all, so it acts as a fixed per-pass sleep, and cutting it 2000 → 500 made rend
 dropped every one of 1,635 review nodes with `outcome=ok` and no error. Add the explicit `waitFor`
 readiness gate first, confirm the content is still there, and only then take the blind dwell down.
 
+### Readiness contracts — deciding a render is complete by asking the page
+
+Every other settle signal is a timer, and a timer cannot be wrong out loud. A render that missed a
+widget, or that serialized pre-hydration markup, reports 200, non-empty and indexable, and nothing
+downstream can tell. A contract states per page type what a complete render CONTAINS; the renderer
+holds until that is true **and** the DOM has gone quiet, and posts the per-clause result back.
+
+Six assertion forms, each of which exists because something else could not express it:
+
+| form                                          | satisfied when                                                       | exists because                                                                                                |
+| --------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
+| `selector` + `minCount`                       | at least N match (shadow-piercing)                                   | the base case                                                                                                 |
+| `anyOf`                                       | any branch matches                                                   | an empty-but-legitimate listing page is structurally identical to one whose grid has not arrived              |
+| `absent`                                      | nothing matches                                                      | skeletons and spinners that a real render replaces                                                            |
+| `selector` + `shed`                           | no element still carries the attribute (`maxRemaining`, `allowNone`) | frameworks drop a marker on hydrate; this is the check that catches a snapshot of pre-hydration markup        |
+| `every` + `contains`                          | every container has content, and at least one exists                 | "at least 3 rails" is a constant someone measured once; "every rail is filled" survives the template changing |
+| `selector` + `nonEmptyText` (+ `textMatches`) | some match has text / matching text                                  | presence is not the same as populated                                                                         |
+
+Any clause can carry `onlyIf` — a guard making it conditional on what the page's **own** data says
+(`jsonLdNumber`) or on what the DOM holds (`present`). This is the difference between a guess and a
+check: a product with reviews and one without are structurally identical apart from the reviews, so
+a presence gate cannot tell "none" from "not yet", and accepting _either_ review items or a rating
+summary fires early on pages that do have reviews (measured: the summary lands 236–263 ms before the
+first review). Keyed on the page's declared `aggregateRating.ratingCount`, the document says what
+should exist and the rendered DOM is held to it.
+
+**Three properties worth knowing before writing one:**
+
+- **The quiet window is not optional, and it is where the speed comes from.** Running the same stop
+  policy with an _empty_ contract saves the same time to within 30 ms on 7 of 8 pages — the win is
+  replacing blind dwells with one real quiescence test. Stopping the instant a contract holds saves
+  more and loses the recommendation rails (measured: 99% of product links on one product page, 100%
+  on an empty facet), because rails have no server-rendered placeholder and no clause can assert one
+  is still coming. The contract's contribution is falsifiability, and that is what makes a short
+  quiet window safe.
+- **A clause must be false on a truncated render.** Anything server-rendered is true before the page
+  finishes and proves nothing; keep those as validity clauses and make sure at least one clause names
+  content that genuinely arrives late, or the contract will stop early.
+- **Templates change, so contracts rot.** A clause that has never been true, once every other clause
+  holds and the DOM is quiet, is stood aside after `unmetGraceMs` and reported unsatisfied — the
+  render falls back to the ordinary settle. Cost degrades to roughly today's behaviour instead of
+  waiting out the timeout on every render forever; without that valve one unsatisfiable clause
+  measured +385% wall.
+
+- **Cap `timeoutMs` low.** A contract that does not satisfy pays its whole timeout **and then the
+  full fallback settle on top**, and it costs CPU rather than only wall, because the page keeps
+  executing while the poll runs. Measured on a concurrency ladder: with a 15s timeout, one render in
+  twelve under contention took 10.8s and CPU/render rose 36%; at 3s the straggler was 4.8s and CPU
+  rose 12%, with the median render unchanged either way. Erring low is the safe direction — giving up
+  early falls back to exactly what the renderer does without a contract, so a too-low timeout costs
+  the optimisation and never the content. Set it from the `firstSatisfiedMs` distribution the results
+  carry rather than from a guess; if p95 approaches the timeout, the contract is being abandoned
+  under load and the win is quietly gone.
+
+Note this is a different failure from the rot valve above, and needs its own control: `unmetGraceMs`
+fires when a clause has never been true **and the page has gone quiet**, which is what a template
+change looks like. A page that is merely slow is still mutating, so the valve does not fire and the
+timeout is what bounds the wait.
+
+An unsatisfied contract always falls through to the normal settle, so a badly written contract can
+cost a render time but never content. `job.readiness` carries `satisfied`, per-clause `ok`/`count`/
+`firstTrueMs`, and any `observe` counts.
+
 ### `postProcess.minifyInlineCss` — re-emitting inline CSS from the CSSOM
 
 Replaces each inline `<style>`'s source text with the browser's own serialization of the parsed

packages/browser/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender-browser",
-	"version": "1.29.0",
+	"version": "1.30.0",
 	"type": "module",
 	"description": "Headless-browser render library for Harper Prerender: claims render jobs from the @harperfast/prerender queue, renders pages in headless Chrome (Puppeteer), and posts the HTML back. Embedded by a render service and configured entirely via startWorker() options.",
 	"keywords": [

packages/browser/src/RenderJob.ts

@@ -6,6 +6,7 @@ import { encode } from './util/encoder.js';
 import { getHostHealth, parseRetryAfter } from './HostHealth.js';
 import { renderPhaseOf } from './util/renderPhase.js';
 import type { JobDocumentCache } from './documentReuse.js';
+import type { ReadinessExpectations, ReadinessResult } from './readiness.js';
 
 /** One `waitFor` rule's outcome for one render. Only rules whose scope MATCHED appear. */
 export interface WaitForResult {
@@ -56,6 +57,12 @@ export type JobConfig = {
 	renderBudget?: number;
 	callbackOrigin: string;
 	isFromSitemap: boolean;
+	/**
+	 * What the last accepted render of this URL produced, so this render can be judged against the
+	 * page's own history rather than against a constant measured once (see readiness.ts). Absent on a
+	 * first render, which is never a regression.
+	 */
+	expectations?: ReadinessExpectations;
 };
 
 /**
@@ -164,6 +171,14 @@ export default class RenderJob {
 	 * simply absent, and nothing says a rule spent its whole `timeoutMs` finding nothing.
 	 */
 	waitForResults: WaitForResult[] = [];
+	/**
+	 * What this page type's readiness contract said, when one governed the render. Posted back so an
+	 * incomplete render is reported rather than silent: today a render that missed its SEO-critical
+	 * content still reports 200, non-empty and indexable.
+	 */
+	readiness?: ReadinessResult;
+	/** Carried from the job so the renderer can judge this render against this URL's history. */
+	expectations?: ReadinessExpectations;
 	acceptLanguage: string | undefined;
 	renderBudget: number | undefined;
 	callbackOrigin: string;
@@ -214,6 +229,7 @@ export default class RenderJob {
 		this.renderBudget = config.renderBudget;
 		this.callbackOrigin = config.callbackOrigin;
 		this.isFromSitemap = config.isFromSitemap;
+		this.expectations = config.expectations;
 	}
 
 	/**
@@ -299,6 +315,23 @@ export default class RenderJob {
 	 * Builds the encoded body too (the expensive gzip) so a caller assembling several variants pays
 	 * it once per variant and retries re-send the same bytes.
 	 */
+	/** The contract's verdict in its wire form, or undefined when no contract governed this render. */
+	private readinessReport(): ReadinessReport | undefined {
+		const r = this.readiness;
+		if (!r) return undefined;
+		return {
+			contract: r.contract,
+			satisfied: r.satisfied,
+			unmet: r.require.filter((c) => !c.ok).map((c) => c.name),
+			skipped: r.require.filter((c) => c.skipped).map((c) => c.name),
+			waitedMs: r.waitedMs,
+			firstSatisfiedMs: r.firstSatisfiedMs,
+			learned: r.learned ?? {},
+			shortfalls: r.shortfalls ?? [],
+			rebaselined: r.rebaselined ?? false,
+		};
+	}
+
 	async resultMetadata(): Promise<{ metadata: VariantMetadata; contentBuffer: Buffer | null }> {
 		const attemptError = this.error;
 		const metadata: VariantMetadata = {
@@ -309,6 +342,7 @@ export default class RenderJob {
 			redirectedTo: this.redirectedTo,
 			isIndexable: this.isIndexable,
 			structuredOffers: this.structuredOffers,
+			readiness: this.readinessReport(),
 			outcome: this.outcome,
 			// Present only when true, so the flat legacy envelope is byte-identical for every render
 			// that did not reuse a document (and an older plugin never sees the key at all).
@@ -369,6 +403,9 @@ export default class RenderJob {
 			redirectedTo: this.redirectedTo,
 			isIndexable: this.isIndexable,
 			structuredOffers: this.structuredOffers,
+			// Carried even on a failed result build: a render that could not be reported is exactly
+			// when knowing whether its contract held is most useful.
+			readiness: this.readinessReport(),
 			outcome: 'error',
 			// Carried through so the plugin still sees where this variant's document came from, even
 			// though its body never made it onto the wire.
@@ -440,6 +477,32 @@ export default class RenderJob {
 }
 
 /** One variant's share of a posted result — see `RenderJob.resultMetadata`. */
+/**
+ * What a readiness contract said, as it travels back to the consumer.
+ *
+ * Deliberately narrower than the in-process `ReadinessResult`: `unmet` carries only the names of
+ * clauses that did not hold, because that is what a metric needs to make an unsatisfiable clause
+ * visible — a gate that quietly fails on every render is the trap this feature exists to avoid, and
+ * it is only avoided if the failure reaches something that counts it. `learned` is the observation
+ * counts the consumer stores as this URL's expectation for next time.
+ */
+export type ReadinessReport = {
+	contract: string;
+	satisfied: boolean;
+	/** Names of the clauses that did not hold. Empty when satisfied. */
+	unmet: string[];
+	/** Names of clauses a guard decided did not apply — never conflate with "checked and passed". */
+	skipped: string[];
+	waitedMs: number;
+	firstSatisfiedMs: number | null;
+	/** Observation counts, for the consumer to store as this URL's expectation. */
+	learned: Record<string, number>;
+	/** Observations that fell far below what this URL last produced. */
+	shortfalls: Array<{ name: string; expected: number; got: number; ratio: number }>;
+	/** The shortfalls repeated enough times to be the page's new shape; re-learn from this render. */
+	rebaselined: boolean;
+};
+
 export type VariantMetadata = {
 	deviceType: string;
 	statusCode: number | undefined;
@@ -448,6 +511,8 @@ export type VariantMetadata = {
 	redirectedTo: string | undefined;
 	isIndexable: boolean | undefined;
 	structuredOffers: Array<string | null> | null | undefined;
+	/** Present only when a contract governed this render, so an older consumer never sees the key. */
+	readiness?: ReadinessReport;
 	outcome: JobOutcome;
 	documentReused: true | undefined;
 	documentPrefetched: true | undefined;

packages/browser/src/config.ts

@@ -13,6 +13,7 @@ import { readFileSync } from 'node:fs';
 import { resolve as resolvePath } from 'node:path';
 import { KnownDevices } from 'puppeteer';
 import type { PuppeteerLifeCycleEvent } from 'puppeteer';
+import { validateReadiness, type ReadinessConfig } from './readiness.js';
 
 export type Viewport = {
 	width: number;
@@ -449,6 +450,13 @@ export type PrerenderConfig = {
 	 * deployments render byte-identically; present → both `renderOnce` and the fleet honor it.
 	 */
 	waitFor?: WaitForRule[];
+	/**
+	 * Per-page-type statements of what a COMPLETE render contains (see readiness.ts). When a contract
+	 * governs a render it replaces the timer-based settle: the renderer stops once the page says it is
+	 * complete AND has gone quiet, and reports which assertions held — so an incomplete render becomes
+	 * a fact on the wire rather than a silence.
+	 */
+	readiness?: ReadinessConfig;
 	/** Inject Web Components (ShadyDOM/ShadyCSS) polyfill-forcing flags before load. */
 	injectWebComponentsPolyfill: boolean;
 	/** Extra request headers added to the navigation request (besides the bypass token and job headers). */
@@ -741,6 +749,7 @@ const validate = (config: PrerenderConfig): PrerenderConfig => {
 	});
 	// waitFor is optional; when present every rule needs a non-empty selector and non-negative
 	// numeric fields (it is API-/JSON-supplied, so validate before it reaches the in-page waits).
+	validateReadiness(config.readiness);
 	if (config.waitFor !== undefined) {
 		if (!Array.isArray(config.waitFor)) {
 			throw new Error('prerender config: waitFor must be an array of rules');
@@ -905,8 +914,19 @@ export type ResolvedConfig = {
 // matching override names), not by URL: every product page resolves the same config, so the cache
 // holds one entry per distinct combination rather than one per URL. Bounded by construction — the
 // number of combinations is a property of the config, not of the corpus.
-const resolvedCache = new Map<string, PrerenderConfig>();
-let resolvedCacheFor: ConfigOverride[] | undefined;
+// Keyed by the BASE CONFIG OBJECT, not by its overrides array.
+//
+// The previous key was the identity of `config.overrides`, which is wrong whenever two configs share
+// an overrides array — exactly what happens when one config is derived from another by spreading it
+// and changing a field. Every URL matching an override then resolved to the FIRST config's cached
+// result, silently and for the rest of the process; a URL matching none was unaffected, because that
+// path returns before the cache. It cost two wasted measurement runs before it was found, and it
+// presents as "this feature works on the home page and nowhere else", which is not a shape anyone
+// debugs quickly.
+//
+// A WeakMap on the config itself cannot have that failure: a different base config is a different
+// key by construction, and an old config's entries are collected with it.
+const resolvedCache = new WeakMap<PrerenderConfig, Map<string, PrerenderConfig>>();
 
 /**
  * The effective config for one render. Matches `config.overrides` against this job's URL path and
@@ -924,9 +944,10 @@ export const resolveConfigForJob = (
 
 	// The cache is keyed by name-signature, so it must be dropped when the config itself is replaced
 	// (a live config reload). Identity of the overrides array is the cheapest correct witness.
-	if (resolvedCacheFor !== overrides) {
-		resolvedCache.clear();
-		resolvedCacheFor = overrides;
+	let cache = resolvedCache.get(config);
+	if (!cache) {
+		cache = new Map<string, PrerenderConfig>();
+		resolvedCache.set(config, cache);
 	}
 
 	let path = '';
@@ -945,14 +966,14 @@ export const resolveConfigForJob = (
 	if (!applied.length) return { config, applied };
 
 	const key = `${deviceType}\u0000${applied.join('\u0000')}`;
-	let resolved = resolvedCache.get(key);
+	let resolved = cache.get(key);
 	if (!resolved) {
 		const names = new Set(applied);
 		resolved = overrides.reduce(
 			(acc, override) => (names.has(override.name) ? deepMerge(acc, override.config) : acc),
 			config
 		);
-		resolvedCache.set(key, resolved);
+		cache.set(key, resolved);
 	}
 	return { config: resolved, applied };
 };