Surface raw-document caching, sitemap conditional fetching and explain cadence in the console (v0.14.0)

packages/console/README.md

@@ -158,7 +158,8 @@ API contract; the short version: **Overview** (scale, serve health, backlog shap
 floor, schedule repair, and the discovered-target purge), **Traffic** (offload/hit-rate charts from
 one bounded analytics scan per node, freshness reported relative to each route's own render cadence,
 the non-hit verdicts broken out by what would fix them — coverage stated net of URLs the origin does
-not have — the discovery gate, and a client-side bot filter), **Sitemaps**, **Page cache**,
+not have — the discovery gate, the raw-document cache's refusals, and a client-side bot filter),
+**Sitemaps** (per-root ingest and check state, plus 24h walk counters), **Page cache**,
 **Queue** (render/claim health and the backlog), **Nodes**, **Invalidations** (preview-first
 record/clear), **Change probe**, **URL explainer**, **Metrics** (the live catalog), **Config**.
 
@@ -254,6 +255,43 @@ way a row survived a pass — deferred, spared, unreadable, failed — is subtra
 reports what the pass never reached; a missing term there turns "we spared 40% on purpose" into
 "~40% was never reached".
 
+**The raw-document cache reports what it REFUSED**, not what it stored (plugin v0.76.0, console
+v0.14.0). `render.raw` keeps the origin document a miss already fetched, for URLs outside the
+render rotation — the facet and parameter combinations a crawler invents, which own no target and
+therefore miss on every single request. The failure mode is silence: a route that is enabled and
+filling nothing produces the same miss rate, the same origin proxies and the same absence of errors
+as a route nobody enabled, so the panel leads with the reason each candidate was turned away.
+Two of those reasons are findings rather than traffic and are called out above the breakdown —
+`has-cookie` means the origin is personalizing a route that was enabled on the assumption it is
+shared (the refusal is right, the assumption is not), and `oversize` means `render.raw.maxBytes`
+sits below the route's real document size, so the feature is on and structurally cannot fill.
+A raw serve is a **cache serve** and counts toward offload, but it is never a hit and never a
+freshness number: nothing rendered it, it has no cadence to be measured against, and the plugin
+emits no `page_age` for it. The Cache-served tile names the raw share instead of folding it in.
+
+**Ingested is not checked** (plugin v0.69.0, console v0.14.0). A sitemap walk now sends
+`If-Modified-Since`, and a `304` deliberately writes nothing — the stored row and its validator are
+still current. That makes `Sitemap.lastRefreshed` the time that document's _entries_ were last
+ingested, which on a nightly-rebuilt corpus is hours old by design; when it was last _looked at_
+lives on the run row. Both are shown under their own names, because printing the first under the
+second's label turns conditional fetching working into an operator chasing a sitemap that is not
+stale. Beside them, a walk-activity panel sums the per-run counters across roots and nodes over 24h
+(the same range key the Change probe view uses, so the two share one cached scan): `not modified` is
+the only evidence anywhere that conditional fetching is working at all — a walk that re-parses every
+document succeeds exactly like one that skipped — and a flat zero across a day of walks is flagged.
+`rendered soon` is the share of new targets whose first render was pulled inside
+`sitemap.newTargets.window` rather than waiting out a full interval of jitter; it is a _subset_ of
+created, stated with its denominator, and the gap is the per-run cap sending a bulk ingest back to
+the old behaviour.
+
+**A cadence is not its ceiling** (plugin v0.77.0, console v0.14.0). The URL explainer's Target card
+shows `renderInterval`, which is the interval the demand ladder schedules _inside_ — with the ladder
+armed it is not the cadence for most of a corpus, and nothing said so. The explainer now renders the
+plugin's own resolution: the effective interval, all four inputs (route, stored, default, rung), the
+demand floor, and `clampedBy` — the clamp that actually bound. `floor` on one URL is information;
+`floor` across a route means the ladder has no dynamic range there and the promotion machinery is
+running for nothing. `ceiling` means the rung is inert.
+
 Two more changed shape when configuration became editable:
 
 - **Nodes is new**, and it exists because "is this node healthy" had four homes: liveness and the
@@ -271,7 +309,8 @@ Two more changed shape when configuration became editable:
 
 Each domain view owns the options that govern the data it shows — `sitemap.*` under Sitemaps,
 `queue`/`render`/`scan` under Queue, `page`/`cacheKey` under Page cache, `analytics`/`crawlStats`
-under Traffic, `invalidation` under Invalidations, `changeProbe` under Change probe — while Config
+under Traffic, `invalidation` under Invalidations, `changeProbe` under Change probe (`render.raw.*`
+rides the `render` group under Queue) — while Config
 remains exhaustive, so a setting can be found either by where it acts or by name.
 
 ## Development

packages/console/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender-console",
-	"version": "0.13.1",
+	"version": "0.14.0",
 	"type": "module",
 	"description": "Standalone Harper component serving the prerender management console UI, proxying to a prerender deployment's /prerender_admin API",
 	"license": "Apache-2.0",

packages/console/src/admin/charts.js

@@ -66,14 +66,27 @@ export const CACHE_STATUS_COLORS = {
  * `hit`, `swr` and `verified` are the page itself; `peer-rescue` is the owner's copy of it. What
  * is NOT here: `miss`/`stale` (origin), `blob-*` (the local body failed AND no rescue landed, so
  * the request went to origin), `invalidated` (refused), `skip`/`bypass` (never consulted).
+ *
+ * `raw` IS IN THIS SET (plugin v0.76.0): a stored origin document answered the request, so the
+ * origin was spared exactly as it was by a snapshot. Leaving it out would not read as missing —
+ * it would read as a smaller cache-served share, which is the same failure `verified` caused for
+ * one release.
+ *
+ * BUT IT IS NOT AN AGE POPULATION. `page_age` / `route_page_age` are emitted only when the serve
+ * SOURCE is `cache` (`recordServeOutcome`), and a raw serve's source is `raw` — nothing rendered
+ * it, so it has no cadence to be measured against. Anything dividing by "cache serves" to talk
+ * about freshness must therefore count the source, not this set; see the staleness panel.
  */
-// `raw` IS IN THIS SET, and leaving it out would not read as missing — it would read as a smaller
-// hit rate, exactly as `verified` did for one plugin release. A raw serve answered from storage and
-// cost the origin nothing, which is what this set means.
 export const CACHE_SERVED = new Set(['hit', 'swr', 'verified', 'peer-rescue', 'raw']);
 export const isCacheServed = (status) => CACHE_SERVED.has(status);
 
-/** Where the bytes came from (bot_serve.path). `origin` is the one offload counts against. */
+/**
+ * Where the bytes came from (bot_serve.path). `origin` is the one offload counts against.
+ *
+ * `raw` is its own source upstream rather than a flavour of `cache`, and it stays that way here:
+ * "answered from storage" and "a render covers this URL" are different questions, and a raw
+ * document answers only the first.
+ */
 export const SOURCE_COLORS = { cache: OK, rendered: INFO, raw: '#7fd4e8', origin: WARN };
 
 /** What became of a posted render result (render outcome.method). */
@@ -594,7 +607,10 @@ const SITEMAP_FETCHES = 'sitemap_sitemaps';
  *   probes    `probe_probed` — one origin call per attempt, failures included (a refused probe was
  *             still a request). Sweep and canary both emit it. A probe hits a small endpoint, not a
  *             page render, so it is cheaper than the other three; it is still a request.
- *   sitemaps  `sitemap_sitemaps` — one fetch per sitemap a refresh run processed.
+ *   sitemaps  `sitemap_sitemaps` — one fetch per sitemap a refresh run processed. Conditional
+ *             fetching (plugin v0.69.0) does NOT remove these: the counter is attempts, and a
+ *             document the origin answered `304` to still cost a request. What it removes is the
+ *             re-parse and the prune scan on THIS side, which no origin-load term ever counted.
  *
  * `renders` is read with sumCount (one emit = one result); the two pass counters with sumValues
  * (one emit per pass carrying the pass's count — sumCount there would count passes).

packages/console/src/admin/views/inspect.js

@@ -26,6 +26,11 @@
  *   - The same rule covers "below the claim floor" and "leased", which are answers about the
  *     OWNER's node-local shared buffer. The owner computes both and this view consumes them
  *     verbatim; it never compares a row against the querying node's own floor.
+ *   - A CADENCE IS NOT ITS CEILING. The Target's `renderInterval` is the interval the demand
+ *     ladder schedules INSIDE, not the one this URL runs on, and the two differ for most of a
+ *     corpus once the ladder is armed. `cadenceCard` renders the plugin's own resolution of it
+ *     (`explain`'s `cadence` block, plugin v0.77.0) — every input, and the clamp that decided —
+ *     rather than leaving a reader to infer a cadence from a ceiling.
  *
  * THE BROWSE HALF'S QUERY SHAPE, AND THE HONESTY RULE THAT FOLLOWS FROM IT. `PrerenderedPage`
  * has only its primary key. A prefix search is a primary-key range and cheap; anything else —
@@ -234,6 +239,7 @@ function explanation(ctx, data) {
 				verdictPills(data, page),
 			],
 		}),
+		cadenceCard(data),
 		card('Stored rows', {
 			body: [
 				el('h3', { cls: 'subhead', text: 'PrerenderedPage' }),
@@ -301,7 +307,12 @@ function explanation(ctx, data) {
 									: '—',
 							],
 							['Scheduler node', mono(target.schedulerNode ?? '—')],
-							['Render interval', target.renderInterval ? duration(Number(target.renderInterval)) : 'default'],
+							// The stored CEILING, not the cadence: the demand ladder schedules inside it. The
+							// Render cadence card above resolves the two, which is why this row names what it is.
+							[
+								'Render interval (ceiling)',
+								target.renderInterval ? duration(Number(target.renderInterval)) : 'default',
+							],
 							['State', target.state === 'suppressed' ? pill('suppressed', 'warn') : pill('active', 'ok')],
 							data.rows.suppression && ['Suppressed', suppressionSummary(data.rows.suppression)],
 						])
@@ -313,6 +324,120 @@ function explanation(ctx, data) {
 	];
 }
 
+/**
+ * How often this URL actually re-renders, and which input decided it.
+ *
+ * WHY THIS IS NOT ONE NUMBER. The cadence a page runs on is the demand ladder's stored rung, raised
+ * to the route's `demandFloor` and then capped by the base interval — route > stored > default. So
+ * there are four inputs and two clamps, and every one of them is a value an operator can set
+ * somewhere else and then fail to see the effect of. The `Render interval` row on the Target card
+ * below is the CEILING, not the cadence; reading it as the cadence is the specific mistake the
+ * plugin grew `explainCadence` (v0.77.0) to make impossible.
+ *
+ * READ `clampedBy` FIRST. It names the clamp that actually bound:
+ *
+ *   floor     the ladder wanted faster and `demandFloor` refused. One URL is information; `floor`
+ *             across a route means the ladder has NO dynamic range there — it is pinned at the
+ *             floor whatever the traffic does, and the promotion machinery is running for nothing.
+ *   ceiling   the ladder wanted slower than the route grants, so the route's own interval won. The
+ *             page renders at its configured cadence; the rung is inert.
+ *   null      the rung applied as computed, or there is no rung yet.
+ *
+ * The ceiling is tested FIRST upstream, deliberately: a floor above the route's interval reports
+ * `ceiling`, because that is the clamp that produced the answer — and a `demandFloor` larger than
+ * the `renderInterval` it modifies is precisely the misconfiguration this view is opened to find.
+ */
+const CLAMP = {
+	floor: [
+		'warn',
+		'clamped by demandFloor',
+		'The ladder’s rung was raised to the route’s demand floor. Across a route this means the ladder has ' +
+			'no range to work in: it cannot go faster than the floor, so promotion is running for nothing.',
+	],
+	ceiling: [
+		'',
+		'clamped by the route ceiling',
+		'The rung was slower than the route’s own interval, so the route won. The ladder never schedules ' +
+			'slower than the cadence a route already grants — the rung is inert here.',
+	],
+};
+
+/**
+ * An interval as text, or null when there isn't one.
+ *
+ * `duration()` takes `Math.abs(ms)`, so it formats `null` as "0s" and `undefined` as "NaNs" — both
+ * of which read as a real cadence. Every figure on this card comes from a payload that may be
+ * older than this console, so each one goes through here rather than trusting the field to exist.
+ * `Number()` because `storedInterval` rides a schema Long and arrives as a string on some payloads.
+ */
+const intervalText = (value) => {
+	const ms = Number(value);
+	return Number.isFinite(ms) && ms > 0 ? duration(ms) : null;
+};
+
+function cadenceCard(data) {
+	const cadence = data.cadence;
+	// Null for a URL with no target, and that is a real answer rather than a missing one: cadence is
+	// a property of being in the rotation. An older plugin sends no `cadence` at all, which is the
+	// same absence — say which, rather than drawing a card of dashes.
+	if (!cadence) return null;
+
+	const clamp = CLAMP[cadence.clampedBy];
+	const effective = intervalText(cadence.effectiveInterval);
+	const rung = intervalText(cadence.demandInterval);
+	const floor = intervalText(cadence.demandFloor);
+	const row = (text, note) => el('span', null, [mono(text), muted(`  ${note}`)]);
+
+	return card('Render cadence', {
+		head: [
+			clamp ? pill(clamp[1], clamp[0]) : rung ? pill('ladder rung applied', 'ok') : pill('base interval', ''),
+			spacer(),
+			effective && el('span', { cls: 'mono', style: { fontSize: '13px', color: 'var(--fg-0)' }, text: effective }),
+		],
+		body: [
+			clamp && el('div', { cls: `note ${clamp[0]}`.trim(), style: { marginBottom: '10px' } }, [clamp[2]]),
+			kv([
+				[
+					'Effective interval',
+					effective ? row(effective, 'what the scheduler actually files') : muted('— not reported'),
+				],
+				[
+					'Base (ceiling)',
+					intervalText(cadence.baseInterval)
+						? row(intervalText(cadence.baseInterval), `from ${cadence.baseFrom ?? 'unknown'}`)
+						: muted('— not reported'),
+				],
+				[
+					'Route interval',
+					intervalText(cadence.routeInterval)
+						? mono(intervalText(cadence.routeInterval))
+						: muted('— the route sets none'),
+				],
+				[
+					'Stored on the target',
+					intervalText(cadence.storedInterval)
+						? mono(intervalText(cadence.storedInterval))
+						: muted('— sitemap changefreq or an explicit write'),
+				],
+				[
+					'Default interval',
+					intervalText(cadence.defaultInterval) ? mono(intervalText(cadence.defaultInterval)) : muted('—'),
+				],
+				[
+					'Demand rung',
+					rung ? row(rung, 'the ladder’s stored decision') : muted('— the ladder has not evaluated this target'),
+				],
+				['Demand floor', floor ? row(floor, 'the fastest rung this route may reach') : muted('— the route sets none')],
+			]),
+			el('p', { cls: 'muted', style: { margin: '12px 0 0', fontSize: '12px' } }, [
+				'Resolved exactly as the scheduler resolves it: the rung raised to the floor, then capped by the ',
+				'base — so the Target card’s render interval below is the ceiling this is clamped into, never the ',
+				'cadence on its own.',
+			]),
+		],
+	});
+}
+
 function verdictPills(data, page) {
 	return el('div', { cls: 'toolbar', style: { marginTop: '14px' } }, [
 		// A timed-out read must never render as a confident verdict: an unread page row is