Status
The raw-document half of this is implemented in #181. The probe-gating half turned out not to need building. What remains is two narrower things, at the bottom.
Superseded: render.gateOnProbe is a no-op you can already configure
The original proposal was a gate: skip a scheduled render when the probe's signature is unchanged, with maxInterval as a backstop.
That produces exactly the same renders as setting the route's renderInterval to the backstop and letting the existing probe trigger pull renders forward. Same change-driven renders, same backstop, same keys × r / (1 - (1-r)^b) arithmetic — and zero new code on the claim path.
It is also already the running behaviour on the deployment this was written for: the probe is armed, anchored to the origin's nightly change, and triggerRevalidate hard-expires and re-files a page the day its watched fields change.
Two reasons not to build the gate as specified, beyond redundancy:
- It inverts the safety property the probe is documented to have.
configSchema.js states it plainly: the probe is "an accelerator on top of the baseline render cadence, never a gate on it — the failure mode to survive is the origin replatforming under a rule." A gate turns a stale rule from noisy over-rendering into a silent freeze.
- It would cost a per-row read on the hot path. The ready-set sweep already keeps a per-sweep memo to avoid so much as a URL parse; a
ProbeState point read per due row inside that cursor loop is the version to refuse.
What would have been worth building is the part the proposal did not have: a coverage guarantee. Raising an interval to a long backstop silently makes a probe outage, a dropped rule match, or an un-seeded URL into a multi-day staleness event. If this is ever revisited, the shape is a grant evaluated at schedule-write time — the long interval applies only where positive probe coverage exists (enabled, not dry-run, a rule matches, the pass is not cycle-behind) — resolved inside resolveEffectiveInterval, which every schedule writer already passes through. Never a per-row check in the sweep.
Shipped in #181: raw-document caching
Per-route rawCache: true plus render.raw.*. On a miss for an opted-in route the body already being streamed to the crawler is teed, captured to a byte cap, and stored — so the next crawler asking for that URL is a cache hit instead of another origin round trip. No render capacity, no second origin request, one write.
Differences from the sketch in the original proposal, all deliberate:
- Its own table in its own database, not
PrerenderedPage. That table has no expiration and this population is unbounded and long-tailed by construction; it is not residency-pinned, so every row would replicate to every node; and page_cache is the database whose transaction log has been measured pegging worker threads at 85–95% of CPU on the receive side of a copy.
- It is a serve-path tee, not a scheduled fetcher. A
policy: raw that schedules raw refreshes would add a speculative origin fetch per URL per TTL for pages nothing crawls — worse than the misses it replaces. Teeing is demand-driven by construction and converts exactly the misses that already exist.
- It only ever replaces an origin proxy. A stored document is read when
PrerenderedPage held nothing at all. A stale or invalidated row means a render is coming, and the live origin beats a stored copy of it.
skip was not implemented because it already exists: discoverTargets: false gates a route out of the render rotation, and it is the intended pairing for rawCache.
Still open
1. Post-processing raw documents for size
Raw origin documents can be much larger than a rendered snapshot — frameworks that hydrate from serialised props put the majority of the bytes in attributes no consumer reads, and some documents exceed search engines' soft size limits.
#181 caps and refuses oversize documents rather than shrinking them, and the cap is on compressed bytes (the body is stored in the origin's own encoding so the serve path can re-encode from the stored header). So the cap bounds memory and storage; it is not a test against a decompressed size limit.
The existing postProcess.removeAttributes cannot be reused: it runs as document.querySelectorAll in page context, so it needs a browser. A non-DOM streaming rewrite belongs in @harperfast/prerender-browser, which has a build step and where a parser dependency is cheap — not in the plugin, which runs inside Harper.
2. Detecting a front-end deploy
A deploy changes every page's content-hashed assets, and a snapshot that outlives the origin's retention of its assets renders unstyled to bots. The original proposal asked for a documented manual trigger for bulk invalidation.
A better shape is available and cheaper: a deploy is a correlated change, so it does not need a per-URL walk to find — it needs one detection and one bulk invalidation. That is exactly the canary's existing shape (invalidateScope on a trip). A second canary cohort that fetches documents and digests a template-only projection — <script src>, <link href>, asset hashes, explicitly excluding content — would catch a deploy within its interval for a few hundred fetches a day, reusing machinery that already exists and already knows how to invalidate a scope.
It is also unusually well conditioned as a detector: quiescent is ~0% of the cohort and a deploy is ~100% simultaneously, so there is a wide safe band for the threshold — unlike a content canary, where the natural change rate and the event rate can sit close enough together to produce false trips.
Two couplings to get right if this is built:
- A template trip must not reseed.
actOnTrip fires requestSweepReseed unconditionally, which is right for a mass content change but would burn a full corpus probe pass rewriting content baselines a deploy did not touch.
- A trip should file schedule rows, not just an epoch. Invalidation demotes but never reschedules, so healing depends on cadence — and the longer the route's interval, the longer the whole scope serves from origin after a trip.
Constraints (unchanged)
- Every default must be an exact no-op. Unset config produces byte-identical behaviour, the way
config.overrides does.
- Nothing here may assume a particular site's routes, change rate, or SSR completeness.
Related
🤖 Generated with Claude Code
Status
The raw-document half of this is implemented in #181. The probe-gating half turned out not to need building. What remains is two narrower things, at the bottom.
Superseded:
render.gateOnProbeis a no-op you can already configureThe original proposal was a gate: skip a scheduled render when the probe's signature is unchanged, with
maxIntervalas a backstop.That produces exactly the same renders as setting the route's
renderIntervalto the backstop and letting the existing probe trigger pull renders forward. Same change-driven renders, same backstop, samekeys × r / (1 - (1-r)^b)arithmetic — and zero new code on the claim path.It is also already the running behaviour on the deployment this was written for: the probe is armed, anchored to the origin's nightly change, and
triggerRevalidatehard-expires and re-files a page the day its watched fields change.Two reasons not to build the gate as specified, beyond redundancy:
configSchema.jsstates it plainly: the probe is "an accelerator on top of the baseline render cadence, never a gate on it — the failure mode to survive is the origin replatforming under a rule." A gate turns a stale rule from noisy over-rendering into a silent freeze.ProbeStatepoint read per due row inside that cursor loop is the version to refuse.What would have been worth building is the part the proposal did not have: a coverage guarantee. Raising an interval to a long backstop silently makes a probe outage, a dropped rule match, or an un-seeded URL into a multi-day staleness event. If this is ever revisited, the shape is a grant evaluated at schedule-write time — the long interval applies only where positive probe coverage exists (enabled, not dry-run, a rule matches, the pass is not cycle-behind) — resolved inside
resolveEffectiveInterval, which every schedule writer already passes through. Never a per-row check in the sweep.Shipped in #181: raw-document caching
Per-route
rawCache: trueplusrender.raw.*. On a miss for an opted-in route the body already being streamed to the crawler is teed, captured to a byte cap, and stored — so the next crawler asking for that URL is a cache hit instead of another origin round trip. No render capacity, no second origin request, one write.Differences from the sketch in the original proposal, all deliberate:
PrerenderedPage. That table has no expiration and this population is unbounded and long-tailed by construction; it is not residency-pinned, so every row would replicate to every node; andpage_cacheis the database whose transaction log has been measured pegging worker threads at 85–95% of CPU on the receive side of a copy.policy: rawthat schedules raw refreshes would add a speculative origin fetch per URL per TTL for pages nothing crawls — worse than the misses it replaces. Teeing is demand-driven by construction and converts exactly the misses that already exist.PrerenderedPageheld nothing at all. A stale or invalidated row means a render is coming, and the live origin beats a stored copy of it.skipwas not implemented because it already exists:discoverTargets: falsegates a route out of the render rotation, and it is the intended pairing forrawCache.Still open
1. Post-processing raw documents for size
Raw origin documents can be much larger than a rendered snapshot — frameworks that hydrate from serialised props put the majority of the bytes in attributes no consumer reads, and some documents exceed search engines' soft size limits.
#181 caps and refuses oversize documents rather than shrinking them, and the cap is on compressed bytes (the body is stored in the origin's own encoding so the serve path can re-encode from the stored header). So the cap bounds memory and storage; it is not a test against a decompressed size limit.
The existing
postProcess.removeAttributescannot be reused: it runs asdocument.querySelectorAllin page context, so it needs a browser. A non-DOM streaming rewrite belongs in@harperfast/prerender-browser, which has a build step and where a parser dependency is cheap — not in the plugin, which runs inside Harper.2. Detecting a front-end deploy
A deploy changes every page's content-hashed assets, and a snapshot that outlives the origin's retention of its assets renders unstyled to bots. The original proposal asked for a documented manual trigger for bulk invalidation.
A better shape is available and cheaper: a deploy is a correlated change, so it does not need a per-URL walk to find — it needs one detection and one bulk invalidation. That is exactly the canary's existing shape (
invalidateScopeon a trip). A second canary cohort that fetches documents and digests a template-only projection —<script src>,<link href>, asset hashes, explicitly excluding content — would catch a deploy within its interval for a few hundred fetches a day, reusing machinery that already exists and already knows how to invalidate a scope.It is also unusually well conditioned as a detector: quiescent is ~0% of the cohort and a deploy is ~100% simultaneously, so there is a wide safe band for the threshold — unlike a content canary, where the natural change rate and the event rate can sit close enough together to produce false trips.
Two couplings to get right if this is built:
actOnTripfiresrequestSweepReseedunconditionally, which is right for a mass content change but would burn a full corpus probe pass rewriting content baselines a deploy did not touch.Constraints (unchanged)
config.overridesdoes.Related
explainreports the full cadence resolution, which is how "what interval is this URL actually on" stopped being guesswork🤖 Generated with Claude Code