You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A customer deployment reported the change probe's request volume against their per-product price/availability JSON API (the same call the PDP's client JS makes on load) and asked for it to go away. Measured across the four nodes (2026-09-14): ~85% of what they saw was changeProbe (mode: continuous, 12h cycle, ~1.4M PDP URLs); ~15% was the PDP's own client JS during renders (#153). Their edge began throttling that path the same evening (probe failures 1-2% -> 33-52%, all connection timeouts and resets), so the probe was already half-blind.
The customer supplied the facts that decide the design:
Prices change only at their local midnight, never intraday; a refresh right after midnight is good for the day. Our data agrees (83% of canary-detected change events in the 23:00-02:59 local window; promo nights repriced 7.8% and 18.9% of products).
Availability changes intraday, and sold-out products leave the product sitemap the same day (57/58 sold-out products absent from that day's sitemap, 77/77 in-stock present). Sitemap membership is a daily availability signal; per-URL <lastmod> is not (every entry carries the build date).
They offered a better endpoint (2026-09-16): a per-product details API that is cached at the edge with the PDP's own lifetime (purged on known changes, expired at their midnight) and carries price, per-variant availability, canonical slug, promotions, badges, title/meta, images and breadcrumbs. Verified: 8/8 sitemap products and 46/46 variants agree with the PDP's JSON-LD on price and availability; two fetches 2.5 s apart differ in zero fields; sold-out is a 200 with a status word and no price object, not an error status; an unknown id is a 400 with a stable error code; the edge requires a Referer header (any value) or serves a cached 404 page. They also confirmed the page's own price call can be dropped for bot renders.
Resolved
The probe stays; it moved endpoints. With a cached, PDP-coherent endpoint the probe costs the origin one product-service read per product per pass and detects nearly every rich-result field, which removes the reason for the fleet-side "conditional render" this issue originally designed (see "Dropped" below).
Live switch, 2026-09-16, via config-override rows:changeProbe.rules -> the details endpoint (GET, referer header, 20 extract slots: price core, status word, price set, all-variants flag, canonical slug, sale status, percentage off, promotion, badges, new/best-seller/top-rated flags, title, brand, meta title/description, main image, breadcrumbs), with dryRun: true and reprobeAfter: 0 for one full cycle because the signature shape changed for every URL and the plugin had no way to tell a rule edit from a content change (the canary would have read 100% changed and invalidated the route). The in-flight old-rule passes were cancelled with a momentary enabled: false so nothing kept polling the old path.
ProbeState.ruleFingerprint: a baseline records the rule that made it; a mismatch re-baselines (stored, not compared, not triggered, outside the canary verdict; rebaselined stat, probe_rebaselined metric). A rule edit costs one pass without detection for that rule and no dry-run cycle.
pageCheck reads availability words (In Stock, IN_STOCK, schema.org forms) and [*] per-variant lists, with per-rule availableValues / unavailableValues over the built-in vocabulary.
[*] projection in extract paths, so a rule can watch variants[*].availability without signing whole variant objects.
mode: anchored (anchorTime, anchorTimezone, anchorWindow): one full pass a day starting right after the origin's scheduled change, at the ceiling by default; no boot sweep; the canary keeps its cadence. This is the "nightly pass" of the original design, as configuration.
Deployment sequence for the customer: release 0.65.0 -> bump the component and port the rule into config.yaml (identical fingerprint to the override rule, verified) -> clear the override rows (changeProbe.rules, .dryRun, .reprobeAfter, .cycleTarget) once the re-baseline pass has finished on every node -> fleet config with the block pattern.
Dropped: conditional render in the fleet (former stages 2-3)
Pre-fetching the SSR document in the worker, extracting a signature, and seeding #157's document cache was designed to replace price polling with a fetch the render needed anyway. Against a cached endpoint that costs the origin ~nothing and answers in 15-35 KB, that machinery no longer buys origin relief. What it still would buy — ~6-15% render throughput from hiding the document fetch, and ~80% fewer blob writes from making cadence renders conditional — is a render-pipeline optimization that belongs with #155/#156/#157 on its own merits, not to this issue. Its measurements stay valid and are kept here for that work:
Per-device SSR: desktop vs mobile product documents differ in 3 cosmetic hunks, Vary: Accept-Encoding only, so one document fetch per product is valid.
Render is ~11.7 s wall / ~9.6 s Chrome CPU per variant (median 11.5 s, p95 16.3 s); origin SSR 465 ms mean, p95 1.26 s; PDP blob 610 KB raw / 53 KB gzipped, replicated x4; ~1.0M render attempts/day.
Remaining follow-ups (this issue stays open for these)
Field-class lanes. Widening the signature to ratings would re-render a product per new review. Give extract slots a lane (immediate for price/availability/canonical, nightly for copy, images, ratings, breadcrumbs) so the anchored pass absorbs slow-moving classes and the canary/immediate path stays about price. Run the wider signature in dry-run first and read the changed-share split (probe_changed) before turning classes into renders.
Sitemap-membership availability — BUILT, awaiting merge (Re-check URLs that leave a sitemap, per route (v0.68.0) #164, prerender-v0.68.0). Per-route ingress.routes[].departureAction (none | expire | render): a PDP target that becomes unlinked has its pages HARD-expired (past page.swrTtl, so they stop serving rather than serving stale — a plain Date.now() expiry leaves the page swr and still serving, which is the trap here) and is filed to render at the current minute. Route-scoped as this item specified, so catalog pages are excluded: a product URL leaving a product sitemap says something about that product, while a listing URL leaving a paginated sitemap usually means the catalog was re-bucketed. Never deleted — the origin still serves a 200 with out-of-stock markup, so this is a re-check and the render's own verdict decides. Capped per walk (sitemap.departure.maxActions), and dryRun defaults TRUE because the count a real walk would act on is not yet known (see the measurement note below).
The trap that shaped the design: the check must run AFTER the whole walk, not at prune time. The product sitemap is 17 children of exactly 50,000 entries ordered by ascending prd- id, so it shears across boundaries — a URL moving to an EARLIER child is re-attached before the child it left is pruned and never looks departed, but one moving to a LATER child is pruned first and looks departed until the child that now claims it is reached. Acting at prune time would fire on every URL after an insertion. Candidates are re-read once the walk ends; departure_reattached counts the shear, and a large share there means the raw removed count is not a departure count.
Still unmeasured: how many PDPs actually depart per day. The SitemapRefresh row keeps only the last run, and the pass I read (18:00 ET: created 65 / updated 3 / removed 38 / skipped 886,471) is not the one that absorbs the midnight-CT rebuild — that is the 06:00 ET pass. That number sizes maxActions, which is why the feature ships in dry run.
Distinct probe identity.changeProbe.userAgent (or a header) so an origin's CDN can allowlist verification traffic separately from renders; today the probe rides origin.userAgents.desktop.
Canonical slug as a signal. With the canonical in the signature, a slug change can retarget without rendering (today it surfaces as a canonical-mismatch suppressed render, ~153k/day on this deployment).
Console. Surface rebaselined and nextAnchoredRunAt; alert on a steady probe_rebaselined (a rule that keeps changing).
Load (measured on the deployment)
Probe keys are never shopper-warmed (the PDP's client does not call the details endpoint), so each probe is an edge miss and one origin product-service read: ~1.4M/day at the 24h cycle, the same count at one anchored pass a day, delivered at the ratePerSecond ceiling (10/s/node) for ~12 h rather than spread.
Signature ~1.2-1.6 KB/URL (was ~40 B): ProbeState grows to ~600 MB per node, node-local, written only on change after the re-baseline.
The flagged price API drops to the render-side XHR share (~0.6M/day) at the switch and to zero once the fleet block ships.
Acceptance
Zero probe requests to the price API; probe_failed back at the 1-2% floor on the details endpoint (first dry-run canary: 500 probed, 0 failed).
Anchored pass starts within a minute of the anchor and finishes inside the ceiling-rate window; nextAnchoredRunAt published.
A rule edit produces one pass of probe_rebaselined and no canary trip.
Pages of products that leave the sitemap re-render within 24 h of the refresh (follow-up 2). Gated on one dry-run walk reporting departure_would_render and departure_reattached before sitemap.departure.dryRun is turned off.
Notes / risks
Status signals on the details endpoint: sold-out is a body state, not a status, so the "unavailable" signal is dormant there; the id-not-found signal is now trustworthy (the old endpoint also answered it for live products when the cohort cookie was missing).
Analytics: sweep probe_* rows emit per finished pass (smeared); use canary rows or the Invalidation table for flip timing. probe_* are value metrics (mean x count).
Versions:prerender-v0.65.0 = #159 (also claimed by #154; whichever merges second renumbers). The former stage-2 browser version (1.25.0) is released back to the #155-#157 line.
Context
A customer deployment reported the change probe's request volume against their per-product price/availability JSON API (the same call the PDP's client JS makes on load) and asked for it to go away. Measured across the four nodes (2026-09-14): ~85% of what they saw was
changeProbe(mode: continuous, 12h cycle, ~1.4M PDP URLs); ~15% was the PDP's own client JS during renders (#153). Their edge began throttling that path the same evening (probe failures 1-2% -> 33-52%, all connection timeouts and resets), so the probe was already half-blind.The customer supplied the facts that decide the design:
<lastmod>is not (every entry carries the build date).Refererheader (any value) or serves a cached 404 page. They also confirmed the page's own price call can be dropped for bot renders.Resolved
The probe stays; it moved endpoints. With a cached, PDP-coherent endpoint the probe costs the origin one product-service read per product per pass and detects nearly every rich-result field, which removes the reason for the fleet-side "conditional render" this issue originally designed (see "Dropped" below).
changeProbe.rules-> the details endpoint (GET,refererheader, 20 extract slots: price core, status word, price set, all-variants flag, canonical slug, sale status, percentage off, promotion, badges, new/best-seller/top-rated flags, title, brand, meta title/description, main image, breadcrumbs), withdryRun: trueandreprobeAfter: 0for one full cycle because the signature shape changed for every URL and the plugin had no way to tell a rule edit from a content change (the canary would have read 100% changed and invalidated the route). The in-flight old-rule passes were cancelled with a momentaryenabled: falseso nothing kept polling the old path.prerender-v0.65.0) makes that procedure unnecessary and ships the generic pieces:ProbeState.ruleFingerprint: a baseline records the rule that made it; a mismatch re-baselines (stored, not compared, not triggered, outside the canary verdict;rebaselinedstat,probe_rebaselinedmetric). A rule edit costs one pass without detection for that rule and no dry-run cycle.pageCheckreads availability words (In Stock,IN_STOCK, schema.org forms) and[*]per-variant lists, with per-ruleavailableValues/unavailableValuesover the built-in vocabulary.[*]projection in extract paths, so a rule can watchvariants[*].availabilitywithout signing whole variant objects.mode: anchored(anchorTime,anchorTimezone,anchorWindow): one full pass a day starting right after the origin's scheduled change, at the ceiling by default; no boot sweep; the canary keeps its cadence. This is the "nightly pass" of the original design, as configuration.block.urlPatternsleft JSON-LD offers (12/12), aggregate rating, price markup and error-string counts identical on desktop and mobile. That is a fleet configuration change, no code.Deployment sequence for the customer: release 0.65.0 -> bump the component and port the rule into
config.yaml(identical fingerprint to the override rule, verified) -> clear the override rows (changeProbe.rules,.dryRun,.reprobeAfter,.cycleTarget) once the re-baseline pass has finished on every node -> fleet config with the block pattern.Dropped: conditional render in the fleet (former stages 2-3)
Pre-fetching the SSR document in the worker, extracting a signature, and seeding #157's document cache was designed to replace price polling with a fetch the render needed anyway. Against a cached endpoint that costs the origin ~nothing and answers in 15-35 KB, that machinery no longer buys origin relief. What it still would buy — ~6-15% render throughput from hiding the document fetch, and ~80% fewer blob writes from making cadence renders conditional — is a render-pipeline optimization that belongs with #155/#156/#157 on its own merits, not to this issue. Its measurements stay valid and are kept here for that work:
Vary: Accept-Encodingonly, so one document fetch per product is valid.Set-Cookie, so its scripts would run without the bucket and itssampleEverycheck (SSR vs SSR) cannot see it. Rules: pin listed cookies on every variant (cookies.pin), never cross session/cart/visitor/bot-manager cookies, allowlist deterministic bucket cookies per job, comparestructuredOffersbetween variants on sample jobs, and enabledocumentReuseon that deployment only after one PDP render shows the replayed variant matching.Remaining follow-ups (this issue stays open for these)
Field-class lanes. Widening the signature to ratings would re-render a product per new review. Give extract slots a lane (
immediatefor price/availability/canonical,nightlyfor copy, images, ratings, breadcrumbs) so the anchored pass absorbs slow-moving classes and the canary/immediate path stays about price. Run the wider signature in dry-run first and read the changed-share split (probe_changed) before turning classes into renders.Sitemap-membership availability — BUILT, awaiting merge (Re-check URLs that leave a sitemap, per route (v0.68.0) #164,
prerender-v0.68.0). Per-routeingress.routes[].departureAction(none|expire|render): a PDP target that becomesunlinkedhas its pages HARD-expired (pastpage.swrTtl, so they stop serving rather than serving stale — a plainDate.now()expiry leaves the pageswrand still serving, which is the trap here) and is filed to render at the current minute. Route-scoped as this item specified, so catalog pages are excluded: a product URL leaving a product sitemap says something about that product, while a listing URL leaving a paginated sitemap usually means the catalog was re-bucketed. Never deleted — the origin still serves a 200 with out-of-stock markup, so this is a re-check and the render's own verdict decides. Capped per walk (sitemap.departure.maxActions), anddryRundefaults TRUE because the count a real walk would act on is not yet known (see the measurement note below).The trap that shaped the design: the check must run AFTER the whole walk, not at prune time. The product sitemap is 17 children of exactly 50,000 entries ordered by ascending
prd-id, so it shears across boundaries — a URL moving to an EARLIER child is re-attached before the child it left is pruned and never looks departed, but one moving to a LATER child is pruned first and looks departed until the child that now claims it is reached. Acting at prune time would fire on every URL after an insertion. Candidates are re-read once the walk ends;departure_reattachedcounts the shear, and a large share there means the rawremovedcount is not a departure count.Still unmeasured: how many PDPs actually depart per day. The
SitemapRefreshrow keeps only the last run, and the pass I read (18:00 ET:created 65 / updated 3 / removed 38 / skipped 886,471) is not the one that absorbs the midnight-CT rebuild — that is the 06:00 ET pass. That number sizesmaxActions, which is why the feature ships in dry run.Distinct probe identity.
changeProbe.userAgent(or a header) so an origin's CDN can allowlist verification traffic separately from renders; today the probe ridesorigin.userAgents.desktop.Canonical slug as a signal. With the canonical in the signature, a slug change can retarget without rendering (today it surfaces as a canonical-mismatch suppressed render, ~153k/day on this deployment).
Console. Surface
rebaselinedandnextAnchoredRunAt; alert on a steadyprobe_rebaselined(a rule that keeps changing).Load (measured on the deployment)
ratePerSecondceiling (10/s/node) for ~12 h rather than spread.Acceptance
probe_failedback at the 1-2% floor on the details endpoint (first dry-run canary: 500 probed, 0 failed).nextAnchoredRunAtpublished.probe_rebaselinedand no canary trip.departure_would_renderanddeparture_reattachedbeforesitemap.departure.dryRunis turned off.Notes / risks
probe_*rows emit per finished pass (smeared); use canary rows or the Invalidation table for flip timing. probe_* are value metrics (mean x count).Versions:
prerender-v0.65.0= #159 (also claimed by #154; whichever merges second renumbers). The former stage-2 browser version (1.25.0) is released back to the #155-#157 line.