missing_docs: stop the drift-watch triage from silently seeing nothing - #617
Conversation
Found by the first manual validation run of the drift-watch skill. All three defects fail the same way: the run exits 0, reports nothing, and that is indistinguishable from "nothing shipped". 1. Triage baseline ignored the release marker Two independent state files track releases. surface_snapshot.json records what was last *observed*; last_release_processed.json records what was last *triaged*. A bookkeeping PR that regenerates the snapshot advances the first without touching the second, and #603 did exactly that -- on main today the snapshot reads 2026.08.19 while the marker reads 2026.08.18. The gate fires (marker behind) but --diff returns an empty changelog_review (snapshot ahead), so the primary triage input is empty and the release gets marked processed having triaged nothing. The baseline is now the earlier of the two markers. Against current main that moves changelog_review from 0 items to 3. I introduced this when I split the marker out of the snapshot. The reasoning was sound -- the snapshot is regenerated wholesale and would lose the marker -- but it only considered the snapshot falling behind, not running ahead. 2. oz_updates were2. oz_updates were2. oz_updates were2. oz_updatrepo2. oz_updates were2. oz_updates were2. oz_up no2. oz_updates were2. oz_updates were2. oz_updates were2. oz_updatrepo2. nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnntfnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnh was not executable. Texecutable. Texecutable. Texecutable. Texecutable. Texecun. The field nThe field nThe field nThe field nThe field nThe fiel in the currcurrcurrcue pcurrcurrcurrcue pcurrcurrcurrcue pcurrcurrcurrcue pcurrcuchangelog heading would fail silently Section matching is a hardcoded allowlist against a user-facing heading, so the rename would zero out that section with no error. Added the Automation Platform variants, plus a guard that treats bullets under an unrecounrecounrecounrecounrecounrecounrecounrecounrecxes are listed as dededededededededededededededededededededededededededededededededededed ttttttttttttttttttttttttttttttttttttttttttttttttttttter launch posts use prose headings and policing all 277 entries would fail every run. Also adds the __future__ annotations import that check_new_release.py already had. Without it audit_docs.py cannot be imported on Python 3.9,already had. Without it audit_docs.py cannot be imported on Python 3.9,already had. Without it audit_docs.py cannot be importedver the baseline, the normalizer, the tracked/untracked/unknown split, and thtracked/untracked/unknown split, and thtracked/untrp.dev>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![warp-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR makes the missing_docs drift-watch path safer by aligning changelog triage to the earlier of the observed and processed release markers, surfacing oz_updates from the release gate, and failing loudly on unrecognized pending changelog sections. The regression tests cover the marker baseline, version normalization, tracked/untracked section handling, and unknown-section scoping; I did not find blocking correctness, security, or spec-drift issues.
Concerns
- Non-blocking: the new marker reader should defensively handle syntactically valid JSON with the wrong top-level shape before calling
.get, matching the safe fallback described in its docstring.
Verdict
Found: 0 critical, 0 important, 1 suggestions
Approve with nits
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
![warp-agent-staging[bot]](https://avatars.githubusercontent.com/in/2196476?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR improves drift-watch changelog triage by preserving the earlier observed or triaged baseline, surfacing platform-update bullets, and failing visibly for unrecognized changelog sections. The focused regression suites pass, and the audit path reports the expected pending release candidates.
Validation
gh pr checks 617 --required— passed (required CLA check)- Focused missing-docs test suites — passed: 17 audit tests (7 skipped because the sibling repos are not colocated), 17 release-check tests, and 15 reviewer-routing tests
audit_docs.py --warp /workspace/warp --warp-server /workspace/warp-server --diff— completed and reported the expected pending changelog entriescheck_new_release.py --json— emitted the currentoz_updatesbulletstyle_lint --changedand internal-link check — passednpm run build— blocked by the pre-existing_llms-txt/support.txtstack overflow on both this PR andmainunder Node 24;npm run lintcould not run becausetrunkis not installed in the sandbox
Verdict
Found: 0 critical, 1 important, 0 suggestions, 0 nits
Request changes
Review catch from both bots on #617. read_last_processed_release() called .get() on whatever json.loads returned, so a marker containing valid JSON of the wrong top-level type -- [], a bare string, a number -- raised AttributeError rather than taking the documented fallback to None. That aborts diff-mode triage entirely, which is a worse outcome than the desync the reader was added to fix. Guarded with isinstance(data, dict) and a warning, matching how check_new_release.py's read_state() already handles the same file. The inconsistency between two readers of one file was the actual defect. Regression test covers every bad shape: list, bare string, number, null, truncated JSON, empty file, and an object missing the expected key, plus the missing-marker and well-formed cases so the fallback cannot be achieved by breaking the happy path. Co-Authored-By: Warp <agent@warp.dev>
There was a problem hiding this comment.
Overview
The follow-up fixes the release-marker reader so valid JSON with a non-object top level takes the documented safe fallback instead of raising. It adds regression coverage for each malformed marker shape and retains the valid-marker path.
Validation
gh pr checks 617 --required— passed- Focused missing-docs test suites — passed: 18 audit tests (7 skipped because the sibling repos are not colocated), 17 release-check tests, and 15 reviewer-routing tests
audit_docs.py --warp /workspace/warp --warp-server /workspace/warp-server --diff— completed and reported the expected pending changelog entriescheck_new_release.py --json— emitted the currentoz_updatesbulletnpm run build— fails identically on the PR andmainat the existing/_llms-txt/support.txtstack overflow under Node 24;npm run lintcould not run becausetrunkis not installed in the sandbox
Verdict
Found: 0 critical, 0 important, 0 suggestions, 0 nits
Approve
1 participant
Summary
The first manual validation run of the
missing_docsdrift-watch skill (run) found three defects. They fail the same way: the run exits 0, reports nothing, and that is indistinguishable from "nothing shipped."The gate and the audit both worked correctly — nine candidates, zero feature PRs, bookkeeping only. These are gaps in what reached triage in the first place.
The bugs
1. The triage baseline ignored the release marker
Two state files track releases, and they can disagree:
surface_snapshot.json→ what was last observedlast_release_processed.json→ what was last triagedA bookkeeping PR that regenerates the snapshot advances the first without touching the second. #603 did exactly that, so on
maintoday:The gate fires because the marker is behind.
--diffthen returns an emptychangelog_reviewbecause the snapshot is ahead. The primary triage input is empty, nothing is reviewed, and the release gets recorded as processed.The baseline is now the earlier of the two markers. Against current
mainthat moveschangelog_reviewfrom 0 items to 3 — including the release's oneoz_updatesbullet.I introduced this when I split the marker out of the snapshot in #586. The reasoning was sound — the snapshot is regenerated wholesale and would lose the marker — but it only accounted for the snapshot falling behind, never running ahead.
2.
oz_updateswere counted but never showncheck_new_release.pyprintedOz updates: 1.audit_docs.pyhas zero references to the field, and the audit is offline by design, so it never will.The skill names
oz_updatesas the preferred source for platform changes the client changelog cannot see. That instruction was not executable — the validating agent had to fetchclient_versionby hand. The gate now prints the bullets and includes them in--json.The field name stays
oz_updates. It is the API's, verified live in the current release payload, and product naming does not apply to a data contract.3. A renamed changelog heading would fail silently
Section matching is a hardcoded allowlist against a user-facing heading — squarely in scope for the Automation Platform rename. Today it matches
**Oz updates**; the day that heading changes, the section yields zero bullets with no error.Added the Automation Platform variants, plus a guard that treats bullets under an unrecognized heading as an extraction failure (exit 2) rather than an empty result.
Two details that matter:
Bug fixesis listed as deliberately untracked, so an intentional skip cannot read as a rename.**Multithread yourself with agents**), and policing all 277 historical entries would fail every run. Caught by running it.Also
audit_docs.pywas missing thefrom __future__ import annotationsthatcheck_new_release.pyalready has. Without it the module cannot be imported on Python 3.9, sotest_audit_docs.pydied before running a single test on any machine using the system interpreter.Validation
test_check_new_release.py17/17,test_suggest_reviewers.py15/15.test_audit_docs.py16/17. The one failure istest_diff_against_committed_snapshot_is_current, which is environmental: my localwarp-servercheckout is three months stale, so 123api_routes_removedand 16slash_commands_removedare the local repos lagging the committed snapshot, not drift in it. Previously masked by the import error above.Notes for reviewers
Draft PR #615 is closed in favor of a clean re-run after this lands. Its triage was sound — the verdicts are well-reasoned and the surface-map entries correct — but it got there by manually working around the desync, and merging it would advance the marker past the release the post-fix validation run needs to triage. Everything in it regenerates.
Remaining feedback from the validation run, not addressed here:
rgis unavailable in the environment (skill examples use it), ledger rows for non-PR items have no stable key so they will be re-proposed, and "major launch" deferral could use a first-class verdict label distinct from Gate 0.Unverified claims
None — no page content changes. Behavior was verified by running the scripts against the live
client_versionpayload and the committed changelog, and by the regression tests above.Co-Authored-By: Warp agent@warp.dev