docs(content-drive): spike spec for folder-listing query plan instability (#37183)

[ihoffmann-dot]

Spec-Kit PR 1 of 2 — Spike. Carries the spike spec only; this investigation does not prescribe a fix (see below). Needs a developer approval (not a merge) before /speckit-plan proceeds on whichever direction the team picks.

Resolves the spec phase of #37183.

Proposed Changes

• spec.md — problem statement, investigation to date (measured facts), three candidate directions with tradeoffs and open unknowns per direction (none verified — no live instance was available to prototype), a hypothesis-ranking on the unexplained latency bimodality on large folders, and a recommendation for what a follow-up prototype should measure before committing to a design.

Why a spike, not a feature spec

This started as a proposed fix (a folder-first query rewrite) that measured 4.3x faster on the largest folder tested — but 3x slower on a mid-size folder, a real regression verified with identical result sets in both directions. The root cause isn't a bad query, it's PostgreSQL planner instability tied to LIMIT: below ~500 rows the planner walks the sort-column index hoping to satisfy the limit early; above it, it switches to gather+sort, ~10x faster on large folders. The production chunk size (400) sits at the worst point of the unstable range. A configuration-only mitigation (raising chunk size) was also tested and rejected — it regressed a different mid-size folder by 42%.

None of the three candidate directions in the spec is picked as "the" fix. That decision needs either a prototype or a plan-phase design spike, informed by this document.

Coordination note

Shares the underlying query with #37184 (field-filter chunk multiplier), already spec'd separately. Whatever direction is chosen here must preserve that fix's single-scan-per-request assumption.

Checklist

• Direction chosen and a follow-up plan/prototype scoped, before /speckit-plan proceeds
• Latency-bimodality hypothesis (plan-cache generic/custom plan flip on this same query, unconfirmed) investigated with a live instance

Additional Info

Parent epic #36814. Originally investigated as item 1 of #37148 (umbrella investigation issue covering 4 related performance items in Content Drive/Site Browser).

/speckit-adr-context consulted ADR-0018 (database-first search for Content Drive) — none of the three candidate directions moves folder-scoping or ordering resolution to the search index, so no conflict identified. No new ADR proposed at this stage; one of the candidate directions (a) is itself a possible schema change that, if pursued, likely warrants its own ADR before implementation.

🤖 Generated with Claude Code

This PR fixes: #37183

[claude]

Claude finished @ihoffmann-dot's task in 1m 44s —— View job

---

Review: spike spec for folder-listing planner instability

Docs-only PR (one added spec.md). No code, so no runtime/security surface. I verified the spec's factual claims against the current source on this branch — the substance holds up, but the line references are stale.

Verification (substance ✅)

• Folder predicate and id.parent_path=? with no host_inode in that clause — confirmed (BrowserAPIImpl.java:2198).
• Host scoping added as a separate and (id.host_inode = ?) clause via appendSiteQuery — confirmed (BrowserAPIImpl.java:2179). The spec's careful correction of the issue's "adds host filter as a side effect" framing is accurate.
• ORDER BY c.mod_date asc|desc with no tiebreaker — confirmed (BrowserAPIImpl.java:2574–2581); the non-determinism finding is real.
• BROWSER_DB_CHUNK_FACTOR=10, BROWSER_DB_CHUNK_MIN_SIZE=200 ⇒ LIMIT 400 for a 40-row page — confirmed (BrowserAPIImpl.java:729–734, 216).
• Single-chunk exit for the non-ES path (accumulatedContent.size() >= maxRows) — confirmed (BrowserAPIImpl.java:296). The item-2 coordination invariant is correctly described.
• "No index on identifier.parent_path alone; only unique(parent_path, asset_name, host_inode) and the functional idx_identifier_parent_path_trigger" — confirmed (postgres.sql:1033, 2236).

New Issues

• 🟡 Medium: specs/37183-content-drive-folder-listing-plan-instability/spec.md:36-43,113(sic),150-165,327-352 — All line references are off in the current tree. They fall into two consistent bands: the low refs are +5 (e.g. getContentUnderParentFromDB cited :192 → actually 197; chunkSize :211 → 216; buildPaginatedDotConnect :364 → 369; chunk constants :724–729 → 729–734), and the high refs are +61 (selectQuery :1946 → 2007; appendSiteQuery :2113 → 2174; appendFolderQuery :2134/predicate :2137 → 2195/2198; appendOrderByQuery :2513 → 2574; getContentByChunks loop :262–309/exit :291 → 267–314/296). This is consistent with BrowserAPIImpl.java having shifted under the spec (the search-links commit 98bc757d landed just before this branch). Method/symbol names all still resolve, so the doc is navigable — but a spike's whole value is being a decision-ready record that points reviewers at exact code, and every citation currently lands ~5–61 lines early. Suggest citing by method name + relative anchor rather than absolute line numbers, or refreshing them against this branch. Fix this →

• 🟡 Medium: specs/37183-content-drive-folder-listing-plan-instability/spec.md:3 — Feature Branch header says 37148-content-drive-folder-listing-plan-instability but the actual PR branch is issue-37183-content-drive-folder-listing-plan-instability (and the folder/issue is 37183). The header points at the umbrella issue number and omits the issue- prefix, so it won't match the real branch.

• 🟡 Medium: specs/37183-content-drive-folder-listing-plan-instability/spec.md:158 — Unbalanced quote in the host-filtering paragraph: ...the CTE's \where parent_path = ? and host_inode = ?" as a single...` opens with a backtick and closes with a straight double-quote. Cosmetic, but it's inside the most nuanced correctness paragraph, so worth cleaning.

Notes (non-blocking, spec quality)

• The document is unusually disciplined for a spike — every candidate direction is explicitly labeled unverified, tradeoffs and open unknowns are separated, and the "cheapest experiment first" ordering (stats-target → count-probe cost → denormalization) is sound. The ADR-0018 read-your-writes constraint on direction (a) option 3 (materialized view) is correctly identified.
• The two correctness findings (§5: missing host_inode in the folder predicate is not actually missing — it's a separate clause; and the missing ORDER BY tiebreaker) are both accurate and correctly scoped as "flag for the fix designer, not a spike deliverable."

Nothing here blocks the spec approval — the analysis is correct where it counts. The line-number drift is the one thing I'd fix before this becomes the reference other devs navigate from.

· branch issue-37183-content-drive-folder-listing-plan-instability