Skip to content

docs(optimizer): reconcile optimization and sketch-bench formulations #652

Description

@milindsrivastava1997

Summary

Reconcile the overlapping optimization/sketch-bench design documents so there is one clearly identified canonical formulation and an explicit record of what is conceptual, proposed, implemented, or blocked.

Scope

Review and update:

  • .design_docs/optimizer-simplified-formulation.md
  • .design_docs/sketch-config-optimization-formulation.md
  • .design_docs/optimizer-mip-formulation.md
  • .design_docs/optimizer-v1-implementation-plan.md
  • .design_docs/sliding-window-multi-window-merge-blocker.md where needed

Reconciliation points

  • Explain the progression from abstract summarization strategies to AQE/config-level facility-location MIP.
  • Establish the canonical definitions of RQE, QE, AQE, candidate config, deployment cost, and assignment cost.
  • Reconcile the two sliding-window models: current single-window/exact-range engine behavior versus the proposed multi-window sliding extension.
  • Reconcile ingestion-memory versus retained/query-working-memory accounting, including sliding retention overhead.
  • State the assumptions and limits of O(1) subtraction, especially when the repeat interval exceeds the window size.
  • Clarify the role and current data gaps of sketch-bench: atomic cost measurements, accuracy lookup, cardinality/distribution profiling, and batch-versus-streaming validity.
  • Align the documented MIP with the implementation status: current per-AQE greedy selection, missing cross-AQE sharing, placeholder label-group multiplier, and stub costs.
  • Mark superseded or historical formulations clearly, or consolidate them into a single canonical design document with links from the others.

Acceptance criteria

  • A reader can identify which document is authoritative for the mathematical formulation and which describes implementation status.
  • The sliding-window feasibility rules are unambiguous and distinguish supported behavior from future extensions.
  • Cost and accuracy inputs are defined consistently across documents.
  • Known unresolved assumptions remain explicitly tracked rather than appearing as settled behavior.
  • All affected internal links and terminology are updated.

References

  • .design_docs/CAPABILITY_MATCHING_DESIGN.md
  • .design_docs/query-engine-fetch-merge-audit.md
  • .design_docs/sliding-window-multi-window-merge-blocker.md

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions