Skip to content

PurgedKFold::split leaks: test window starts at the first test label's END time - #66

Merged
Sean-Koval merged 4 commits into
mainfrom
work/pkf-purge-window
Sep 19, 2026
Merged

Sean-Koval merged 4 commits into
mainfrom
work/pkf-purge-window

Conversation

@Sean-Koval

Copy link
Copy Markdown
Contributor

Scope

[code / P0] PurgedKFold::split in crates/openquant/src/cross_validation.rs builds its purge window from .1 (label END) of the first test sample instead of .0 (label START), and takes the window end from the last test sample rather than the max end over the fold. Training samples whose labels overlap the leading test labels are therefore not purged (label leakage), and variable-length labels can under-purge the trailing edge. In scope: correct the window, add a regression property test that fails before the fix, check every other caller/copy of the same window logic in the crate (the free ml_get_train_times-style helper near line 95, feature-importance and hyperparameter-tuning CV paths) and the docs page for the module. Out of scope: Python bindings (#42), re-porting 59ac6fa (#33), changing embargo semantics.

Acceptance

  • A property test over random variable-length label spans with pct_embargo = 0 asserts that no training label span intersects any test label span; it fails on main and passes after the fix.
  • A hand-checkable case (daily bars, 3-day labels) asserts the exact expected train indices for each fold.
  • Existing cross_validation, feature_importance and hyperparameter_tuning tests pass; any expectation that changes is listed in the PR with the reason it was wrong before.
  • Every other occurrence of the window logic in the crate is fixed or shown correct.

Closes #65

Sean-Koval and others added 3 commits September 18, 2026 19:14
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n test rows

split() built its purge window from the END time of the first test label and
the end of the last one. Training labels overlapping the first test labels
were kept (leakage), and with variable-length labels the window could invert
(start after end), purge nothing, and leave the test rows themselves in the
training set.

Use the first test label's start and the latest end in the fold (AFML 7.3),
and drop the test indices explicitly. new() now rejects n_splits < 2 or
> n_samples, which previously panicked (division by zero / empty fold).

Tests: an exact 3-day-label case, a seeded property test over variable-length
spans asserting no train label intersects any test label, and the n_splits
bounds. The first two fail before this change. No existing expectation moved.

Fixes #65

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Sean-Koval

Copy link
Copy Markdown
Contributor Author

Review notes

Worse than filed in #65. The property test showed that with variable-length labels the old window could invert (test_start = end of the first test label can be later than test_end = end of the last). Then nothing was purged and the test rows stayed in the training set — on main, the seeded case fails with train 1 and test 0 from the same fold. The original code only ever removed test rows as a side effect of purging.

Blast radius. ml_cross_val_score, feature importance and hyperparameter tuning take their splits from PurgedKFold, so any CV score computed with overlapping labels was optimistic. backtesting_engine::apply_purge_and_embargo compares spans pairwise per test index and is not affected. ml_get_train_times is correct given a correct window.

Behavior changes

  • Fewer training rows per fold whenever labels overlap fold boundaries (the fix).
  • PurgedKFold::new returns Err for n_splits < 2 or > n_samples; both previously panicked.
  • Unchanged: embargo semantics (still applied on both sides of the fold, which is more conservative than AFML's after-only embargo — left alone deliberately).

Evidence. Two of the three new tests fail on main and pass here. cargo test --workspace: 246 passed / 0 failed / 1 ignored; no existing expectation changed. cargo fmt --check clean; clippy clean under the flags CI uses on main.

Merge note. #64 also edits cross_validation.rs (type alias, loop rewrites). Whichever merges second needs a small conflict resolution; I'd merge this one first since it is the correctness fix.

@Sean-Koval

Copy link
Copy Markdown
Contributor Author

CI note: benchmark-regression failed once on synthetic_ticker_pipeline_end_to_end (+53.2% vs a 50% limit) and passed on re-run with no code change. That bench imports only risk_metrics, sampling and fast_ewma, none of which this PR touches, and the same job passed on the previous push of this branch. Shared-runner timing noise — the gate's flakiness is worth its own look under #41.

@Sean-Koval
Sean-Koval merged commit 54c38f2 into main Sep 19, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PurgedKFold::split leaks: test window starts at the first test label's END time

1 participant