Problem
crates/openquant/src/microstructural_features.rs, MicrostructuralFeaturesGenerator::new_from_csv:
tick_num_iter: tick_num_series.to_vec().into_iter(),
current_bar_tick: tick_num_series.first().copied().unwrap_or(0),
first() reads the first threshold without advancing the iterator. In get_features_from_csv, after the first bar closes, tick_num_iter.next() therefore returns the first threshold again, and the very next tick satisfies tick_num >= current_bar_tick.
With thresholds [10, 20, 30] the generator emits bars at ticks 10, 11, 20, 30 — four bars instead of three. The extra bar holds a single tick, and the bar that should cover ticks 11–20 covers 12–20.
mlfinlab's generator consumes the first element when it initialises (self.current_bar_tick_num = next(self.tick_num_generator)), so features align one-to-one with the bars that produced tick_num_series.
Consequence
Feature rows do not align with the bars they are supposed to describe: joining them to a bar frame by position is off by one from the second row on, and row 2 is a degenerate one-tick bar (VWAP = last price, zero-variance inputs to Kyle/Amihud/Hasbrouck).
Why tests missed it
test_feature_generator_function asserts only that the output is non-empty and the first bar's average tick size is positive. It never checks the number of rows.
Fix
- Advance the iterator when taking the first threshold.
- Test: number of feature rows equals
tick_num_series.len(); a synthetic CSV with thresholds [10, 20, 30] yields exactly 3 bars whose tick counts are 10, 10, 10.
Found while reviewing #64 (left unchanged there; that PR is behavior-preserving).
Problem
crates/openquant/src/microstructural_features.rs,MicrostructuralFeaturesGenerator::new_from_csv:first()reads the first threshold without advancing the iterator. Inget_features_from_csv, after the first bar closes,tick_num_iter.next()therefore returns the first threshold again, and the very next tick satisfiestick_num >= current_bar_tick.With thresholds
[10, 20, 30]the generator emits bars at ticks 10, 11, 20, 30 — four bars instead of three. The extra bar holds a single tick, and the bar that should cover ticks 11–20 covers 12–20.mlfinlab's generator consumes the first element when it initialises (
self.current_bar_tick_num = next(self.tick_num_generator)), so features align one-to-one with the bars that producedtick_num_series.Consequence
Feature rows do not align with the bars they are supposed to describe: joining them to a bar frame by position is off by one from the second row on, and row 2 is a degenerate one-tick bar (VWAP = last price, zero-variance inputs to Kyle/Amihud/Hasbrouck).
Why tests missed it
test_feature_generator_functionasserts only that the output is non-empty and the first bar's average tick size is positive. It never checks the number of rows.Fix
tick_num_series.len(); a synthetic CSV with thresholds[10, 20, 30]yields exactly 3 bars whose tick counts are 10, 10, 10.Found while reviewing #64 (left unchanged there; that PR is behavior-preserving).