Allow Dredge AP to use online motion resolution method

[chrishalcrow]

The motion correction algorithms in SpikeInterface have two steps. You first compute many histograms-in-depth (using e.g. peak detection or LFP) over time. Second, you compare the histograms at different times to extract a time vector.

In the current implementation, if you use peak detection the comparison part of the algorithm compares ALL histograms against all others. This is fine if you have a standard recording, but fills up RAM quickly if you get towards one day long recordings. If you use LFP there is an "online" method to compare the histograms. This only compares histograms which are nearby in time, meaning that you don't need to load them all into memory. Hence, this can be used for very long recordings.

This PR refactors the "all at once" and "online" methods to extract motion vectors from histograms, allowing either method to be used regardless of histogram-construction method. Most of the pain is in how parameters are dealt with - they were packaged a bit differently for the two algorithms. We attempt to unify the arguments in the two methods as much as possible, and abstract some language.

Docs, tests, etc incoming.

I spent a while understanding the parameters and all the kwarg dicts. Here's a summary:

Parameters

User can give dredge_ap and dredge_online_lpf the following dicts:

thomas_kw: passed to thomas_solve
xcorr_kw: passed to xcorr_windows

Inside the functions, we set the following kwarg dicts:

full_xcorr_kw

• rigid
• bin_um
• max_disp_um
• progress_bar
• device

weights_kw (passed to weight_correlation_matrixin AP, just used to get mincorr in LFP)

• mincorr
• time_horizon_s
• do_window_weights
• weights_threshold_low
• weights_threshold_high

threshold_kw (passed to threshold_correlation_matrix). We set:

• mincorr_percentile_nneighbs
• in_place
• soft
• time_horizon_s
• bin_s

[cwindolf]

This is amazing! I'll take a look at the code.

[cwindolf]

Can't see any issues here, this is great!

[chrishalcrow]

I think this is working. I still want to add tests and docs, but if anyone wants to try it out on some real data, have a go! @alejoe91 @cwindolf

Here's an hour chronic recording estimated motion:

on main:

this PR, full:
motion, motion_info = si.compute_motion(sliced_rec, output_motion_info=True, estimate_motion_kwargs={'rigid': True, 'batching_mode': 'full'})

this PR, online:
motion, motion_info = si.compute_motion(sliced_rec, output_motion_info=True, estimate_motion_kwargs={'rigid': True, 'batching_mode': 'online'})

[alejoe91]

Suggested change

	chunk_len_s=None,
	chunk_len_s: float | None, default: None,

[alejoe91]

I think that a better behavior would be to fail if mode is online and chunk_len_s is None. You want online? Give me chunk size!

[alejoe91]

Add comment about longer chunks is better.

[samuelgarcia]

You are a hero.

[samuelgarcia]

Does the raster fit in memory for long recording ?
Should we also add some kind of lazy raster on demand given peaks and locations ? (that could mimic the recording API ... maybe ugly)

@cwindolf @chrishalcrow what do you think ?

[cwindolf]

I think it's okay to keep the raster in memory. Even 100 days' recording is <10G on NP1. Oops total lie. It's like 140G. Hmm...

Chris is a hero!! Next up is multi-segment :)

[cwindolf]

Maybe this can be merged with the raster in memory for now. I think it's more natural to do the rasters separately in a multi-segment setting, so maybe when we implement multi-segment we can improve this?