Skip to content

Add CCKR adaptive log reservoir sampler to Microsoft.Extensions.Telem… - #7676

Draft
evgenyfedorov2 wants to merge 1 commit into
dotnet:mainfrom
evgenyfedorov2:evgenyfedorov2/cckr-log-sampling
Draft

evgenyfedorov2 wants to merge 1 commit into
dotnet:mainfrom
evgenyfedorov2:evgenyfedorov2/cckr-log-sampling

Conversation

@evgenyfedorov2

@evgenyfedorov2 evgenyfedorov2 commented Aug 3, 2026

Copy link
Copy Markdown
Member

…etry

Implements the CCKR (Chao-Cohen-Kaplan-Reservoir) adaptive log sampler as implementations of the existing LoggingSampler and LogBuffer seams, so no changes to the logging pipeline are required:

  • Cckr<TCallsite,TPayload>: bottom-(K+1) weighted reservoir with EXP ranks, cross-period inverse-frequency feedback, Chao1 unseen-weight, bounded novelty preserve, and Horvitz-Thompson sampling weights (faithful port of the reference algorithm).

  • CckrLoggingSampler (LoggingSampler): the admit/drop decision. CckrLogBuffer (LogBuffer): holds admitted records per category and emits the kept records carrying the sampling.count weight at each flush, reusing SerializedLogRecord/DeserializedLogRecord/IBufferedLogger.

  • AddCckrLogSampling registers one reservoir instance as both seams. Public API: AddCckrLogSampling, ReservoirSamplingConfig, UnseenWeightMode.

Microsoft Reviewers: Open in CodeFlow

…etry

Implements the CCKR (Chao-Cohen-Kaplan-Reservoir) adaptive log sampler as implementations of the existing LoggingSampler and LogBuffer seams, so no changes to the logging pipeline are required:

- Cckr<TCallsite,TPayload>: bottom-(K+1) weighted reservoir with EXP ranks, cross-period inverse-frequency feedback, Chao1 unseen-weight, bounded novelty preserve, and Horvitz-Thompson sampling weights (faithful port of the reference algorithm).

- CckrLoggingSampler (LoggingSampler): the admit/drop decision. CckrLogBuffer (LogBuffer): holds admitted records per category and emits the kept records carrying the sampling.count weight at each flush, reusing SerializedLogRecord/DeserializedLogRecord/IBufferedLogger.

- AddCckrLogSampling registers one reservoir instance as both seams. Public API: AddCckrLogSampling, ReservoirSamplingConfig, UnseenWeightMode.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
state.HeapCount++;
HeapPush(new HeapEntry(key, callsite, payload));

if (_heap.Count > _reservoirCapacity)

@Yun-Ting Yun-Ting Sep 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (_heap.Count > _reservoirCapacity)
// Retain T+1 entries: the root is the (K+1)-th smallest rank, the first item excluded from the bottom-K sample (tau)
if (_heap.Count > _reservoirCapacity + 1)

In Bottom (K + 1) algorithm, tau is the rank of the first excluded item.
That's what keeps tau independent of every retained item's own rank; the condition the HT weights to be correct.

{
_ = Throw.IfNull(output);

double finalTau = Tau;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
double finalTau = Tau;
// tau is the (K+1)-th smallest rank (the first excluded item), so take it from the heap before draining. i.e.: leaving exactly K records to emit.
double finalTau = double.PostiveInfinity;
if (_heap.Count > _reservoirCapcity)
{
finalTau = HeapPopMax().Key;
}

}
}

public void Dispose() => _pending.Dispose();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question - do we need to call Flush() here before disposing, so we don't lose whatever's still in the reservoirs up to a full FlushInterval?

I don't think the providers are gone by this point. And ExtendedLoggerFactory.cs:76 registers DI-supplied providers with dispose: false, so the factory skips them and the container disposes them itself (in reverse creation order.)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants