Skip to content

feat(changelog): support full-compaction mode changelog producer - #266

Open
lszskye wants to merge 3 commits into
apache:mainfrom
lszskye:full_compaction_changelog
Open

feat(changelog): support full-compaction mode changelog producer#266
lszskye wants to merge 3 commits into
apache:mainfrom
lszskye:full_compaction_changelog

Conversation

@lszskye

@lszskye lszskye commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #174

Support changelog-producer=full-compaction for primary-key tables.

This change:

  • adds FullChangelogMergeFunctionWrapper to compare max-level records with merged results and generate INSERT, UPDATE_BEFORE, UPDATE_AFTER, and DELETE records;
  • adds FullChangelogMergeTreeCompactRewriter and wires it into the compact manager factory;
  • generates changelog files when compaction outputs to the maximum level;
  • supports changelog row deduplication and ignored fields;
  • enables streaming reads of full-compaction changelog manifests;
  • limits the initial full streaming scan to max-level files so pending lower-level changes are not emitted twice;
  • removes the unused output-level argument from the regular MergeFunctionWrapperFactory;
  • removes the validation that previously rejected the full-compaction changelog producer.

Tests

Added UT for:

  • single-record full-compaction results;
  • INSERT, UPDATE, and DELETE changelog generation;
  • changelog row deduplication;
  • row-deduplication ignored fields;
  • rejecting multiple max-level records for the same key;

Added integration coverage for:
WriteAndReadInteTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant