Skip to content

[FLINK-40466][table] Refactor defining table arguments in PTF Test Harness for more complex table arguments - #29018

Open
autophagy wants to merge 1 commit into
apache:masterfrom
autophagy:FLINK-40466-harness-table-argument-builder
Open

[FLINK-40466][table] Refactor defining table arguments in PTF Test Harness for more complex table arguments#29018
autophagy wants to merge 1 commit into
apache:masterfrom
autophagy:FLINK-40466-harness-table-argument-builder

Conversation

@autophagy

Copy link
Copy Markdown
Contributor

What is the purpose of the change

While working on extending ProcessTableFunctionTestHarness to better support changelog processing modes and order by/late data handling, I noticed the current way of building table arguments started suffer from duplicated effort. For example, it started resulting in an API surface shape that looked like:

    ProcessTableFunctionTestHarness.ofClass(UpsertConsumerPTF.class)
    .withTableArgument("input", DataTypes.of("ROW<key STRING, value INT>"))
    .withPartitionBy("input", "key")
    .withTableArgumentUpsertKey("input", "key")
    .withTableArgumentChangelogMode("input", ChangelogMode.upsert(true))
    .build())

Defining these features of the table argument, reusing the same string table argument id over and over again, seemed fragile and repetitious. This PR replaces this way of defining table arguments with a builder, so the above example might look like this instead:

ProcessTableFunctionTestHarness.ofClass(UpsertConsumerPTF.class)
.withTableArgument(
        TableArgument.forArgument("input")
            .type(DataTypes.of("ROW<key STRING, value INT>"))
            .partitionBy("key")
            .withUpsertKey(key")
            .withChangelogMode(ChangelogMode.upsert(true))
            .build())

This change removes the previous API surface for defining a table argument, plus removes the old partition by method. I figured since this is still public evolving and unreleased, mutating the api surface here is still okay. Happy to be corrected though!

Brief change log

  • Added a builder based configuration option for defining table arguments in the PTF test harness.

Verifying this change

This change is already covered by existing tests, such as the ProcessTableFunctionTestHarnessTest class.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (yes(ish))
  • If yes, how is the feature documented? (docs / JavaDocs)

@flinkbot

flinkbot commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@autophagy
autophagy force-pushed the FLINK-40466-harness-table-argument-builder branch from c3647aa to d7160ff Compare August 25, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants