[build] Relocate shaded jackson and other internal deps in fluss-fs-s3

[Jackeyzhe]

Purpose

Linked issue: close #4072

The fluss-fs-s3 uber-jar bundles hadoop-common and aws-java-sdk-*
dependencies without relocations for jackson and several other bundled
third-party packages, leaking 1,050 unshaded jackson class files
(jackson-databind, jackson-core,
jackson-annotations, jackson-dataformat-cbor) at com/fasterxml/*
paths. These can shadow a downstream application's own jackson-core
(e.g. 2.16+) and cause runtime failures such as NoSuchMethodError.

Brief change log

fluss-filesystems/fluss-fs-s3/pom.xml:

• Added shade relocations for the transitively-bundled third-party
  packages, consolidated into the single <relocations> block and
  namespaced under org.apache.fluss.shaded.* (aligned with the
  pre-existing org.apache.commons relocation on main):
  • com.fasterxml → org.apache.fluss.shaded.com.fasterxml
  • org.apache.htrace → org.apache.fluss.shaded.org.apache.htrace
  • com.ctc → org.apache.fluss.shaded.com.ctc
  • com.google.common / com.google.re2j / com.google.j2objc /
    com.google.thirdparty → org.apache.fluss.shaded.com.google.*
    (narrowed from the broad com.google namespace so references to
    un-bundled protobuf/gson are not rewritten to a dangling path)
  • org.apache.commons → only the 9 actually-bundled packages:
    beanutils, codec, collections, compress, configuration2,
    io, lang3, logging, text (narrowed from the broad
    org.apache.commons namespace so references to un-bundled
    commons-cli / commons-jxpath / commons-math3 etc. are not
    rewritten to a dangling path)
  • org.codehaus → only stax2 and mojo, the two org.codehaus
    packages actually bundled in this jar (124 + 1 classes).
    org.codehaus.jackson is referenced by Hadoop (e.g.
    MetricsJsonBuilder) but is not packaged here, so it is left
    un-relocated, preserving the pre-change external classpath resolution
• Added a <filter> excluding META-INF/versions/**/com/fasterxml/**:
  the Maven Shade Plugin relocates base-path classes but not
  Multi-Release JAR entries (same issue class as [fluss-client][ fluss-flink-*] Uber-jars leak com.fasterxml.jackson.core alongside the shaded copy #3553, which affected
  fluss-shaded-jackson). The relocated base-path implementation is
  sufficient for all Java versions.

Tests

Build-only change (pom.xml), verified by inspecting the produced jar:

• Before fix: 1,050 unshaded com/fasterxml class entries
• After fix: 0 unshaded com/fasterxml entries (base path and MRJ);
  1,050 jackson classes correctly relocated to
  org/apache/fluss/shaded/com/fasterxml/*
• No dangling class references remain: protobuf/gson,
  commons-cli/jxpath/math3, and org.codehaus.jackson are not
  rewritten. The checker reports two non-class commons-codec resource
  strings (language/bm/ and language/dmrules); both resources are present
  at their relocated paths in the produced jar
• Bytecode references inside the bundled AWS SDK / Hadoop classes are
  rewritten to the shaded path (verified via javap on sampled classes)
• Intentional jaxb-api MRJ entries under META-INF/versions/** are
  preserved untouched
• ./mvnw spotless:check passes

API and Format

No API or storage format changes. This only affects the internal shading
layout of the fluss-fs-s3 uber-jar; all relocated classes move under
the org.apache.fluss.shaded namespace.

Documentation

No new feature introduced; no documentation changes required.

[litiliu]

would it be better to shaded the whole com.google(including re2j and common)?

[Jackeyzhe]

Good catch — agreed, widened to the whole com.google namespace
(a6215ed). It turns out the jar was still leaking 1,974 unshaded
com.google.common (Guava) entries plus j2objc / thirdparty, which
is the same shadowing bug class as the jackson leak in #4072.

Verified: 0 unshaded com/google entries (base path and MRJ), all
constant-pool references rewritten to the shaded path, and hadoop's
own org.apache.hadoop.thirdparty.com.google namespace left untouched.

While verifying I also noticed fluss-fs-hdfs has the same problem
at a larger scale (1,073 unshaded com/fasterxml + 2,731 com/google
in its uber-jar) — I'll file a separate issue for that so this PR
stays scoped to fluss-fs-s3.

[Jackeyzhe]

Filed #4116 for the fluss-filesystems shading leak (all 7 plugins verified leaked, gs worst)
so #4073 stays scoped to s3 — thanks again for the catch, @litiliu.

[binary-signal]

@Jackeyzhe thank you for addressing this issue, I've left some comments

[binary-signal]

I applied this PR locally on top of main (4e372d5c7) and can confirm it does what it says: fluss-fs-s3 goes from 1050 unshaded com/fasterxml classes to 0, with 1974 guava, 48 re2j, 205 ctc and 125 codehaus classes correctly relocated under org.apache.fluss.fs.shaded.s3.

One thing worth addressing before merge, though.

com.google is broader than what the jar actually bundles

The shade plugin rewrites every reference matching a <relocation> pattern, including references to classes the jar does not contain. fluss-fs-s3 bundles only four com.google.* packages:

package	classes bundled
com.google.common	1974
com.google.re2j	48
com.google.j2objc	14
com.google.thirdparty	5

But the bundled Hadoop and AWS SDK code references com.google.protobuf and com.google.gson, neither of which is packaged (both are <exclusion>s in this pom). After this PR, those references are rewritten to org.apache.fluss.fs.shaded.s3.com.google.protobuf.* — about 40 distinct classes, e.g.:

org/apache/fluss/fs/shaded/s3/com/google/protobuf/Message                (11 refs)
org/apache/fluss/fs/shaded/s3/com/google/protobuf/Descriptors            (5 refs)
org/apache/fluss/fs/shaded/s3/com/google/protobuf/GeneratedMessage       (5 refs)
org/apache/fluss/fs/shaded/s3/com/google/gson/stream/JsonReader          (1 ref)

Before relocation, com.google.protobuf.Message is a soft dependency: absent from the uber-jar, but resolvable if the surrounding classpath or plugin parent classloader provides it. After relocation the reference points at a coordinate nothing will ever provide, because no artifact anywhere publishes org.apache.fluss.fs.shaded.s3.com.google.protobuf. A previously-satisfiable link becomes a guaranteed NoClassDefFoundError on any code path that touches it.

This is not hypothetical. I hit exactly this mechanism via the sibling org.apache.commons relocation (added in #3960, already on main) while extending this PR's approach to the other filesystem plugins:

java.lang.NoClassDefFoundError: org/apache/fluss/shaded/org/apache/commons/cli/ParseException
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1713)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1374)
    ...
    at org.apache.fluss.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBehaviorTest.java:62)

commons-cli was on the classpath. The reference had simply been rewritten to a name that does not exist.

To make this checkable , #4144 adds tools/ci/check_shaded_jars.py, whose --dangling mode reports exactly this: relocated class references a jar does not contain. Running it on fluss-fs-s3 built from this PR lists 92 such classes, 83 of them real classes rather than annotations.

Suggested change

Name the packages that are actually bundled rather than the com.google prefix:

<relocation>
    <pattern>com.google.common</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.common</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.j2objc</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.j2objc</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.thirdparty</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.thirdparty</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.re2j</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.re2j</shadedPattern>
</relocation>

This keeps the full guava fix (the actual win here — 1974 classes) and drops the dangling protobuf/gson links. It also matches what fluss-fs-hadoop-shaded already does, which the PR description cites as the pattern being mirrored: that module relocates com.google.re2j specifically, not com.google.

The same applies to the existing org.apache.commons relocation from #3960, which leaves dangling references to commons-cli, commons-codec, commons-math3, commons-net, commons-jxpath, commons-jexl2 and commons-vfs2 in every filesystem plugin. That is pre-existing rather than something this PR introduces; it is written up in #4143 along with the rest of the shading leaks.

#4144 applies the narrow-pattern approach across the modules #4143 covers, if a worked example is useful — every relocation there names the packages actually bundled, and io.netty.internal.tcnative and org.apache.arrow are excluded for the JNI reason above. It deliberately leaves fluss-fs-s3 to this PR.

Two notes for anyone verifying locally

• clean is required. An incremental build reuses already-relocated classes in target/classes, so a narrowed pattern appears to have no effect until you run mvn clean. This cost me a couple of confusing debug cycles.
• mvn test -pl fluss-filesystems -amd fails on HadoopUtilsTest independently of any of this: in a reactor containing fluss-fs-hadoop-shaded, dependent modules get its unshaded target/classes because shade runs at package. Use verify or install instead. Reproduces on unmodified main.

The MRJ exclusion in this PR is also worth keeping even alongside #3884's global one — #3884 puts it in the root pom under <filters combine.children="append">, so it already cascades here, but the local copy keeps this PR correct on its own.

[Jackeyzhe]

I applied this PR locally on top of main (4e372d5c7) and can confirm it does what it says: fluss-fs-s3 goes from 1050 unshaded com/fasterxml classes to 0, with 1974 guava, 48 re2j, 205 ctc and 125 codehaus classes correctly relocated under org.apache.fluss.fs.shaded.s3.

One thing worth addressing before merge, though.

com.google is broader than what the jar actually bundles

The shade plugin rewrites every reference matching a <relocation> pattern, including references to classes the jar does not contain. fluss-fs-s3 bundles only four com.google.* packages:

package classes bundled
com.google.common 1974
com.google.re2j 48
com.google.j2objc 14
com.google.thirdparty 5
But the bundled Hadoop and AWS SDK code references com.google.protobuf and com.google.gson, neither of which is packaged (both are <exclusion>s in this pom). After this PR, those references are rewritten to org.apache.fluss.fs.shaded.s3.com.google.protobuf.* — about 40 distinct classes, e.g.:

org/apache/fluss/fs/shaded/s3/com/google/protobuf/Message                (11 refs)
org/apache/fluss/fs/shaded/s3/com/google/protobuf/Descriptors            (5 refs)
org/apache/fluss/fs/shaded/s3/com/google/protobuf/GeneratedMessage       (5 refs)
org/apache/fluss/fs/shaded/s3/com/google/gson/stream/JsonReader          (1 ref)

Before relocation, com.google.protobuf.Message is a soft dependency: absent from the uber-jar, but resolvable if the surrounding classpath or plugin parent classloader provides it. After relocation the reference points at a coordinate nothing will ever provide, because no artifact anywhere publishes org.apache.fluss.fs.shaded.s3.com.google.protobuf. A previously-satisfiable link becomes a guaranteed NoClassDefFoundError on any code path that touches it.

This is not hypothetical. I hit exactly this mechanism via the sibling org.apache.commons relocation (added in #3960, already on main) while extending this PR's approach to the other filesystem plugins:

java.lang.NoClassDefFoundError: org/apache/fluss/shaded/org/apache/commons/cli/ParseException
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1713)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1374)
    ...
    at org.apache.fluss.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBehaviorTest.java:62)

commons-cli was on the classpath. The reference had simply been rewritten to a name that does not exist.

To make this checkable , #4144 adds tools/ci/check_shaded_jars.py, whose --dangling mode reports exactly this: relocated class references a jar does not contain. Running it on fluss-fs-s3 built from this PR lists 92 such classes, 83 of them real classes rather than annotations.

Suggested change

Name the packages that are actually bundled rather than the com.google prefix:

<relocation>
    <pattern>com.google.common</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.common</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.j2objc</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.j2objc</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.thirdparty</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.thirdparty</shadedPattern>
</relocation>
<relocation>
    <pattern>com.google.re2j</pattern>
    <shadedPattern>org.apache.fluss.fs.shaded.s3.com.google.re2j</shadedPattern>
</relocation>

This keeps the full guava fix (the actual win here — 1974 classes) and drops the dangling protobuf/gson links. It also matches what fluss-fs-hadoop-shaded already does, which the PR description cites as the pattern being mirrored: that module relocates com.google.re2j specifically, not com.google.

The same applies to the existing org.apache.commons relocation from #3960, which leaves dangling references to commons-cli, commons-codec, commons-math3, commons-net, commons-jxpath, commons-jexl2 and commons-vfs2 in every filesystem plugin. That is pre-existing rather than something this PR introduces; it is written up in #4143 along with the rest of the shading leaks.

#4144 applies the narrow-pattern approach across the modules #4143 covers, if a worked example is useful — every relocation there names the packages actually bundled, and io.netty.internal.tcnative and org.apache.arrow are excluded for the JNI reason above. It deliberately leaves fluss-fs-s3 to this PR.

Two notes for anyone verifying locally

• clean is required. An incremental build reuses already-relocated classes in target/classes, so a narrowed pattern appears to have no effect until you run mvn clean. This cost me a couple of confusing debug cycles.
• mvn test -pl fluss-filesystems -amd fails on HadoopUtilsTest independently of any of this: in a reactor containing fluss-fs-hadoop-shaded, dependent modules get its unshaded target/classes because shade runs at package. Use verify or install instead. Reproduces on unmodified main.

The MRJ exclusion in this PR is also worth keeping even alongside #3884's global one — #3884 puts it in the root pom under <filters combine.children="append">, so it already cascades here, but the local copy keeps this PR correct on its own.

Thanks @binary-signal for the detailed analysis and for adding the checker in
#4144. I've addressed the broad-relocation issue in 46fd4970e.

The broad com.google relocation is now limited to the four packages that
fs-s3 actually bundles: com.google.common, com.google.re2j,
com.google.j2objc, and com.google.thirdparty. I also narrowed
org.codehaus to stax2 and mojo, and the pre-existing
org.apache.commons relocation to the nine commons packages present in the
jar. All relocated packages now use the shared org.apache.fluss.shaded.*
namespace.

After a clean rebuild, the jar contains 0 unshaded jackson entries and all
1,954 bundled guava classes remain relocated. References to non-bundled
protobuf/gson, commons-cli/jxpath/math3, and org.codehaus.jackson are no
longer rewritten to shaded paths. The checker's only two remaining reports
are commons-codec resource strings, and both resources are present at their
relocated paths in the jar.

I'll keep #4073 scoped to fs-s3 and leave the broader filesystem-family work
to #4143/#4144.