HIVE-29483: Implement an integration testing mechanism with real Tez on Yarn including resource localization by hemanthumashankar0511 · Pull Request #6640 · apache/hive

hemanthumashankar0511 · 2026-07-27T07:37:16Z

What changes were proposed in this pull request?

Adds an opt-in itests/tez-yarn-it module with a Docker/Testcontainers-based HDFS+YARN cluster and an end-to-end test that verifies hive-exec.jar is localized when Tez runs on YARN.

Why are the changes needed?

MiniTezCluster / TestMiniTezCliDriver do not exercise real YARN resource localization, so regressions like skipping hive-exec.jar from commonLocalResources are not caught today.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Ran locally with mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it -Dtest=TestTezYarnLocalization#testQuerySucceedsWithAppJar, and also validated on GitHub Actions (ubuntu-latest, native Linux amd64) using the same Maven command after mvn clean install -DskipTests -Pitests -pl itests/tez-yarn-it -am; the full end-to-end test is reliable on native Linux and macOS

ramitg254 · 2026-07-30T07:43:59Z

+import java.nio.file.Files;
+import java.nio.file.Path;
+
+/**


I think we don't need this comment entirely naming itself quite self explainatory

ramitg254 · 2026-07-30T07:44:53Z

+ *
+ * Run with: mvn -Pitests,tez-yarn -pl itests/tez-yarn-it -Dtest=TezYarnClusterContainerTest test
+ */
+public class TezYarnClusterContainerTest {


please use TestTezYarnClusterContainer

ramitg254 · 2026-07-30T08:08:42Z

+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**


please get rid of these large chunk of explainatory comments
try to keep them coincise to convey the purpose wherever the methods and classes are complex enough and need some explicit explanation
please avoid these * Run with: kind of comments

ramitg254 · 2026-07-30T08:12:20Z

+import java.util.Set;
+import java.util.stream.Stream;
+
+/**


same https://github.com/apache/hive/pull/6640/changes#r3681063334
and here comments are explaining the same redundant thing over and over so please remove those

Hey @ramitg254 Thanks for pointing it out..I'll get rid of these unwanted comments

Copilot

Pull request overview

Adds a new opt-in integration-test module that runs Hive-on-Tez against a real (containerized) HDFS + YARN cluster, specifically to validate YARN resource localization behavior (e.g., ensuring hive-exec.jar is localized for Tez AM/task containers).

Changes:

Introduces itests/tez-yarn-it Maven module (profile-gated execution) with Testcontainers-based HDFS/YARN cluster orchestration.
Adds end-to-end HS2 + Tez query test intended to catch regressions in Tez-on-YARN jar localization.
Adds Docker build assets and minimal Hadoop/YARN/Hive config used by the integration tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
itests/pom.xml	Adds the new `tez-yarn-it` module to the itests reactor.
itests/tez-yarn-it/pom.xml	Defines the module, test dependencies, surefire args, and generation of a custom hosts file for stable container hostname resolution.
itests/tez-yarn-it/src/test/java/org/apache/hive/tez/yarn/TezYarnClusterContainer.java	Testcontainers orchestration for NameNode/DataNode/RM/NM plus helper methods for staging jars to HDFS.
itests/tez-yarn-it/src/test/java/org/apache/hive/tez/yarn/TestTezYarnLocalization.java	End-to-end HS2 + Tez test and diagnostics collection.
itests/tez-yarn-it/src/test/java/org/apache/hive/tez/yarn/TestTezYarnClusterContainer.java	Container “smoke test” coverage for HDFS/YARN basics and jar upload helper.
itests/tez-yarn-it/src/test/java/org/apache/hive/tez/yarn/TestHiveServer2Connectivity.java	HS2 connectivity smoke test against the containerized cluster.
itests/tez-yarn-it/src/test/resources/hive-site-yarn-it.xml	Hive configuration overrides for the Tez-on-YARN integration tests.
itests/tez-yarn-it/src/test/resources/yarn-site.xml	YARN configuration overrides (notably disabling container memory checks for Docker).
itests/tez-yarn-it/src/test/resources/custom_hosts_file	Static host mappings used via `-Djdk.net.hosts.file` to make container hostnames resolvable from the test JVM.
itests/tez-yarn-it/src/test/docker/hadoop-yarn/Dockerfile	Custom Hadoop image that adds a Java 21 runtime for Tez AM/task container launch.
itests/tez-yarn-it/src/test/docker/hadoop-yarn/config	Environment-driven Hadoop/YARN configuration used by the Dockerized cluster.

Suppressed comments (1)

itests/tez-yarn-it/src/test/java/org/apache/hive/tez/yarn/TestTezYarnLocalization.java:66

These additional HDFS setup commands also ignore exit codes, which can hide early failures (e.g., namenode not ready, permissions issues) and make the failing signal less direct.

    nn.execInContainer("hdfs", "dfs", "-mkdir", "-p", HDFS_ROOT + "/warehouse");
    nn.execInContainer("hdfs", "dfs", "-mkdir", "-p", HDFS_ROOT + "/scratch");
    nn.execInContainer("hdfs", "dfs", "-mkdir", "-p", HDFS_ROOT + "/user-install");
    nn.execInContainer("hdfs", "dfs", "-chmod", "-R", "777", HDFS_ROOT);

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+  public void start() {
+    namenode.start();
+    datanode.start();
+    resourcemanager.start();
+    nodemanager.start();


+  private static String buildHadoopImage() {
+    String basedir = System.getProperty("basedir", ".");
+    Path dockerfile = Paths.get(basedir, "src/test/docker/hadoop-yarn/Dockerfile");
+    return new ImageFromDockerfile("hive-it-hadoop-jdk21", false)
+        .withDockerfile(dockerfile)
+        .get();
+  }


+  private static boolean isTezFrameworkJar(String name) {
+    if (name.endsWith("-tests.jar")) {
+      return false;
+    }
+    return name.contains("tez")
+        || name.startsWith("hadoop-shim")
+        || name.startsWith("hadoop-mapreduce-client-core")
+        || name.startsWith("hadoop-mapreduce-client-common");
+  }


+    nn.execInContainer("hdfs", "dfs", "-mkdir", "-p", "/tmp");
+    nn.execInContainer("hdfs", "dfs", "-chmod", "-R", "777", "/tmp");


+    Path localScratch = Files.createDirectories(
+            Path.of("/tmp", "hive-tez-loc-" + System.currentTimeMillis()));


+    cluster.namenodeContainer().execInContainer("hdfs", "dfs", "-mkdir", "-p", "/tmp/hive-29483/warehouse");
+    cluster.namenodeContainer().execInContainer("hdfs", "dfs", "-mkdir", "-p", "/tmp/hive-29483/scratch");
+    cluster.namenodeContainer().execInContainer("hdfs", "dfs", "-chmod", "-R", "777", "/tmp/hive-29483");


…og lives

… RPC port

…2Connectivity

…kflow

…ers in tez-yarn-it

abstractdog

thanks @hemanthumashankar0511 , tried it locally, seems to work, I left some comments regarding the design
once you address those and/or we discussed them, I'm ready to test it again

abstractdog · 2026-08-24T09:18:42Z

+    if (hiveSite != null) conf.addResource(hiveSite);
+    if (yarnSite != null) conf.addResource(yarnSite);


use proper {} blocks for single line blocks as well

abstractdog · 2026-08-24T09:19:35Z

+    cluster.namenodeContainer().execInContainer("hdfs", "dfs", "-mkdir", "-p", "/tmp/hive-29483/scratch");
+    cluster.namenodeContainer().execInContainer("hdfs", "dfs", "-chmod", "-R", "777", "/tmp/hive-29483");
+
+    Path localScratch = Files.createTempDirectory("hive-29483-local-");


using "29483" explicitly looks strange, it should be handled by the test code under the hood

abstractdog · 2026-08-24T09:56:45Z

+  private static final String HDFS_BASE      = "hdfs://namenode:8020";
+  private static final String HDFS_WAREHOUSE = HDFS_BASE + "/tmp/hive-tez-loc/warehouse";
+  private static final String HDFS_SCRATCH   = HDFS_BASE + "/tmp/hive-tez-loc/scratch";
+  private static final String HDFS_ROOT      = "/tmp/hive-tez-loc";


no need for extra indendation for aligning values

abstractdog · 2026-08-24T10:00:29Z

+          <!-- Resolve docker-network hostnames (namenode, resourcemanager, …) to 127.0.0.1
+               so HDFS URIs embedded in Tez LocalResources are reachable from both the host JVM
+               and inside YARN containers (via the shared docker network alias). -->
+          <argLine>${maven.test.jvm.args} -Xshare:off -Djdk.net.hosts.file=${test.conf.dir}/custom_hosts_file</argLine>


is there a specific reason to disable class data sharing via Xshare? if the base java image supports class metadata sharing for the base JDK classes, it can reduce container startup time a bit

abstractdog · 2026-08-24T10:06:38Z

+    return nodemanager;
+  }
+
+  private void waitForNodeManagerRegistration() {


consider using Awaitility for this purpose, which is already used in the hive project

abstractdog · 2026-08-24T11:34:23Z

+    conf.set("tez.lib.uris", tezLibUris);
+    conf.setBoolean("tez.use.cluster.hadoop-libs", true);
+    conf.setBoolVar(HiveConf.ConfVars.HIVE_SERVER2_TEZ_INITIALIZE_DEFAULT_SESSIONS, false);
+    conf.setIntVar(HiveConf.ConfVars.HIVE_SERVER2_TEZ_SESSIONS_PER_DEFAULT_QUEUE, 0);
+
+    conf.set("tez.am.client.am.port-range",
+        TezYarnClusterContainer.AM_CLIENT_PORT + "-" + TezYarnClusterContainer.AM_CLIENT_PORT);
+
+    String containerEnv = "JAVA_HOME=" + TezYarnClusterContainer.CONTAINER_JAVA_21_HOME
+        + ",HADOOP_HOME=/opt/hadoop"
+        + ",HADOOP_MAPRED_HOME=/opt/hadoop";
+    conf.set("tez.am.launch.env", containerEnv);
+    conf.set("tez.task.launch.env", containerEnv);
+
+    hs2Port = findFreePort();
+    conf.setIntVar(HiveConf.ConfVars.HIVE_SERVER2_THRIFT_PORT, hs2Port);
+    conf.setVar(HiveConf.ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST, "localhost");
+    conf.setIntVar(HiveConf.ConfVars.HIVE_SERVER2_WEBUI_PORT, findFreePort());
+    conf.setVar(HiveConf.ConfVars.HIVE_SERVER2_TRANSPORT_MODE, "binary");
+    conf.setVar(HiveConf.ConfVars.HIVE_SERVER2_AUTHENTICATION, "NOSASL");
+    conf.setBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ENABLE_DOAS, false);


is it possible to move whatever is constant to hive-site-yarn-it.xml?

abstractdog · 2026-08-24T11:37:34Z

+      dumpNmCommand("NodeManager daemon log (tail 200)",
+          "find /var/log/hadoop -maxdepth 1 -name '*.log' 2>/dev/null | head -3 "
+          + "| xargs -I{} sh -c 'echo \"--- {} ---\"; tail -200 {}' 2>/dev/null || true");


after a successful run of TestTezYarnLocalization I saw empty nodemanager logs, is this expected?

===== NM: NodeManager daemon log (tail 200) ===== (no output found) ########## END NodeManager diagnostics ##########

abstractdog · 2026-08-24T11:44:03Z

+  }
+
+  @Test
+  public void testQuerySucceedsWithAppJar() throws Exception {


this unit test might want to become even smarter by asserting 2 more things:

hive-exec.jar is present on HDFS somewhere (result of localization step 1)

hive-exec.jar is present in the yarn container's local folder (result of localization step 2)

abstractdog · 2026-08-24T11:55:56Z

+  <name>Hive Integration - Tez on YARN localization tests</name>
+  <properties>
+    <hive.path.to.root>../..</hive.path.to.root>
+    <skip.tez.yarn.tests>true</skip.tez.yarn.tests>


instead of skip.tez.yarn.tests I would use run.tez.yarn.tests, because skip + false is always a strange form of double-negative

abstractdog · 2026-08-24T12:09:22Z

+import java.util.Set;
+import java.util.stream.Stream;
+
+public class TezYarnClusterContainer {


design consideration: would you consider moving all this stuff to a docker-compose file for users' convenience (see my other comment regarding readme: https://github.com/apache/hive/pull/6640/changes#r3842926313)
I think this PR now handles a lot of the Java code when it comes to docker setup, which otherwise would be a simple docker-compose

and then TezYarnClusterContainer can leverage ComposeContainers like:

File composeFile = ... "src/test/docker/hadoop-yarn/docker-compose.yml"); ComposeContainer c = new ComposeContainer(composeFile)

given that "docker compose" is not available on precommit pods, and we don't want to make it a hard dependency for devs, withLocalCompose(false) will work

I'm hoping that a docker-compose will save us from using workarounds/hacks like the "custom_hosts_file"

for docker compose and the surrounding templates and stuff, feel free to take a look at whatever we already have in the packaging module:
https://github.com/apache/hive/blob/master/packaging/src/docker/docker-compose.yml

sonarqubecloud · 2026-08-24T12:45:31Z

Quality Gate passed

Issues
13 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

hemanthumashankar0511 · 2026-08-25T09:01:12Z

Hey @abstractdog thanks for the review!! I'll go through all the suggestions and keep you updated

asf-ci-hive added tests pending tests passed and removed tests pending labels Jul 27, 2026

hemanthumashankar0511 changed the title ~~[WIP]HIVE-29483: Implement an integration testing mechanism with real Tez on Yarn including resource localization (DO NOT REVIEW)~~ [WIP]HIVE-29483: Implement an integration testing mechanism with real Tez on Yarn including resource localization Jul 28, 2026

hemanthumashankar0511 changed the title ~~[WIP]HIVE-29483: Implement an integration testing mechanism with real Tez on Yarn including resource localization~~ HIVE-29483: Implement an integration testing mechanism with real Tez on Yarn including resource localization Jul 28, 2026

hemanthumashankar0511 marked this pull request as ready for review July 28, 2026 09:52

ramitg254 reviewed Jul 30, 2026

View reviewed changes

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Jul 30, 2026

abstractdog requested a lite review from Copilot August 24, 2026 07:14

Copilot started reviewing on behalf of abstractdog August 24, 2026 07:15 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

hemanthumashankar0511 added 16 commits August 24, 2026 17:12

HIVE-29483: Add Tez-on-YARN resource localization integration test

c95d6ab

Add GitHub Actions workflow for the Tez-on-YARN localization IT

0a15e01

Point NM diagnostics at /var/log/hadoop/userlogs where container sysl…

bcaf80d

…og lives

Stage Tez hadoop-shim jar into tez.lib.uris to fix AM startup

93d9c6a

Make Tez AM reachable from host: fixed hostname + pinned/published AM…

4b88667

… RPC port

mapreduce conf

97e4261

change query

d9a1ed2

file permission denied fix

f31852b

permissions fix

ca55318

reduce phase fix

1804c7e

final fix

2eabe3d

set METASTORE_TRY_DIRECT_SQL to false

c4bf0ef

cleanup task

76ae733

Drop duplicate TestTezYarnLocalizationBase in favor of TestHiveServer…

ae45ae9

…2Connectivity

HIVE-29483: Clean up comments and remove fork-only GitHub Actions wor…

6207db7

…kflow

HIVE-29483: Strip HTML tags from TezYarnClusterContainer Javadoc

f5867a7

hemanthumashankar0511 added 3 commits August 24, 2026 17:12

HIVE-29483: Fix ASF license header in pom.xml to use full attribution

cd6327d

Remove redundant comments and docstring

71349d1

HIVE-29483: Add safemode wait, HDFS exit checks, and correct ASF head…

362e95b

…ers in tez-yarn-it

hemanthumashankar0511 force-pushed the tez_on_yarn_it branch from c45e9a5 to 362e95b Compare August 24, 2026 11:50

asf-ci-hive added tests pending and removed tests passed labels Aug 24, 2026

abstractdog requested changes Aug 24, 2026

View reviewed changes

asf-ci-hive added tests unstable and removed tests pending labels Aug 24, 2026

		nn.execInContainer("hdfs", "dfs", "-mkdir", "-p", "/tmp");
		nn.execInContainer("hdfs", "dfs", "-chmod", "-R", "777", "/tmp");

		Path localScratch = Files.createDirectories(
		Path.of("/tmp", "hive-tez-loc-" + System.currentTimeMillis()));

		if (hiveSite != null) conf.addResource(hiveSite);
		if (yarnSite != null) conf.addResource(yarnSite);

Uh oh!

Conversation

hemanthumashankar0511 commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

abstractdog left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Aug 24, 2026

Quality Gate passed

Uh oh!

hemanthumashankar0511 commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hemanthumashankar0511 commented Jul 27, 2026 •

edited

Loading