HIVE-29825:Fix stale CM root cache in ReplChangeManager after HDFS

[Manya0407]

What changes were proposed in this pull request?

A minimal fix in ReplChangeManager.getCmRoot() plus a regression test.

On a cache hit in encryptionZoneToCmrootMapping, the code now checks whether the cached CM root path still exists on HDFS. If it was deleted, it recreates the directory using the existing createCmRoot() helper (mkdirs + CM-root permissions), then returns the path.

Also added testRecycleAfterCmRootDeletedFromHdfs in TestReplChangeManager to cover the stale-cache scenario.

Why are the changes needed?

ReplChangeManager keeps an in-memory map from encryption zone → CM root path. That map survives across requests in a long-lived HMS JVM, but the actual HDFS directory can be removed externally — for example when DROP DATABASE ... CASCADE deletes a database directory that contains .cmroot.

After that, later MOVE recycle operations still get a cache hit and use the stale path without verifying HDFS. Because the parent .cmroot directory no longer exists, rename() fails and the failure path calls setTimes() on a non-existent destination file, causing:

FileNotFoundException: .../.cmroot/bucket_XXXXX_ does not exist

Does this PR introduce any user-facing change?

No

How was this patch tested?

Local Testing

[Copilot]

Pull request overview

Fixes a stale cache scenario in ReplChangeManager.getCmRoot() where the in-memory CM-root mapping can outlive the actual HDFS directory (e.g., after external deletion), causing subsequent recycle MOVE operations to fail. The change ensures that a cached CM-root path is validated against HDFS and recreated if missing, and adds a regression test to prevent recurrence.

Changes:

• On encryptionZoneToCmrootMapping cache hits, verify the CM-root path exists on HDFS and recreate it (via createCmRoot) if it was deleted.
• Add testRecycleAfterCmRootDeletedFromHdfs to cover the “directory deleted but cache still populated” scenario.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java	Validates cached CM-root existence on HDFS and recreates it when missing to prevent stale-cache failures.
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestReplChangeManager.java	Adds a regression test that deletes the CM root after an initial recycle and verifies subsequent recycle recreates it and succeeds.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[ayushtkn]

The fix is wrong, so is the test.

The test doesn't stimulate the actual scenario claimed to be broken. There are tests around encryption, like TestMetaStoreMultipleEncryptionZones extend a test in a Encryption class with this DROP Database cascade scenario.

Now regarding your scenario: DROP DATABASE CASCADE

In Replication the Database can not be dropped only if it is source of replication. The DropDatabaseHandler will throw you out here

hive/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/handler/DropDatabaseHandler.java

Lines 204 to 205 in 1bcf6d4

	if (!MetastoreConf.getBoolVar(handler.getConf(), HIVE_IN_TEST) && ReplChangeManager.isSourceOfReplication(db)) {
	throw new InvalidOperationException("can not drop a database which is a source of replication");

Someone going and deleting the directory directly on storage layer isn't a valid scenario, if he can delete any directory on storage layer, they can wipe out the entire cluster itself.

So, the use case you mentioned here itself is moot and not valid in my eyes.

Thinking bit further, some how if you find a valid use case, then itself the fix is wrong.

Your cmRoot is deleted and you are creating an empty directory, like for what? You lost the data itself, You shouldn't have let the directory itself get deleted if u need it in future? DropTableHandler handles this scenario here

hive/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/handler/DropTableHandler.java

Lines 214 to 216 in 1bcf6d4

	for (Path path : pathsToDelete) {
	deleteDataExcludeCmroot(path, ifPurge, shouldEnableCm);
	}

Similar would have been required in DatabaseDropHandler IMO or DropDatabase ideally should invalidate the cache if it contains such a Encryption path. FWIW exists
isn't free it is an RPC to Namenode and takes a read call.