fix(cfg): enable finalize-on-close for aiml-checkpointing profile - #5003
fix(cfg): enable finalize-on-close for aiml-checkpointing profile#5003vipnydav wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds the "aiml-checkpointing" profile optimization rule to the configuration. However, the changes in cfg/config.go and cfg/config_test.go were made manually. Since these files are auto-generated, they should not be edited directly; instead, you should modify cfg/params.yaml and run the generation command to regenerate them automatically.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5003 +/- ##
==========================================
+ Coverage 83.81% 83.82% +0.01%
==========================================
Files 174 174
Lines 21376 21376
==========================================
+ Hits 17916 17919 +3
+ Misses 2779 2776 -3
Partials 681 681
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Hi @charith87, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
3 similar comments
|
Hi @charith87, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
|
Hi @charith87, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
|
Hi @charith87, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you! |
Description
This PR enables
finalize-on-closeby default for theaiml-checkpointingprofile to prevent checkpoint loading failures caused by stale metadata.Background:
As discovered in b/540250863 during checkpoint benchmarking, when a directory rename occurs, GCSFuse invalidates cached entries under the old folder prefix. A subsequent
ListObjectscall on the new directory re-populates the cache.However, because zonal buckets use the appendable API where objects are not finalized on write by default,
ListObjectscan return a stale, intermediate object size for these unfinalized objects. Since theaiml-checkpointingprofile sets an infinite metadata cache TTL (metadata-cache.ttl-secs = -1), this stale size is cached permanently in RAM, which ultimately causesOUT_OF_RANGEerrors during checkpoint loading.Enabling finalize on close explicitly finalizes the object, avoiding the asynchronous metadata propagation delay for unfinalized appendable objects.
Link to the issue in case of a bug fix.
b/544523257
Testing details
aiml-checkpointingprofile incfg/config_test.goAny backward incompatible change? If so, please explain.
NA