Fix multiprocess dataloader CPU iteration leak via weakref sweep - #4983
Fix multiprocess dataloader CPU iteration leak via weakref sweep#4983snehalv2002 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces iterator cleanup mechanisms for multihost and remote iterators during elastic training events, and simplifies elastic training utilities by removing unused functions and migrating to custom badput event recording. The review feedback highlights a critical missing import of weakref in multihost_dataloading.py and points out that self.dataloader is not set as an instance attribute, which prevents it from being properly closed.
| _GLOBAL_REMOTE_ITERATORS = weakref.WeakSet() | ||
| _GLOBAL_MULTIHOST_ITERATORS = weakref.WeakSet() |
There was a problem hiding this comment.
The module uses weakref.WeakSet to track global iterators, but weakref is never imported in this file. This will result in a NameError: name 'weakref' is not defined immediately when the module is imported. Please import weakref at the top of the file or right before its usage.
import weakref
_GLOBAL_REMOTE_ITERATORS = weakref.WeakSet()
_GLOBAL_MULTIHOST_ITERATORS = weakref.WeakSet()| if hasattr(self, "dataloader") and hasattr(self.dataloader, "close") and callable(self.dataloader.close): | ||
| self.dataloader.close() |
There was a problem hiding this comment.
In RemoteIterator, self.dataloader is never set as an instance attribute (it is only a local variable dataloader in the reset() method). Consequently, hasattr(self, "dataloader") will always evaluate to False, and the dataloader will never be explicitly closed. Consider storing self.dataloader = dataloader in reset() so that it can be properly cleaned up.
Description
Start with a short description of what the PR does and how this is a change from
the past.
The rest of the description includes relevant details and context, examples:
If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456
You can also provide a comma-separated list. If you don't want to close a bug but
simply to reference it, use BUGS, e.g.:
BUGS: b/123456
Notice 1: Once all tests pass, the "pull ready" label will automatically be assigned.
This label is used for administrative purposes. Please do not add it manually.
Notice 2: For external contributions, our settings currently require an approval from a MaxText maintainer to trigger CI tests.
Tests
Please describe how you tested this change, and include any instructions and/or
commands to reproduce.
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.