Skip to content

Fix multiprocess dataloader CPU iteration leak via weakref sweep - #4983

Draft
snehalv2002 wants to merge 1 commit into
mainfrom
fix-iterator-leak
Draft

Fix multiprocess dataloader CPU iteration leak via weakref sweep#4983
snehalv2002 wants to merge 1 commit into
mainfrom
fix-iterator-leak

Conversation

@snehalv2002

Copy link
Copy Markdown
Collaborator

Description

Start with a short description of what the PR does and how this is a change from
the past.

The rest of the description includes relevant details and context, examples:

  • why is this change being made,
  • the problem being solved and any relevant context,
  • why this is a good solution,
  • some information about the specific implementation,
  • shortcomings of the solution and possible future improvements.

If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456

You can also provide a comma-separated list. If you don't want to close a bug but
simply to reference it, use BUGS, e.g.:
BUGS: b/123456

Notice 1: Once all tests pass, the "pull ready" label will automatically be assigned.
This label is used for administrative purposes. Please do not add it manually.

Notice 2: For external contributions, our settings currently require an approval from a MaxText maintainer to trigger CI tests.

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces iterator cleanup mechanisms for multihost and remote iterators during elastic training events, and simplifies elastic training utilities by removing unused functions and migrating to custom badput event recording. The review feedback highlights a critical missing import of weakref in multihost_dataloading.py and points out that self.dataloader is not set as an instance attribute, which prevents it from being properly closed.

Comment on lines +49 to +50
_GLOBAL_REMOTE_ITERATORS = weakref.WeakSet()
_GLOBAL_MULTIHOST_ITERATORS = weakref.WeakSet()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The module uses weakref.WeakSet to track global iterators, but weakref is never imported in this file. This will result in a NameError: name 'weakref' is not defined immediately when the module is imported. Please import weakref at the top of the file or right before its usage.

import weakref

_GLOBAL_REMOTE_ITERATORS = weakref.WeakSet()
_GLOBAL_MULTIHOST_ITERATORS = weakref.WeakSet()

Comment on lines +235 to +236
if hasattr(self, "dataloader") and hasattr(self.dataloader, "close") and callable(self.dataloader.close):
self.dataloader.close()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In RemoteIterator, self.dataloader is never set as an instance attribute (it is only a local variable dataloader in the reset() method). Consequently, hasattr(self, "dataloader") will always evaluate to False, and the dataloader will never be explicitly closed. Consider storing self.dataloader = dataloader in reset() so that it can be properly cleaned up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant