Respect sparse_threshold for pandas SparseDtype columns - #537
Open
Danila Pechenev (Danila-Pechenev) wants to merge 2 commits into
Open
Respect sparse_threshold for pandas SparseDtype columns#537Danila Pechenev (Danila-Pechenev) wants to merge 2 commits into
Danila Pechenev (Danila-Pechenev) wants to merge 2 commits into
Conversation
Danila Pechenev (Danila-Pechenev)
marked this pull request as ready for review
September 4, 2026 13:41
Member
|
When we wrote this function, the intention was that sparse_threshold would be used to sparsify columns and not densify already sparse ones. The documentation is not clear on this, so I understand why this PR was made. In my mind, users that pass in sparse columns already have checked that it made sense for their column to be sparse, so we don't want to overwrite this. When looking at #378, the issue had actually been fixed by moving to narwhal. Bypassing the density check to keep the sparse columns sparse solved the inefficiency. I see 3 questions/observations from this PR:
Martin Stancsics (@stanmart), I would love to get your input here. What do you think? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #378.
Summary
Update
from_dfso pandasSparseDtypecolumns respectsparse_thresholdinstead of being unconditionally routed to sparse storage.
Why
Issue #378 predates the current Narwhals-based
from_dfimplementation.In current
main, pandas sparse columns are handled separately because theirdtype is not supported by Narwhals, but this special case bypasses
sparse_threshold.The updated logic preserves logical non-zero density semantics while avoiding
unnecessary scans for low-density zero-fill sparse columns.
For zero-fill
SparseDtypecolumns, pandas sparse storage density is used as afast first check. If it is already at or below the threshold, the column is
known to be sparse without scanning its logical values. If it is above the
threshold, logical density is evaluated so explicitly stored zeros are handled
correctly.
For non-zero fill values, classification uses logical non-zero density because
pandas storage density does not represent non-zero density in that case.
Empty and missing-fill sparse columns preserve the existing sparse behavior.
Tests
Added regression coverage for:
sparse_threshold;Checklist
CHANGELOG.rstentry