feat(fs): add file-system.max-dir-entries write guard (ENOSPC above limit) - #4977
Conversation
Add a `file-system.max-dir-entries` flag/config field. When it is greater than zero, creating a new file in a directory that already contains at least this many entries is rejected with ENOSPC and a warning is logged. 0 (the default) keeps the guard disabled. This addresses the write-guard request in GoogleCloudPlatform#4696 (the corruption-prevention piece): read-path mitigations reduce listing cost but do not stop a directory from growing further into the size range that degrades listing performance and risks metadata corruption. Returning ENOSPC gives applications a standard, actionable "no space left on device" signal. The count is bounded via ReadDescendants (it stops once it has seen the limit), mirroring the existing rename-dir-limit idiom, so directories comfortably below the limit only pay to list their own entries.
There was a problem hiding this comment.
Code Review
This pull request introduces a max-dir-entries configuration limit to prevent directories from growing too large, returning ENOSPC when the limit is exceeded. Feedback on the implementation highlights a correctness bug where recursive rather than flat counting is used, and a performance bottleneck caused by holding an exclusive lock during GCS network calls. Additionally, the reviewer recommends extending this guard to other entry-creating operations (such as MkDir, CreateSymlink, MkNode, and Rename) and expanding test coverage to verify these scenarios.
- Count direct entries (non-recursive) instead of ReadDescendants, so a parent with few direct entries is not blocked because its subdirectories collectively hold many files. Adds a bounded, lock-free CountDirEntriesUpTo (Delimiter "/") to DirInode. - Perform the GCS listing without holding the parent inode's exclusive lock, so entry creation is no longer serialized behind the network call. - Extend the guard to MkDir, MkNode, CreateSymlink, and cross-directory Rename so the limit cannot be bypassed. - Expand tests: direct-vs-recursive counting, and MkDir/CreateSymlink blocking once a directory is full.
|
Thanks for the review — addressed all three findings in the latest commit: 1. Recursive vs. flat counting + lock contention. Replaced 2. Completeness. The guard is now enforced in 3. Test coverage. Added:
|
…F1008 golangci-lint (staticcheck QF1008) flagged t.fsTest.SetUpTestSuite/ TearDownTestSuite/TearDown as redundant embedded-field selectors. Call the promoted methods directly (t.SetUpTestSuite() etc.) to clear the Lint CI failure. No behavior change.
|
Thanks for the review — all three points are addressed in the current implementation ( 1. Recursive counting + lock contention (both fixed). The count no longer uses 2. Completeness (fixed). The guard is enforced at every entry-creating op, not just 3. Test coverage (added). CI. Fixed the |
Description
Adds a
file-system.max-dir-entriesflag / config field. When it is greater than zero, creating a new file in a directory that already contains at least this many entries is rejected withENOSPCand aWARNINGis logged.0(the default) keeps the guard disabled, so behavior is unchanged unless a user opts in.This implements the write-guard requested in #4696 — the corruption-prevention piece. Read-path mitigations (kernel list cache TTL, larger stat cache) reduce listing cost but do not stop a directory from growing further into the size range where listing degrades and metadata state can corrupt. Returning
ENOSPC("no space left on device") gives applications and scripts a standard, actionable signal at the source.Implementation notes:
ReadDescendants(ctx, limit), which stops once it has seen the limit — mirroring the existingfile-system.rename-dir-limitidiom. A directory comfortably below the limit only pays to list its own (few) entries; the cost is only incurred for directories at/near the limit, which are exactly the ones to protect.CreateFileop (a single choke point covering both the empty-object and streaming/local create paths).ENOSYS) are left to the normal create path.Scope: this PR covers requested feature #1 (write guard) only. The warning-threshold log (#2) is #4736; streaming readdir (#3) and documented limits (#4) are tracked in #4976.
Link to the issue in case of a bug fix.
Implements part of #4696. Remaining-scope tracking issue: #4976.
Testing details
go test ./cfgpasses with the new flag. Addedinternal/fs/max_dir_entries_test.go(fsTest-harness behavioral test: creates into a directory until a create is rejected, asserts the rejection isENOSPCand occurs after no more thanlimitsuccesses; plus a below-limit success case). Verified it compiles underGOOS=linux go test -c ./internal/fs/.internal/fsFUSE-backed suite runs in the project's Linux CI.Any backward incompatible change? If so, please explain.
No. The guard is off by default (
max-dir-entries: 0); existing behavior is unchanged unless a user sets a positive limit.