Skip to content

perf(storage): optimize Folder.UpdateTime to int64 nanoseconds and direct proto conversion - #5001

Draft
kislaykishore wants to merge 1 commit into
masterfrom
pr/opt-gcs-folder-timestamp
Draft

perf(storage): optimize Folder.UpdateTime to int64 nanoseconds and direct proto conversion#5001
kislaykishore wants to merge 1 commit into
masterfrom
pr/opt-gcs-folder-timestamp

Conversation

@kislaykishore

Copy link
Copy Markdown
Collaborator

Description

This PR optimizes gcs.Folder.UpdateTime by refactoring it from a 24-byte time.Time struct to an 8-byte int64 (unix nanoseconds), and enhances GCSFolder() to perform direct mathematical conversion from protobuf controlpb.Folder timestamp fields (seconds * 1e9 + nanos) without allocating intermediate time.Time objects.

Struct Sizing & Go Heap Class Reduction

  • gcs.Folder shrinks from 40 B to 24 B, dropping across the Go heap allocator class boundary from 48 B to 24 B.
  • Saves 24 bytes (-50.0%) of real heap memory per folder, yielding 24.0 MB saved per 1M folders (and 240.0 MB per 10M folders).
  • Slice allocations (e.g. folder listings) save 40.0% allocated bytes.
Struct / Type Baseline Size Optimized Size Struct Delta Go Heap Class Heap Allocation Delta RAM Saved (per 1M folders) RAM Saved (per 10M folders)
gcs.Folder 40 B 24 B -16 B (-40.0%) 48 B -> 24 B -24 B (-50.0%) 24.0 MB 240.0 MB

Benchmark Performance

Measured via benchstat (n=10, p < 0.05):

Benchmark Target Master Baseline Optimized Branch CPU Latency Delta Baseline Memory Branch Memory Allocation Delta
GCSFolder_WithTimestamp 95.60 ns/op 83.42 ns/op -12.75% (p=0.000) 96 B/op, 2 allocs 72 B/op, 2 allocs -25.00% B/op
GCSFolder_NilTimestamp 88.29 ns/op 79.38 ns/op -10.10% (p=0.000) 96 B/op, 2 allocs 72 B/op, 2 allocs -25.00% B/op
Alloc_Folder 32.74 ns/op 25.95 ns/op -20.75% (p=0.000) 48 B/op, 1 alloc 24 B/op, 1 alloc -50.00% B/op
Alloc_FolderSlice_1000 4.387 µs/op 5.035 µs/op ~0.0% 40.00 KiB/op, 1 alloc 24.00 KiB/op, 1 alloc -40.00% B/op

Link to the issue in case of a bug fix.

N/A

Testing details

  1. Manual - Executed make build and verified formatting/linter with 0 issues.
  2. Unit tests - Executed unit tests: go test -v ./internal/storage/gcs/... and go test -v ./internal/storage/fake/....
  3. Integration tests - N/A

Any backward incompatible change? If so, please explain.

N/A

…rect proto conversion

## Summary
This commit refactors `gcs.Folder.UpdateTime` from a 24-byte `time.Time` struct to an 8-byte `int64` (unix nanoseconds) and optimizes `GCSFolder()` to perform direct mathematical conversion from protobuf `controlpb.Folder` timestamp fields (`seconds * 1e9 + nanos`) without allocating intermediate `time.Time` objects.

## Struct Sizing & Go Heap Class Reduction
`gcs.Folder` shrinks from 40 B to 24 B, dropping across the Go heap allocator class boundary from 48 B to 24 B—saving **24 bytes (-50.0%) of real heap memory per folder**.

| Struct / Type | Baseline Size | Optimized Size | Struct Delta | Go Heap Class | Heap Allocation Delta | RAM Saved (per 1M folders) | RAM Saved (per 10M folders) |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| `gcs.Folder` | 40 B | **24 B** | -16 B (-40.0%) | 48 B -> **24 B** | **-24 B (-50.0%)** | **24.0 MB** | **240.0 MB** |

## Benchmark Performance
Measured via `benchstat` (n=10, p < 0.05):

| Benchmark Target | Master Baseline | Optimized Branch | CPU Latency Delta | Baseline Memory | Branch Memory | Allocation Delta |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| `GCSFolder_WithTimestamp` | 95.60 ns/op | **83.42 ns/op** | **-12.75% (p=0.000)** | 96 B/op, 2 allocs | **72 B/op, 2 allocs** | **-25.00% B/op** |
| `GCSFolder_NilTimestamp` | 88.29 ns/op | **79.38 ns/op** | **-10.10% (p=0.000)** | 96 B/op, 2 allocs | **72 B/op, 2 allocs** | **-25.00% B/op** |
| `Alloc_Folder` | 32.74 ns/op | **25.95 ns/op** | **-20.75% (p=0.000)** | 48 B/op, 1 alloc | **24 B/op, 1 alloc** | **-50.00% B/op** |
| `Alloc_FolderSlice_1000` | 4.387 µs/op | 5.035 µs/op | ~0.0% | 40.00 KiB/op, 1 alloc | **24.00 KiB/op, 1 alloc** | **-40.00% B/op** |

## Verification
- Unit tests: `go test -v ./internal/storage/gcs/...` & `go test -v ./internal/storage/fake/...`
- Benchmarks: `go test -bench=BenchmarkGCSFolder -benchmem ./internal/storage/gcs/...`

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Folder struct's UpdateTime field from time.Time to int64 (representing nanoseconds) and updates the fake storage bucket and GCS folder implementations accordingly. It also introduces mutex locking in RenameFolder and adds new tests and a benchmark. The review feedback suggests using an explicit integer constant 1000000000 instead of 1e9 for clarity in nanosecond conversions, and replacing b.Loop() in the benchmark with a traditional loop to ensure compatibility with Go versions older than 1.24.

// Setting the parameters in Folder and doing conversions as necessary.
var updateTime int64
if ts := attrs.GetUpdateTime(); ts != nil {
updateTime = ts.GetSeconds()*1e9 + int64(ts.GetNanos())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using 1e9 (which is an untyped float constant) in integer arithmetic can be less clear and potentially confusing. It is more idiomatic in Go to use an untyped integer constant like 1_000_000_000 or 1000000000 for nanosecond conversions.

Suggested change
updateTime = ts.GetSeconds()*1e9 + int64(ts.GetNanos())
updateTime = ts.GetSeconds()*1000000000 + int64(ts.GetNanos())

Comment thread internal/storage/gcs/folder_test.go
@codecov

codecov Bot commented Aug 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.83%. Comparing base (0673d19) to head (755eb23).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5001      +/-   ##
==========================================
+ Coverage   83.81%   83.83%   +0.01%     
==========================================
  Files         174      174              
  Lines       21376    21381       +5     
==========================================
+ Hits        17916    17924       +8     
+ Misses       2779     2777       -2     
+ Partials      681      680       -1     
Flag Coverage Δ
unittests 83.83% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant