Skip to content

fix(sql): align S3 Select behavior with RustFS - #370

Merged
overtrue merged 1 commit into
rustfs:mainfrom
GatewayJ:codex/s3-select-compatibility
Sep 9, 2026
Merged

fix(sql): align S3 Select behavior with RustFS#370
overtrue merged 1 commit into
rustfs:mainfrom
GatewayJ:codex/s3-select-compatibility

Conversation

@GatewayJ

@GatewayJ GatewayJ commented Sep 9, 2026

Copy link
Copy Markdown
Member

Related issues

  • Follow-up to the S3 Select compatibility audit; no GitHub issue is linked.

Background and user impact

The rc sql command diverged from the current RustFS S3 Select implementation in three places: CSV input record delimiters were not configurable, Parquet scan ranges were rejected locally even though RustFS supports them, and scan ranges on compressed input reached the server before failing. Select service errors could also collapse to the unhelpful service error text.

Root cause

S3 Select validation was duplicated between the CLI and AWS SDK adapter, the core CSV input model did not expose RecordDelimiter, and service-error classification preferred the generic SDK wrapper over modeled error metadata.

Solution

  • Add the additive --csv-input-record-delimiter option with RustFS-compatible one- or two-byte validation.
  • Centralize S3 Select option validation in rc-core and validate before alias lookup or network access.
  • Allow Parquet ScanRange requests.
  • Reject non-noop ScanRange requests for GZIP/BZIP2 input locally while preserving RustFS's accepted { start: 0, end: none } no-op form.
  • Preserve service error codes and modeled messages, and classify Busy/SlowDown as retryable network failures.
  • Add focused core, CLI, and S3 adapter regression tests.
  • Apply three behavior-preserving boolean simplifications required for the workspace to pass Clippy on stable Rust 1.96.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace
  • Live RustFS integration: two-byte CSV input record delimiter
  • Live RustFS integration: Parquet with a non-empty full-object scan range
  • Live RustFS integration: compressed scan range rejected locally with exit code 2
  • Live RustFS integration: Select parser errors retain the service code and message

@overtrue
overtrue merged commit fbf5e9f into rustfs:main Sep 9, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants