fix(ilm): reject unknown import fields and accept S3-shaped rule JSON - #367
Conversation
`rc ilm rule import` silently dropped unknown rule fields, so a JSON file using S3-shaped keys (nested filter/abortIncompleteMultipartUpload, snake_case) imported a degraded, actionless rule that the server then refused with a confusing "rule must have an action" error. Import now accepts the rc camelCase dialect plus the common alternative spellings (snake_case, S3/PascalCase, and the S3 JSON shape with a nested Filter and AbortIncompleteMultipartUpload action), rejects ambiguous flat+nested combinations, and fails loudly on unknown fields instead of dropping them.
|
cc @overtrue |
overtrue
left a comment
There was a problem hiding this comment.
Requesting changes because valid S3 size-only lifecycle filters are rejected during JSON import. Both standalone ObjectSizeGreaterThan and ObjectSizeLessThan cases were reproduced against this commit. The 11 existing lifecycle unit tests pass, but both regression probes fail with unknown-field errors. Please support these predicates and add regression coverage.
| #[derive(Debug, Deserialize)] | ||
| #[serde(rename_all = "camelCase")] | ||
| #[serde(rename_all = "camelCase", deny_unknown_fields)] | ||
| struct LifecycleFilterInput { |
There was a problem hiding this comment.
[P2] Support standalone S3 object-size filter predicates. S3 permits ObjectSizeGreaterThan or ObjectSizeLessThan directly under Filter, alongside the existing Prefix, Tag, and And alternatives. For example, {"Rules":[{"ID":"size-filter","Status":"Enabled","Filter":{"ObjectSizeGreaterThan":1024},"Expiration":{"Days":30}}]} is valid S3-shaped input, but this new input type rejects it with an unknown-field error because neither size predicate is defined. ObjectSizeLessThan fails identically. This prevents importing valid size-only lifecycle rules even though the existing XML writer already supports them. Please add both fields to LifecycleFilterInput, include them in predicate counting and flattening, and cover both standalone forms with tests.
S3 permits ObjectSizeGreaterThan and ObjectSizeLessThan directly under Filter, alongside Prefix, Tag, and And. The strict input dialect rejected these valid size-only filters with an unknown-field error. Add both predicates to LifecycleFilterInput (with the usual PascalCase and snake_case aliases), count them in the Filter predicate check, and flatten them onto rc's flat size fields. Update the multi-predicate error message and its test, and cover both standalone forms with regression tests.
|
Addressed in e19c6dc: |
Problem
rc ilm rule importsilently drops unknown rule fields. A JSON file whose keys don't match rc's camelCase dialect imports successfully, reports✓ Imported 1 lifecycle rule(s), and sends a degraded rule to the server — which then rejects it with a confusing error.Real-world sequence (this is how I hit it, originally misdiagnosed as a server bug in rustfs/rustfs#7456):
$ cat lifecycle.json {"rules":[{"id":"cleanup","status":"Enabled","filter":{"prefix":"v1/"},"abort_incomplete_multipart_upload":{"days_after_initiation":1}}]} $ rc ilm rule import alias/bucket lifecycle.json ✗ Failed to set lifecycle rules: set_bucket_lifecycle: Network error: HTTP 400: InvalidArgument: Rule must have at least one of Expiration, Transition, NoncurrentVersionExpiration, NoncurrentVersionTransition, or DelMarkerExpirationThe server is correct — rc parsed the file, dropped
filterandabort_incomplete_multipart_uploadas unknown fields, and PUT an actionless rule. Anyone pasting S3-shaped JSON (a natural thing to do, since it mirrorsPutBucketLifecycleConfiguration) gets this exact failure, and nothing points at the real cause.Fix
LifecycleRuleInput(and the nested input structs) now:abort_incomplete_multipart_upload_days) and S3/PascalCase (AbortIncompleteMultipartUploadDays) for every field;filter(Prefix/Tag/Andwith tags and size bounds) flattened onto rc's flat filter fields, nestedabortIncompleteMultipartUpload: { daysAfterInitiation }, andexpiration.ExpiredObjectDeleteMarker.deny_unknown_fields), naming the offending key — a wrong-dialect file is now a parse error, never a silent degradation.prefixandfilter.prefix, flat and nested abort days) error unless identical; afilterwith more than one top-level predicate (Prefix+Tag) is rejected like the server does.An empty
filter: {}stays valid (the server documents it as "applies to every object").Verification
crates/core/tests/lifecycle_import_dialect.rs: 10 tests covering the camelCase dialect (regression guard), snake_case, PascalCase, S3-shaped filter/abort/marker acceptance,Andflattening, unknown-field rejection, and both conflict cases.lifecycle_xml.rs(the existing round-trip test constructed an abort value but never asserted its XML).cargo fmt --all --check✅,cargo clippy -p rc-core --all-targets -- -D warnings✅,cargo test --workspace✅ (all suites green).rustfs/rustfs:v1.0.0-rc.5node: the exact JSON from the problem section now imports and round-trips:Notes
cargo clippy --workspacewith Rust 1.95 fails oncrates/s3/src/admin/catalog.rs:305(nonminimal_bool) — that failure exists on unmodifiedmainunder this toolchain and is unrelated to this change; I left it alone.