fix(query-frontend): don't vertically shard queries using vector() - #7806
Open
CharlieTLe wants to merge 2 commits into
Open
fix(query-frontend): don't vertically shard queries using vector()#7806CharlieTLe wants to merge 2 commits into
CharlieTLe wants to merge 2 commits into
Conversation
`<expr> or vector(0)` returns the vector() fallback instead of the left hand
side when -frontend.query-vertical-shard-size > 1.
Vertical sharding appends a shard matcher to every vector selector, so each
shard evaluates the whole query over only the series it owns and the shard
results are concatenated. That is correct only while every output series is
derived from a selector: then the shard that owns the series is the only shard
that can produce it, and the analyzer's job is just to pick sharding labels
that preserve the invariant.
vector(s) breaks it. It synthesises a series with an empty labelset out of a
scalar and has no selector behind it, so every shard produces it. For
max without (__name__, job, series) ({__name__="test_series_a",job="test"})
or vector(-0.3587905225767787)
the analyzer shards without (__name__, job, series). The test_series_a series
carry no other labels, so all of them hash to the same shard:
* that shard's LHS yields {} => 57, the vector() sample collides with it and
`or` correctly drops it, so the shard returns {} => 57;
* the other shard's LHS is empty, nothing collides, `or` lets the vector()
sample through and the shard returns {} => -0.3587905225767787.
Merging the shard results yields two {} series at the same timestamp and the
wrong one can win. Observed:
unsharded: {} => 57
sharded : {} => -0.3587905225767787
PromQL `or` returns all LHS series and from the RHS only series whose labelset
is absent on the left, so 57 is the correct answer and the sharded result is
wrong.
The upstream Thanos analyzer already refuses to shard absent, absent_over_time
and scalar for the same underlying reason - their results depend on data an
individual shard cannot see - but `vector` is missing from that list, and
Cortex's own disableBinaryExpressionAnalyzer wrapper does not cover it and is
only installed when the parquet queryable is enabled.
Since the function list lives in vendored Thanos, add a small Cortex-side
wrapper analyzer that marks any query using vector() as not shardable, and
install it unconditionally in initQueryFrontendTripperware. Queries that do not
use vector() are unaffected, so this costs sharding only for queries that
cannot be sharded correctly today. Adding `vector` to the Thanos analyzer
upstream would let the wrapper be dropped again.
Filtering `vector(` out of the fuzz corpus instead - the approach proposed in
cortexproject#7547 and implemented in the still-open cortexproject#7551 - would have hidden a real,
user-visible wrong-results bug, so it is deliberately not done here.
Verified by hand (unsharded vs -frontend.query-vertical-shard-size=2, same
build) and with the fuzz seed from the failing CI run:
CORTEX_IMAGE=<local build> FUZZ_SEED=1787336350 go test -v \
-tags "integration,requires_docker,integration_query_fuzz" \
-timeout 2400s -count=1 ./integration/ \
-run '^TestVerticalShardingFuzz$'
fails before the change and passes after it, as do seeds 1, 424242 and
987654321.
Fixes cortexproject#7804
Signed-off-by: Charlie Le <charlie_le@apple.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Fixes a vertical-sharding correctness bug:
<expr> or vector(0)— the idiomatic "default value" pattern in dashboards and alerting rules — returns thevector()fallback instead of the left-hand side when-frontend.query-vertical-shard-size > 1.Vertical sharding appends a shard matcher to every vector selector (
InjectShardingInfo), so each shard evaluates the whole query over only the series it owns and the shard results are concatenated. That is correct only while every output series is derived from a selector — then the shard that owns the series is the only shard that can produce it, and the analyzer's job is just to pick sharding labels that preserve the invariant.vector(s)breaks the invariant: it synthesises a series with an empty labelset out of a scalar and has no selector behind it, so every shard produces it. Forthe analyzer shards without (
__name__,job,series). Thetest_series_aseries carry no other labels, so all of them hash to the same shard:{} => 57, thevector()sample collides with it andorcorrectly drops it, so the shard returns{} => 57;orlets thevector()sample through, and the shard returns{} => -0.3587905225767787.Merging the two shard results yields two
{}series at the same timestamp and the wrong one can win. PromQLorreturns all LHS series and, from the RHS, only series whose labelset is absent on the left — so57is the correct answer and the sharded result is wrong.The upstream Thanos analyzer already refuses to shard
absent,absent_over_timeandscalarfor the same underlying reason (their results depend on data an individual shard cannot see), butvectoris missing from that list. Cortex's owndisableBinaryExpressionAnalyzerwrapper does not cover it either, and is only installed when the parquet queryable is enabled.Since the function list lives in vendored Thanos, this PR adds a small Cortex-side wrapper analyzer that marks any query using
vector()as not shardable, and installs it unconditionally ininitQueryFrontendTripperware. Queries that do not usevector()are unaffected, so this costs sharding only for queries that cannot be sharded correctly today. Addingvectorto the Thanos analyzer upstream would let the wrapper be dropped again — worth a follow-up.Filtering
vector(out of the fuzz corpus instead — the approach proposed in #7547 and implemented in the still-open #7551 — would have hidden a real, user-visible wrong-results bug, so it is deliberately not done here.Verification
Unit tests, including new cases for the wrapper (
or vector(0)and nestedvector()become non-shardable; ordinary aggregations and a selector merely namedvector_seriesstay shardable):Reproduced by hand first — same build on both sides, one unsharded and one with
-frontend.query-vertical-shard-size=2, querying directly with no fuzzer involved.Before:
After:
And from the fuzz test at the seed the failing CI run logged, against a locally built image (
linux/arm64, macOS/Docker Desktop):With this change:
To avoid racing another build sharing this Docker daemon, the image was built under a private tag rather than via
make ./cmd/cortex/.uptodate(which retags:latest).No config or flags changed, so
make docwas not required.Related
The other failure in the same CI job —
TestExpandedPostingsCacheFuzz— is a different root cause (a legitimate Prometheus 3.8-vs-3.9 semantic difference across the released image andmaster, not a Cortex bug). It is tracked in #7803 and fixed test-side in #7805.Fixes #7804