Skip to content

fix(query-frontend): don't vertically shard queries using vector() - #7806

Open
CharlieTLe wants to merge 2 commits into
cortexproject:masterfrom
CharlieTLe:fix-vertical-sharding-or-vector
Open

fix(query-frontend): don't vertically shard queries using vector()#7806
CharlieTLe wants to merge 2 commits into
cortexproject:masterfrom
CharlieTLe:fix-vertical-sharding-or-vector

Conversation

@CharlieTLe

Copy link
Copy Markdown
Member

What this PR does

Fixes a vertical-sharding correctness bug: <expr> or vector(0) — the idiomatic "default value" pattern in dashboards and alerting rules — returns the vector() fallback instead of the left-hand side when -frontend.query-vertical-shard-size > 1.

Vertical sharding appends a shard matcher to every vector selector (InjectShardingInfo), so each shard evaluates the whole query over only the series it owns and the shard results are concatenated. That is correct only while every output series is derived from a selector — then the shard that owns the series is the only shard that can produce it, and the analyzer's job is just to pick sharding labels that preserve the invariant.

vector(s) breaks the invariant: it synthesises a series with an empty labelset out of a scalar and has no selector behind it, so every shard produces it. For

max without (__name__, job, series) ({__name__="test_series_a",job="test"}) or vector(-0.3587905225767787)

the analyzer shards without (__name__, job, series). The test_series_a series carry no other labels, so all of them hash to the same shard:

  • that shard's LHS yields {} => 57, the vector() sample collides with it and or correctly drops it, so the shard returns {} => 57;
  • the other shard's LHS is empty, nothing collides, or lets the vector() sample through, and the shard returns {} => -0.3587905225767787.

Merging the two shard results yields two {} series at the same timestamp and the wrong one can win. PromQL or returns all LHS series and, from the RHS, only series whose labelset is absent on the left — so 57 is the correct answer and the sharded result is wrong.

The upstream Thanos analyzer already refuses to shard absent, absent_over_time and scalar for the same underlying reason (their results depend on data an individual shard cannot see), but vector is missing from that list. Cortex's own disableBinaryExpressionAnalyzer wrapper does not cover it either, and is only installed when the parquet queryable is enabled.

Since the function list lives in vendored Thanos, this PR adds a small Cortex-side wrapper analyzer that marks any query using vector() as not shardable, and installs it unconditionally in initQueryFrontendTripperware. Queries that do not use vector() are unaffected, so this costs sharding only for queries that cannot be sharded correctly today. Adding vector to the Thanos analyzer upstream would let the wrapper be dropped again — worth a follow-up.

Filtering vector( out of the fuzz corpus instead — the approach proposed in #7547 and implemented in the still-open #7551 — would have hidden a real, user-visible wrong-results bug, so it is deliberately not done here.

Verification

Unit tests, including new cases for the wrapper (or vector(0) and nested vector() become non-shardable; ordinary aggregations and a selector merely named vector_series stay shardable):

$ go test -tags "netgo slicelabels" -count=1 ./pkg/querysharding/... ./pkg/cortex/... ./pkg/querier/tripperware/...
ok  	github.com/cortexproject/cortex/pkg/querysharding	0.193s
ok  	github.com/cortexproject/cortex/pkg/cortex	1.375s
ok  	github.com/cortexproject/cortex/pkg/querier/tripperware	0.355s
ok  	github.com/cortexproject/cortex/pkg/querier/tripperware/instantquery	0.577s
ok  	github.com/cortexproject/cortex/pkg/querier/tripperware/queryrange	0.543s

Reproduced by hand first — same build on both sides, one unsharded and one with -frontend.query-vertical-shard-size=2, querying directly with no fuzzer involved.

Before:

query: max without (__name__, job, series) ({__name__="test_series_a",job="test"}) or vector(-0.3587905225767787)
  unsharded: {} => 57
  sharded  : {} => -0.3587905225767787

query: (max without (__name__, job, series) ({__name__="test_series_a",job="test"}) or (-{__name__="test_series_b"} or vector(-0.3587905225767787)))
  unsharded: {} => 57
  sharded  : {} => -0.3587905225767787

After:

query: max without (__name__, job, series) ({__name__="test_series_a",job="test"}) or vector(-0.3587905225767787)
  unsharded: {} => 57
  sharded  : {} => 57

query: (max without (__name__, job, series) ({__name__="test_series_a",job="test"}) or (-{__name__="test_series_b"} or vector(-0.3587905225767787)))
  unsharded: {} => 57
  sharded  : {} => 57

And from the fuzz test at the seed the failing CI run logged, against a locally built image (linux/arm64, macOS/Docker Desktop):

$ CORTEX_IMAGE=<local build of master> FUZZ_SEED=1787336350 go test -v \
    -tags "integration,requires_docker,integration_query_fuzz" -timeout 2400s \
    -count=1 ./integration/ -run '^TestVerticalShardingFuzz$'

    query_fuzz_test.go:2246: case 92 results mismatch.
        instant query: (
            max without (__name__, job, series) ({__name__="test_series_a",job="test"})
          or
            (-{__name__="test_series_b"} or vector(-0.3587905225767787))
        )
        res1 len: 4 data: {} => 57 @[1787336921.48]
        ...
        res2 len: 4 data: {} => -0.3587905225767787 @[1787336921.48]
--- FAIL: TestVerticalShardingFuzz (11.81s)

With this change:

FUZZ_SEED=1787336350 --- PASS: TestVerticalShardingFuzz (11.58s)
FUZZ_SEED=1          --- PASS: TestVerticalShardingFuzz (12.06s)
FUZZ_SEED=424242     --- PASS: TestVerticalShardingFuzz (12.15s)
FUZZ_SEED=987654321  --- PASS: TestVerticalShardingFuzz (12.02s)

To avoid racing another build sharing this Docker daemon, the image was built under a private tag rather than via make ./cmd/cortex/.uptodate (which retags :latest).

No config or flags changed, so make doc was not required.

Related

The other failure in the same CI job — TestExpandedPostingsCacheFuzz — is a different root cause (a legitimate Prometheus 3.8-vs-3.9 semantic difference across the released image and master, not a Cortex bug). It is tracked in #7803 and fixed test-side in #7805.

Fixes #7804

`<expr> or vector(0)` returns the vector() fallback instead of the left hand
side when -frontend.query-vertical-shard-size > 1.

Vertical sharding appends a shard matcher to every vector selector, so each
shard evaluates the whole query over only the series it owns and the shard
results are concatenated. That is correct only while every output series is
derived from a selector: then the shard that owns the series is the only shard
that can produce it, and the analyzer's job is just to pick sharding labels
that preserve the invariant.

vector(s) breaks it. It synthesises a series with an empty labelset out of a
scalar and has no selector behind it, so every shard produces it. For

        max without (__name__, job, series) ({__name__="test_series_a",job="test"})
          or vector(-0.3587905225767787)

the analyzer shards without (__name__, job, series). The test_series_a series
carry no other labels, so all of them hash to the same shard:

  * that shard's LHS yields {} => 57, the vector() sample collides with it and
    `or` correctly drops it, so the shard returns {} => 57;
  * the other shard's LHS is empty, nothing collides, `or` lets the vector()
    sample through and the shard returns {} => -0.3587905225767787.

Merging the shard results yields two {} series at the same timestamp and the
wrong one can win. Observed:

        unsharded: {} => 57
        sharded  : {} => -0.3587905225767787

PromQL `or` returns all LHS series and from the RHS only series whose labelset
is absent on the left, so 57 is the correct answer and the sharded result is
wrong.

The upstream Thanos analyzer already refuses to shard absent, absent_over_time
and scalar for the same underlying reason - their results depend on data an
individual shard cannot see - but `vector` is missing from that list, and
Cortex's own disableBinaryExpressionAnalyzer wrapper does not cover it and is
only installed when the parquet queryable is enabled.

Since the function list lives in vendored Thanos, add a small Cortex-side
wrapper analyzer that marks any query using vector() as not shardable, and
install it unconditionally in initQueryFrontendTripperware. Queries that do not
use vector() are unaffected, so this costs sharding only for queries that
cannot be sharded correctly today. Adding `vector` to the Thanos analyzer
upstream would let the wrapper be dropped again.

Filtering `vector(` out of the fuzz corpus instead - the approach proposed in
 cortexproject#7547 and implemented in the still-open cortexproject#7551 - would have hidden a real,
user-visible wrong-results bug, so it is deliberately not done here.

Verified by hand (unsharded vs -frontend.query-vertical-shard-size=2, same
build) and with the fuzz seed from the failing CI run:

        CORTEX_IMAGE=<local build> FUZZ_SEED=1787336350 go test -v \
          -tags "integration,requires_docker,integration_query_fuzz" \
          -timeout 2400s -count=1 ./integration/ \
          -run '^TestVerticalShardingFuzz$'

fails before the change and passes after it, as do seeds 1, 424242 and
987654321.

Fixes cortexproject#7804

Signed-off-by: Charlie Le <charlie_le@apple.com>
@dosubot dosubot Bot added component/query-frontend go Pull requests that update Go code type/bug labels Aug 21, 2026
Signed-off-by: Charlie Le <charlie_le@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: TestVerticalShardingFuzz — or vector(...) returns the vector() fallback instead of the LHS when vertical sharding is enabled

1 participant