perf(unique): make verifyUniqueWithinMutation linear - #9822
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The new linear algorithm should be backed by targeted unit tests asserting the core within-mutation @unique semantics to guard against regressions.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR optimizes the in-request duplicate check for @unique predicates by replacing the previous quadratic pairwise scan with a linear, map-based pass keyed by (predicate, value), reducing CPU and allocations for large batched mutations.
Changes:
- Introduces a
uniqueValueKeyto represent(predicate, value)identity for within-mutation duplicate detection. - Rewrites
verifyUniqueWithinMutationto track first-seen subjects in aseenmap, making the check O(N) in the number of unique edges. - Preserves prior semantics around same-subject duplicates, nil
ObjectValueskipping, and pruned-mutation handling.
File summaries
| File | Description |
|---|---|
edgraph/server.go |
Replaces O(N²) within-mutation @unique duplicate detection with a single-pass seen-map keyed by (predicate, value). |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Fixes #9814. The in-request duplicate check for @unique predicates compared every unique-predicate edge against every other, calling dql.TypeValFrom once per pair: O(N^2) time and allocations in the number of edges per mutation. At 8k edges one check took 1.67s and 64M allocations, dominating batched writes on @unique predicates. Replace the nested scan with a single pass over a seen-map keyed on (predicate, value), remembering the first subject that set each value. Semantics are unchanged: duplicate values from the same subject remain allowed, nil ObjectValues are skipped, entries pruned by updateMutations are still ignored, and the error message is identical. Value identity still uses the interface{} produced by TypeValFrom, so type identity participates in the comparison exactly as it did with ==. Measured (M4 Pro, benchstat over 6 runs, all p=0.002): 5.20ms -> 32.6us at 500 edges, 1.67s -> 585us at 8000 edges (-99.96%); allocs/op drops from N^2 (64M at 8k) to ~N (8k). The after curve doubles per doubling of N, i.e. linear.
…d tests
Review catch: uniqueValueKey held whatever dql.TypeValFrom returned,
and five of its branches return slice types ([]byte for
bytes/geo/datetime/bigfloat, []float32 for vfloat) - hashing one
panics, there is no recover on the mutation path, and the chunker makes
it reachable from a plain JSON mutation ("[1.0, 2.0]" on a string
@unique predicate parses as Vfloat32Val before the schema is
consulted). Worse than the old code, whose == comparison only ran once
two edges shared a predicate.
Slice values are now keyed by exact byte content
(string(v) / FloatArrayAsBytes), and types.TypeID joins the key so
equal bytes of different types never collide. The previous code
panicked on any two same-predicate slice values, so content equality
replaces a crash rather than changing working behavior.
Tests added as requested, next to the existing bounds checks:
- TestVerifyUniqueWithinMutationSemantics: different-subject duplicate
rejected with the exact established error message; same-subject
repeats, distinct values/predicates/types, nil ObjectValues and
cross-mutation duplicates in one request.
- TestVerifyUniqueWithinMutationNonScalarValues: panic regression
driving the reviewer's JSON repro through the real chunker (guarded
against going vacuous), plus []byte content equality and
string-vs-equal-bytes non-collision. Verified to panic with "hash of
unhashable type: []float32" on the previous commit.
Perf holds: 38us @500 edges to 651us @8k, growth 2.0x per doubling
(linear); still -99.96% vs the O(N^2) code at 8k edges.
f8e3cd7 to
5bd79b1
Compare
| default: | ||
| return uniqueValueKey{predicate: predicate, lang: lang, tid: tv.Tid, value: tv.Value} |
There was a problem hiding this comment.
This is the trap I was pointing at with the %T note last round, and the default arm is where it hits: tid is now part of the key for every value, not just the slice branches.
DefaultID and StringID both produce a Go string, so == treated them as equal. Keying on Tid splits them:
_:a <email> "x" . → DefaultVal → Tid=DefaultID, Value=string("x")
_:b <email> "x"^^<xs:string> . → StrVal → Tid=StringID, Value=string("x")
Same worktree, same test, both commits:
| commit | result |
|---|---|
014fe43 (base) |
could not insert duplicate value [x] for predicate [email] |
5bd79b1 (this) |
<nil> — both edges written |
And the everyday trigger isn't typed RDF literals, it's mixing RDF and JSON mutations in one api.Request: chunker/rdf_parser.go:215 emits DefaultVal for untyped literals while the JSON path emits StrVal. For two new nodes both injected eq() queries come back empty, so this check is the only guard, and it now passes. Two nodes land on a @unique predicate with the same value.
So the commit message's "preserving the type identity the previous ==-based check had" holds for int64(1) vs "1", but not here.
Fix is to keep type separation only where it's actually needed. For everything that isn't a slice the interface{} already carries its own Go dynamic type, which is exactly what == compared:
type uniqueValueKey struct {
predicate string
lang string
value interface{}
}
// byteContent keys slice-typed values by exact bytes. Its own Go type keeps them from
// colliding with a plain string, and Tid separates the four []byte-backed types.
type byteContent struct {
tid types.TypeID
bytes string
}
func uniqueValueKeyFrom(predicate, lang string, tv types.Val) uniqueValueKey {
value := tv.Value
switch v := tv.Value.(type) {
case []byte:
value = byteContent{tid: tv.Tid, bytes: string(v)}
case []float32:
value = byteContent{tid: tv.Tid, bytes: string(types.FloatArrayAsBytes(v))}
}
return uniqueValueKey{predicate: predicate, lang: lang, value: value}
}I prototyped that on this commit: the DefaultVal/StrVal duplicate is caught again and every subtest you added still passes, a string never collides with equal bytes and the whole vfloat group included.
| ))) | ||
| }) | ||
|
|
||
| t.Run("value type participates in identity", func(t *testing.T) { |
There was a problem hiding this comment.
This subtest passes under both the old == and the new tid key, so it doesn't actually pin the behavior it's named for. Every string in these tests is built with Value_StrVal, which is why the DefaultVal regression slipped through green CI.
Worth adding a case that puts both string representations in one request, since that's the one the key encoding can break:
t.Run("equal strings from different representations collide", func(t *testing.T) {
require.Error(t, verifyUniqueWithinMutation(qcFor(
nquad("_:a", "email", &api.Value{Val: &api.Value_DefaultVal{DefaultVal: "x"}}),
nquad("_:b", "email", str("x")),
)))
})
Fixes #9814.
The in-request duplicate check for @unique predicates compared every unique-predicate edge against every other, calling dql.TypeValFrom once per pair: O(N^2) time and allocations in the number of edges per mutation. At 8k edges one check took 1.67s and 64M allocations, dominating batched writes on @unique predicates.
Replace the nested scan with a single pass over a seen-map keyed on (predicate, value), remembering the first subject that set each value. Semantics are unchanged: duplicate values from the same subject remain allowed, nil ObjectValues are skipped, entries pruned by updateMutations are still ignored, and the error message is identical. Value identity still uses the interface{} produced by TypeValFrom, so type identity participates in the comparison exactly as it did with ==.
Measured (M4 Pro, benchstat over 6 runs, all p=0.002): 5.20ms -> 32.6us at 500 edges, 1.67s -> 585us at 8000 edges (-99.96%); allocs/op drops from N^2 (64M at 8k) to ~N (8k).
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.