Skip to content

perf(unique): make verifyUniqueWithinMutation linear - #9822

Open
shiva-istari wants to merge 2 commits into
mainfrom
shiva/unique-perf
Open

perf(unique): make verifyUniqueWithinMutation linear#9822
shiva-istari wants to merge 2 commits into
mainfrom
shiva/unique-perf

Conversation

@shiva-istari

@shiva-istari shiva-istari commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Fixes #9814.

The in-request duplicate check for @unique predicates compared every unique-predicate edge against every other, calling dql.TypeValFrom once per pair: O(N^2) time and allocations in the number of edges per mutation. At 8k edges one check took 1.67s and 64M allocations, dominating batched writes on @unique predicates.

Replace the nested scan with a single pass over a seen-map keyed on (predicate, value), remembering the first subject that set each value. Semantics are unchanged: duplicate values from the same subject remain allowed, nil ObjectValues are skipped, entries pruned by updateMutations are still ignored, and the error message is identical. Value identity still uses the interface{} produced by TypeValFrom, so type identity participates in the comparison exactly as it did with ==.

Measured (M4 Pro, benchstat over 6 runs, all p=0.002): 5.20ms -> 32.6us at 500 edges, 1.67s -> 585us at 8000 edges (-99.96%); allocs/op drops from N^2 (64M at 8k) to ~N (8k).


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new linear algorithm should be backed by targeted unit tests asserting the core within-mutation @unique semantics to guard against regressions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR optimizes the in-request duplicate check for @unique predicates by replacing the previous quadratic pairwise scan with a linear, map-based pass keyed by (predicate, value), reducing CPU and allocations for large batched mutations.

Changes:

  • Introduces a uniqueValueKey to represent (predicate, value) identity for within-mutation duplicate detection.
  • Rewrites verifyUniqueWithinMutation to track first-seen subjects in a seen map, making the check O(N) in the number of unique edges.
  • Preserves prior semantics around same-subject duplicates, nil ObjectValue skipping, and pruned-mutation handling.
File summaries
File Description
edgraph/server.go Replaces O(N²) within-mutation @unique duplicate detection with a single-pass seen-map keyed by (predicate, value).
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread edgraph/server.go
Comment thread edgraph/server.go
Comment thread edgraph/server.go
Fixes #9814.

The in-request duplicate check for @unique predicates compared every
unique-predicate edge against every other, calling dql.TypeValFrom once
per pair: O(N^2) time and allocations in the number of edges per
mutation. At 8k edges one check took 1.67s and 64M allocations,
dominating batched writes on @unique predicates.

Replace the nested scan with a single pass over a seen-map keyed on
(predicate, value), remembering the first subject that set each value.
Semantics are unchanged: duplicate values from the same subject remain
allowed, nil ObjectValues are skipped, entries pruned by
updateMutations are still ignored, and the error message is identical.
Value identity still uses the interface{} produced by TypeValFrom, so
type identity participates in the comparison exactly as it did with ==.

Measured (M4 Pro, benchstat over 6 runs, all p=0.002): 5.20ms -> 32.6us
at 500 edges, 1.67s -> 585us at 8000 edges (-99.96%); allocs/op drops
from N^2 (64M at 8k) to ~N (8k). The after curve doubles per doubling
of N, i.e. linear.
…d tests

Review catch: uniqueValueKey held whatever dql.TypeValFrom returned,
and five of its branches return slice types ([]byte for
bytes/geo/datetime/bigfloat, []float32 for vfloat) - hashing one
panics, there is no recover on the mutation path, and the chunker makes
it reachable from a plain JSON mutation ("[1.0, 2.0]" on a string
@unique predicate parses as Vfloat32Val before the schema is
consulted). Worse than the old code, whose == comparison only ran once
two edges shared a predicate.

Slice values are now keyed by exact byte content
(string(v) / FloatArrayAsBytes), and types.TypeID joins the key so
equal bytes of different types never collide. The previous code
panicked on any two same-predicate slice values, so content equality
replaces a crash rather than changing working behavior.

Tests added as requested, next to the existing bounds checks:
- TestVerifyUniqueWithinMutationSemantics: different-subject duplicate
  rejected with the exact established error message; same-subject
  repeats, distinct values/predicates/types, nil ObjectValues and
  cross-mutation duplicates in one request.
- TestVerifyUniqueWithinMutationNonScalarValues: panic regression
  driving the reviewer's JSON repro through the real chunker (guarded
  against going vacuous), plus []byte content equality and
  string-vs-equal-bytes non-collision. Verified to panic with "hash of
  unhashable type: []float32" on the previous commit.

Perf holds: 38us @500 edges to 651us @8k, growth 2.0x per doubling
(linear); still -99.96% vs the O(N^2) code at 8k edges.
Comment thread edgraph/server.go
Comment on lines +2479 to +2480
default:
return uniqueValueKey{predicate: predicate, lang: lang, tid: tv.Tid, value: tv.Value}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the trap I was pointing at with the %T note last round, and the default arm is where it hits: tid is now part of the key for every value, not just the slice branches.

DefaultID and StringID both produce a Go string, so == treated them as equal. Keying on Tid splits them:

_:a <email> "x" .                 → DefaultVal → Tid=DefaultID, Value=string("x")
_:b <email> "x"^^<xs:string> .    → StrVal     → Tid=StringID,  Value=string("x")

Same worktree, same test, both commits:

commit result
014fe43 (base) could not insert duplicate value [x] for predicate [email]
5bd79b1 (this) <nil> — both edges written

And the everyday trigger isn't typed RDF literals, it's mixing RDF and JSON mutations in one api.Request: chunker/rdf_parser.go:215 emits DefaultVal for untyped literals while the JSON path emits StrVal. For two new nodes both injected eq() queries come back empty, so this check is the only guard, and it now passes. Two nodes land on a @unique predicate with the same value.

So the commit message's "preserving the type identity the previous ==-based check had" holds for int64(1) vs "1", but not here.

Fix is to keep type separation only where it's actually needed. For everything that isn't a slice the interface{} already carries its own Go dynamic type, which is exactly what == compared:

type uniqueValueKey struct {
	predicate string
	lang      string
	value     interface{}
}

// byteContent keys slice-typed values by exact bytes. Its own Go type keeps them from
// colliding with a plain string, and Tid separates the four []byte-backed types.
type byteContent struct {
	tid   types.TypeID
	bytes string
}

func uniqueValueKeyFrom(predicate, lang string, tv types.Val) uniqueValueKey {
	value := tv.Value
	switch v := tv.Value.(type) {
	case []byte:
		value = byteContent{tid: tv.Tid, bytes: string(v)}
	case []float32:
		value = byteContent{tid: tv.Tid, bytes: string(types.FloatArrayAsBytes(v))}
	}
	return uniqueValueKey{predicate: predicate, lang: lang, value: value}
}

I prototyped that on this commit: the DefaultVal/StrVal duplicate is caught again and every subtest you added still passes, a string never collides with equal bytes and the whole vfloat group included.

Comment thread edgraph/server_test.go
)))
})

t.Run("value type participates in identity", func(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This subtest passes under both the old == and the new tid key, so it doesn't actually pin the behavior it's named for. Every string in these tests is built with Value_StrVal, which is why the DefaultVal regression slipped through green CI.

Worth adding a case that puts both string representations in one request, since that's the one the key encoding can break:

t.Run("equal strings from different representations collide", func(t *testing.T) {
	require.Error(t, verifyUniqueWithinMutation(qcFor(
		nquad("_:a", "email", &api.Value{Val: &api.Value_DefaultVal{DefaultVal: "x"}}),
		nquad("_:b", "email", str("x")),
	)))
})

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

perf(unique): verifyUniqueWithinMutation is O(N^2), dominates @unique cost on batched mutations

3 participants