Skip to content

Call the bandit with no cache lock held - #14

Merged
sshaplygin merged 2 commits into
mainfrom
v04/b1-unlock-bandit
Sep 13, 2026
Merged

Call the bandit with no cache lock held#14
sshaplygin merged 2 commits into
mainfrom
v04/b1-unlock-bandit

Conversation

@sshaplygin

Copy link
Copy Markdown
Owner

An epoch used to run entirely under the cache's write lock, bandit call
included. Go's RWMutex queues new readers behind a waiting writer, so for as
long as SelectPolicy ran, every Get and Add in the process waited on
it
— a bandit reading a network store turned a store timeout into a cache
outage. The interface managed this by forbidding it ("an implementation must
not block"), which pushed the problem onto every implementer and is why the
distributed bandit had to buffer its reports and serve selections from an
atomic.

The split

1. write lock    close any gradual window, read and reset every arm's
                 counters into an epochSnapshot
2. no lock       deliver the snapshot, ask for a selection
3. write lock    check the selection is still current, apply

Phases 1 and 3 are each atomic, so no caller observes a half-applied switch.
Between them the cache is fully usable.

Two things holding one lock gave for free

Serialisation. Calls into the bandit are serialised by a new banditMu,
taken only while no cache lock is held. A Bandit is caller-supplied code with
no stated concurrency contract, and it used to be entered under the write lock;
that guarantee is kept. A slow bandit is now paid for by the next epoch instead
of by every reader.

Freshness. A selection is dropped if another epoch collected while it was
being made: it was computed from evidence two epochs old, and the stability
gates would check it against c.epochStats the newer epoch has already
overwritten.

epochsCollected is a second counter and the reason is not decorative.
epochID advances when an epoch is applied and the stability gates are
specified against it, so reusing it for staleness moved the cooldown arithmetic
by one epoch. TestSwitchStability_CooldownRearmsAfterSwitch caught that.

Red → green

The first commit (798f9f0) is the failing tests. Against its parent:

Test Failure before
SlowBanditDoesNotBlockReaders Get blocked behind the bandit: a slow bandit is a cache outage
StaleSelectionIsDiscarded Condition never satisfied — the second epoch cannot collect while the first holds the write lock
TickerAndRequestEpochsDoNotLoseCounts conservation under both epoch clocks at once

The tests were retargeted after that commit — the first version used
EpochRequests: 1, under which every Get runs an epoch, so there was no
non-triggering reader to measure. The final versions were re-run against the
pre-refactor implementation and still fail there
, which is the property that
matters.

Contract change

The Bandit godoc and docs/design.md no longer forbid blocking. They state
what it costs: the switch is delayed and may be dropped, concurrent Get and
Add are unaffected, calls are serialised, and under EpochRequests the
caller completing the epoch still pays — as it already did for the switch and
any migration it triggers.

Verification

go test -race -short -count=5 .   -> ok
make all                          -> Release checks passed, nine modules
make evidence                     -> regenerated

Hit rates move within run-to-run noise (zipf 68.14% against a recorded 67.88%,
with otter moving 73.36% against 73.25% in the same run); no table is
rewritten. Serial hot-path benchmarks unchanged — Get 102/153 ns, sampled
41/44; Add 98/144, sampled 53/55 — as expected, since the new lock is on the
epoch path and Get's path is untouched.

Class H — please read epoch.go in full, plus the new docs/design.md
paragraph. This is a concurrency core; the reviewer is the weak link here.

Three properties the epoch does not have yet. All three fail against this
commit's parent, which is the point of committing them first.

  SlowBanditDoesNotBlockReaders    a bandit parked in SelectPolicy must not
                                   stall Get. Fails: "Get blocked behind the
                                   bandit: a slow bandit is a cache outage".
  StaleSelectionIsDiscarded        a decision superseded by a later epoch must
                                   be dropped. Fails by deadlock -- the second
                                   epoch cannot start while the first holds the
                                   write lock, so the package times out.
  TickerAndRequestEpochsDoNotLose  every counted request reaches the bandit or
  Counts                           is still live in its arm, under both epoch
                                   clocks at once.

Go's RWMutex queues new readers behind a waiting writer, so a bandit called
while the cache's write lock is held stalls every Get in the process for as
long as it runs: a store timeout becomes a cache outage. The godoc on Bandit
currently manages this by forbidding it -- "an implementation must not block"
-- which pushes the problem onto every implementer and is why the distributed
bandit had to buffer and serve selections from an atomic.
An epoch used to run entirely under the cache's write lock, bandit call
included. Go's RWMutex queues new readers behind a waiting writer, so for as
long as SelectPolicy ran, every Get and Add in the process waited on it: a
bandit that read a network store turned a store timeout into a cache outage.
The interface managed this by forbidding it -- "an implementation must not
block" -- which pushed the problem onto every implementer, and is why the
distributed bandit had to buffer its reports and serve selections from an
atomic.

The epoch now runs in three phases:

  1. under the write lock: close any gradual window, read and reset every
     arm's counters into an epochSnapshot;
  2. holding no cache lock: deliver that snapshot and ask for a selection;
  3. under the write lock again: check the selection is still current, apply.

Phases 1 and 3 are each atomic, so no caller can observe a half-applied
switch. Between them the cache is fully usable.

Two things the split needs that holding one lock gave for free.

Calls into the bandit are serialised by a new banditMu, taken only while no
cache lock is held. A Bandit is caller-supplied code with no stated
concurrency contract and it used to be entered under the write lock; that
guarantee is kept, and the cost of a slow bandit is now paid by the next
epoch rather than by every reader.

A selection is dropped if another epoch collected while it was being made.
Such a decision was computed from evidence two epochs old, and the stability
gates would check it against c.epochStats the newer epoch has since
overwritten -- it would be admitted or rejected on numbers that are not its
own.

epochsCollected is a second counter, and the reason is not decorative: epochID
advances when an epoch is applied and the stability gates are specified
against it, so reusing it for staleness moved the cooldown arithmetic by one
epoch. TestSwitchStability_CooldownRearmsAfterSwitch caught that.

The Bandit godoc and docs/design.md no longer forbid blocking. They state what
it costs instead: the switch is delayed and may be dropped, concurrent Get and
Add are not affected, and under EpochRequests the caller completing the epoch
still pays -- as it already did for the switch and any migration.

Verified: go test -race -short -count=5 on the root module, make all across
nine modules, make evidence regenerated. Hit rates move within run-to-run
noise (zipf 68.14% against a recorded 67.88%, with otter moving 73.36% against
73.25% in the same run) and no table is rewritten. Serial hot-path benchmarks
are unchanged -- Get 102/153 ns, sampled 41/44, Add 98/144, sampled 53/55 --
which is expected: the new lock is on the epoch path, and Get's path is
untouched.
@sshaplygin
sshaplygin merged commit 25f27de into main Sep 13, 2026
25 checks passed
@sshaplygin
sshaplygin deleted the v04/b1-unlock-bandit branch September 13, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant