Backport to 7.4: add retry limit to startMoveKeys to prevent infinite loops (#13176) - #13782
Conversation
…(PR 13176) Add START_MOVE_KEYS_MAX_RETRIES knob (default 50, BUGGIFY=10) and throw start_move_keys_too_many_retries when exceeded. Wire the new error into DDRelocationQueue (same light-weight re-queue handler as finish_move_keys_too_many_retries and move_to_removed_server) and normalDDQueueErrors so DD does not restart. Also propagate actor_cancelled before the retry limit check in startMoveKeys.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
RandomSeed="1264945544" SourceVersion="27f7192f47c366c7f668f79f1de23e05fecd8959" Time="1786731900" BuggifyEnabled="1" DeterminismCheck="0" OldBinary="fdbserver-7.4.5" FaultInjectionEnabled="1" TestFile="tests/restarting/from_7.3.0/UpgradeAndBackupRestore-1.toml" RandomSeed="1264945545" SourceVersion="1d9b495d6ba7f4390bba1bb4d51c38f4ed4a2055" Time="1786731930" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/restarting/from_7.3.0/UpgradeAndBackupRestore-2.toml" |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Result of foundationdb-pr-clang on Linux RHEL 9
|
Result of foundationdb-pr-clang-arm on Linux RHEL 9
|
Result of foundationdb-pr on Linux RHEL 9
|
Result of foundationdb-pr-cluster-tests on Linux RHEL 9
|
This comment has been minimized.
This comment has been minimized.
Result of foundationdb-pr-macos-m1 on macOS 14.x
|
Result of foundationdb-pr-macos on macOS 14.x
|
Add START_MOVE_KEYS_MAX_RETRIES knob (default 50, BUGGIFY=10) and throw start_move_keys_too_many_retries when exceeded. Wire the new error into DDRelocationQueue (same light-weight re-queue handler as finish_move_keys_too_many_retries and move_to_removed_server) and normalDDQueueErrors so DD does not restart. Also propagate actor_cancelled before the retry limit check in startMoveKeys.
20260724-003619-stack-backport-7.4-eac563f902694355 compressed=True data_size=41563174 duration=3238562 ended=100000 fail=5 fail_fast=10 max_runs=100000 pass=99995 priority=100 remaining=0 runtime=0:31:27 sanity=False started=100000 stopped=20260724-010746 submitted=20260724-003619 timeout=5400 username=stack-backport-7.4The 5 failures are all 'RandomSeed="1520625388" SourceVersion="ac41ec01f6c2b74a6c1432bb75f874a9c3ab2dd2" Time="1784853672" BuggifyEnabled="0" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/fast/MinimumThroughput.toml"'. I'll fix in a different PR. The failures are not related to here.
(They are fixed over in #13783)