Delay verify-replay retries without holding a SQL thread

[emelialei88]

🎯 The "Why" (Intent)

When a transaction loses the commit-time verify check, it is replayed immediately. Under contention every loser retries in lockstep and collides again, so replays climb into the dozens and latency accumulates into seconds. Only distributed transactions had a backoff, and it sleeps on the SQL thread, starving unrelated clients when the pool is small.

🛠️ The "What" (Critical Changes)

• After verify_retry_poll_after (default 3) immediate replays, a verify-failed txn is parked on a timer for a random [0, verify_retry_poll) ms (default 50), then re-queued. The SQL thread is released while it waits. verify_retry_poll 0 restores lockstep retries.
• Distributed transactions keep their existing on-thread poll.
• tests/verify_retry_poll.test: a benchmark (not pass/fail).

📊 Benchmark

4-node cluster, 32 writers × 200 upserts on one row, SQL pool of 8 threads. Bystander = an unrelated write to another table during the storm (~45 ms idle, mostly connection setup). "On-thread" sleeps on the SQL worker; "parked" is this PR. Both pause from the 2nd retry on (threshold 1, see below), so only the thread use differs.

max wait	storm time	replays	bystander avg / p95, on-thread	bystander avg / p95, parked
0 (today)	~19 s	~165k	164 / 661 ms	169 / 662 ms
10 ms	5.7–6.0 s	24–32k	76 / 128 ms	52 / 54 ms
25 ms	4.8–5.1 s	12–14k	79 / 147 ms	50 / 53 ms
50 ms	4.4–4.6 s	7–8k	64 / 62 ms	50 / 52 ms
100 ms	4.5–4.6 s	4–5.5k	52 / 55 ms	50 / 52 ms

Threshold: how many retries go back immediately before pausing (verify_retry_poll_after)

The original attempt is not a retry. With threshold N, retries 1..N are re-queued immediately and the pause starts before retry N+1:

threshold 1:  attempt ✗ → retry 1 ✗ → pause → retry 2 ...
threshold 3:  attempt ✗ → retry 1 ✗ → retry 2 ✗ → retry 3 ✗ → pause → retry 4 ...

Parked, 50 ms max wait, pool 48, 32 writers. Each cell is total time / total replays.

writers per row	threshold 1	threshold 3	threshold 5
32 (one row, 200 upserts each, 3 runs)	4.5 s / 8.0k	4.4 s / 13.8k	4.6 s / 18.6k
~8 (4 rows, 1000 upserts each, 2 runs)	13.8 s / 24k	15.8 s / 44k	16.0 s / 55k
~2 (16 rows, 1000 upserts each, 2 runs)	13.5 s / 15k	14.0 s / 26k	16.1 s / 35k
~0.5 (64 rows, 1000 upserts each, 2 runs)	12.2 s / 0.6k	12.4 s / 0.7k	12.2 s / 0.9k

Each extra immediate retry mostly collides again: replays grow with the threshold everywhere, and time never improves beyond noise (±0.6 s). Threshold 1 is fastest or tied. The default here is 3; the data favours 1, pending a re-run on a larger cluster.

With writers spread over 2000 rows (little conflict), every setting gives 2.2–2.6 s and under 1k replays. With a 48-thread pool, the bystander is unaffected either way.

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
analyze_partial_index_off_generated [failed with core dumped] **quarantined**
analyze [failed with core dumped] **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
tunables
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**
skipscan [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_truncate_multiddl_generated [db unavailable at finish] **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
tunables
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: 2/730 tests failed ⚠.

The first 10 failing tests are:
comdb2sys_queueodh_generated
consumer_non_atomic_default_consumer_generated **quarantined**