Hand the replication-ack wait to a background thread (port of #1963)

[emelialei88]

🎯 The "Why" (Intent)

Every block processor blocks until all replicants ack its commit, so the 8 writer threads cap in-flight commits at 8. This moves that wait to a background thread without changing what clients see. Port of #1963.

🛠️ The "What" (Critical Changes)

• Split the inline ack wait into shared steps plus a non-blocking per-node poll.
• New seqnum-wait thread (async_dist_commit, off by default) applies the same timeouts, incoherent marking, durability and lease rules, then replies on a thread of its own.
• Only plain REP_SYNC_FULL blkseq commits (no durable LSNs, 2PC, rowlocks or schema change) are handed off; everything else, or a full queue, waits inline.
• A request can now be finished, and its log written, from another thread.
• Hand-off metrics, and an async_dist_commit test comparing each case against the inline wait.

async_dist_commit off (today)
  block processor: [apply + commit] ─▶ [wait for all acks ......] ─▶ reply ─▶ next request
                                        ▲ thread held the whole time

async_dist_commit on
  block processor: [apply + commit] ─▶ hand off ─▶ next request
                                          │
                     queue full, or not   │
                     plain REP_SYNC_FULL ─┼─▶ wait inline (old path)
                                          ▼
  seqnum-wait thread: queue (≤ max_outstanding_trans) ─▶ poll each node's ack
                                          ├─ node too slow ─▶ mark incoherent (same rules)
                                          └─ all acked ───▶ reply on its own thread (one per reply, ≤ cap) ─▶ client

📊 Results

8-node cluster, single-row autocommit inserts (each client has one commit in flight), 8 writer threads. Release build (RelWithDebInfo); sections marked Debug weren't rerun because they're diagnostics or replicant-bound. Old locking is forced with a benchmark-only switch, so both columns of each table use the same binary. Compare numbers within a table.

1. Off vs on (32 clients, cap 128, 2 runs of 180 s, inserts/s)

	off	on	gain
old locking	12,105	12,920	+7%
with #6293	14,933	16,679	+12%

Both PRs together: +38% over today. No commit fell back to the inline wait.

2. Gain by client count (with #6293, cap = clients, 2 runs of 120 s)

clients	off	on	on + early stop (prototype, section 7)
32	13,945	16,563 (+19%)	16,952 (+22%)
64	13,792	14,291 (+4%)	14,997 (+9%)
128	13,394	11,828 (−12%)	13,371 (0%)
256	13,963	13,360 (−4%)	14,390 (+3%)

Off stays at ~14k whatever the client count. On peaks at 32 clients; past that each waiter pass checks more commits in flight and the gain shrinks. Early stop recovers most of it.

3. More writer threads (32 clients, old locking, cap 64, 2 runs of 120 s)

	8 threads	32 threads
off	12,014 (6.6 cores)	4,751 (15.1 cores)
on	13,223 (6.9)	11,292 (6.2)

Every ack broadcasts on one shared condition, waking every writer thread asleep in the inline wait (bdb_wait_for_seqnum_from_all_int); each re-takes the same mutex, checks, and sleeps again. Debug perf profile at 32 threads: ~40% of master CPU in that loop (waking 12.8%, "acked yet?" check 11.0%, mutex re-take 10.0%, check + LSN compares 5.0%), and the ack thread queues on the same mutex. With this change only the waiter sleeps on it.

4. Where writer threads spend their time (Debug, 32 clients, stack samples of the master)

	asleep waiting for acks	waiting on repo lock	other	idle
8 threads	31%	36%	33%	0
32 threads	86%	6%	3%	5%
32 threads + this	0	55%	8%	37%

With the waiter, the next bottleneck is the repo lock (#6293).

5. async_dist_commit_max_outstanding_trans (64 clients, 2 runs of 120 s; % = commits that waited inline)

cap	old locking	with #6293
off	11,839	13,257
16	11,256 (6%)	12,167 (22%)
32	10,984 (3%)	11,353 (12%)
64	11,148 (0)	14,313 (+8%)
128	10,791 (0)	14,453 (+9%)

• Cap below the client count is worse than off. A commit that doesn't fit waits inline and holds a writer thread for a whole ack round trip (Debug stack samples: ~4% of commits took 42% of writer-thread time).
• Cap ≥ clients: loses a little with old locking and gains with Take the trn repo lock only for logical commits #6293. Not root-caused with old locking; slowing the waiter down raised throughput (Debug), so its passes rather than its speed seem to be the cost.

6. Slow replicants (Debug)

• Every replicant sleeping 1 ms per message in its receive thread (16 clients): 187 off / 187 on. Commits finish at the replicants' pace; the waiter changes who waits, not how long.
• Every apply thread sleeping 2 ms per transaction (32 / 64 clients): 4 apply threads (default) 1,892 / 1,897 off vs 1,878 / 1,889 on; 32 apply threads 10,753 / 10,684 off vs 11,954 / 12,191 on (+11% / +14%). With enough apply threads the early ack hides the apply time and the usual gain returns.

7. Prototype: cheaper waiter passes (cap = clients, 2 runs of 120 s; not in this PR)

Skip reads every node's acks once per pass and skips commits that clearly aren't acked yet; early stop stops a pass at the first commit not yet acked by everyone (acks are cumulative per node).

clients	locking	off	on	on + skip	on + early stop
64	old	11,107	10,894 (−2%)	11,581 (+4%)	11,270 (+1%)
64	#6293	12,647	14,874 (+18%)	14,412 (+14%)	15,285 (+21%)
128	old	11,053	8,768 (−21%)	9,047 (−18%)	9,266 (−16%)
128	#6293	12,717	12,363 (−3%)	12,777 (+0.5%)	13,120 (+3%, 1 run)

With #6293 and early stop, against today's code (old locking, off; cap 128, 3 runs of 180 s): 32 clients 12,220 → 17,301 (+42%), 64 clients 11,282 → 15,665 (+39%).

8. Late acks (network latency) (cap 128, 2 runs of 120 s, inserts/s)

Benchmark-only patch on every replicant: apply each commit at full speed, but send its ack D ms later from a separate thread, as if over a long link.

ack delay	clients	off	on, old locking	on, with #6293
1 ms	32	5,950	12,815 (2.2×)	15,165 (2.6×)
1 ms	64	5,871	11,124 (1.9×)	14,411 (2.5×)
2 ms	32	3,433	11,051 (3.2×)	11,152 (3.3×)
2 ms	64	3,423	11,185 (3.3×)	14,337 (4.2×)
5 ms	32	1,502	5,598 (3.7×)	5,613 (3.7×)
5 ms	64	1,500	10,078 (6.7×)	10,406 (6.9×)

Off, throughput is capped at about 8 writer threads ÷ ack delay, however many clients there are; on, it grows with clients ÷ delay until the master's other limits. At 2 ms / 32 and 5 ms / 32 the clients are the limit (32 commits per round trip), so #6293 adds nothing there. No commit waited inline.

9. Short replicant pauses (Debug; one replicant frozen 1 s every 15 s, 32 clients): off 10,687, on 11,140. No clear effect, likely because the slow-replicant rule stops waiting for the frozen node.

10. Cost (Debug profile, 64 clients)

waiter thread	busiest thread on the master: 15–17% of its user CPU, ~42k passes/s
reply threads	one per reply in flight, ≤ cap; idle ones exit after 10 s

⚠️ Caveats

• Default cap is still 8 in code. Each client has at most one commit in flight, so writers beyond the cap fall back to the inline wait, which is slower than off (section 5). Any cap ≥ clients behaves the same; 64 or 128 would cover most cases.
• Locks drop earlier. With the waiter on, the block processor finishes before the acks arrive, so the locks it holds until it finishes (serializable commit lock, qconsume lock, time-partition views_lk, javasp lock) and the post-commit callbacks (analyze, genid48) are released/run after the local commit, before replication is confirmed. Client replies still wait for every ack.
• No gain when replicants are slow to receive (section 6), and quorum (sync atleast N) / durable-LSN commits always use the inline wait.
• A single slow replicant isn't a win case either: by default the master marks it incoherent (MAKE_SLOW_REPLICANTS_INCOHERENT) and stops waiting for it.
• Many clients cost throughput: with Take the trn repo lock only for logical commits #6293 the gain falls past 32 clients and turns into a loss at 128 (section 2). Cheaper passes (section 7, early stop) recover most of it; not in this PR.
• Needs Take the trn repo lock only for logical commits #6293: with old locking the waiter gains little without ack delay and loses at 128 clients (−21%).

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
reco-ddlk-sql **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
tunables
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
osql_cleanup [failed with core dumped]
analyze_partial_index_off_generated [failed with core dumped] **quarantined**
analyze [failed with core dumped] **quarantined**
tsa
consumer_non_atomic_default_consumer_generated **quarantined**
tunables
sc_downgrade [timeout] **quarantined**
skipscan [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: harness did not run - no test results were produced ⚠.

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
comdb2sys_queueodh_generated [db unavailable at finish]
reco-ddlk-sql **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**