Speed up mtree table-diff by fixing CDC-drain overhead

[danolivo]

Problem

mtree table-diff was reported ~2x slower than plain table-diff on a 499,999-row, 3-node cluster. mtree table-diff first reconciles each node's cached Merkle tree with current data by draining that node's CDC (Change Data Capture) stream (UpdateMtree); table-diff reads the live tables directly and has no cache to refresh, so it never drains. The drain was paying two avoidable costs on every diff:

1. Sequential per-node drain. Each node's CDC stream was drained one after another. With N nodes that multiplied the per-node drain latency N times.
2. Idle-timeout termination. In non-continuous mode the drain stopped only when it reached the target LSN via a data message, or after a ~1s idle receive-timeout. Because the publication streams only the tracked table, any unrelated WAL activity advances the global flush LSN while the table itself sees no data messages — so the "reached target" path rarely fires and the drain falls through to the full ~1s idle wait. This was paid per node, on every diff, even when nothing relevant had changed.

On a 3-node cluster that is a ~3s fixed floor (3 × ~1s) on every mtree table-diff, independent of whether anything changed — which is exactly why a brute-force table-diff could finish first.

Changes

1. Drain CDC streams concurrently across nodes (internal/consistency/mtree/merkle.go). Each UpdateFromCDC targets a distinct node over its own connection and writes only that node's CDC metadata, so the calls are independent. They now run concurrently; per-node errors are collected and the first (in node order) is returned, preserving prior semantics. Collapses N sequential drains into ~one drain's wall-clock.
2. Stop the drain promptly at the target LSN (internal/infra/cdc/listen.go). PostgreSQL keepalive messages carry ServerWALEnd ("I have sent you everything up to here"). When ServerWALEnd >= targetFlushLSN we are caught up and stop immediately, instead of waiting out the ~1s idle timeout. This is the correct catch-up signal for a publication-filtered stream (where the target often can't be reached via table data messages). Applies to non-continuous drains only; the long-running listener is unchanged.

Safety of (2): replication messages arrive in LSN order, so a keepalive past the target arrives only after every data message ≤ target was already delivered; received changes are applied in goroutines that the post-loop wg.Wait()/metaWg.Wait() await before returning, so setting the stop flag cannot drop a received change.

Correctness

table-diff path is not touched. The full mtree + CDC integration suites — including the under-reporting guards (TestUpdateMtreeReflectsInserts/Deletes/Updates, bidirectional/3-node diffs) and the CDC/slot tests — pass under go test -race.

Not in this PR

A follow-up bounded-drain / --quick mode (cap the drain when the WAL backlog is large, report per-node staleness, and gate table-repair from acting on an incomplete diff) is proposed separately; it targets the widespread/large-backlog case this PR does not fully close.

[coderabbitai]

📝 Walkthrough

Walkthrough

CDC replication now stops non-continuous streams when keepalive messages show the target flush LSN has been reached. UpdateMtree now starts per-node CDC updates concurrently, waits for all of them, and returns the first node error in node order.

Changes

CDC termination flow

Layer / File(s)	Summary
Keepalive stop condition internal/infra/cdc/listen.go	processReplicationStream sets stopStreaming when a keepalive ServerWALEnd reaches targetFlushLSN during non-continuous runs.
Concurrent CDC drain internal/consistency/mtree/merkle.go	UpdateMtree launches cdc.UpdateFromCDC for each cluster node in parallel, waits for all goroutines, and returns the first non-nil error in node order.

Poem

A rabbit hopped through LSN light,
and nodded when the stream felt right.
One node, then many, sprang in sync —
the burrow hummed with softer brink.
🐇

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: reducing CDC-drain overhead to speed up mtree table-diff.
Description check	✅ Passed	The description matches the changeset and explains the CDC-drain optimizations and their motivation.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ace-192

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 duplication

Metric	Results
Duplication	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/infra/cdc/listen.go`:
- Around line 332-335: Advance lastLSN before setting stopStreaming in the
keepalive catch-up path inside the replication loop. When pkm.ServerWALEnd has
reached targetFlushLSN in the non-continuous branch, make sure the value that
later feeds the final standby status and UpdateCDCMetadata is updated to the
target/observed WAL end before exiting, so the CDC checkpoint reflects the
drained position. Locate the fix near the keepalive catch-up check in listen.go
and adjust the replication-state variables used by the final status persistence.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0c2c0905-430f-48f1-ae5f-0a79f549147b

📥 Commits

Reviewing files that changed from the base of the PR and between d460f9e and 67ac95d.

📒 Files selected for processing (2)

• internal/consistency/mtree/merkle.go
• internal/infra/cdc/listen.go

[mason-sharp]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}