test: cover embedded fulltext first pass and memtable release

[cms-cms]

Summary

• Add native embedded full-text stability integration tests for Intermittent 4016 on first full-text pass after fresh ingestion in embedded seekdb 1.4.0 seekdb#1382: immediate full-text queries after fresh SDK ingestion, and eight concurrent scans overlapping continuous inserts and frequent minor freeze.
• Each scenario uses an isolated process and fresh database. The scan/freeze table has a full-text index and no vector index.
• Preserve the first error, exception chain, best-effort same-connection trace ID and bounded log context immediately. Large rotated logs use a total byte budget; exact failed traces take priority over routine memtable messages.
• Keep database errors as failures. No query retry, xfail, error suppression, reduced pressure, or kernel changes.

The deterministic synthetic corpus is not the original SciFact benchmark. The first-pass test uses supplied 8-dimensional embeddings. The scan/freeze case preloads 24,515 documents, runs eight scanning connections, inserts 32 rows per freeze round, and requests MINOR FREEZE every 120ms for 75 seconds.

Native embedded comparison

These results used test snapshot abab612eab3d41a8be289fc12f67be68959ae32e, bindings 5e8eece63205dccfa9500f98893c91d1f50eb4f1, old SeekDB dc88204e55100056acbfd64e213cd1ac2a256cd9 and fixed SeekDB 8a8d6f5a8dd48cdbf859305b3f7eb050149a5bd1 (includes #1384).

Platform / kernel	First-pass queries	Scan/freeze successful queries	Freezes	Scan errors
macOS arm64 / old	1200, no errors	169481	590	80, all 4016 in retained error records
macOS arm64 / fixed	1200, no errors	176592	589	0
Linux x86_64 / old	1200, no errors	64970	625	67 errors; first explicitly 4016
Linux x86_64 / fixed	1200, no errors	53351	625	0

The actual embedded client was used, not a server endpoint or mock. Original conditional backend/version skips were retained.

Evidence boundaries and other results

• macOS old: the first failed client trace matches server end_stmt / send_error_packet records returning -4016. Memtable release/read-barrier messages are present, but the complete original issue's data/schema-mismatch → memtable-iterator → block-stat chain was not recovered. The error code alone is not proof of an identical kernel stack.
• Linux old: the CI console preserves the failed scenario report and the first real OperationalError(4016, 'Internal error'). Uploading the oversized full-database artifact failed with HTTP 413. Raw server logs and the other 66 individual error records are therefore not verified; do not label every error as confirmed 4016 or claim a matched server stack.
• macOS fixed: full applicable suite passed (599 unit, 3 neutral integration, 112 embedded integration, plus 1 existing XPASS). The old Mac also hit a separate collection-fork 4012 timeout; that operation passed with the fixed kernel.
• Linux fixed: both new FTS cases and all applicable embedded integrations passed, but the whole job failed in a fake-client harness unit test: a 30ms mock window ended before one simulated thread ran. This is separate from the real database result.

Diagnostic follow-up validation (before the CI source-wheel change)

The follow-up changes only diagnostics and fake-client unit scheduling, not the real SQL, workload duration, data, or failure assertions:

• 18 harness tests passed, including >256MiB rotated logs, immediate capture and noisy-context budget exhaustion.
• The fake-client test now uses a bounded first-round barrier and a test-local logical clock; 100 consecutive local iterations passed.
• Full upstream unit suite in the actual Linux CI image: 601 passed, 221 existing conditional skips, no failures.
• Ruff checks passed.
• A retrospective read of retained macOS raw logs confirmed that the prioritized 2MiB output keeps the first failed trace's server error records.
• A fresh Linux fixed full-suite CI run for the follow-up snapshot is pending. Old product-failure runs are not retried just to improve diagnostics.

This is a regression test addition, not a database fix. A finite fixed-version pass does not prove that every timing interleaving is eliminated.

GitHub CI source-wheel correction

The previous GitHub run 35204502666 installed the released pylibseekdb 1.3.0 from uv.lock, not the source-built fixed wheel used above. It correctly failed both new tests: one explicit 4016 in the first pass and 72 scan errors, with the first scan error explicitly 4016. These failures are not hidden or reclassified as success.

Commit 4a5d0d0c4f2f2c0d5fb37daa934a38097ff5165b adds a pinned source build inside the existing embedded job (no extra preflight job). The kernel and bindings use the same exact fixed revisions listed above. A native Ubuntu 22.04 / CPython 3.11 wheel is cached by source pins and build recipe, checked against wheel/binary SHA-256 and full REVISION, and verified after installation. Tests and examples use uv run --no-sync to prevent replacement by the released wheel. All embedded test assertions remain intact. Other jobs retain the existing dependency lock. This CI-local wheel is not a PyPI release or a portable manylinux package.

The failed quality check was Ruff 0.14.11 formatting; the source now matches that version. Local validation: all 165 Python files pass Ruff check/format; 33 focused unit/contract tests pass; Bash syntax, actionlint 1.7.12 and uv 0.9.22 locked resolution pass. The latest GitHub run 35224727865 has passed quality; embedded source build/full integration validation is still pending. This does not replace the pending internal full-suite/artifact-collector follow-up.

All PR commit authors and committers are now cms-cms; the accidentally machine-attributed diagnostic commit was corrected without changing its code tree. CLA status is being rechecked. Keep this PR draft until the new CI results are established.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[CLAassistant]

All committers have signed the CLA.