Skip to content

Add EQL v3 Drizzle support#565

Open
tobyhede wants to merge 25 commits into
james/cip-3291-bigint-stackfrom
eql-v3-drizzle-concrete-types
Open

Add EQL v3 Drizzle support#565
tobyhede wants to merge 25 commits into
james/cip-3291-bigint-stackfrom
eql-v3-drizzle-concrete-types

Conversation

@tobyhede

@tobyhede tobyhede commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the EQL v3 Drizzle integration (@cipherstash/stack/eql/v3/drizzle) plus hardened live test coverage, on top of the v3 typed client / schema DSL.

  • A Drizzle-native types namespace whose columns are the semantic eql_v3.<domain> Postgres types.
  • createEncryptionOperatorsV3 — capability-checked operators (eq/ne/gt/gte/lt/lte/between/notBetween/contains/inArray/notInArray/asc/desc/and/or/not/isNull/isNotNull/exists/notExists) that emit the two-argument eql_v3 term-function SQL. The concrete column type drives which operators are legal (e.g. contains needs a match index; asc/between need order).
  • extractEncryptionSchemaV3 rebuilds the encrypt schema for EncryptionV3.
  • Live test coverage — a type-driven 35-domain matrix drives live encrypt/decrypt + SQL query proofs. Recent additions strengthen assertion quality: range exclusion (not just inclusion), type-boundary values through real domain columns, NULL persistence across capability tiers, and lock-context querying through the operator path — plus a CI guard that fails loudly if the live suites silently skip.

The existing v2 @cipherstash/stack/drizzle integration is unchanged. v3 free-text search is token containment, so contains replaces SQL like/ilike (deliberately not exposed).

Usage

import { drizzle } from "drizzle-orm/postgres-js"
import { integer, pgTable } from "drizzle-orm/pg-core"
import postgres from "postgres"
import { EncryptionV3 } from "@cipherstash/stack/v3"
import {
  createEncryptionOperatorsV3,
  extractEncryptionSchemaV3,
  types,
} from "@cipherstash/stack/eql/v3/drizzle"

const users = pgTable("users", {
  id: integer("id").primaryKey().generatedAlwaysAsIdentity(),
  email: types.TextSearch("email"),   // free-text search
  nickname: types.TextEq("nickname"), // equality
  age: types.IntegerOrd("age"),       // order + range
})

const client = await EncryptionV3({ schemas: [extractEncryptionSchemaV3(users)] })
const e = createEncryptionOperatorsV3(client)
const db = drizzle({ client: postgres(process.env.DATABASE_URL!) })

// Filter operators are async (they encrypt the query term); asc/desc are sync.
const rows = await db
  .select()
  .from(users)
  .where(await e.contains(users.email, "example.com"))
  .orderBy(e.asc(users.age))

Test Plan

  • cd packages/stack && pnpm biome check src/eql/v3/drizzle __tests__/{drizzle-v3,v3-matrix}
  • cd packages/stack && pnpm vitest run __tests__/{drizzle-v3,v3-matrix}
  • cd packages/stack && pnpm test:types && pnpm build
  • Live suites (Postgres + eql_v3) run when DATABASE_URL + CS_* are set and self-install the eql_v3 extension at beforeAll. The Drizzle lock-context suite additionally needs USER_JWT and soft-skips without it.

Notes

  • Live tests are credential-gated; a new REQUIRE_LIVE / REQUIRE_LIVE_PG guard turns a silent skip into a CI failure so the live matrix can't quietly go green.
  • The two-argument eql_v3.* function forms are interim (tracked by CIP-3402 / CIP-3423).

Summary by CodeRabbit

  • New Features

    • Added support for a new encrypted database integration with richer query capabilities, including equality, ranges, pattern matching, sorting, and membership checks.
    • Introduced a new set of column types for working with encrypted data while keeping query behavior aligned with the underlying field type.
    • Existing encrypted database integration remains available unchanged.
  • Bug Fixes

    • Improved handling of null and undefined values when reading and writing encrypted JSON-backed fields.

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5e72d5bd-9762-47c4-b5d8-a4e4b0b938e3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds a new EQL v3 Drizzle integration under @cipherstash/stack/eql/v3/drizzle, including a JSONB codec, a column adapter with domain discovery/stashing, a PascalCase types namespace, an eql_v3 SQL dialect, capability-checked encryption operators, and schema extraction, plus package export wiring, build config updates, design docs, and comprehensive tests.

Changes

EQL v3 Drizzle Integration

Layer / File(s) Summary
Design docs and changeset
docs/superpowers/specs/2026-07-06-eql-v3-drizzle-concrete-types-design.md, .changeset/eql-v3-drizzle.md
Documents the concrete-type v3 architecture, module layout, operator/error semantics, and testing plan; declares a minor release.
JSONB codec
packages/stack/src/eql/v3/drizzle/codec.ts, packages/stack/__tests__/drizzle-v3/codec.test.ts
Adds v3ToDriver/v3FromDriver for JSONB serialization with null/undefined handling.
Column adapter
packages/stack/src/eql/v3/drizzle/column.ts, packages/stack/__tests__/drizzle-v3/column.test.ts
Adds EQL_V3_DOMAINS, makeEqlV3Column, getEqlV3Column, isEqlV3Column for stashing/recovering v3 builders on Drizzle columns.
Types namespace
packages/stack/src/eql/v3/drizzle/types.ts, packages/stack/__tests__/drizzle-v3/types.test.ts, types.test-d.ts
Exposes a types object mirroring @/eql/v3 types, wrapping factories with makeEqlV3Column.
SQL dialect
packages/stack/src/eql/v3/drizzle/sql-dialect.ts, packages/stack/__tests__/drizzle-v3/sql-dialect.test.ts
Adds v3Dialect generating eql_v3 term-function SQL for equality, comparison, range, match, orderBy.
Schema extraction
packages/stack/src/eql/v3/drizzle/schema-extraction.ts, packages/stack/__tests__/drizzle-v3/schema-extraction.test.ts
Adds extractEncryptionSchemaV3 to rebuild EncryptionV3 schemas from Drizzle tables.
Encryption operators
packages/stack/src/eql/v3/drizzle/operators.ts, packages/stack/__tests__/drizzle-v3/operators.test.ts, operators-live-pg.test.ts
Adds createEncryptionOperatorsV3 and EncryptionOperatorError with capability-checked eq/ne/comparison/range/match/inArray/order/boolean operators.
Export wiring and build
packages/stack/src/eql/v3/drizzle/index.ts, packages/stack/package.json, packages/stack/tsup.config.ts, packages/stack/__tests__/drizzle-v3/exports.test.ts
Adds the barrel export and wires new ./eql/v3/drizzle package export/build entry.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant App
  participant createEncryptionOperatorsV3
  participant EncryptionClient
  participant v3Dialect
  participant Postgres

  App->>createEncryptionOperatorsV3: eq(column, value)
  createEncryptionOperatorsV3->>createEncryptionOperatorsV3: resolve column context and requireIndex
  createEncryptionOperatorsV3->>EncryptionClient: encryptQuery(value, queryType)
  EncryptionClient-->>createEncryptionOperatorsV3: encrypted term
  createEncryptionOperatorsV3->>v3Dialect: equality(term)
  v3Dialect-->>createEncryptionOperatorsV3: SQL fragment
  createEncryptionOperatorsV3-->>App: SQL
  App->>Postgres: execute query with SQL
Loading

Possibly related issues

Suggested reviewers: coderdan

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding EQL v3 Drizzle support.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch eql-v3-drizzle-concrete-types

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@changeset-bot

changeset-bot Bot commented Jul 6, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0fa0793

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 6 packages
Name Type
@cipherstash/stack Minor
@cipherstash/bench Patch
@cipherstash/prisma-next Patch
@cipherstash/basic-example Patch
@cipherstash/prisma-next-example Patch
@cipherstash/e2e Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@tobyhede tobyhede changed the base branch from main to feat/eql-v3-text-search-schema July 6, 2026 06:56
@tobyhede tobyhede marked this pull request as ready for review July 6, 2026 06:57
@tobyhede tobyhede requested a review from a team as a code owner July 6, 2026 06:57

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (3)
packages/stack/__tests__/drizzle-v3/column.test.ts (1)

11-18: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test reaches into Drizzle's internal column config instead of the public API.

(col as any).config.customTypeParams.dataType() asserts on an internal Drizzle builder shape rather than exercising a public entry point (e.g., col.getSQLType(), which is Drizzle's documented API for the emitted SQL domain type). This couples the test to internal customTypeParams/config shape called out as fragile in column.ts.
As per coding guidelines, "prefer testing through the public API rather than private internals."

♻️ Proposed fix using public API
   it('sets dataType() to the concrete eql_v3 domain', () => {
     const col = makeEqlV3Column(v3Types.Int4Ord('age'))
-    // biome-ignore lint/suspicious/noExplicitAny: reading drizzle internals in test
-    expect((col as any).config.customTypeParams.dataType()).toBe(
-      'eql_v3.int4_ord',
-    )
+    expect((col as unknown as { getSQLType(): string }).getSQLType()).toBe(
+      'eql_v3.int4_ord',
+    )
   })
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/stack/__tests__/drizzle-v3/column.test.ts` around lines 11 - 18, The
`makeEqlV3Column` test is asserting against Drizzle internals via `(col as
any).config.customTypeParams.dataType()`, so update the test to verify the
emitted SQL domain through the public `getSQLType()` API on the column returned
by `makeEqlV3Column` instead. Keep the test focused on the same
`makeEqlV3Column`/`v3Types.Int4Ord` path, but remove the `any` cast and internal
`config` access so it only depends on documented behavior.

Source: Coding guidelines

packages/stack/src/eql/v3/drizzle/operators.ts (2)

188-200: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Unbounded concurrent encryption calls for large inArray/notInArray inputs.

Each element triggers its own equality()encryptTerm() round trip, and all are fired concurrently via Promise.all with no batching or concurrency cap. A large values array could spike load on the encryption/KMS backend.

Consider chunking or capping concurrency (e.g. via p-limit or manual batching) for large arrays.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/stack/src/eql/v3/drizzle/operators.ts` around lines 188 - 200, The
inArrayOp helper currently fans out one equality() call per value through
Promise.all, which can overwhelm the encryption/KMS path for large arrays.
Update inArrayOp to limit concurrency by batching or capping parallel equality()
work (for example with a small worker pool or chunked awaits) while preserving
the existing negate/or/and behavior and the empty-array true/false handling.

49-56: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider Drizzle's is() guard instead of blind any casts for column internals.

drizzleTableOf/resolveContext reach into (column as any).table / .name. Drizzle-orm exposes is(value, Column) specifically to narrow such values safely without any, which would also guard against non-Column SQLWrapper inputs being misread.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/stack/src/eql/v3/drizzle/operators.ts` around lines 49 - 56, The
`drizzleTableOf` and `resolveContext` helpers are reading `.table` and `.name`
via `any` casts on `SQLWrapper`, which should be replaced with Drizzle’s `is()`
guard for `Column` instead. Update these helpers to first narrow the input with
`is(value, Column)` and only then access column internals, falling back safely
for non-Column wrappers. Keep the changes localized to the `drizzleTableOf` and
`resolveContext` logic so the context resolution remains type-safe without
relying on blind casts.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/superpowers/specs/2026-07-06-eql-v3-drizzle-concrete-types-design.md`:
- Line 113: The fenced example blocks in this spec are unlabeled, which triggers
docs lint and hurts readability. Update both fenced blocks in the affected
sections to include a language tag using the existing markdown fences in this
document, and keep the content otherwise unchanged. Use the surrounding
fenced-example markup in the spec to locate the two bare fences that need the
same labeling fix.

In `@packages/stack/src/eql/v3/drizzle/operators.ts`:
- Line 47: The table cache in operators.ts is keyed by table name, which can
collide for pgSchema() tables that share names across schemas and merge
unrelated entries via the 'unknown' fallback. Update the cache used by the v3
operator logic to key by the table object itself, replacing the current
Map<string, AnyV3Table> with a WeakMap<PgTable, AnyV3Table>, and adjust the
lookup/insert logic in the table caching path so the AnyV3Table metadata stays
attached to the correct table instance.

In `@packages/stack/src/eql/v3/drizzle/schema-extraction.ts`:
- Around line 20-25: In schema-extraction.ts, the column identifier passed into
getEqlV3Column() is using the JS property key instead of the Drizzle column
name, which can rebuild v3 columns under the wrong name in the fallback path.
Update the loop that populates columns to prefer column.name when available and
only fall back to property if no Drizzle name exists, keeping the rest of the
logic in getEqlV3Column() unchanged.

---

Nitpick comments:
In `@packages/stack/__tests__/drizzle-v3/column.test.ts`:
- Around line 11-18: The `makeEqlV3Column` test is asserting against Drizzle
internals via `(col as any).config.customTypeParams.dataType()`, so update the
test to verify the emitted SQL domain through the public `getSQLType()` API on
the column returned by `makeEqlV3Column` instead. Keep the test focused on the
same `makeEqlV3Column`/`v3Types.Int4Ord` path, but remove the `any` cast and
internal `config` access so it only depends on documented behavior.

In `@packages/stack/src/eql/v3/drizzle/operators.ts`:
- Around line 188-200: The inArrayOp helper currently fans out one equality()
call per value through Promise.all, which can overwhelm the encryption/KMS path
for large arrays. Update inArrayOp to limit concurrency by batching or capping
parallel equality() work (for example with a small worker pool or chunked
awaits) while preserving the existing negate/or/and behavior and the empty-array
true/false handling.
- Around line 49-56: The `drizzleTableOf` and `resolveContext` helpers are
reading `.table` and `.name` via `any` casts on `SQLWrapper`, which should be
replaced with Drizzle’s `is()` guard for `Column` instead. Update these helpers
to first narrow the input with `is(value, Column)` and only then access column
internals, falling back safely for non-Column wrappers. Keep the changes
localized to the `drizzleTableOf` and `resolveContext` logic so the context
resolution remains type-safe without relying on blind casts.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c223fed7-0a1c-4838-90db-88844e2a9f9f

📥 Commits

Reviewing files that changed from the base of the PR and between 63fe076 and 863e521.

📒 Files selected for processing (20)
  • .changeset/eql-v3-drizzle.md
  • docs/superpowers/specs/2026-07-06-eql-v3-drizzle-concrete-types-design.md
  • packages/stack/__tests__/drizzle-v3/codec.test.ts
  • packages/stack/__tests__/drizzle-v3/column.test.ts
  • packages/stack/__tests__/drizzle-v3/exports.test.ts
  • packages/stack/__tests__/drizzle-v3/operators-live-pg.test.ts
  • packages/stack/__tests__/drizzle-v3/operators.test.ts
  • packages/stack/__tests__/drizzle-v3/schema-extraction.test.ts
  • packages/stack/__tests__/drizzle-v3/sql-dialect.test.ts
  • packages/stack/__tests__/drizzle-v3/types.test-d.ts
  • packages/stack/__tests__/drizzle-v3/types.test.ts
  • packages/stack/package.json
  • packages/stack/src/eql/v3/drizzle/codec.ts
  • packages/stack/src/eql/v3/drizzle/column.ts
  • packages/stack/src/eql/v3/drizzle/index.ts
  • packages/stack/src/eql/v3/drizzle/operators.ts
  • packages/stack/src/eql/v3/drizzle/schema-extraction.ts
  • packages/stack/src/eql/v3/drizzle/sql-dialect.ts
  • packages/stack/src/eql/v3/drizzle/types.ts
  • packages/stack/tsup.config.ts

Comment thread docs/superpowers/specs/2026-07-06-eql-v3-drizzle-concrete-types-design.md Outdated
Comment thread packages/stack/src/eql/v3/drizzle/operators.ts Outdated
Comment thread packages/stack/src/eql/v3/drizzle/schema-extraction.ts
freshtonic added a commit that referenced this pull request Jul 6, 2026
#565 ships the sibling integration but on the protect-ffi 0.26 /
pre-rename-domain bundle line, while main (this module's base) is 0.27 +
SQL-standard names with term constructors in eql_v3_internal and scalar
encryptQuery unsupported — so its exact SQL and term-operand path do not
transfer. Adopt its dialect shape instead: gating on build().indexes,
equality-via-ORE fallback, encrypted ORDER BY (now in scope as a fragment
builder), whereIn with bounded operand-encryption concurrency, and the
no-silent-fallback error convention. Free-text match pinned to the two-arg
eql_v3.contains form; sequencing no longer blocks on #565.
@tobyhede tobyhede force-pushed the eql-v3-drizzle-concrete-types branch 3 times, most recently from 0a99c6a to c43777e Compare July 7, 2026 03:38
tobyhede added 19 commits July 7, 2026 14:03
Resolves pending review feedback from #541 (merged). Seven findings:

- test: restore the live ORE proof for text order domains. text_ord /
  text_ord_ore carry both hm (unique) and ob (ore), so the single-kind
  classifier ran only the eq_term/hmac_256 proof and silently skipped
  ord_term/ore_block_256 -- a wrong-valued ob would pass the matrix green.
  Run both proofs; build the ob term with queryType:'orderAndRange' since
  equality resolves to hm on these domains.
- schema: extract a shared resolveMatchOpts() merge+clone helper used by
  both the v2 and v3 freeTextSearch builders, giving v2 the clone-on-write
  protection it lacked (it stored caller opts by reference).
- eql/v3: derive EncryptedTextSearchColumn.build()'s unique/ore blocks via
  indexesForCapabilities (override only match) so the emitted index shape
  cannot drift from the shared capability mapping.
- encryption/v3: precompute row reconstructors per schema table at
  typedClient construction and return a DecryptionError failure for unknown
  tables, so table.build()'s duplicate-column throw can no longer escape the
  decrypt Result contract as a promise rejection.
- eql/v3: alias V3DecryptedModel to V3ModelInput (character-identical) to
  stop the input/output model shapes silently drifting.
- ci: add src/schema/match-defaults.ts to the fta-v3 path scope -- it shapes
  every emitted v3 match block but sat outside the gate.
- ci: re-baseline FTA --score-cap 72 -> 69 after the eql/v3 file split and
  refresh the spec doc's stale 71.08 baseline claim.
…e casts

Follow-ups from PR review (non-blocking):

- catalog: give v3 timestamp domains a non-midnight sample set (TS_S) so the
  live matrix actually detects a cast_as 'timestamp' -> 'date' truncation
  regression; midnight-only DATE_S survived truncation silently. Add a runnable
  matrix.test.ts guard so the detection power can't be reverted unnoticed.
- schema: document the 'timestamp' cast_as on both dataType() JSDoc @param
  lists (and restore the dropped 'text' on EncryptedColumn), noting 'timestamp'
  preserves time-of-day where 'date' truncates.
- columns/encryption: extract DATE_LIKE_CASTS as the single source of truth for
  the date-like set, shared by the type-level PlaintextFromKind and the runtime
  rowReconstructor (previously hand-synced in two places).
tobyhede added 5 commits July 7, 2026 14:09
…/null)

Close assertion-strength gaps in the EQL v3 live suites where a real
regression could pass green:

- between/notBetween: add narrow single-point range cases that prove
  EXCLUSION (the spanning cases only ever proved inclusion).
- ordering oracle: compare text bytewise, not via localeCompare, to match
  ORE byte order.
- inArray/notInArray: exercise the >4-value concurrency pool.
- matrix-boundary-live-pg: drive catalog boundary samples (INT4/smallint
  bounds, 1e15) through real eql_v3.<type> columns, not just the FFI.
- operators-lock-context-live-pg: query lock-context-bound rows through the
  Drizzle operator path (ops.eq + lockContext/audit); soft-skips on USER_JWT.
- operators-null-live-pg: NULL persistence across storage/eq/ord/match tiers.
- live-gate-required guard + CI: fail loudly (REQUIRE_LIVE/REQUIRE_LIVE_PG)
  instead of silently skipping the live matrices when creds are absent.
@tobyhede tobyhede force-pushed the eql-v3-drizzle-concrete-types branch from c43777e to dbe1ec0 Compare July 7, 2026 04:28
@tobyhede tobyhede changed the base branch from feat/eql-v3-text-search-schema to james/cip-3291-bigint-stack July 7, 2026 04:29
@tobyhede tobyhede requested a review from freshtonic July 7, 2026 04:33
Comment on lines +235 to +240
// Pure-ORE domains (numeric/date) answer equality via ORE, so
// `queryType:'equality'` resolves to the `ob` term — the exact
// equality-via-ORE path Part A fixed. Text order domains ALSO carry `hm`,
// where `equality` resolves to HMAC by the shared `unique > … > ore`
// priority; force `orderAndRange` there so the term still carries the `ob`
// this proof extracts with `ore_block_256`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does Drizzle care if its ORE or HMAC under the hood? I don't understand.

Comment on lines +89 to +105
* Which proofs a domain's configured indexes call for. Unlike a single-kind
* classifier these lists are NOT mutually exclusive — a domain runs EVERY proof
* its indexes support:
*
* - eq (`hm`): every domain carrying a `unique` index → `eq_term`/`hmac_256`.
* - ord (`ob`): every domain carrying an `ore` index → `ord_term`/`ore_block_256`.
* - match (`bf`): every domain carrying a `match` index → `match_term`/`bloom_filter`.
* - storage: a domain with NO index — only the ciphertext round-trip proof.
*
* Text order domains (`text_ord`/`text_ord_ore`) carry BOTH `unique` and `ore`,
* so they appear in `eqDomains` AND `ordDomains` and run both proofs — a
* wrong-valued `ob` would otherwise slip through an eq-only check (text equality
* is HMAC-based, so `queryType:'equality'` on them resolves to `hm`, never `ob`;
* their ord term is built with `queryType:'orderAndRange'` below). `text_search`
* also carries all three indexes but is deliberately exercised by the match
* proof ALONE — its distinguishing, richest capability and the one canonical
* example per tier — so it is excluded from the eq/ord lists via `!match`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO checking for the presence of specific SEM is brittle and breaks the abstraction. The tests should be based on properties - (e.g. do values equal each other or not)

Comment on lines +15 to +20
it('sets dataType() to the concrete eql_v3 domain', () => {
const col = makeEqlV3Column(v3Types.IntegerOrd('age'))
const table = pgTable('users', { age: col })

expect(table.age.getSQLType()).toBe('eql_v3.integer_ord')
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any way to set the SEM specifier? i.e. distinguish between ORE and OPE variants? Can be a follow-up if not.

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test coverage review — PR #565 (EQL v3 Drizzle integration)

This PR lands a large, well-covered v3 Drizzle integration (codec, columns, operators, schema extraction, dialect) plus a typedClient reconstructor refactor. Happy paths and most gating errors are thoroughly tested via the V3_MATRIX-driven suites. The gaps below are all new negative / fallback branches introduced by this diff that no test currently exercises — none are blocking.

Verdict: COMMENT — 5 coverage gaps flagged inline.

Out of scope (noted, not posted inline)

  • v3FromDriver (src/eql/v3/drizzle/codec.ts) calls JSON.parse on the driver string with no guard, so a malformed/corrupt jsonb value throws rather than surfacing a typed error. It's a thin wrapper and low priority, but there is no malformed-input case in codec.test.ts — worth a one-line test if you want the throw pinned.

No overflow beyond the cap.

decrypt: (encrypted) => client.decrypt(encrypted),
decryptModel: async (input, table, lockContext) => {
const reconstruct = reconstructors.get(table)
if (!reconstruct) return unknownTableFailure as never

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gap: The new reconstructors.get(table) miss branch — returning unknownTableFailure when decryptModel/bulkDecryptModels is handed a table the client was not initialized with — is untested. typed-client-v3.test.ts only ever passes the same table given to typedClient(...), so this negative arm (and its mirror at bulkDecryptModels, line 226) never runs. This is the classic missing-negative-case pattern: the happy reconstruct path is covered, the wrong-table path is not.

// __tests__/typed-client-v3.test.ts — mirrors the existing fakeClient pattern
it('fails with a DecryptionError when given a table it was not initialized with', async () => {
  const other = encryptedTable('other', { x: types.Text('x') })
  const client = typedClient(fakeClient({ when: null }), table) // NOT `other`

  const single = await client.decryptModel({}, other as never)
  expect(single.failure?.type).toBe('DecryptionError')
  expect(single.failure?.message).toMatch(/not initialized with/i)

  const bulk = await client.bulkDecryptModels([{}], other as never)
  expect(bulk.failure?.type).toBe('DecryptionError')
})

Expected: both calls resolve to a { failure } Result (never throw / reject), pinning the contract the comment above the branch describes.

const tableName = (table as any)[Symbol.for('drizzle:Name')] as
| string
| undefined
if (!tableName) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gap: The missing-table-name throw is untested. schema-extraction.test.ts covers the sibling throw (/no encrypted v3/i) but not this one — a lopsided negative case (one throw arm tested, the other not). Every current test builds the table via pgTable(...), which always sets the drizzle:Name symbol, so the !tableName arm never runs.

// __tests__/drizzle-v3/schema-extraction.test.ts
it('throws when the drizzle table has no name symbol', () => {
  const table = { secret: { getSQLType: () => 'eql_v3.text_eq', name: 'secret' } }
  expect(() => extractEncryptionSchemaV3(table as never)).toThrow(
    /read table name/i,
  )
})

Expected: throws "Unable to read table name from Drizzle table." (the check must fire before the no-columns check).

: undefined
if (typeof sqlType === 'string') return sqlType
const dt = columnAny.dataType
const domain = typeof dt === 'function' ? dt() : (columnAny.sqlName ?? dt)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gap: getSqlType's fallback branch — recovering the domain from dataType() or sqlName when getSQLType() is absent — is never exercised. column.test.ts and schema-extraction.test.ts only feed columns that expose getSQLType() (line 74 path), so the dataType/sqlName recovery at lines 78–79 is untested even though getEqlV3Column/isEqlV3Column are public API.

// __tests__/drizzle-v3/column.test.ts
it('recovers a v3 column exposing only dataType()/sqlName (no getSQLType)', () => {
  expect(isEqlV3Column({ dataType: () => 'eql_v3.text_eq' })).toBe(true)
  expect(isEqlV3Column({ sqlName: 'eql_v3.integer_ord' })).toBe(true)
  expect(
    getEqlV3Column('nickname', { dataType: () => 'eql_v3.text_eq' })?.getEqlType(),
  ).toBe('eql_v3.text_eq')
})

Expected: both shapes are recognised and rebuilt to their concrete domain; a regression that drops the fallback would return false/undefined.

}
const conditions = await mapWithConcurrency(
values,
MAX_IN_ARRAY_CONCURRENCY,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gap: inArray/notInArray are only tested with 2 values, so the mapWithConcurrency worker-pool loop (lines 73–80) that engages when values.length > MAX_IN_ARRAY_CONCURRENCY (4) never runs — a single worker handling ≤2 items never iterates the while (true) queue. A boundary case (>4 values) is needed to prove every value produces exactly one term with none dropped or duplicated.

// __tests__/drizzle-v3/operators.test.ts
it('inArray fans out every value past the concurrency limit', async () => {
  const { ops, encrypt, render } = setup()
  const values = [1, 2, 3, 4, 5, 6, 7] // > MAX_IN_ARRAY_CONCURRENCY (4)
  const q = render(await ops.inArray(users.age, values))
  expect((q.sql.match(/eql_v3\.eq/g) ?? []).length).toBe(values.length)
  expect(encrypt).toHaveBeenCalledTimes(values.length)
})

Expected: 7 eql_v3.eq terms and 7 encrypt calls — an off-by-one in the worker's index handling would drop or repeat a term.

}
}

function applyOperationOptions(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gap: applyOperationOptions merges per-call opts over constructor defaults (opts?.lockContext ?? defaults.lockContext, same for audit), but only the default-forwarding arm is tested (ops.eq(users.nickname, 'ada') with no opts). The override arm — a per-call lockContext/audit winning over a constructor default — is unexercised.

// __tests__/drizzle-v3/operators.test.ts (setup() defaults: lockContext user-123, audit actor:test)
it('per-call lockContext / audit override the constructor defaults', async () => {
  const { ops, encrypt } = setup()
  const callLock = { identityClaim: 'call-override' }
  const callAudit = { metadata: { actor: 'call' } }
  await ops.eq(users.nickname, 'ada', { lockContext: callLock, audit: callAudit })
  const op = encrypt.mock.results[0]?.value
  expect(op.withLockContext).toHaveBeenCalledWith(callLock)
  expect(op.audit).toHaveBeenCalledWith(callAudit)
})

Expected: the call-site values are forwarded, not the constructor defaults; a swap of the ?? operands would fail this.

Comment on lines +73 to +76
function unwrap<T>(result: { data?: T; failure?: { message: string } }): T {
if (result.failure) throw new Error(result.failure.message)
return result.data as T
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side note: @byteslice/result 0.5.0 includes an unwrap function.

Comment on lines +142 to +148
const skipUnlessJwt = (): boolean => {
if (!userJwt) {
console.log('Skipping lock-context operator test - no USER_JWT provided')
return true
}
return false
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd probably miss this is no JWT set. In fact, I'm dubious that there is one set now.

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. A couple of items that should be addressed before merge. Also, the docs are almost non-existent. Should there be a section in the README and some typedoc?

@coderdan

coderdan commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Code review — EQL v3 Drizzle integration + bigint domains

Reviewed at extra-high effort (multi-angle finder pass → verify → sweep) against dbe1ec0. The change is well-tested (deterministic + live-pg matrix), which refuted several plausible bugs (see bottom). 12 findings survived, correctness first.

Correctness / robustness

1. bigint domains are type-blessed but have no runtime guard → opaque low-level errorpackages/stack/src/eql/v3/columns.ts:166 (also types.ts:128, Plaintext in types.ts:96)
types.Bigint/BigintEq/… are exported publicly and PlaintextForColumn resolves them to bigint, but the only value validator (assertValidNumericValue, helpers/validation.ts:43) checks typeof === 'number', so a bigint passes every guard to protect-ffi 0.27, which cannot marshal a JS BigInt across the Neon boundary.
Failure: client.encrypt(123n, { table, column }) on a types.BigintEq column type-checks and passes unit CI (FFI mocked), then fails at runtime with a cryptic Neon/serialization error instead of a clear "bigint not yet supported" SDK error. On the drizzle operator path the throw isn't wrapped in EncryptionOperatorError. The gate is prose comments only — an explicit runtime guard on cast_as === 'bigint' (or not exporting the factories until the pin bumps) would keep the type/runtime contract from diverging silently.

2. decryptModel/bulkDecryptModels key the reconstructor by table object identity → a valid, correctly-typed call can return a failurepackages/stack/src/encryption/v3.ts:217 (Map built at :190)
EncryptedTable is structurally typed (tableName: string, brand is only the phantom _columnType), so two separately-constructed but identical tables are the same type yet different objects. The old code built rowReconstructor(table) from whatever table was passed; the new code does reconstructors.get(table) and returns unknownTableFailure on a miss.
Failure: const c = await EncryptionV3({schemas:[usersA]}) then c.decryptModel(row, usersB) where usersB = encryptedTable('users', {…same…}) (a re-import, HMR reload, or table built in another module). Compiles (structural match), but at runtime returns DecryptionError: "…a table this client was not initialized with…" instead of decrypting — a regression vs. prior behavior. Keying by tableName would match the semantic identity the FFI/build() already use.

3. Drizzle customType declares plaintext data types but toDriver only JSON-serializes (no encryption)packages/stack/src/eql/v3/drizzle/codec.ts:10 (type declared in column.ts:88)
The column is typed data: PlaintextForColumn<C> (Date/bigint/number/string) with toDriver(value) = JSON.stringify(value). The intended flow inserts pre-encrypted envelopes (live tests cast through as never), but nothing in the types enforces that.
Failure: For types.Bigint, the type-valid db.insert(t).values({ n: 1n }) calls JSON.stringify(1n) → throws TypeError: Do not know how to serialize a BigInt synchronously. For date/string, the same type-valid plaintext insert bypasses encryption and JSON-stringifies plaintext into the eql_v3 column (a plaintext-exposure footgun — some domain CHECK constraints may reject the non-envelope shape, but the type actively invites the write). Consider typing data as the encrypted envelope, or documenting/guarding the insert path.

4. Drizzle types mirror can't reach EncryptedTextSearchColumn.freeTextSearch(opts) → tokenization silently locked to defaultspackages/stack/src/eql/v3/drizzle/types.ts:11
Each factory is (name) => makeEqlV3Column(factory(name)), returning a Drizzle column — there's no way to call the chainable .freeTextSearch(opts) tuner. The canonical flow (extractEncryptionSchemaV3(table)EncryptionV3({schemas}), per the live tests) bakes the recovered builder's build().indexes.match into the real encrypt config.
Failure: A drizzle user who needs custom tokenizer/k/m (short-substring or CJK free-text) has no drizzle-path way to set it; every text_search column gets defaultMatchOpts() ngram settings, and contains() behaves against the default tokenization with no error or warning. The JSDoc claims to "mirror" the v3 types namespace but omits this capability.

5. bigint decrypt is unsound once the encrypt gate liftspackages/stack/src/eql/v3/columns.ts:681 (reconstructor at encryption/v3.ts:147)
PlaintextFromKind types bigint columns as bigint, and rowReconstructor intentionally passes them through (only DATE_LIKE_CASTS are rebuilt). The FFI cannot return a JS bigint, so a decrypted bigint column would hold a number/string while typed bigint. Currently masked because encrypt throws first (finding 1).
Failure: After the FFI pin bumps, decryptModel returns the FFI's serialized form for a bigint column but the type asserts bigint; downstream value + 1n throws Cannot mix BigInt and other types. Worth a reconstruction step (or a test) landing alongside the pin bump.

Efficiency

6. inArray/notInArray re-resolve the column context and call build() once per array elementpackages/stack/src/eql/v3/drizzle/operators.ts:269
inArrayOp maps every value through equality(...), and each equality runs resolveContextgetEqlV3Column + builder.build().indexes (:132). The context is identical for all N values.
Failure: inArray(col, [...1000 ids]) triggers 1000 getEqlV3Column calls and 1000 build() invocations (for a text_search column each build() deep-clones the match block). Resolve context once, then reuse it per operand (keep the concurrency-4 encryption). A WeakMap<builder, indexes> memo on build() would also fix per-operator rebuilds.

7. between/notBetween encrypt min then max sequentiallypackages/stack/src/eql/v3/drizzle/operators.ts:237
const encMin = await encryptOperand(...) then const encMax = await encryptOperand(...) — the two operands are independent.
Failure: Every range predicate pays 2× encrypt latency (round-trips to the crypto backend). const [encMin, encMax] = await Promise.all([...]) halves it.

Cleanup / conventions

8. EncryptionOperatorError is redeclared, forking a class that already exists and is exported by v2packages/stack/src/eql/v3/drizzle/operators.ts:29
v2 (src/drizzle/operators.ts:82, re-exported from src/drizzle/index.ts) defines an identical-shaped EncryptionOperatorError. Two distinct runtime class identities: a consumer's catch (e) { if (e instanceof EncryptionOperatorError) } against one import silently fails to match the other, and the shared context shape must be edited in two places. Lift to one shared module.

9. A Linear issue ID appears in a source comment (and in the changeset)packages/stack/src/eql/v3/columns.ts:165 (also .changeset/eql-v3-bigint-domains.md lines 6 and 10)
Internal issue trackers shouldn't leak into this public repo's source or its published CHANGELOG. Drop the IDs or reword. (IDs omitted here for the same reason.)

10. Drizzle table-name extraction via Symbol.for('drizzle:Name') is hand-inlined across ~5 sitespackages/stack/src/eql/v3/drizzle/schema-extraction.ts:11 (again at operators.ts:115; v2 has getDrizzleTableName)
The Drizzle-internal drizzle:Name contract is hard-coded in multiple places with no single source of truth; a Drizzle upgrade that changes it breaks all of them independently. Share one getDrizzleTableName(table) helper.

11. EQL_V3_DOMAINS and buildersByDomain run the same probe iteration twice and can driftpackages/stack/src/eql/v3/drizzle/column.ts:10
Both do Object.values(v3Types).map(f => f('__probe__').getEqlType()), constructing throwaway probe columns per domain. EQL_V3_DOMAINS is exactly buildersByDomain's key set — derive it: new Set(buildersByDomain.keys()).

12. Dead EQL_V3_COLUMN_LEGACY_PARAM (_eqlv3Column) redundancypackages/stack/src/eql/v3/drizzle/column.ts:30
_eqlv3Column is never read outside this file; writeBuilder always sets it together with the Symbol, and readBuilder returns the Symbol first via ??, so the legacy branch is unreachable (the Symbol carrier is confirmed to survive pgTable via config.customTypeParams). The const, type member, extra write, and ?? branch can be deleted; "legacy" also implies a nonexistent external contract.


Cleared during verification (not bugs): eql_v3.eq/neq on ORE-only *_ord columns (live-pg matrix proves exact/complement selection); applyOperationOptions discarding withLock.audit()'s return (audit() mutates this and returns this); resolveMatchOpts vs. the old inline merge (behavior-identical, now safer — it clones for v2 too); EncryptedTextSearchColumn.build() super.build()+spread (byte-identical index block); V3DecryptedModel = V3ModelInput alias (textually identical to the prior mapped type); DATE_LIKE_CASTS filter (same set as === 'date' || === 'timestamp'); new Date(value) reconstruction (correct instant for the UTC samples; unchanged by this PR); mapWithConcurrency (no race, no unhandled-rejection leak); stash recovery before/after pgTable; and the batch-encrypt-query.ts change (type-only, runtime unchanged).

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants