Migrate docsite embed/search to Postgres + pgvector (WIP)

[IZO-Ong]

Short Description

Migrates embed_docsite/search_docsite from Pinecone to Postgres + pgvector (0.8.1 extension), keeping the legacy Pinecone path available. Both indexing and searching support either backend, selected independently — the write target and read backend are separate settings. WIP — comments are welcome!

Addresses #305

Implementation Details

Using both backends

Indexing — embed_docsite's payload takes a target param:

{ "target": "postgres" }   // or "pinecone" (default)

Searching — search_docsite reads DOCSITE_SEARCH_BACKEND (env, default pinecone), overridable per-request via a backend payload field:

{ "query": "...", "backend": "postgres" }

job_chat's retrieve_docs.py and the search_documentation tool both resolve the backend the same way, through a shared resolve_backend() in search_docsite.py.

No new required env vars for existing setups, Two new optional vars in .env.example:

• DOCSITE_SEARCH_BACKEND — pinecone | postgres (Defaults to pinecone if not provided)
• POSTGRES_TEST_URL — separate DB for the integration suite only (see Tests below)

Search strategies

Postgres adds three strategies: semantic (pgvector cosine), keyword (Postgres full-text search), and hybrid (Reciprocal Rank Fusion of both).

Production consumers (job_chat, search_documentation) use strategy='semantic' on both backends, at the same 0.8 cosine threshold Pinecone already used. hybrid/keyword are available via the search_docsite service directly and via run_eval, but aren't wired into any production consumer yet.

Migrations

Schema lives in services/migrations/20260728000000_docsite_batches_and_chunks.sql (timestamped per @stuartc's request for predictable ordering) — docsite_batches (versioned batch lifecycle: building -> complete/failed) and docsite_chunks (chunk text + vector(1536) embedding + generated tsvector for FTS), plus the vector extension and four indexes.

Applied by services/db_migrations.py, run explicitly from embed_docsite's Postgres write path. Migrations are idempotent (tracked in _migrations_docs) and lexically ordered. No manual step needed, as the first embed_docsite run with target=postgres sets up the schema.

Running run_eval

Compares both backends (strategy='semantic' on each) over a golden query set: recall@5, p50/p95 latency, and for the Postgres-semantic-vs-Pinecone pairing specifically, per-query doc-title agreement.

Setup:

docker run -d --name apollo-postgres \
  -e POSTGRES_USER=apollo -e POSTGRES_PASSWORD=apollo -e POSTGRES_DB=apollo_dev \
  -p 5432:5432 pgvector/pgvector:pg16

Set OPENAI_API_KEY, PINECONE_API_KEY, POSTGRES_URL in services/.env (this is the file that's actually loaded — see note below).

Populate Pinecone and all three Postgres chunk sizes.

bun py embed_docsite --input tmp/payload_pinecone.json          # {"target": "pinecone"}
bun py embed_docsite --input tmp/payload_postgres_1000.json     # {"target": "postgres", "chunk_target_length": 1000, "chunk_min_length": 700, "keep_batches": 3}
bun py embed_docsite --input tmp/payload_postgres_1800.json     # {"target": "postgres", "chunk_target_length": 1800, "chunk_min_length": 1260, "keep_batches": 3}
bun py embed_docsite --input tmp/payload_postgres_2500.json     # {"target": "postgres", "chunk_target_length": 2500, "chunk_min_length": 1750, "keep_batches": 3}

Run the eval (must run from services/, the Python path root):

cd services
poetry run python -m search_docsite.tests.eval.run_eval

Results

30 golden queries now (15 conceptual, 15 keyword), and the comparison now covers hybrid vs semantic across three chunk sizes (1000/1800/2500 chars). Labels for all 30 queries were derived from the general_docs corpus via full-text search on each query's key terms.

configuration	conceptual	keyword	overall	p50	p95
Pinecone semantic	1.00	1.00	1.00	1.055s	1.965s
Postgres semantic (1000)	1.00	1.00	1.00	0.457s	1.184s
Postgres hybrid (1000)	0.93	1.00	0.97	0.462s	0.605s
Postgres semantic (1800)	0.93	1.00	0.97	0.566s	0.914s
Postgres hybrid (1800)	0.93	1.00	0.97	0.483s	0.713s
Postgres semantic (2500)	1.00	1.00	1.00	0.464s	1.157s
Postgres hybrid (2500)	1.00	1.00	1.00	0.549s	0.918s

Jaccard similarity between Pinecone and Postgres semantic (1000) is 0.970.

Full run_eval output

configuration                 conceptual   keyword   overall      p50      p95
Pinecone semantic                   1.00      1.00      1.00   1.055s   1.965s
Postgres semantic (1000)            1.00      1.00      1.00   0.457s   1.184s
Postgres hybrid (1000)              0.93      1.00      0.97   0.462s   0.605s
Postgres semantic (1800)            0.93      1.00      0.97   0.566s   0.914s
Postgres hybrid (1800)              0.93      1.00      0.97   0.483s   0.713s
Postgres semantic (2500)            1.00      1.00      1.00   0.464s   1.157s
Postgres hybrid (2500)              1.00      1.00      1.00   0.549s   0.918s

Scored 30 queries, skipped 0

Migration fidelity (Pinecone semantic vs Postgres semantic (1000)):
  mean doc-title Jaccard=0.970 across 30 queries
  1.00  overlap=4/4  how do I configure a webhook trigger
  1.00  overlap=3/3  how do I set up a cron trigger
  0.75  overlap=3/4  what is a run in OpenFn
  1.00  overlap=2/2  how do I use collections to store state between runs
  1.00  overlap=4/4  how do I configure a credential for an adaptor
  1.00  overlap=4/4  what is the difference between a job and a workflow
  1.00  overlap=1/1  how do I deploy a project using the CLI
  1.00  overlap=4/4  how do I debug a failed run
  0.75  overlap=3/4  how do I write a data transform function
  1.00  overlap=5/5  how do I control who can access a project
  1.00  overlap=1/1  how do I set up a sandbox environment
  1.00  overlap=3/3  how long is my run data kept
  1.00  overlap=4/4  how do I get notified when a workflow fails
  1.00  overlap=3/3  how do I version control my project with GitHub
  1.00  overlap=3/3  what security measures does OpenFn have
  1.00  overlap=1/1  state.cursor
  1.00  overlap=1/1  --force flag on project push
  0.60  overlap=3/5  webhook auth method
  1.00  overlap=2/2  workflow snapshots
  1.00  overlap=2/2  lazy state operator
  1.00  overlap=2/2  dataValue
  1.00  overlap=3/3  each operation
  1.00  overlap=3/3  rerun a workflow
  1.00  overlap=4/4  collections CLI commands
  1.00  overlap=4/4  activity history
  1.00  overlap=3/3  project.yaml
  1.00  overlap=4/4  step editor
  1.00  overlap=2/2  git branch
  1.00  overlap=2/2  sandbox
  1.00  overlap=3/3  security compliance

Discussions

Hybrid ties semantic on keyword recall everywhere (both hit 1.00 at every chunk size) rather than beating it, and semantic is at least as good as hybrid on conceptual recall at every chunk size tested. Hybrid's p95 is consistently the best in its chunk-size group, but on recall alone this data doesn't support switching the production default.

2500 chars is the only size where Postgres matches Pinecone's 1.00 conceptual recall for both strategies; 1000 and 1800 both dip to 0.93 on hybrid (1800 dips on semantic too).

Tests

Unit — 113 passing, fully mocked, no live services required:

• embed_docsite: test_docsite_processor.py, test_docsite_indexer.py, test_embed_docsite.py, test_db_migrations.py
• search_docsite: test_docsite_search.py (new Postgres backend), test_pinecone_legacy_search.py (legacy, preserved), test_search_docsite_main.py, test_run_eval.py (now covers per-category recall aggregation and chunk-size batch resolution too)
• Plus updated consumer tests in job_chat and search_documentation for the new resolve_backend() delegation

Integration — 6 passing, against a real ephemeral Postgres + pgvector:

• services/embed_docsite/tests/integration/test_postgres_docsite_roundtrip.py — fresh-DB migrate→index→promote, search across all three strategies, reindex prunes the previous batch, reader gets a clear 503 without a schema

docker run -d --name apollo-pgvector-test -e POSTGRES_PASSWORD=postgres -p 5433:5432 pgvector/pgvector:pg16
export POSTGRES_TEST_URL=postgresql://postgres:postgres@127.0.0.1:5433/postgres
poetry run pytest services/embed_docsite/tests/integration -q

Caution: Do not use POSTGRES_URL for POSTGRES_TEST_URL — the fixtures DROP the docsite tables and the vector extension between tests, so pointing this at a database would destroy its data.

Not done by this PR

• Wiring hybrid into job_chat's general-docs retrieval and the planner's search tool.
• Deciding what replaces the similarity threshold at the call sites
• Adaptor-docs search

AI Usage

Please disclose whether you've used AI in this work (it's cool, we just want to
know!):

• Yes, I have used AI
• No, I have not used AI

You can read more details in our
Responsible AI Policy

[hanna-paasivirta]

Amazing, thanks so much Isaac! I will review this on Monday.

[hanna-paasivirta]

Hi @IZO-Ong this looks brilliant, thank you so much!

Great to see an eval set in here. For the final version it would be nice to expand it to ~30 queries, including some keyword-heavy ones (exact function or term lookups where hybrid should shine on), and add hybrid vs semantic on Postgres to the comparison so we have evidence for switching strategies later.

Just a clarification on scope from here onward since the original issue was written before some things changed: the prompt.py item is already done (we replaced the adaptor TypeScript definitions with a function list) and adaptor-docs search is now being explored separately. So please treat your scope as Postgres backend + general-docs search only.

The hybrid wiring into job_chat's general_docs retrieval and the planner's search tool (the two call sites you already touched) can be a follow-up PR, gated on the eval showing parity or better. One thing to keep in mind: both call sites currently filter results with a similarity threshold, which hybrid intentionally doesn't support, so part of that follow-up is deciding what replaces it (top_k only, an RRF floor, etc.). Don't invest in optimising search methods beyond that; if hybrid needs heavy tuning to beat semantic, flag it rather than iterating. Adaptor docs will land in the same Postgres store later (via your migrations system), but how they're searched is out of scope here.

[hanna-paasivirta]

Could we rename this to a timestamped filename to stick to Stuart's request in #305

[IZO-Ong]

Is the convention for the migration filename being YYYYMMDDHHMMSS ok to make the ordering lexical? So for instance the file name might be 20260728000000

[IZO-Ong]

Hi all, I've updated based on the feedback above:

• Migration renamed to a timestamped filename.
• Chunk size tested at three points (1000/1800/2500 chars, originally only 1000 char)
• Hybrid vs semantic now compared on Postgres across all three chunk sizes.
• Query set expanded 10 to 30 (15 conceptual questions, 15 keyword)

Full results table + per query breakdown in the Results section above.