Skip to content
This repository was archived by the owner on Aug 18, 2026. It is now read-only.

perf(postgresql): add INSERT batch mode: values & unnest - #113

Closed
tin-thebc wants to merge 7 commits into
streamingfast:developfrom
RoverInc:loc/feat/allow-insert-unnest-with-buffer-for-postgres-dialect
Closed

tin-thebc wants to merge 7 commits into
streamingfast:developfrom
RoverInc:loc/feat/allow-insert-unnest-with-buffer-for-postgres-dialect

Conversation

@tin-thebc

Copy link
Copy Markdown
  • Load nullability and defaults from pg_catalog
  • Extend ColumnInfo with nullable, hasDefault, defaultExpr
  • Inline defaults in UNNEST projections (scalars and arrays)
  • Refuse UNNEST when NOT NULL without default is absent
  • Improve batching logs; set default batch size to 50
  • Update PRD task 11c with implementation details

vinhloc30796 and others added 7 commits August 30, 2025 17:33
- Load nullability and defaults from pg_catalog
- Extend ColumnInfo with nullable, hasDefault, defaultExpr
- Inline defaults in UNNEST projections (scalars and arrays)
- Refuse UNNEST when NOT NULL without default is absent
- Improve batching logs; set default batch size to 50
- Update PRD task 11c with implementation details
@maoueh

maoueh commented Sep 29, 2025

Copy link
Copy Markdown
Contributor

@tin-thebc This is a follow up of #105 correct?

@tin-thebc

Copy link
Copy Markdown
Author

@maoueh yes that's correct, can you help me to close the duplicate one?

@GabrielCartier

GabrielCartier commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Hi @tin-thebc — thanks for this, and sorry it sat so long.

This repository is being deprecated (see #133). The SQL sink now lives in the main substreams CLI as substreams sink postgres / substreams sink clickhouse, shipped in substreams v1.20.2. All future work happens there, so this PR can't land here.

The code moved wholesale, so the port is mostly a path rewrite:

here substreams repo
db_changes/db/ sink/sql/db_changes/db/
db_changes/sinker/ sink/sql/db_changes/sinker/
cmd/substreams-sink-sql/run.go cmd/substreams/sink_postgres.go

The batch/unnest work isn't in the CLI yet, and we'd still like it — would you be up for reopening it against streamingfast/substreams (branch off develop)? The branch conflicts here anyway, so rebasing onto the new tree is roughly the same effort. If you'd rather not redo it, say so and we'll port it ourselves with credit to you.

Closing this one out. (#105, the earlier duplicate, is already closed.)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants