Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ jobs:
fail-fast: true
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
timeout-minutes: 30
timeout-minutes: 45
steps:
- name: Check out repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down
17 changes: 17 additions & 0 deletions docs/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,16 @@ v0.63.0 - Transactions, table fixtures, SQL fragments, storage, and kwargs param

**Added:**

* Native AlloyDB and PostgreSQL BM25 full-text search support via the ``pg_textsearch`` extension.
Includes the ``PGTextSearch`` dialect registered under ``sqlglot.dialects``, custom AST operator
support for the BM25 relevance ranking operator (``<@>``), automatic extension detection, and
``enable_pg_textsearch`` configuration across all PostgreSQL adapters (AsyncPG, Psycopg, ADBC, and PsqlPy).
* Public :func:`~sqlspec.core.config_runtime.is_postgres_extension_active` helper in ``sqlspec.core.config_runtime``
and adapter modules, with ``active_extensions`` capability tracking on runtime driver features.
* Exposed ``pg_textsearch_available`` property across ``AsyncpgConfig``, ``PsycopgSyncConfig``,
``PsycopgAsyncConfig``, ``AdbcConfig``, and ``PsqlpyConfig``.
(`#782 <https://github.com/litestar-org/sqlspec/pull/782>`_)

* Public table-queue primitives extracted to :mod:`sqlspec.extensions.events.primitives`
and exported from :mod:`sqlspec.extensions.events`: :func:`~sqlspec.extensions.events.lock_clause`,
:func:`~sqlspec.extensions.events.row_limit_clause`,
Expand Down Expand Up @@ -143,6 +153,13 @@ v0.63.0 - Transactions, table fixtures, SQL fragments, storage, and kwargs param

**Changed:**

* Decoupled the ``ParadeDB`` dialect so it inherits directly from ``Postgres`` rather than ``PGVector``,
allowing clean independent combinations of vector search and BM25 extensions.
* Standardized PostgreSQL extension detection across all adapters on a single first-connection probe
via :func:`~sqlspec.core.config_runtime.build_postgres_extension_probe_names`, removing ad-hoc ADK
probe branches.
(`#782 <https://github.com/litestar-org/sqlspec/pull/782>`_)

* Driver execution methods (:meth:`~sqlspec.driver.SyncDriverAdapterBase.execute`,
:meth:`~sqlspec.driver.SyncDriverAdapterBase.select`, etc.) enforce keyword argument parameter passing
(``execute(sql, a=1, b=2)`` or ``execute(sql, **params)``). Passing positional dictionary literals
Expand Down
12 changes: 7 additions & 5 deletions docs/reference/adapters/adbc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Supported Backends
* - PostgreSQL
- ``postgres``, ``postgresql``, ``pg``
- ``adbc_driver_postgresql``
- Numeric parameters; optional pgvector and ParadeDB dialect detection.
- Numeric parameters; optional pgvector, pg_textsearch, and ParadeDB dialect detection.
* - SQLite
- ``sqlite``, ``sqlite3``
- ``adbc_driver_sqlite``
Expand Down Expand Up @@ -212,13 +212,15 @@ first connection and upgrades the SQL dialect accordingly:

- **pgvector** — If the ``vector`` extension is installed, switches to the ``pgvector``
dialect which supports distance operators (``<->``, ``<=>``, ``<#>``, ``<+>``, ``<~>``, ``<%>``).
- **ParadeDB** — If the ``pg_search`` extension is installed (alongside ``vector``),
switches to the ``paradedb`` dialect which adds BM25 search operators (``@@@``, ``&&&``,
``|||``, ``===``) on top of pgvector operators.
- **pg_textsearch** — If the ``pg_textsearch`` extension is installed, switches to the
``pg_textsearch`` dialect which supports BM25 score ranking (``<@>``).
- **ParadeDB** — If the ``pg_search`` extension is installed, switches to the ``paradedb``
dialect which adds BM25 search operators (``@@@``, ``&&&``, ``|||``, ``===``).

Detection is controlled by two driver feature flags:
Detection is controlled by driver feature flags:

- ``enable_pgvector`` — Defaults to ``True`` when the ``pgvector`` Python package is installed.
- ``enable_pg_textsearch`` — Defaults to ``True``.
- ``enable_paradedb`` — Defaults to ``True``.

Detection runs once per config instance and caches the result. Non-PostgreSQL backends
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/adapters/asyncpg.rst
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Driver
Extension Dialects
==================

AsyncPG supports the :doc:`pgvector and ParadeDB dialects <../dialects>` for vector
AsyncPG supports the :doc:`pgvector, pg_textsearch, and ParadeDB dialects <../dialects>` for vector
similarity search and full-text search operators. See the :doc:`Dialects <../dialects>`
reference for operator details.

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/adapters/psqlpy.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Driver Features
Extension Dialects
==================

PsqlPy supports the :doc:`pgvector and ParadeDB dialects <../dialects>` for vector
PsqlPy supports the :doc:`pgvector, pg_textsearch, and ParadeDB dialects <../dialects>` for vector
similarity search and full-text search operators. See the :doc:`Dialects <../dialects>`
reference for operator details.

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/adapters/psycopg.rst
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Async Driver
Extension Dialects
==================

Psycopg supports the :doc:`pgvector and ParadeDB dialects <../dialects>` for vector
Psycopg supports the :doc:`pgvector, pg_textsearch, and ParadeDB dialects <../dialects>` for vector
similarity search and full-text search operators. See the :doc:`Dialects <../dialects>`
reference for operator details.

Expand Down
43 changes: 41 additions & 2 deletions docs/reference/dialects.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ group, so ``sqlglot.parse_one(..., dialect="pgvector")`` resolves them in any
environment where SQLSpec is installed; importing ``sqlspec`` alone does not load
them. Import the classes directly when you need them as objects::

from sqlspec.dialects import PGVector, ParadeDB, Spanner, Spangres
from sqlspec.dialects import PGTextSearch, PGVector, ParadeDB, Spanner, Spangres

Performance builds compile the custom dialect helper modules alongside
``sqlglot[c]``: generator transforms, operator registries, and compatibility
Expand Down Expand Up @@ -50,6 +50,45 @@ Adds support for pgvector distance operators:
* - ``<%>``
- Jaccard distance (binary vectors)

PGTextSearch
------------

.. autoclass:: sqlspec.dialects.postgres.PGTextSearch
:members:
:show-inheritance:
:no-index:

Adds support for PostgreSQL deployments with the ``pg_textsearch`` BM25 extension installed.
AlloyDB currently provides this extension in preview on PostgreSQL 17 and 18; see
`AlloyDB BM25 requirements <https://docs.cloud.google.com/alloydb/docs/ai/create-bm25-index>`_.
Server packaging and version support are independent of SQLSpec dialect availability.

.. list-table::
:header-rows: 1

* - Operator
- Description
* - ``<@>``
- BM25 score ranking operator (returns negative score for ASC index scans)

Indexes are created with ``USING bm25 (column) WITH (text_config='english')`` and queries order by ``column <@> 'query' ASC``.
Enable the extension in the database before using its operators or index method.
The dialect also supports pgvector distance operators for hybrid queries.

Asyncpg, psycopg, psqlpy, and PostgreSQL-backed ADBC configurations probe enabled
extensions on first connection. ``enable_pg_textsearch`` defaults to ``True``;
setting it to ``False`` disables detection, not the installed server extension.
``pg_textsearch_available`` and ``is_postgres_extension_active()`` report the cached,
enabled-and-detected state, and remain false before a successful probe.
ADK ``enable_bm25`` requires successful pg_textsearch detection.

The dialect label remains selected in priority order: ``paradedb``,
``pg_textsearch``, then ``pgvector`` for an otherwise default PostgreSQL
configuration. The active extension set records all enabled discoveries independently.
An explicitly selected non-default dialect is preserved; select a dialect that
supports the operators your queries use. Extension detection does not override
that choice, and a dialect label alone does not mark an extension as available.

ParadeDB
--------

Expand All @@ -58,7 +97,7 @@ ParadeDB
:show-inheritance:
:no-index:

Extends PGVector with ParadeDB pg_search operators:
Extends PostgreSQL with ParadeDB (pg_search) operators:

.. list-table::
:header-rows: 1
Expand Down
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ sqlspec = "sqlspec.__main__:run_cli"

[project.entry-points."sqlglot.dialects"]
paradedb = "sqlspec.dialects.postgres:ParadeDB"
pg_textsearch = "sqlspec.dialects.postgres:PGTextSearch"
pgvector = "sqlspec.dialects.postgres:PGVector"
spangres = "sqlspec.dialects.spanner:Spangres"
spanner = "sqlspec.dialects.spanner:Spanner"
Expand Down Expand Up @@ -236,6 +237,7 @@ exclude = [
"sqlspec/migrations/commands.py", # interpreted: inspect.signature/functools.wraps on coroutines
"sqlspec/dialects/postgres/_pgvector.py", # interpreted: Dialect subclass needs sqlglot's metaclass
"sqlspec/dialects/postgres/_paradedb.py", # interpreted: Dialect subclass needs sqlglot's metaclass
"sqlspec/dialects/postgres/_pg_textsearch.py", # interpreted: Dialect subclass needs sqlglot's metaclass
"sqlspec/dialects/spanner/_spanner.py", # interpreted: Dialect subclass needs sqlglot's metaclass
"sqlspec/dialects/spanner/_spangres.py", # interpreted: Dialect subclass needs sqlglot's metaclass
"sqlspec/storage/_arrow_payload.py", # PyArrow conversion boundary stays interpreted
Expand Down
28 changes: 25 additions & 3 deletions sqlspec/adapters/adbc/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
detect_postgres_extensions,
get_statement_config,
is_postgres_dialect,
is_postgres_extension_active,
resolve_dialect_from_config,
resolve_dialect_name,
resolve_driver_connect_func,
Expand Down Expand Up @@ -104,6 +105,10 @@ class AdbcDriverFeatures(TypedDict):
When True and the resolved dialect is PostgreSQL, queries ``pg_extension``
on the first connection to check for the ``pg_search`` extension.
Defaults to True. Independent of enable_pgvector.
enable_pg_textsearch: Enable pg_textsearch extension detection for BM25 search.
When True and the resolved dialect is PostgreSQL, queries ``pg_extension``
on the first connection to check for the ``pg_textsearch`` extension.
Defaults to True.
enable_events: Enable database event channel support.
Defaults to True when extension_config["events"] is configured.
Provides pub/sub capabilities via table-backed queue (ADBC has no native pub/sub).
Expand All @@ -124,6 +129,7 @@ class AdbcDriverFeatures(TypedDict):
arrow_extension_types: NotRequired[bool]
enable_pgvector: NotRequired[bool]
enable_paradedb: NotRequired[bool]
enable_pg_textsearch: NotRequired[bool]
enable_events: NotRequired[bool]
on_connection_create: "NotRequired[Callable[[AdbcConnection], None]]"
events_backend: NotRequired[Literal["poll_queue"]]
Expand Down Expand Up @@ -197,6 +203,7 @@ class AdbcConfig(NoPoolSyncConfig[AdbcConnection, AdbcDriver]):
__slots__ = (
"_default_session_config",
"_paradedb_available",
"_pg_textsearch_available",
"_pgvector_available",
"_resolved_dialect",
"_user_connection_hook",
Expand Down Expand Up @@ -231,6 +238,7 @@ def __init__(
self.connection_config = normalize_connection_config(connection_config)
self._pgvector_available: bool | None = None
self._paradedb_available: bool | None = None
self._pg_textsearch_available: bool | None = None

self._resolved_dialect = resolve_dialect_from_config(self.connection_config)

Expand Down Expand Up @@ -288,7 +296,7 @@ def create_connection(self) -> AdbcConnection:
def _update_dialect_for_extensions(self) -> None:
"""Update statement_config dialect based on detected extensions.

Priority: paradedb > pgvector > postgres (default).
Priority: paradedb > pg_textsearch > pgvector > postgres (default).
Only switches when current dialect is ``postgres``.
"""
current_dialect = self.statement_config.dialect or "postgres"
Expand All @@ -297,9 +305,16 @@ def _update_dialect_for_extensions(self) -> None:

if self._paradedb_available:
self.statement_config = self.statement_config.replace(dialect="paradedb")
elif self._pg_textsearch_available:
self.statement_config = self.statement_config.replace(dialect="pg_textsearch")
elif self._pgvector_available:
self.statement_config = self.statement_config.replace(dialect="pgvector")

@property
def pg_textsearch_available(self) -> bool:
"""Return True if the pg_textsearch extension is available."""
return bool(self._pg_textsearch_available)

def _detect_extensions_if_needed(self) -> None:
"""Detect postgres extensions on first call, caching results.

Expand All @@ -313,13 +328,17 @@ def _detect_extensions_if_needed(self) -> None:
if not is_postgres_dialect(dialect):
self._pgvector_available = False
self._paradedb_available = False
self._pg_textsearch_available = False
return

connection = self.create_connection()
try:
probe_names = build_postgres_extension_probe_names(self.driver_features)
pgvector_available, paradedb_available = detect_postgres_extensions(
connection, enable_pgvector="vector" in probe_names, enable_paradedb="pg_search" in probe_names
pgvector_available, paradedb_available, pg_textsearch_available = detect_postgres_extensions(
connection,
enable_pgvector="vector" in probe_names,
enable_paradedb="pg_search" in probe_names,
enable_pg_textsearch="pg_textsearch" in probe_names,
)
finally:
connection.close()
Expand All @@ -329,9 +348,12 @@ def _detect_extensions_if_needed(self) -> None:
detected_extensions.add("vector")
if paradedb_available:
detected_extensions.add("pg_search")
if pg_textsearch_available:
detected_extensions.add("pg_textsearch")
self.statement_config, self._pgvector_available, self._paradedb_available = resolve_postgres_extension_state(
self.statement_config, self.driver_features, detected_extensions
)
self._pg_textsearch_available = is_postgres_extension_active(self.driver_features, "pg_textsearch")

def provide_connection(self, *args: Any, **kwargs: Any) -> "AdbcConnectionContext":
"""Provide a connection context manager.
Expand Down
37 changes: 25 additions & 12 deletions sqlspec/adapters/adbc/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
)
from sqlspec.core.config_runtime import (
build_postgres_extension_probe_names,
is_postgres_extension_active,
resolve_postgres_extension_state,
resolve_runtime_statement_config,
)
Expand Down Expand Up @@ -74,6 +75,7 @@
"get_statement_config",
"handle_postgres_rollback",
"is_postgres_dialect",
"is_postgres_extension_active",
"normalize_driver_path",
"normalize_postgres_empty_parameters",
"normalize_script_rowcount",
Expand Down Expand Up @@ -231,41 +233,44 @@ def detect_dialect(connection: Any, logger: Any | None = None, *, fallback_diale


def detect_postgres_extensions(
connection: Any, *, enable_pgvector: bool = False, enable_paradedb: bool = False
) -> "tuple[bool, bool]":
"""Detect pgvector and paradedb extensions on a postgres connection.
connection: Any, *, enable_pgvector: bool = False, enable_paradedb: bool = False, enable_pg_textsearch: bool = False
) -> "tuple[bool, bool, bool]":
"""Detect pgvector, paradedb, and pg_textsearch extensions on a postgres connection.

Queries ``pg_extension`` for the ``vector`` and ``pg_search`` extensions.
Queries ``pg_extension`` for the ``vector``, ``pg_search``, and ``pg_textsearch`` extensions.
Returns cached-friendly booleans suitable for storing on the config instance.

Args:
connection: ADBC connection to a PostgreSQL database.
enable_pgvector: Whether to check for the pgvector extension.
enable_paradedb: Whether to check for the pg_search extension.
enable_pg_textsearch: Whether to check for the pg_textsearch extension.

Returns:
Tuple of ``(pgvector_available, paradedb_available)``.
Tuple of ``(pgvector_available, paradedb_available, pg_textsearch_available)``.
"""
extensions: list[str] = []
if enable_pgvector:
extensions.append("vector")
if enable_paradedb:
extensions.append("pg_search")
if enable_pg_textsearch:
extensions.append("pg_textsearch")

if not extensions:
return False, False
return False, False, False

try:
cursor = connection.cursor()
try:
cursor.execute("SELECT extname FROM pg_extension WHERE extname = ANY($1::text[])", [extensions])
rows = cursor.fetchall()
detected: set[str] = {row[0] for row in rows} if rows else set()
return "vector" in detected, "pg_search" in detected
return "vector" in detected, "pg_search" in detected, "pg_textsearch" in detected
finally:
cursor.close()
except Exception:
return False, False
return False, False, False


def normalize_driver_path(driver_name: str) -> str:
Expand Down Expand Up @@ -455,15 +460,21 @@ def resolve_dialect_name(dialect: Any) -> str:
"""Return the normalized dialect name string."""
if dialect is None:
return ""
if isinstance(dialect, str):
return dialect.lower()
if isinstance(dialect, type) and issubclass(dialect, sqlglot.Dialect):
return dialect.__name__.lower()
if isinstance(dialect, sqlglot.Dialect):
return type(dialect).__name__.lower()
return str(dialect)


def is_postgres_dialect(dialect_name: str) -> bool:
"""Return True when the dialect indicates PostgreSQL.

Includes pgvector and paradedb which are PostgreSQL extension dialects.
Includes pgvector, paradedb, and pg_textsearch extension dialects.
"""
return dialect_name in {"postgres", "postgresql", "pgvector", "paradedb"}
return dialect_name in {"postgres", "postgresql", "pgvector", "paradedb", "pg_textsearch", "pgtextsearch"}


def handle_postgres_rollback(dialect: str, cursor: Any, logger: Any | None = None) -> None:
Expand Down Expand Up @@ -726,7 +737,8 @@ def build_profile() -> "DriverParameterProfile":
def get_statement_config(detected_dialect: str) -> StatementConfig:
"""Create statement configuration for the specified dialect."""
default_style, supported_styles = DIALECT_PARAMETER_STYLES.get(
detected_dialect, (ParameterStyle.QMARK, [ParameterStyle.QMARK])
"postgres" if is_postgres_dialect(detected_dialect) else detected_dialect,
(ParameterStyle.QMARK, [ParameterStyle.QMARK]),
)

sqlglot_dialect = "postgres" if detected_dialect == "postgresql" else detected_dialect
Expand All @@ -744,7 +756,7 @@ def get_statement_config(detected_dialect: str) -> StatementConfig:
parameter_overrides["preserve_parameter_format"] = False
parameter_overrides["supported_execution_parameter_styles"] = {ParameterStyle.QMARK, ParameterStyle.NUMERIC}

if detected_dialect in {"postgres", "postgresql"}:
if is_postgres_dialect(detected_dialect):
parameter_overrides["ast_transformer"] = build_null_pruning_transform(dialect=sqlglot_dialect)

return build_statement_config_from_profile(
Expand Down Expand Up @@ -775,6 +787,7 @@ def apply_driver_features(
processed_features.setdefault("enable_arrow_extension_types", processed_features["arrow_extension_types"])
processed_features.setdefault("enable_pgvector", PGVECTOR_INSTALLED)
processed_features.setdefault("enable_paradedb", True)
processed_features.setdefault("enable_pg_textsearch", True)

if json_serializer is not None:
statement_config = _apply_adbc_json_serializer(statement_config, json_serializer)
Expand Down
Loading
Loading