diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..1eb68f4 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,68 @@ +# Changelog + +All notable changes to `arc-tsdb-client` are documented here. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.2.0] - 2026-07-03 + +A large feature-parity release bringing the SDK up to date with the Arc server, +plus bug fixes and the first real HTTP-layer test coverage. + +### Added + +- **MessagePack query** (`query.query_msgpack`, experimental): decodes the + columnar MessagePack response with native timestamp handling and returns the + same `QueryResponse` shape as `query()` (with per-column Arrow `types`). Raises + `ArcNotSupportedError` when the server lacks the `duckdb_arrow` build tag (501). +- **Bulk import client** (`client.import_`): `import_csv`, `import_parquet`, + `import_lp`, `import_tle`, and `import_stats`. Accepts paths, file-like objects, + or bytes. New models: `ImportResult`, `LPImportResult`, `TLEImportResult`, + `ImportStats`. +- **TLE streaming write** (`write.write_tle`) and `write.tle_stats`. +- **Databases management** (`client.databases`): `list`, `get`, + `list_measurements`, `create`, `delete`. New models under `models.database`. +- **Enterprise sub-clients** (sync + async): `rbac`, `governance`, `audit`, + `queries` (query management), `cluster`, `compaction`, `tiering`, `backup`, + `mqtt`. +- **zstd write compression** (now the default; gzip and none still supported). + Compression is configurable via a `Compression` enum, a string, or a bool. +- **Automatic retries** with exponential backoff + jitter for connection errors, + 429 (honoring `Retry-After`), and 503. Configurable via `max_retries` / + `retry_backoff`. +- New exceptions: `ArcNotSupportedError` (501) and `ArcLicenseError` (403 for a + missing enterprise-license feature; subclass of `ArcAuthenticationError`). +- Real HTTP round-trip test coverage (via `pytest-httpx`) for headers, error + mapping, compression on the wire, retries, msgpack decode, imports, databases, + and enterprise wrappers — sync and async. Opt-in live integration tests under + `tests/integration` (gated on `ARC_TEST_URL`). + +### Changed + +- Default write compression is now **zstd** (was gzip). Passing + `compression=True` still maps to gzip for backward compatibility. +- Writes no longer send a `Content-Encoding` header; Arc auto-detects the codec + from the payload's magic bytes. +- `User-Agent` is now derived from the package version instead of a hardcoded + string, and the package version is a single source of truth read by the build. + +### Fixed + +- Polars datetime → columnar conversion used a fragile private attribute that + raised at runtime; it now casts via the public `polars` API and normalizes the + series' `time_unit` to microseconds. +- Pandas `datetime64[us]` (and `ms`/`s`) timestamps were mis-scaled by assuming + nanoseconds; conversion now respects the column's resolution — including + tz-naive columns, whose numpy `datetime64` dtype exposes no `.unit` (the unit + is read via `numpy.datetime_data`). Previously these landed ~1000x too small. +- The buffered writer's time-based flush was passive (fired only on the next + write). It now runs a background timer (thread for sync, task for async) so + idle buffers still flush on `flush_interval`. +- `show_tables` interpolated the database name into SQL without validation; the + identifier is now validated against Arc's naming rules. +- Removed the dead, empty `dataframe` package stub. + +## [0.1.2] + +- Pad missing columns with `None` in the buffered columnar merge (#202). diff --git a/README.md b/README.md index 2076c04..8a4b6da 100644 --- a/README.md +++ b/README.md @@ -53,19 +53,27 @@ with ArcClient(host="localhost", token="your-token") as client: ## Features - **High-performance ingestion**: MessagePack columnar format (9M+ records/sec) -- **Multiple formats**: MessagePack columnar/row, InfluxDB Line Protocol +- **Multiple formats**: MessagePack columnar/row, InfluxDB Line Protocol, TLE +- **Compression**: gzip **and zstd** (zstd is the default and recommended codec) - **DataFrame integration**: Pandas, Polars, PyArrow - **Sync and async APIs**: Full async support with httpx -- **Buffered writes**: Automatic batching with size and time thresholds -- **Query support**: SQL queries with JSON or Arrow IPC streaming +- **Buffered writes**: Automatic batching with a background flush timer +- **Automatic retries**: Exponential backoff for connection errors, 429, and 503 +- **Query support**: SQL queries with JSON, Arrow IPC streaming, or MessagePack +- **Bulk import**: CSV, Parquet, line-protocol files, and TLE files +- **Databases**: List, create, delete databases and list measurements - **Management**: Retention policies, continuous queries, delete operations +- **Enterprise**: RBAC, query governance, audit logs, query management, cluster, + compaction, storage tiering, backup/restore, and MQTT ingestion - **Authentication**: Token management (create, rotate, revoke) ## Data Ingestion ### Columnar Format (Recommended) -The fastest way to write data. Uses MessagePack with gzip compression: +The fastest way to write data. Uses MessagePack with zstd compression by +default (the server recommends zstd; gzip and no compression are also supported — +see [Configuration](#configuration)): ```python client.write.write_columnar( @@ -134,6 +142,56 @@ lines = [ client.write.write_line_protocol(lines) ``` +### TLE (Two-Line Element) Streaming + +Stream satellite TLE text; the server parses it into orbital-element fields: + +```python +tle = """ISS (ZARYA) +1 25544U 98067A 24001.50000000 .00016717 00000-0 10270-3 0 9994 +2 25544 51.6416 247.4627 0006703 130.5360 325.0288 15.49309239432416""" + +client.write.write_tle(tle, measurement="satellite_tle") +``` + +## Bulk Import + +Import files directly (admin token required). Each returns a typed result with +the number of rows imported. Files may be gzip-compressed for LP/TLE. + +```python +# CSV +result = client.import_.import_csv("metrics.csv", measurement="cpu", + time_column="time", time_format="epoch_ms") +print(result.rows_imported) + +# Parquet +client.import_.import_parquet("data.parquet", measurement="cpu") + +# InfluxDB line-protocol file +client.import_.import_lp("dump.lp", precision="ns") + +# TLE file +client.import_.import_tle("catalog.tle", measurement="satellite_tle") +``` + +`import_csv`/`import_parquet`/`import_lp`/`import_tle` accept a path, a binary +file-like object, or raw `bytes`. + +## Databases + +```python +client.databases.create("metrics") +for db in client.databases.list().databases: + print(db.name, db.measurement_count) + +info = client.databases.get("metrics") +measurements = client.databases.list_measurements("metrics") + +# Destructive — requires confirm=True and server-side delete enabled +client.databases.delete("metrics", confirm=True) +``` + ## Querying Data ### JSON Response @@ -163,6 +221,21 @@ pl_df = client.query.query_polars("SELECT * FROM default.cpu LIMIT 1000") table = client.query.query_arrow("SELECT * FROM default.cpu LIMIT 1000") ``` +### MessagePack Response (experimental) + +A compact columnar MessagePack response with native timestamp decoding. Returns +the same `QueryResponse` shape as `query()` (row-major `data`), plus per-column +Arrow types in `.types`: + +```python +result = client.query.query_msgpack("SELECT * FROM default.cpu LIMIT 1000") +print(result.columns, result.types) +``` + +> **Experimental.** This uses `/api/v1/query/msgpack`, which requires the Arc +> server to be built with the `duckdb_arrow` build tag. Against a server without +> it, the call raises `ArcNotSupportedError` (HTTP 501). + ### Query Estimation Preview query cost before execution: @@ -313,33 +386,73 @@ print(f"New token: {rotated.new_token}") client.auth.revoke_token(token_id=123) ``` +## Enterprise Operations + +These require an Arc server with the corresponding enterprise license feature. +Without it, calls raise `ArcLicenseError` (a subclass of `ArcAuthenticationError`). +Each sub-client is a thin wrapper that returns the server's JSON. + +```python +# RBAC: organizations -> teams -> roles -> measurement permissions +org = client.rbac.create_organization("acme") +client.rbac.add_token_to_team(token_id=123, team_id=1) + +# Query governance (per-token rate limits & quotas) +client.governance.create_policy(token_id=123, rate_limit_per_minute=600) +print(client.governance.usage(123)) + +# Audit logs +client.audit.logs(event_type="query", since="2024-01-01T00:00:00Z", limit=100) + +# Query management +client.queries.active() +client.queries.kill("query-id") + +# Cluster / compaction / tiering / backup / MQTT +client.cluster.status() +client.compaction.trigger(tier="hourly,daily") +client.tiering.set_policy("metrics", hot_only=True, hot_max_age_days=7) +client.backup.create() +client.mqtt.list_subscriptions() +``` + ## Configuration ```python +from arc_client import ArcClient, Compression + client = ArcClient( - host="localhost", # Arc server hostname - port=8000, # Arc server port (default: 8000) - token="your-token", # API token - database="default", # Default database - timeout=30.0, # Request timeout in seconds - compression=True, # Enable gzip compression for writes - ssl=False, # Use HTTPS - verify_ssl=True, # Verify SSL certificates + host="localhost", # Arc server hostname + port=8000, # Arc server port (default: 8000) + token="your-token", # API token + database="default", # Default database + timeout=30.0, # Request timeout in seconds + compression=Compression.ZSTD, # "zstd" (default), "gzip", "none", or a bool + ssl=False, # Use HTTPS + verify_ssl=True, # Verify SSL certificates + max_retries=3, # Retries for connection errors / 429 / 503 + retry_backoff=0.5, # Base backoff (seconds) for exponential retry ) ``` +`compression` accepts a `Compression` enum, a string (`"zstd"`/`"gzip"`/`"none"`), +or a bool for backward compatibility (`True` → gzip, `False` → none). Arc detects +the codec from the payload's magic bytes, so no `Content-Encoding` header is sent. + ## Error Handling ```python from arc_client.exceptions import ( ArcError, # Base exception ArcConnectionError, # Connection failures - ArcAuthenticationError,# Auth failures (401) + ArcAuthenticationError,# Auth failures (401/403) + ArcLicenseError, # Enterprise feature not licensed (403) ArcQueryError, # Query execution errors ArcIngestionError, # Write failures ArcValidationError, # Invalid input ArcNotFoundError, # Resource not found (404) ArcRateLimitError, # Rate limited (429) + ArcNotSupportedError, # Operation unsupported by server (501) ArcServerError, # Server errors (5xx) ) diff --git a/pyproject.toml b/pyproject.toml index 71cc6e3..860982e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "arc-tsdb-client" -version = "0.1.2" +dynamic = ["version"] description = "Python SDK for Arc time-series database" readme = "README.md" license = { text = "MIT" } @@ -31,6 +31,7 @@ dependencies = [ "msgpack>=1.0.7", "pyarrow>=15.0.0", "pydantic>=2.6.0", + "zstandard>=0.22.0", ] [project.urls] @@ -60,6 +61,9 @@ docs = [ "nbsphinx>=0.9.0", ] +[tool.hatch.version] +path = "src/arc_client/__init__.py" + [tool.hatch.build.targets.sdist] include = ["/src"] @@ -88,4 +92,4 @@ warn_unused_ignores = true [tool.pytest.ini_options] asyncio_mode = "auto" testpaths = ["tests"] -addopts = "-v --cov=arc_client --cov-report=term-missing" +addopts = "-v --cov=src/arc_client --cov-report=term-missing" diff --git a/src/arc_client/__init__.py b/src/arc_client/__init__.py index 9d1a663..11e22c3 100644 --- a/src/arc_client/__init__.py +++ b/src/arc_client/__init__.py @@ -1,28 +1,35 @@ """Arc Python SDK - High-performance client for Arc time-series database.""" +# Defined before submodule imports so modules that read arc_client.__version__ +# (e.g. the User-Agent header builder) can import it without a circular-import +# failure during package initialization. +__version__ = "0.2.0" + from arc_client.async_client import AsyncArcClient from arc_client.client import ArcClient +from arc_client.compression import Compression from arc_client.config import ClientConfig from arc_client.exceptions import ( ArcAuthenticationError, ArcConnectionError, ArcError, ArcIngestionError, + ArcLicenseError, ArcNotFoundError, + ArcNotSupportedError, ArcQueryError, ArcRateLimitError, ArcServerError, ArcValidationError, ) -__version__ = "0.1.2" - __all__ = [ # Clients "ArcClient", "AsyncArcClient", # Config "ClientConfig", + "Compression", # Exceptions "ArcError", "ArcConnectionError", @@ -33,6 +40,8 @@ "ArcNotFoundError", "ArcRateLimitError", "ArcServerError", + "ArcNotSupportedError", + "ArcLicenseError", # Version "__version__", ] diff --git a/src/arc_client/async_client.py b/src/arc_client/async_client.py index 27a3f3c..969a421 100644 --- a/src/arc_client/async_client.py +++ b/src/arc_client/async_client.py @@ -4,6 +4,7 @@ from typing import Any, Optional +from arc_client.compression import Compression from arc_client.config import ClientConfig from arc_client.http.async_http import AsyncHTTPClient from arc_client.models.common import HealthResponse @@ -36,9 +37,11 @@ def __init__( token: Optional[str] = None, database: str = "default", timeout: float = 30.0, - compression: bool = True, + compression: Compression | str | bool = Compression.ZSTD, ssl: bool = False, verify_ssl: bool = True, + max_retries: int = 3, + retry_backoff: float = 0.5, ) -> None: """Initialize the async Arc client. @@ -48,9 +51,13 @@ def __init__( token: API token for authentication. database: Default database name. timeout: Request timeout in seconds. - compression: Enable gzip compression for writes. + compression: Write compression codec (none/gzip/zstd, or bool for + back-compat). Defaults to zstd. ssl: Use HTTPS. verify_ssl: Verify SSL certificates. + max_retries: Max automatic retries for transient failures + (connection errors, 429, 503). 0 disables retries. + retry_backoff: Base backoff (seconds) for exponential retry. """ self._config = ClientConfig( host=host, @@ -61,6 +68,8 @@ def __init__( compression=compression, ssl=ssl, verify_ssl=verify_ssl, + max_retries=max_retries, + retry_backoff=retry_backoff, ) self._http: Optional[AsyncHTTPClient] = None @@ -71,6 +80,17 @@ def __init__( self._retention: Any = None self._continuous_queries: Any = None self._delete: Any = None + self._import: Any = None + self._databases: Any = None + self._rbac: Any = None + self._governance: Any = None + self._audit: Any = None + self._queries: Any = None + self._cluster: Any = None + self._compaction: Any = None + self._tiering: Any = None + self._backup: Any = None + self._mqtt: Any = None def _get_http(self) -> AsyncHTTPClient: """Get or create the HTTP client.""" @@ -137,6 +157,107 @@ def delete(self) -> Any: self._delete = AsyncDeleteClient(self._get_http(), self._config) return self._delete + @property + def import_(self) -> Any: + """Get the async bulk import client (CSV, Parquet, line protocol, TLE).""" + if self._import is None: + from arc_client.ingestion.async_importer import AsyncImportClient + + self._import = AsyncImportClient(self._get_http(), self._config) + return self._import + + @property + def databases(self) -> Any: + """Get the async databases management client.""" + if self._databases is None: + from arc_client.management.async_databases import AsyncDatabaseClient + + self._databases = AsyncDatabaseClient(self._get_http(), self._config) + return self._databases + + @property + def rbac(self) -> Any: + """Get the async RBAC management client (enterprise).""" + if self._rbac is None: + from arc_client.management.async_rbac import AsyncRBACClient + + self._rbac = AsyncRBACClient(self._get_http(), self._config) + return self._rbac + + @property + def governance(self) -> Any: + """Get the async query governance client (enterprise).""" + if self._governance is None: + from arc_client.management.async_governance import AsyncGovernanceClient + + self._governance = AsyncGovernanceClient(self._get_http(), self._config) + return self._governance + + @property + def audit(self) -> Any: + """Get the async audit log client (enterprise).""" + if self._audit is None: + from arc_client.management.async_audit import AsyncAuditClient + + self._audit = AsyncAuditClient(self._get_http(), self._config) + return self._audit + + @property + def queries(self) -> Any: + """Get the async query management client (active/history/kill).""" + if self._queries is None: + from arc_client.management.async_query_management import ( + AsyncQueryManagementClient, + ) + + self._queries = AsyncQueryManagementClient(self._get_http(), self._config) + return self._queries + + @property + def cluster(self) -> Any: + """Get the async cluster management client (enterprise).""" + if self._cluster is None: + from arc_client.management.async_cluster import AsyncClusterClient + + self._cluster = AsyncClusterClient(self._get_http(), self._config) + return self._cluster + + @property + def compaction(self) -> Any: + """Get the async compaction management client (enterprise).""" + if self._compaction is None: + from arc_client.management.async_compaction import AsyncCompactionClient + + self._compaction = AsyncCompactionClient(self._get_http(), self._config) + return self._compaction + + @property + def tiering(self) -> Any: + """Get the async storage tiering client (enterprise).""" + if self._tiering is None: + from arc_client.management.async_tiering import AsyncTieringClient + + self._tiering = AsyncTieringClient(self._get_http(), self._config) + return self._tiering + + @property + def backup(self) -> Any: + """Get the async backup/restore client (enterprise).""" + if self._backup is None: + from arc_client.management.async_backup import AsyncBackupClient + + self._backup = AsyncBackupClient(self._get_http(), self._config) + return self._backup + + @property + def mqtt(self) -> Any: + """Get the async MQTT ingestion management client (enterprise).""" + if self._mqtt is None: + from arc_client.management.async_mqtt import AsyncMQTTClient + + self._mqtt = AsyncMQTTClient(self._get_http(), self._config) + return self._mqtt + async def health(self) -> HealthResponse: """Check server health. diff --git a/src/arc_client/client.py b/src/arc_client/client.py index 1595ad1..9bc623f 100644 --- a/src/arc_client/client.py +++ b/src/arc_client/client.py @@ -4,6 +4,7 @@ from typing import Any, Optional +from arc_client.compression import Compression from arc_client.config import ClientConfig from arc_client.http.sync_http import SyncHTTPClient from arc_client.models.common import HealthResponse @@ -36,9 +37,11 @@ def __init__( token: Optional[str] = None, database: str = "default", timeout: float = 30.0, - compression: bool = True, + compression: Compression | str | bool = Compression.ZSTD, ssl: bool = False, verify_ssl: bool = True, + max_retries: int = 3, + retry_backoff: float = 0.5, ) -> None: """Initialize the Arc client. @@ -48,9 +51,13 @@ def __init__( token: API token for authentication. database: Default database name. timeout: Request timeout in seconds. - compression: Enable gzip compression for writes. + compression: Write compression codec (none/gzip/zstd, or bool for + back-compat). Defaults to zstd. ssl: Use HTTPS. verify_ssl: Verify SSL certificates. + max_retries: Max automatic retries for transient failures + (connection errors, 429, 503). 0 disables retries. + retry_backoff: Base backoff (seconds) for exponential retry. """ self._config = ClientConfig( host=host, @@ -61,6 +68,8 @@ def __init__( compression=compression, ssl=ssl, verify_ssl=verify_ssl, + max_retries=max_retries, + retry_backoff=retry_backoff, ) self._http: Optional[SyncHTTPClient] = None @@ -71,6 +80,17 @@ def __init__( self._retention: Any = None self._continuous_queries: Any = None self._delete: Any = None + self._import: Any = None + self._databases: Any = None + self._rbac: Any = None + self._governance: Any = None + self._audit: Any = None + self._queries: Any = None + self._cluster: Any = None + self._compaction: Any = None + self._tiering: Any = None + self._backup: Any = None + self._mqtt: Any = None def _get_http(self) -> SyncHTTPClient: """Get or create the HTTP client.""" @@ -137,6 +157,105 @@ def delete(self) -> Any: self._delete = DeleteClient(self._get_http(), self._config) return self._delete + @property + def import_(self) -> Any: + """Get the bulk import client (CSV, Parquet, line protocol, TLE).""" + if self._import is None: + from arc_client.ingestion.importer import ImportClient + + self._import = ImportClient(self._get_http(), self._config) + return self._import + + @property + def databases(self) -> Any: + """Get the databases management client.""" + if self._databases is None: + from arc_client.management.databases import DatabaseClient + + self._databases = DatabaseClient(self._get_http(), self._config) + return self._databases + + @property + def rbac(self) -> Any: + """Get the RBAC management client (enterprise).""" + if self._rbac is None: + from arc_client.management.rbac import RBACClient + + self._rbac = RBACClient(self._get_http(), self._config) + return self._rbac + + @property + def governance(self) -> Any: + """Get the query governance client (enterprise).""" + if self._governance is None: + from arc_client.management.governance import GovernanceClient + + self._governance = GovernanceClient(self._get_http(), self._config) + return self._governance + + @property + def audit(self) -> Any: + """Get the audit log client (enterprise).""" + if self._audit is None: + from arc_client.management.audit import AuditClient + + self._audit = AuditClient(self._get_http(), self._config) + return self._audit + + @property + def queries(self) -> Any: + """Get the query management client (active/history/kill).""" + if self._queries is None: + from arc_client.management.query_management import QueryManagementClient + + self._queries = QueryManagementClient(self._get_http(), self._config) + return self._queries + + @property + def cluster(self) -> Any: + """Get the cluster management client (enterprise).""" + if self._cluster is None: + from arc_client.management.cluster import ClusterClient + + self._cluster = ClusterClient(self._get_http(), self._config) + return self._cluster + + @property + def compaction(self) -> Any: + """Get the compaction management client (enterprise).""" + if self._compaction is None: + from arc_client.management.compaction import CompactionClient + + self._compaction = CompactionClient(self._get_http(), self._config) + return self._compaction + + @property + def tiering(self) -> Any: + """Get the storage tiering client (enterprise).""" + if self._tiering is None: + from arc_client.management.tiering import TieringClient + + self._tiering = TieringClient(self._get_http(), self._config) + return self._tiering + + @property + def backup(self) -> Any: + """Get the backup/restore client (enterprise).""" + if self._backup is None: + from arc_client.management.backup import BackupClient + + self._backup = BackupClient(self._get_http(), self._config) + return self._backup + + @property + def mqtt(self) -> Any: + """Get the MQTT ingestion management client (enterprise).""" + if self._mqtt is None: + from arc_client.management.mqtt import MQTTClient + + self._mqtt = MQTTClient(self._get_http(), self._config) + return self._mqtt + def health(self) -> HealthResponse: """Check server health. diff --git a/src/arc_client/compression.py b/src/arc_client/compression.py new file mode 100644 index 0000000..68e9b91 --- /dev/null +++ b/src/arc_client/compression.py @@ -0,0 +1,196 @@ +"""Compression utilities for Arc ingestion. + +Arc auto-detects the compression codec of an ingestion payload from its leading +magic bytes (gzip = ``1f 8b``, zstd = ``28 b5 2f fd``) rather than from the +``Content-Encoding`` header, so clients should send the raw compressed bytes and +must **not** set a ``Content-Encoding`` header. + +The server recommends **zstd** for writes; it is the client default. +""" + +from __future__ import annotations + +import gzip +import io +from enum import Enum + + +class Compression(str, Enum): + """Compression codec used for ingestion payloads. + + Values are lowercase strings so they can be passed directly as config + (``compression="zstd"``) or as enum members (``Compression.ZSTD``). + """ + + NONE = "none" + GZIP = "gzip" + ZSTD = "zstd" + + @classmethod + def coerce(cls, value: Compression | str | bool) -> Compression: + """Coerce a config value into a :class:`Compression`. + + Accepts a :class:`Compression`, a string name, or a bool for backward + compatibility (``True`` -> gzip, ``False`` -> none). ``True`` maps to + gzip rather than zstd to preserve the pre-0.2.0 default behavior. + """ + if isinstance(value, Compression): + return value + if isinstance(value, bool): + return cls.GZIP if value else cls.NONE + if isinstance(value, str): + try: + return cls(value.lower()) + except ValueError as exc: + raise ValueError( + f"Invalid compression {value!r}; expected one of {[c.value for c in cls]}" + ) from exc + raise TypeError(f"Cannot coerce {type(value).__name__} to Compression") + + +def compress_gzip(data: bytes, level: int = 6) -> bytes: + """Compress data using gzip. + + Args: + data: Raw bytes to compress. + level: Compression level (0-9). Default 6 is a good balance. + 0 = no compression, 9 = maximum compression. + + Returns: + Gzip compressed bytes. + + Example: + >>> raw = b'hello world' + >>> compressed = compress_gzip(raw) + >>> compressed[:2] # Magic bytes + b'\\x1f\\x8b' + """ + buf = io.BytesIO() + with gzip.GzipFile(fileobj=buf, mode="wb", compresslevel=level) as f: + f.write(data) + return buf.getvalue() + + +def decompress_gzip(data: bytes) -> bytes: + """Decompress gzip data. + + Args: + data: Gzip compressed bytes. + + Returns: + Decompressed bytes. + + Raises: + ValueError: If data is not valid gzip. + """ + if not is_gzipped(data): + raise ValueError("Data is not gzip compressed") + + buf = io.BytesIO(data) + with gzip.GzipFile(fileobj=buf, mode="rb") as f: + return f.read() + + +def is_gzipped(data: bytes) -> bool: + """Check if data is gzip compressed. + + Gzip data starts with magic bytes 0x1f 0x8b. + + Args: + data: Bytes to check. + + Returns: + True if data appears to be gzip compressed. + """ + return len(data) >= 2 and data[0] == 0x1F and data[1] == 0x8B + + +def compress_zstd(data: bytes, level: int = 3) -> bytes: + """Compress data using zstandard. + + Args: + data: Raw bytes to compress. + level: Compression level (1-22). Default 3 is zstd's own default and a + good speed/ratio balance for ingestion payloads. + + Returns: + Zstd compressed bytes (magic ``28 b5 2f fd``). + + Raises: + ImportError: If the ``zstandard`` package is not installed. + """ + try: + import zstandard + except ImportError as exc: # pragma: no cover - zstandard is a hard dep + raise ImportError( + "zstd compression requires the 'zstandard' package. " + "Install it with: pip install zstandard" + ) from exc + + return zstandard.ZstdCompressor(level=level).compress(data) + + +def decompress_zstd(data: bytes) -> bytes: + """Decompress zstandard data. + + Args: + data: Zstd compressed bytes. + + Returns: + Decompressed bytes. + + Raises: + ImportError: If the ``zstandard`` package is not installed. + ValueError: If data is not valid zstd. + """ + try: + import zstandard + except ImportError as exc: # pragma: no cover - zstandard is a hard dep + raise ImportError( + "zstd decompression requires the 'zstandard' package. " + "Install it with: pip install zstandard" + ) from exc + + if not is_zstd(data): + raise ValueError("Data is not zstd compressed") + + return zstandard.ZstdDecompressor().decompress(data) + + +def is_zstd(data: bytes) -> bool: + """Check if data is zstd compressed. + + Zstd frames start with magic bytes 0x28 0xB5 0x2F 0xFD. + + Args: + data: Bytes to check. + + Returns: + True if data appears to be zstd compressed. + """ + return ( + len(data) >= 4 + and data[0] == 0x28 + and data[1] == 0xB5 + and data[2] == 0x2F + and data[3] == 0xFD + ) + + +def compress(data: bytes, codec: Compression) -> bytes: + """Compress ``data`` with the given codec. + + Args: + data: Raw bytes to compress. + codec: The :class:`Compression` codec to apply. + + Returns: + Compressed bytes, or ``data`` unchanged for :attr:`Compression.NONE`. + """ + if codec is Compression.NONE: + return data + if codec is Compression.GZIP: + return compress_gzip(data) + if codec is Compression.ZSTD: + return compress_zstd(data) + raise ValueError(f"Unknown compression codec: {codec!r}") diff --git a/src/arc_client/config.py b/src/arc_client/config.py index 8701124..c3f7c83 100644 --- a/src/arc_client/config.py +++ b/src/arc_client/config.py @@ -2,9 +2,11 @@ from __future__ import annotations -from typing import Optional +from typing import Optional, Union -from pydantic import BaseModel, Field +from pydantic import BaseModel, Field, field_validator + +from arc_client.compression import Compression class ClientConfig(BaseModel): @@ -15,9 +17,29 @@ class ClientConfig(BaseModel): token: Optional[str] = Field(default=None, description="API token for authentication") database: str = Field(default="default", description="Default database name") timeout: float = Field(default=30.0, description="Request timeout in seconds") - compression: bool = Field(default=True, description="Enable gzip compression for writes") + compression: Union[Compression, str, bool] = Field( + default=Compression.ZSTD, + description="Compression codec for writes (none/gzip/zstd). " + "True is treated as gzip and False as none for backward compatibility.", + ) ssl: bool = Field(default=False, description="Use HTTPS") verify_ssl: bool = Field(default=True, description="Verify SSL certificates") + max_retries: int = Field( + default=3, + ge=0, + description="Max automatic retries for transient failures " + "(connection errors, 429, 503). 0 disables retries.", + ) + retry_backoff: float = Field( + default=0.5, + ge=0.0, + description="Base backoff in seconds for exponential retry with jitter.", + ) + + @field_validator("compression") + @classmethod + def _coerce_compression(cls, value: Union[Compression, str, bool]) -> Compression: + return Compression.coerce(value) @property def base_url(self) -> str: diff --git a/src/arc_client/dataframe/__init__.py b/src/arc_client/dataframe/__init__.py deleted file mode 100644 index ad23b2d..0000000 --- a/src/arc_client/dataframe/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""DataFrame adapters for Arc client.""" diff --git a/src/arc_client/exceptions.py b/src/arc_client/exceptions.py index cd18c9a..a7b36bf 100644 --- a/src/arc_client/exceptions.py +++ b/src/arc_client/exceptions.py @@ -59,3 +59,23 @@ class ArcServerError(ArcError): def __init__(self, message: str, status_code: Optional[int] = None) -> None: super().__init__(message) self.status_code = status_code + + +class ArcNotSupportedError(ArcError): + """The server does not support the requested operation. + + Raised, for example, when calling ``query_msgpack`` against an Arc server + that was not built with the ``duckdb_arrow`` build tag (HTTP 501). + """ + + pass + + +class ArcLicenseError(ArcAuthenticationError): + """The requested feature requires an enterprise license. + + Subclasses :class:`ArcAuthenticationError` so existing ``except`` blocks + that catch permission failures keep working. + """ + + pass diff --git a/src/arc_client/http/async_http.py b/src/arc_client/http/async_http.py index d72cb39..cb9e32d 100644 --- a/src/arc_client/http/async_http.py +++ b/src/arc_client/http/async_http.py @@ -2,15 +2,20 @@ from __future__ import annotations +import asyncio from typing import Any, Optional import httpx from arc_client.config import ClientConfig from arc_client.http.base import ( + RETRYABLE_TRANSPORT_ERRORS, HTTPClientBase, + compute_backoff, handle_connection_error, handle_response_error, + is_retryable_status, + parse_retry_after, ) @@ -37,6 +42,48 @@ async def close(self) -> None: await self._client.aclose() self._client = None + async def _request( + self, + method: str, + path: str, + *, + headers: Optional[dict[str, str]] = None, + **kwargs: Any, + ) -> httpx.Response: + """Send a request with retry/backoff for transient failures. + + Retries connection/timeout errors and 429/503 responses up to + ``config.max_retries`` times, honoring ``Retry-After`` when present. + """ + request_kwargs = self._prepare_request_kwargs(headers=headers, **kwargs) + client = self._get_client() + last_exc: Optional[Exception] = None + + for attempt in range(self.config.max_retries + 1): + try: + response = await client.request(method, path, **request_kwargs) + except RETRYABLE_TRANSPORT_ERRORS as e: + last_exc = e + if attempt >= self.config.max_retries: + break + await asyncio.sleep(compute_backoff(attempt, self.config.retry_backoff)) + continue + + if is_retryable_status(response.status_code) and attempt < self.config.max_retries: + retry_after = parse_retry_after(response) + await asyncio.sleep( + compute_backoff(attempt, self.config.retry_backoff, retry_after) + ) + continue + + handle_response_error(response) + return response + + # Exhausted retries on a transport error. + assert last_exc is not None + handle_connection_error(last_exc, self._build_url(path)) + raise last_exc # Never reached, but satisfies the type checker. + async def get( self, path: str, @@ -44,14 +91,7 @@ async def get( headers: Optional[dict[str, str]] = None, ) -> httpx.Response: """Make a GET request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) - try: - response = await self._get_client().get(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise # Never reached, but satisfies type checker + return await self._request("GET", path, headers=headers, params=params) async def post( self, @@ -60,20 +100,20 @@ async def post( content: Optional[bytes] = None, params: Optional[dict[str, Any]] = None, headers: Optional[dict[str, str]] = None, + files: Optional[Any] = None, + data: Optional[Any] = None, ) -> httpx.Response: """Make a POST request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) + extra: dict[str, Any] = {} if json is not None: - kwargs["json"] = json + extra["json"] = json if content is not None: - kwargs["content"] = content - try: - response = await self._get_client().post(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise + extra["content"] = content + if files is not None: + extra["files"] = files + if data is not None: + extra["data"] = data + return await self._request("POST", path, headers=headers, params=params, **extra) async def put( self, @@ -83,16 +123,10 @@ async def put( headers: Optional[dict[str, str]] = None, ) -> httpx.Response: """Make a PUT request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) + extra: dict[str, Any] = {} if json is not None: - kwargs["json"] = json - try: - response = await self._get_client().put(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise + extra["json"] = json + return await self._request("PUT", path, headers=headers, params=params, **extra) async def patch( self, @@ -102,16 +136,10 @@ async def patch( headers: Optional[dict[str, str]] = None, ) -> httpx.Response: """Make a PATCH request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) + extra: dict[str, Any] = {} if json is not None: - kwargs["json"] = json - try: - response = await self._get_client().patch(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise + extra["json"] = json + return await self._request("PATCH", path, headers=headers, params=params, **extra) async def delete( self, @@ -120,14 +148,7 @@ async def delete( headers: Optional[dict[str, str]] = None, ) -> httpx.Response: """Make a DELETE request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) - try: - response = await self._get_client().delete(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise + return await self._request("DELETE", path, headers=headers, params=params) async def __aenter__(self) -> AsyncHTTPClient: return self diff --git a/src/arc_client/http/base.py b/src/arc_client/http/base.py index ade032e..d92e359 100644 --- a/src/arc_client/http/base.py +++ b/src/arc_client/http/base.py @@ -2,26 +2,32 @@ from __future__ import annotations +import os from typing import Any, Optional import httpx +from arc_client import __version__ from arc_client.config import ClientConfig from arc_client.exceptions import ( ArcAuthenticationError, ArcConnectionError, + ArcLicenseError, ArcNotFoundError, + ArcNotSupportedError, ArcRateLimitError, ArcServerError, ) +USER_AGENT = f"arc-client-python/{__version__}" + def build_headers( config: ClientConfig, extra_headers: Optional[dict[str, str]] = None ) -> dict[str, str]: """Build request headers with authentication.""" headers: dict[str, str] = { - "User-Agent": "arc-client-python/0.1.0", + "User-Agent": USER_AGENT, } if config.token: headers["Authorization"] = f"Bearer {config.token}" @@ -47,11 +53,18 @@ def handle_response_error(response: httpx.Response) -> None: if status_code == 401: raise ArcAuthenticationError(f"Authentication failed: {message}") elif status_code == 403: + # Arc returns 403 for missing enterprise-license features with a + # distinctive message; surface those as ArcLicenseError (a subclass of + # ArcAuthenticationError, so plain permission-denied handling still works). + if isinstance(message, str) and "enterprise license" in message.lower(): + raise ArcLicenseError(f"Enterprise license required: {message}") raise ArcAuthenticationError(f"Permission denied: {message}") elif status_code == 404: raise ArcNotFoundError(f"Resource not found: {message}") elif status_code == 429: raise ArcRateLimitError(f"Rate limit exceeded: {message}") + elif status_code == 501: + raise ArcNotSupportedError(f"Operation not supported by server: {message}") elif status_code >= 500: raise ArcServerError(f"Server error: {message}", status_code=status_code) else: @@ -63,6 +76,54 @@ def handle_connection_error(error: Exception, url: str) -> None: raise ArcConnectionError(f"Failed to connect to {url}: {error}") from error +# Transport errors that are safe to retry (connection refused, DNS, timeouts). +RETRYABLE_TRANSPORT_ERRORS = ( + httpx.ConnectError, + httpx.ConnectTimeout, + httpx.ReadTimeout, + httpx.WriteTimeout, + httpx.PoolTimeout, + httpx.RemoteProtocolError, +) + +# HTTP status codes worth retrying: 429 rate-limit, 503 schema-churn / +# replication catch-up (both explicitly retryable per the Arc server). +RETRYABLE_STATUS_CODES = frozenset({429, 503}) + + +def is_retryable_status(status_code: int) -> bool: + """Return True if a response status should be retried.""" + return status_code in RETRYABLE_STATUS_CODES + + +def parse_retry_after(response: httpx.Response) -> Optional[float]: + """Parse a ``Retry-After`` header (seconds only) into a float, if present.""" + value = response.headers.get("Retry-After") + if not value: + return None + try: + seconds = float(value) + except ValueError: + # HTTP-date form is not honored here; fall back to backoff. + return None + return max(0.0, seconds) + + +def compute_backoff(attempt: int, base: float, retry_after: Optional[float] = None) -> float: + """Compute the delay before the next retry. + + Honors an explicit ``Retry-After`` when provided; otherwise uses capped + exponential backoff (``base * 2**attempt``) with full jitter. ``attempt`` is + 0-indexed (0 = delay before the first retry). + """ + if retry_after is not None: + return retry_after + ceiling: float = min(base * (2**attempt), 30.0) + # Full jitter in [0, ceiling]; os.urandom avoids seeding a global RNG. + frac: float = int.from_bytes(os.urandom(4), "big") / 0xFFFFFFFF + return ceiling * frac + + class HTTPClientBase: """Base class for HTTP clients with common functionality.""" diff --git a/src/arc_client/http/sync_http.py b/src/arc_client/http/sync_http.py index 1cf6db9..b4beb7c 100644 --- a/src/arc_client/http/sync_http.py +++ b/src/arc_client/http/sync_http.py @@ -2,15 +2,20 @@ from __future__ import annotations +import time from typing import Any, Optional import httpx from arc_client.config import ClientConfig from arc_client.http.base import ( + RETRYABLE_TRANSPORT_ERRORS, HTTPClientBase, + compute_backoff, handle_connection_error, handle_response_error, + is_retryable_status, + parse_retry_after, ) @@ -37,6 +42,46 @@ def close(self) -> None: self._client.close() self._client = None + def _request( + self, + method: str, + path: str, + *, + headers: Optional[dict[str, str]] = None, + **kwargs: Any, + ) -> httpx.Response: + """Send a request with retry/backoff for transient failures. + + Retries connection/timeout errors and 429/503 responses up to + ``config.max_retries`` times, honoring ``Retry-After`` when present. + """ + request_kwargs = self._prepare_request_kwargs(headers=headers, **kwargs) + client = self._get_client() + last_exc: Optional[Exception] = None + + for attempt in range(self.config.max_retries + 1): + try: + response = client.request(method, path, **request_kwargs) + except RETRYABLE_TRANSPORT_ERRORS as e: + last_exc = e + if attempt >= self.config.max_retries: + break + time.sleep(compute_backoff(attempt, self.config.retry_backoff)) + continue + + if is_retryable_status(response.status_code) and attempt < self.config.max_retries: + retry_after = parse_retry_after(response) + time.sleep(compute_backoff(attempt, self.config.retry_backoff, retry_after)) + continue + + handle_response_error(response) + return response + + # Exhausted retries on a transport error. + assert last_exc is not None + handle_connection_error(last_exc, self._build_url(path)) + raise last_exc # Never reached, but satisfies the type checker. + def get( self, path: str, @@ -44,14 +89,7 @@ def get( headers: Optional[dict[str, str]] = None, ) -> httpx.Response: """Make a GET request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) - try: - response = self._get_client().get(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise # Never reached, but satisfies type checker + return self._request("GET", path, headers=headers, params=params) def post( self, @@ -60,20 +98,20 @@ def post( content: Optional[bytes] = None, params: Optional[dict[str, Any]] = None, headers: Optional[dict[str, str]] = None, + files: Optional[Any] = None, + data: Optional[Any] = None, ) -> httpx.Response: """Make a POST request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) + extra: dict[str, Any] = {} if json is not None: - kwargs["json"] = json + extra["json"] = json if content is not None: - kwargs["content"] = content - try: - response = self._get_client().post(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise + extra["content"] = content + if files is not None: + extra["files"] = files + if data is not None: + extra["data"] = data + return self._request("POST", path, headers=headers, params=params, **extra) def put( self, @@ -83,16 +121,10 @@ def put( headers: Optional[dict[str, str]] = None, ) -> httpx.Response: """Make a PUT request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) + extra: dict[str, Any] = {} if json is not None: - kwargs["json"] = json - try: - response = self._get_client().put(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise + extra["json"] = json + return self._request("PUT", path, headers=headers, params=params, **extra) def patch( self, @@ -102,16 +134,10 @@ def patch( headers: Optional[dict[str, str]] = None, ) -> httpx.Response: """Make a PATCH request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) + extra: dict[str, Any] = {} if json is not None: - kwargs["json"] = json - try: - response = self._get_client().patch(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise + extra["json"] = json + return self._request("PATCH", path, headers=headers, params=params, **extra) def delete( self, @@ -120,14 +146,7 @@ def delete( headers: Optional[dict[str, str]] = None, ) -> httpx.Response: """Make a DELETE request.""" - kwargs = self._prepare_request_kwargs(headers=headers, params=params) - try: - response = self._get_client().delete(path, **kwargs) - handle_response_error(response) - return response - except httpx.ConnectError as e: - handle_connection_error(e, self._build_url(path)) - raise + return self._request("DELETE", path, headers=headers, params=params) def __enter__(self) -> SyncHTTPClient: return self diff --git a/src/arc_client/ingestion/__init__.py b/src/arc_client/ingestion/__init__.py index 013e904..c85f59b 100644 --- a/src/arc_client/ingestion/__init__.py +++ b/src/arc_client/ingestion/__init__.py @@ -9,7 +9,16 @@ from arc_client.ingestion.async_buffered import AsyncBufferedWriter from arc_client.ingestion.async_writer import AsyncWriteClient from arc_client.ingestion.buffered import BufferedWriter -from arc_client.ingestion.compression import compress_gzip, decompress_gzip, is_gzipped +from arc_client.ingestion.compression import ( + Compression, + compress, + compress_gzip, + compress_zstd, + decompress_gzip, + decompress_zstd, + is_gzipped, + is_zstd, +) from arc_client.ingestion.line_protocol import ( format_columnar_as_lines, format_line_protocol, @@ -41,7 +50,12 @@ "format_lines", "format_columnar_as_lines", # Compression + "Compression", + "compress", "compress_gzip", + "compress_zstd", "decompress_gzip", + "decompress_zstd", "is_gzipped", + "is_zstd", ] diff --git a/src/arc_client/ingestion/async_buffered.py b/src/arc_client/ingestion/async_buffered.py index 5f0c4bc..c984115 100644 --- a/src/arc_client/ingestion/async_buffered.py +++ b/src/arc_client/ingestion/async_buffered.py @@ -3,6 +3,7 @@ from __future__ import annotations import asyncio +import contextlib import time from collections import defaultdict from typing import TYPE_CHECKING, Any @@ -38,6 +39,9 @@ def __init__( self._lock = asyncio.Lock() self._closed = False + # Background task that flushes on flush_interval even when idle. + self._flush_task: asyncio.Task[None] | None = None + async def write(self, record: dict[str, Any]) -> None: """Write a single record to the buffer.""" measurement = record.get("measurement") @@ -96,6 +100,22 @@ async def flush(self) -> None: async with self._lock: await self._flush_all_unlocked() + async def _flush_loop(self) -> None: + """Periodically flush measurements whose data has aged past the interval.""" + try: + while True: + await asyncio.sleep(self._flush_interval) + async with self._lock: + if self._closed: + return + if ( + self._buffers + and time.monotonic() - self._last_flush_time >= self._flush_interval + ): + await self._flush_all_unlocked() + except asyncio.CancelledError: + return + async def _flush_measurement_unlocked(self, measurement: str) -> None: """Flush a single measurement. Must hold lock.""" if measurement not in self._buffers or not self._buffers[measurement]: @@ -156,11 +176,21 @@ async def close(self) -> None: if self._closed: return + # Cancel the timer task before the final flush so it can't race. + task = self._flush_task + if task is not None: + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + self._flush_task = None + async with self._lock: await self._flush_all_unlocked() self._closed = True async def __aenter__(self) -> AsyncBufferedWriter: + if self._flush_task is None and self._flush_interval > 0: + self._flush_task = asyncio.ensure_future(self._flush_loop()) return self async def __aexit__(self, *args: Any) -> None: diff --git a/src/arc_client/ingestion/async_importer.py b/src/arc_client/ingestion/async_importer.py new file mode 100644 index 0000000..30f48db --- /dev/null +++ b/src/arc_client/ingestion/async_importer.py @@ -0,0 +1,149 @@ +"""Asynchronous bulk import client for Arc. + +Async twin of :mod:`arc_client.ingestion.importer`. Wraps the +``/api/v1/import/*`` endpoints (CSV, Parquet, line-protocol file, TLE). +""" + +from __future__ import annotations + +from typing import Any, Optional, cast + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcIngestionError +from arc_client.http.async_http import AsyncHTTPClient +from arc_client.ingestion.importer import FileSource, _resolve_file +from arc_client.models.import_ import ( + ImportResult, + ImportStats, + LPImportResult, + TLEImportResult, +) + + +class AsyncImportClient: + """Asynchronous client for Arc bulk import endpoints. + + Example: + >>> result = await client.import_.import_csv("data.csv", measurement="cpu") + >>> print(result.rows_imported) + """ + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _post_import( + self, + path: str, + source: FileSource, + params: dict[str, Any], + database: Optional[str], + extra_headers: Optional[dict[str, str]] = None, + ) -> dict[str, Any]: + """POST a multipart import request and return the ``result`` payload.""" + filename, fileobj = _resolve_file(source) + headers = dict(extra_headers or {}) + if database: + headers["x-arc-database"] = database + + clean_params = {k: v for k, v in params.items() if v is not None} + + try: + response = await self._http.post( + path, + files={"file": (filename, fileobj)}, + params=clean_params or None, + headers=headers or None, + ) + body = response.json() + except ArcIngestionError: + raise + except Exception as e: + raise ArcIngestionError(f"Import failed: {e}") from e + + result = body.get("result") if isinstance(body, dict) else None + if result is None: + raise ArcIngestionError(f"Unexpected import response: {body!r}") + return cast("dict[str, Any]", result) + + async def import_csv( + self, + source: FileSource, + measurement: str, + database: Optional[str] = None, + time_column: str = "time", + time_format: str = "", + delimiter: str = ",", + skip_rows: int = 0, + ) -> ImportResult: + """Import a CSV file. See ImportClient.import_csv for details.""" + params = { + "measurement": measurement, + "time_column": time_column, + "time_format": time_format or None, + "delimiter": delimiter, + "skip_rows": skip_rows, + } + result = await self._post_import( + "/api/v1/import/csv", source, params, database or self._config.database + ) + return ImportResult.model_validate(result) + + async def import_parquet( + self, + source: FileSource, + measurement: str, + database: Optional[str] = None, + time_column: str = "time", + time_format: str = "", + ) -> ImportResult: + """Import a Parquet file. See ImportClient.import_parquet for details.""" + params = { + "measurement": measurement, + "time_column": time_column, + "time_format": time_format or None, + } + result = await self._post_import( + "/api/v1/import/parquet", source, params, database or self._config.database + ) + return ImportResult.model_validate(result) + + async def import_lp( + self, + source: FileSource, + database: Optional[str] = None, + precision: str = "ns", + measurement: Optional[str] = None, + ) -> LPImportResult: + """Import a line-protocol file. See ImportClient.import_lp for details.""" + params = {"precision": precision, "measurement": measurement} + result = await self._post_import( + "/api/v1/import/lp", source, params, database or self._config.database + ) + return LPImportResult.model_validate(result) + + async def import_tle( + self, + source: FileSource, + database: Optional[str] = None, + measurement: str = "satellite_tle", + ) -> TLEImportResult: + """Import a TLE file. See ImportClient.import_tle for details.""" + result = await self._post_import( + "/api/v1/import/tle", + source, + {}, + database or self._config.database, + extra_headers={"x-arc-measurement": measurement}, + ) + return TLEImportResult.model_validate(result) + + async def import_stats(self) -> ImportStats: + """Get cumulative import statistics.""" + try: + response = await self._http.get("/api/v1/import/stats") + body = response.json() + except Exception as e: + raise ArcIngestionError(f"Failed to get import stats: {e}") from e + stats = body.get("stats", body) if isinstance(body, dict) else {} + return ImportStats.model_validate(stats) diff --git a/src/arc_client/ingestion/async_writer.py b/src/arc_client/ingestion/async_writer.py index f0cfaf3..985a4bd 100644 --- a/src/arc_client/ingestion/async_writer.py +++ b/src/arc_client/ingestion/async_writer.py @@ -2,12 +2,13 @@ from __future__ import annotations -from typing import TYPE_CHECKING, Any, Optional +from typing import TYPE_CHECKING, Any, Optional, Union +from arc_client.compression import Compression from arc_client.config import ClientConfig from arc_client.exceptions import ArcIngestionError, ArcValidationError from arc_client.http.async_http import AsyncHTTPClient -from arc_client.ingestion.compression import compress_gzip +from arc_client.ingestion.compression import compress as compress_data from arc_client.ingestion.line_protocol import format_line_protocol from arc_client.ingestion.msgpack import ( dataframe_to_columnar, @@ -15,6 +16,10 @@ encode_records, ) +# Compression argument accepted by write methods: a codec, its name, a bool +# (back-compat), or None to use the client default. +CompressArg = Optional[Union[Compression, str, bool]] + if TYPE_CHECKING: from arc_client.ingestion.async_buffered import AsyncBufferedWriter @@ -47,7 +52,7 @@ async def write_columnar( measurement: str, columns: dict[str, list[Any]], database: Optional[str] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, time_unit: str = "us", ) -> None: """Write columnar data to Arc (highest performance method). @@ -72,7 +77,7 @@ async def write_records( self, records: list[dict[str, Any]], database: Optional[str] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, ) -> None: """Write records to Arc using MessagePack row format.""" try: @@ -91,7 +96,7 @@ async def write_dataframe( database: Optional[str] = None, time_column: str = "time", tag_columns: Optional[list[str]] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, ) -> None: """Write a DataFrame to Arc.""" try: @@ -107,7 +112,7 @@ async def write_line_protocol( self, lines: str | list[str], database: Optional[str] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, ) -> None: """Write data using InfluxDB Line Protocol format.""" data = "\n".join(lines) if isinstance(lines, list) else lines @@ -120,12 +125,43 @@ async def write_point( tags: Optional[dict[str, str]] = None, timestamp: Optional[int] = None, database: Optional[str] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, ) -> None: """Write a single data point using Line Protocol.""" line = format_line_protocol(measurement, fields, tags, timestamp) await self._write_line_protocol(line.encode("utf-8"), database, compress) + async def write_tle( + self, + tle_text: str | bytes, + measurement: str = "satellite_tle", + database: Optional[str] = None, + compress: CompressArg = None, + ) -> None: + """Stream raw TLE (Two-Line Element) text to Arc. + + See WriteClient.write_tle for details. For large files prefer + ``client.import_.import_tle``. + """ + data = tle_text.encode("utf-8") if isinstance(tle_text, str) else tle_text + await self._post_write( + "/api/v1/write/tle", + data, + "text/plain", + database, + compress, + extra_headers={"x-arc-measurement": measurement}, + ) + + async def tle_stats(self) -> dict[str, Any]: + """Get TLE streaming write statistics from ``/api/v1/write/tle/stats``.""" + try: + response = await self._http.get("/api/v1/write/tle/stats") + body = response.json() + except Exception as e: + raise ArcIngestionError(f"Failed to get TLE stats: {e}") from e + return body.get("stats", body) if isinstance(body, dict) else {} + def buffered( self, batch_size: int = 10000, @@ -136,33 +172,37 @@ def buffered( return AsyncBufferedWriter(self, batch_size, flush_interval) - async def _write_msgpack( + def _resolve_codec(self, compress: CompressArg) -> Compression: + """Resolve the effective compression codec for a write.""" + if compress is None: + return Compression.coerce(self._config.compression) + return Compression.coerce(compress) + + async def _post_write( self, + path: str, data: bytes, + content_type: str, database: Optional[str], - compress: Optional[bool], + compress: CompressArg, + extra_headers: Optional[dict[str, str]] = None, ) -> None: - """Send MessagePack data to Arc.""" - should_compress = compress if compress is not None else self._config.compression + """Compress (if requested) and POST a write payload. - if should_compress: - data = compress_gzip(data) - - headers = { - "Content-Type": "application/msgpack", - } - if should_compress: - headers["Content-Encoding"] = "gzip" + Arc auto-detects gzip/zstd from the payload's magic bytes, so no + ``Content-Encoding`` header is sent. + """ + codec = self._resolve_codec(compress) + data = compress_data(data, codec) + headers = {"Content-Type": content_type} + if extra_headers: + headers.update(extra_headers) if database: headers["x-arc-database"] = database try: - response = await self._http.post( - "/api/v1/write/msgpack", - content=data, - headers=headers, - ) + response = await self._http.post(path, content=data, headers=headers) if response.status_code not in (200, 204): raise ArcIngestionError( f"Write failed with status {response.status_code}: {response.text}" @@ -172,38 +212,24 @@ async def _write_msgpack( except Exception as e: raise ArcIngestionError(f"Failed to write data: {e}") from e + async def _write_msgpack( + self, + data: bytes, + database: Optional[str], + compress: CompressArg, + ) -> None: + """Send MessagePack data to Arc.""" + await self._post_write( + "/api/v1/write/msgpack", data, "application/msgpack", database, compress + ) + async def _write_line_protocol( self, data: bytes, database: Optional[str], - compress: Optional[bool], + compress: CompressArg, ) -> None: """Send Line Protocol data to Arc.""" - should_compress = compress if compress is not None else self._config.compression - - if should_compress: - data = compress_gzip(data) - - headers = { - "Content-Type": "text/plain", - } - if should_compress: - headers["Content-Encoding"] = "gzip" - - if database: - headers["x-arc-database"] = database - - try: - response = await self._http.post( - "/api/v1/write/line-protocol", - content=data, - headers=headers, - ) - if response.status_code not in (200, 204): - raise ArcIngestionError( - f"Write failed with status {response.status_code}: {response.text}" - ) - except ArcIngestionError: - raise - except Exception as e: - raise ArcIngestionError(f"Failed to write data: {e}") from e + await self._post_write( + "/api/v1/write/line-protocol", data, "text/plain", database, compress + ) diff --git a/src/arc_client/ingestion/buffered.py b/src/arc_client/ingestion/buffered.py index df3f9dc..d666fbd 100644 --- a/src/arc_client/ingestion/buffered.py +++ b/src/arc_client/ingestion/buffered.py @@ -59,6 +59,11 @@ def __init__( self._lock = threading.Lock() self._closed = False + # Background timer that flushes on flush_interval even when no writes + # arrive. Started on __enter__ and torn down on close(). + self._stop_event = threading.Event() + self._flush_thread: threading.Thread | None = None + def write(self, record: dict[str, Any]) -> None: """Write a single record to the buffer. @@ -138,6 +143,28 @@ def flush(self) -> None: with self._lock: self._flush_all() + def _start_flush_thread(self) -> None: + """Start the background flush timer (idempotent).""" + if self._flush_thread is not None or self._flush_interval <= 0: + return + self._stop_event.clear() + self._flush_thread = threading.Thread( + target=self._flush_loop, name="arc-buffered-flush", daemon=True + ) + self._flush_thread.start() + + def _flush_loop(self) -> None: + """Periodically flush measurements whose data has aged past the interval.""" + while not self._stop_event.wait(self._flush_interval): + with self._lock: + if self._closed: + return + if ( + self._buffers + and time.monotonic() - self._last_flush_time >= self._flush_interval + ): + self._flush_all() + def _flush_measurement(self, measurement: str) -> None: """Flush a single measurement's buffer. Must hold lock.""" if measurement not in self._buffers or not self._buffers[measurement]: @@ -202,12 +229,20 @@ def close(self) -> None: if self._closed: return + # Stop the timer thread first so it can't race the final flush. + self._stop_event.set() + thread = self._flush_thread + if thread is not None: + thread.join(timeout=self._flush_interval + 1.0) + self._flush_thread = None + with self._lock: self._flush_all() self._closed = True def __enter__(self) -> BufferedWriter: """Enter context manager.""" + self._start_flush_thread() return self def __exit__(self, *args: Any) -> None: diff --git a/src/arc_client/ingestion/compression.py b/src/arc_client/ingestion/compression.py index 68971cb..d85868b 100644 --- a/src/arc_client/ingestion/compression.py +++ b/src/arc_client/ingestion/compression.py @@ -1,67 +1,30 @@ -"""Compression utilities for Arc ingestion. +"""Compression utilities for Arc ingestion (compatibility shim). -Arc supports gzip compression for ingestion payloads. Compressed payloads -are auto-detected by the magic bytes (0x1f 0x8b) at the start. +The canonical implementation now lives in :mod:`arc_client.compression`. This +module re-exports it so existing imports from ``arc_client.ingestion.compression`` +keep working. """ from __future__ import annotations -import gzip -import io - - -def compress_gzip(data: bytes, level: int = 6) -> bytes: - """Compress data using gzip. - - Args: - data: Raw bytes to compress. - level: Compression level (0-9). Default 6 is a good balance. - 0 = no compression, 9 = maximum compression. - - Returns: - Gzip compressed bytes. - - Example: - >>> raw = b'hello world' - >>> compressed = compress_gzip(raw) - >>> compressed[:2] # Magic bytes - b'\\x1f\\x8b' - """ - buf = io.BytesIO() - with gzip.GzipFile(fileobj=buf, mode="wb", compresslevel=level) as f: - f.write(data) - return buf.getvalue() - - -def decompress_gzip(data: bytes) -> bytes: - """Decompress gzip data. - - Args: - data: Gzip compressed bytes. - - Returns: - Decompressed bytes. - - Raises: - ValueError: If data is not valid gzip. - """ - if not is_gzipped(data): - raise ValueError("Data is not gzip compressed") - - buf = io.BytesIO(data) - with gzip.GzipFile(fileobj=buf, mode="rb") as f: - return f.read() - - -def is_gzipped(data: bytes) -> bool: - """Check if data is gzip compressed. - - Gzip data starts with magic bytes 0x1f 0x8b. - - Args: - data: Bytes to check. - - Returns: - True if data appears to be gzip compressed. - """ - return len(data) >= 2 and data[0] == 0x1F and data[1] == 0x8B +from arc_client.compression import ( + Compression, + compress, + compress_gzip, + compress_zstd, + decompress_gzip, + decompress_zstd, + is_gzipped, + is_zstd, +) + +__all__ = [ + "Compression", + "compress", + "compress_gzip", + "compress_zstd", + "decompress_gzip", + "decompress_zstd", + "is_gzipped", + "is_zstd", +] diff --git a/src/arc_client/ingestion/importer.py b/src/arc_client/ingestion/importer.py new file mode 100644 index 0000000..0039ecf --- /dev/null +++ b/src/arc_client/ingestion/importer.py @@ -0,0 +1,217 @@ +"""Synchronous bulk import client for Arc. + +Wraps the ``/api/v1/import/*`` endpoints (CSV, Parquet, line-protocol file, TLE) +which accept a ``multipart/form-data`` upload under the field name ``file`` and +respond with ``{"status": "ok", "result": {...}}``. All require an admin token. +""" + +from __future__ import annotations + +import os +from typing import IO, Any, Optional, Union, cast + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcIngestionError +from arc_client.http.sync_http import SyncHTTPClient +from arc_client.models.import_ import ( + ImportResult, + ImportStats, + LPImportResult, + TLEImportResult, +) + +# A file source: a path, a binary file-like object, or raw bytes. +FileSource = Union[str, "os.PathLike[str]", bytes, bytearray, IO[bytes]] + + +def _resolve_file(source: FileSource) -> tuple[str, Any]: + """Resolve a file source into an httpx multipart ``(filename, fileobj)`` pair. + + Returns a filename (best-effort, for the multipart part) and a bytes/file + object that httpx can stream. + """ + if isinstance(source, (bytes, bytearray)): + return ("upload", bytes(source)) + if isinstance(source, (str, os.PathLike)): + path = os.fspath(source) + # Read the file into memory; imports are capped at 500MB server-side. + with open(path, "rb") as fh: + return (os.path.basename(path) or "upload", fh.read()) + # Assume a binary file-like object. + name = getattr(source, "name", "upload") + filename = os.path.basename(name) if isinstance(name, str) else "upload" + return (filename, source.read()) + + +class ImportClient: + """Synchronous client for Arc bulk import endpoints. + + Example: + >>> result = client.import_.import_csv("data.csv", measurement="cpu") + >>> print(result.rows_imported) + """ + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _post_import( + self, + path: str, + source: FileSource, + params: dict[str, Any], + database: Optional[str], + extra_headers: Optional[dict[str, str]] = None, + ) -> dict[str, Any]: + """POST a multipart import request and return the ``result`` payload.""" + filename, fileobj = _resolve_file(source) + headers = dict(extra_headers or {}) + if database: + headers["x-arc-database"] = database + + # Drop params with None values so server defaults apply. + clean_params = {k: v for k, v in params.items() if v is not None} + + try: + response = self._http.post( + path, + files={"file": (filename, fileobj)}, + params=clean_params or None, + headers=headers or None, + ) + body = response.json() + except ArcIngestionError: + raise + except Exception as e: + raise ArcIngestionError(f"Import failed: {e}") from e + + result = body.get("result") if isinstance(body, dict) else None + if result is None: + raise ArcIngestionError(f"Unexpected import response: {body!r}") + return cast("dict[str, Any]", result) + + def import_csv( + self, + source: FileSource, + measurement: str, + database: Optional[str] = None, + time_column: str = "time", + time_format: str = "", + delimiter: str = ",", + skip_rows: int = 0, + ) -> ImportResult: + """Import a CSV file. + + Args: + source: Path, file-like object, or bytes of the CSV file. + measurement: Target measurement (required). + database: Target database (defaults to the client database). + time_column: Name of the timestamp column. Default "time". + time_format: "" (auto) or one of epoch_s/epoch_ms/epoch_us/epoch_ns. + delimiter: Field delimiter (single character). Default ",". + skip_rows: Rows to skip before the header. Default 0. + + Returns: + ImportResult with rows imported and time range. + """ + params = { + "measurement": measurement, + "time_column": time_column, + "time_format": time_format or None, + "delimiter": delimiter, + "skip_rows": skip_rows, + } + result = self._post_import( + "/api/v1/import/csv", source, params, database or self._config.database + ) + return ImportResult.model_validate(result) + + def import_parquet( + self, + source: FileSource, + measurement: str, + database: Optional[str] = None, + time_column: str = "time", + time_format: str = "", + ) -> ImportResult: + """Import a Parquet file. + + Args: + source: Path, file-like object, or bytes of the Parquet file. + measurement: Target measurement (required). + database: Target database (defaults to the client database). + time_column: Name of the timestamp column. Default "time". + time_format: "" (auto) or one of epoch_s/epoch_ms/epoch_us/epoch_ns. + + Returns: + ImportResult with rows imported and time range. + """ + params = { + "measurement": measurement, + "time_column": time_column, + "time_format": time_format or None, + } + result = self._post_import( + "/api/v1/import/parquet", source, params, database or self._config.database + ) + return ImportResult.model_validate(result) + + def import_lp( + self, + source: FileSource, + database: Optional[str] = None, + precision: str = "ns", + measurement: Optional[str] = None, + ) -> LPImportResult: + """Import an InfluxDB line-protocol file. + + Args: + source: Path, file-like object, or bytes of the LP file (may be gzip). + database: Target database (defaults to the client database). + precision: Timestamp precision - ns/us/ms/s. Default "ns". + measurement: Optional measurement filter; only matching lines import. + + Returns: + LPImportResult with rows imported and measurements touched. + """ + params = {"precision": precision, "measurement": measurement} + result = self._post_import( + "/api/v1/import/lp", source, params, database or self._config.database + ) + return LPImportResult.model_validate(result) + + def import_tle( + self, + source: FileSource, + database: Optional[str] = None, + measurement: str = "satellite_tle", + ) -> TLEImportResult: + """Import a TLE (Two-Line Element) file. + + Args: + source: Path, file-like object, or bytes of the TLE file (may be gzip). + database: Target database (defaults to the client database). + measurement: Target measurement (sent via x-arc-measurement header). + Default "satellite_tle". + + Returns: + TLEImportResult with satellite and row counts. + """ + result = self._post_import( + "/api/v1/import/tle", + source, + {}, + database or self._config.database, + extra_headers={"x-arc-measurement": measurement}, + ) + return TLEImportResult.model_validate(result) + + def import_stats(self) -> ImportStats: + """Get cumulative import statistics.""" + try: + response = self._http.get("/api/v1/import/stats") + body = response.json() + except Exception as e: + raise ArcIngestionError(f"Failed to get import stats: {e}") from e + stats = body.get("stats", body) if isinstance(body, dict) else {} + return ImportStats.model_validate(stats) diff --git a/src/arc_client/ingestion/msgpack.py b/src/arc_client/ingestion/msgpack.py index 7c61315..900ae4c 100644 --- a/src/arc_client/ingestion/msgpack.py +++ b/src/arc_client/ingestion/msgpack.py @@ -249,7 +249,11 @@ def dataframe_to_columnar( df: DataFrame to convert (pandas, polars, or pyarrow). measurement: Measurement name (not used in output, just for validation). time_column: Name of the timestamp column. - tag_columns: Columns to treat as tags (string dimensions). + tag_columns: Accepted for API stability but currently a no-op. The + columnar MessagePack write format has no tag/field distinction on the + wire (every non-time column is sent as a plain column); Arc derives + tag columns server-side from the data. Kept so callers can pass it + without breakage and so a future wire format can honor it. Returns: Dictionary of column name to list of values. @@ -291,8 +295,7 @@ def _pandas_to_columnar(df: Any, time_column: str, tag_columns: list[str]) -> di # Handle timestamp column if col_name == time_column: if pd.api.types.is_datetime64_any_dtype(col): - # Convert datetime to microseconds since epoch - columns["time"] = (col.astype("int64") // 1000).tolist() + columns["time"] = _pandas_datetime_to_micros(col) else: # Assume already numeric timestamps columns["time"] = col.tolist() @@ -303,8 +306,42 @@ def _pandas_to_columnar(df: Any, time_column: str, tag_columns: list[str]) -> di return columns +def _pandas_datetime_to_micros(col: Any) -> list[Any]: + """Convert a pandas datetime64 column to microseconds since epoch. + + The int64 view of a datetime64 column is in the column's own resolution + (ns/us/ms/s), so scale from that unit rather than assuming nanoseconds. + ``datetime64[us]`` must not be divided by 1000 the way ``datetime64[ns]`` is. + """ + import numpy as np + + dtype = col.dtype + # tz-aware columns use pandas' DatetimeTZDtype, which exposes ``.unit``. + # tz-naive columns use numpy datetime64, which does NOT — extract the unit + # from the numpy dtype instead of silently defaulting to nanoseconds. + unit = getattr(dtype, "unit", None) + if unit is None: + try: + unit = np.datetime_data(dtype)[0] + except (TypeError, ValueError): # pragma: no cover - non-datetime dtype + unit = "ns" + as_int = col.astype("int64") + # Scale factor to reach microseconds; negative means multiply (upscale). + scale = {"ns": 1_000, "us": 1, "ms": -1_000, "s": -1_000_000}.get(unit, 1_000) + if scale == 1: + values = as_int + elif scale > 0: + values = as_int // scale + else: + values = as_int * (-scale) + result: list[Any] = values.tolist() + return result + + def _polars_to_columnar(df: Any, time_column: str, tag_columns: list[str]) -> dict[str, list[Any]]: """Convert Polars DataFrame to columnar format.""" + import polars as pl + if time_column not in df.columns: raise ArcValidationError(f"Time column '{time_column}' not found in DataFrame") @@ -315,10 +352,21 @@ def _polars_to_columnar(df: Any, time_column: str, tag_columns: list[str]) -> di # Handle timestamp column if col_name == time_column: - # Check if it's a datetime type - if "datetime" in str(col.dtype).lower() or "date" in str(col.dtype).lower(): - # Convert to microseconds - columns["time"] = col.cast(df.__class__._from_arrow.__self__.Int64).to_list() + dtype = col.dtype + if dtype == pl.Datetime: + # Casting a Datetime to Int64 yields the underlying integer in + # the series' own time_unit. Normalize that to microseconds. + as_int = col.cast(pl.Int64) + time_unit = getattr(dtype, "time_unit", "us") + if time_unit == "ns": + columns["time"] = (as_int // 1_000).to_list() + elif time_unit == "ms": + columns["time"] = (as_int * 1_000).to_list() + else: # "us" + columns["time"] = as_int.to_list() + elif dtype == pl.Date: + # Date is days since epoch; convert to microseconds. + columns["time"] = (col.cast(pl.Int64) * 86_400_000_000).to_list() else: columns["time"] = col.to_list() else: diff --git a/src/arc_client/ingestion/writer.py b/src/arc_client/ingestion/writer.py index 5533a22..8b99f5a 100644 --- a/src/arc_client/ingestion/writer.py +++ b/src/arc_client/ingestion/writer.py @@ -2,12 +2,13 @@ from __future__ import annotations -from typing import TYPE_CHECKING, Any, Optional +from typing import TYPE_CHECKING, Any, Optional, Union +from arc_client.compression import Compression from arc_client.config import ClientConfig from arc_client.exceptions import ArcIngestionError, ArcValidationError from arc_client.http.sync_http import SyncHTTPClient -from arc_client.ingestion.compression import compress_gzip +from arc_client.ingestion.compression import compress as compress_data from arc_client.ingestion.line_protocol import format_line_protocol from arc_client.ingestion.msgpack import ( dataframe_to_columnar, @@ -15,6 +16,10 @@ encode_records, ) +# Compression argument accepted by write methods: a codec, its name, a bool +# (back-compat), or None to use the client default. +CompressArg = Optional[Union[Compression, str, bool]] + if TYPE_CHECKING: from arc_client.ingestion.buffered import BufferedWriter @@ -53,7 +58,7 @@ def write_columnar( measurement: str, columns: dict[str, list[Any]], database: Optional[str] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, time_unit: str = "us", ) -> None: """Write columnar data to Arc (highest performance method). @@ -98,7 +103,7 @@ def write_records( self, records: list[dict[str, Any]], database: Optional[str] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, ) -> None: """Write records to Arc using MessagePack row format. @@ -134,7 +139,7 @@ def write_dataframe( database: Optional[str] = None, time_column: str = "time", tag_columns: Optional[list[str]] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, ) -> None: """Write a DataFrame to Arc. @@ -175,7 +180,7 @@ def write_line_protocol( self, lines: str | list[str], database: Optional[str] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, ) -> None: """Write data using InfluxDB Line Protocol format. @@ -205,7 +210,7 @@ def write_point( tags: Optional[dict[str, str]] = None, timestamp: Optional[int] = None, database: Optional[str] = None, - compress: Optional[bool] = None, + compress: CompressArg = None, ) -> None: """Write a single data point using Line Protocol. @@ -222,6 +227,48 @@ def write_point( line = format_line_protocol(measurement, fields, tags, timestamp) self._write_line_protocol(line.encode("utf-8"), database, compress) + def write_tle( + self, + tle_text: str | bytes, + measurement: str = "satellite_tle", + database: Optional[str] = None, + compress: CompressArg = None, + ) -> None: + """Stream raw TLE (Two-Line Element) text to Arc. + + Sends the TLE text as-is to ``/api/v1/write/tle``; the server parses it + into orbital-element fields. For large historical files, prefer + ``client.import_.import_tle`` (bulk import). + + Args: + tle_text: TLE text (str or bytes). May contain many satellites. + measurement: Target measurement, sent via x-arc-measurement header. + Default "satellite_tle". + database: Target database. Uses client default if not specified. + compress: Compression codec. Uses client default if not specified. + + Raises: + ArcIngestionError: If the write fails. + """ + data = tle_text.encode("utf-8") if isinstance(tle_text, str) else tle_text + self._post_write( + "/api/v1/write/tle", + data, + "text/plain", + database, + compress, + extra_headers={"x-arc-measurement": measurement}, + ) + + def tle_stats(self) -> dict[str, Any]: + """Get TLE streaming write statistics from ``/api/v1/write/tle/stats``.""" + try: + response = self._http.get("/api/v1/write/tle/stats") + body = response.json() + except Exception as e: + raise ArcIngestionError(f"Failed to get TLE stats: {e}") from e + return body.get("stats", body) if isinstance(body, dict) else {} + def buffered( self, batch_size: int = 10000, @@ -251,33 +298,42 @@ def buffered( return BufferedWriter(self, batch_size, flush_interval) - def _write_msgpack( + def _resolve_codec(self, compress: CompressArg) -> Compression: + """Resolve the effective compression codec for a write. + + ``compress`` overrides the client default when provided. Accepts a + :class:`Compression`, a string codec name, or a bool (True->gzip, + False->none) for backward compatibility. + """ + if compress is None: + return Compression.coerce(self._config.compression) + return Compression.coerce(compress) + + def _post_write( self, + path: str, data: bytes, + content_type: str, database: Optional[str], - compress: Optional[bool], + compress: CompressArg, + extra_headers: Optional[dict[str, str]] = None, ) -> None: - """Send MessagePack data to Arc.""" - should_compress = compress if compress is not None else self._config.compression - - if should_compress: - data = compress_gzip(data) + """Compress (if requested) and POST a write payload. - headers = { - "Content-Type": "application/msgpack", - } - if should_compress: - headers["Content-Encoding"] = "gzip" + Arc auto-detects gzip/zstd from the payload's magic bytes, so no + ``Content-Encoding`` header is sent. + """ + codec = self._resolve_codec(compress) + data = compress_data(data, codec) + headers = {"Content-Type": content_type} + if extra_headers: + headers.update(extra_headers) if database: headers["x-arc-database"] = database try: - response = self._http.post( - "/api/v1/write/msgpack", - content=data, - headers=headers, - ) + response = self._http.post(path, content=data, headers=headers) # Arc returns 204 No Content on success if response.status_code not in (200, 204): raise ArcIngestionError( @@ -288,39 +344,20 @@ def _write_msgpack( except Exception as e: raise ArcIngestionError(f"Failed to write data: {e}") from e + def _write_msgpack( + self, + data: bytes, + database: Optional[str], + compress: CompressArg, + ) -> None: + """Send MessagePack data to Arc.""" + self._post_write("/api/v1/write/msgpack", data, "application/msgpack", database, compress) + def _write_line_protocol( self, data: bytes, database: Optional[str], - compress: Optional[bool], + compress: CompressArg, ) -> None: """Send Line Protocol data to Arc.""" - should_compress = compress if compress is not None else self._config.compression - - if should_compress: - data = compress_gzip(data) - - headers = { - "Content-Type": "text/plain", - } - if should_compress: - headers["Content-Encoding"] = "gzip" - - if database: - headers["x-arc-database"] = database - - try: - response = self._http.post( - "/api/v1/write/line-protocol", - content=data, - headers=headers, - ) - # Arc returns 204 No Content on success - if response.status_code not in (200, 204): - raise ArcIngestionError( - f"Write failed with status {response.status_code}: {response.text}" - ) - except ArcIngestionError: - raise - except Exception as e: - raise ArcIngestionError(f"Failed to write data: {e}") from e + self._post_write("/api/v1/write/line-protocol", data, "text/plain", database, compress) diff --git a/src/arc_client/management/__init__.py b/src/arc_client/management/__init__.py index e0090bb..f48e3af 100644 --- a/src/arc_client/management/__init__.py +++ b/src/arc_client/management/__init__.py @@ -1,17 +1,64 @@ -"""Management module for Arc client (retention, continuous queries, delete).""" +"""Management module for Arc client. +Retention, continuous queries, delete, databases, and the enterprise surface +(RBAC, governance, audit, query management, cluster, compaction, tiering, +backup, MQTT). +""" + +from arc_client.management.async_audit import AsyncAuditClient +from arc_client.management.async_backup import AsyncBackupClient +from arc_client.management.async_cluster import AsyncClusterClient +from arc_client.management.async_compaction import AsyncCompactionClient from arc_client.management.async_continuous_query import AsyncContinuousQueryClient +from arc_client.management.async_databases import AsyncDatabaseClient from arc_client.management.async_delete import AsyncDeleteClient +from arc_client.management.async_governance import AsyncGovernanceClient +from arc_client.management.async_mqtt import AsyncMQTTClient +from arc_client.management.async_query_management import AsyncQueryManagementClient +from arc_client.management.async_rbac import AsyncRBACClient from arc_client.management.async_retention import AsyncRetentionClient +from arc_client.management.async_tiering import AsyncTieringClient +from arc_client.management.audit import AuditClient +from arc_client.management.backup import BackupClient +from arc_client.management.cluster import ClusterClient +from arc_client.management.compaction import CompactionClient from arc_client.management.continuous_query import ContinuousQueryClient +from arc_client.management.databases import DatabaseClient from arc_client.management.delete import DeleteClient +from arc_client.management.governance import GovernanceClient +from arc_client.management.mqtt import MQTTClient +from arc_client.management.query_management import QueryManagementClient +from arc_client.management.rbac import RBACClient from arc_client.management.retention import RetentionClient +from arc_client.management.tiering import TieringClient __all__ = [ - "AsyncContinuousQueryClient", - "AsyncDeleteClient", - "AsyncRetentionClient", + # Core management "ContinuousQueryClient", + "AsyncContinuousQueryClient", "DeleteClient", + "AsyncDeleteClient", "RetentionClient", + "AsyncRetentionClient", + "DatabaseClient", + "AsyncDatabaseClient", + # Enterprise + "RBACClient", + "AsyncRBACClient", + "GovernanceClient", + "AsyncGovernanceClient", + "AuditClient", + "AsyncAuditClient", + "QueryManagementClient", + "AsyncQueryManagementClient", + "ClusterClient", + "AsyncClusterClient", + "CompactionClient", + "AsyncCompactionClient", + "TieringClient", + "AsyncTieringClient", + "BackupClient", + "AsyncBackupClient", + "MQTTClient", + "AsyncMQTTClient", ] diff --git a/src/arc_client/management/async_audit.py b/src/arc_client/management/async_audit.py new file mode 100644 index 0000000..0da3ae6 --- /dev/null +++ b/src/arc_client/management/async_audit.py @@ -0,0 +1,64 @@ +"""Asynchronous audit-log client for Arc (enterprise). + +Wraps ``/api/v1/audit``. Requires an admin token and the ``audit_logging`` +license feature. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.async_http import AsyncHTTPClient + + +class AsyncAuditClient: + """Query the audit log (enterprise).""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = await getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Audit request failed ({method.upper()} {path}): {e}") from e + + async def logs( + self, + event_type: Optional[str] = None, + actor: Optional[str] = None, + database: Optional[str] = None, + since: Optional[str] = None, + until: Optional[str] = None, + limit: Optional[int] = None, + offset: Optional[int] = None, + ) -> Any: + """Query audit log entries. + + Args: + event_type, actor, database: Optional exact-match filters. + since, until: RFC3339 time bounds. + limit, offset: Pagination. + """ + params = { + "event_type": event_type, + "actor": actor, + "database": database, + "since": since, + "until": until, + "limit": limit, + "offset": offset, + } + params = {k: v for k, v in params.items() if v is not None} + return await self._json("get", "/api/v1/audit/logs", params=params or None) + + async def stats(self, since: Optional[str] = None) -> Any: + """Get audit statistics (optionally since an RFC3339 timestamp).""" + params = {"since": since} if since else None + return await self._json("get", "/api/v1/audit/stats", params=params) diff --git a/src/arc_client/management/async_backup.py b/src/arc_client/management/async_backup.py new file mode 100644 index 0000000..16bb797 --- /dev/null +++ b/src/arc_client/management/async_backup.py @@ -0,0 +1,85 @@ +"""Asynchronous backup/restore client for Arc (enterprise). + +Wraps ``/api/v1/backup``. Requires an admin token. Backup/restore run +asynchronously server-side; poll :meth:`status` for progress. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError, ArcValidationError +from arc_client.http.async_http import AsyncHTTPClient + + +class AsyncBackupClient: + """Create, list, inspect, delete backups and restore from them (enterprise).""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = await getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Backup request failed ({method.upper()} {path}): {e}") from e + + async def create( + self, + include_metadata: Optional[bool] = None, + include_config: Optional[bool] = None, + ) -> Any: + """Start a backup (async). Returns 202 with a running status.""" + body: dict[str, Any] = {} + if include_metadata is not None: + body["include_metadata"] = include_metadata + if include_config is not None: + body["include_config"] = include_config + return await self._json("post", "/api/v1/backup/", json=body or None) + + async def list(self) -> Any: + """List available backups.""" + return await self._json("get", "/api/v1/backup/") + + async def status(self) -> Any: + """Get the current backup/restore progress (or idle).""" + return await self._json("get", "/api/v1/backup/status") + + async def get(self, backup_id: str) -> Any: + """Get a backup manifest by id.""" + return await self._json("get", f"/api/v1/backup/{backup_id}") + + async def delete(self, backup_id: str) -> Any: + """Delete a backup by id.""" + return await self._json("delete", f"/api/v1/backup/{backup_id}") + + async def restore( + self, + backup_id: str, + confirm: bool = False, + restore_data: Optional[bool] = None, + restore_metadata: Optional[bool] = None, + restore_config: Optional[bool] = None, + ) -> Any: + """Restore from a backup (async). Requires confirm=True. + + Raises: + ArcValidationError: If confirm is not True (restore is destructive). + """ + if not confirm: + raise ArcValidationError( + "Refusing to restore without confirm=True. This overwrites data." + ) + body: dict[str, Any] = {"backup_id": backup_id, "confirm": True} + if restore_data is not None: + body["restore_data"] = restore_data + if restore_metadata is not None: + body["restore_metadata"] = restore_metadata + if restore_config is not None: + body["restore_config"] = restore_config + return await self._json("post", "/api/v1/backup/restore", json=body) diff --git a/src/arc_client/management/async_cluster.py b/src/arc_client/management/async_cluster.py new file mode 100644 index 0000000..44e0327 --- /dev/null +++ b/src/arc_client/management/async_cluster.py @@ -0,0 +1,70 @@ +"""Asynchronous cluster-management client for Arc (enterprise). + +Wraps ``/api/v1/cluster``. Reads are available to any valid token; ``files`` and +node removal require an admin token. When clustering is disabled the server +returns 200 with ``{"enabled": false, ...}`` rather than an error. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.async_http import AsyncHTTPClient + + +class AsyncClusterClient: + """Inspect and manage the Arc cluster (enterprise).""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = await getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Cluster request failed ({method.upper()} {path}): {e}") from e + + async def status(self) -> Any: + """Get overall cluster status (includes enabled/mode/license).""" + return await self._json("get", "/api/v1/cluster") + + async def nodes(self, role: Optional[str] = None, state: Optional[str] = None) -> Any: + """List cluster nodes, optionally filtered by role and/or state.""" + params = {k: v for k, v in {"role": role, "state": state}.items() if v} + return await self._json("get", "/api/v1/cluster/nodes", params=params or None) + + async def node(self, node_id: str) -> Any: + """Get a single node by id.""" + return await self._json("get", f"/api/v1/cluster/nodes/{node_id}") + + async def local(self) -> Any: + """Get the local node's info and capabilities.""" + return await self._json("get", "/api/v1/cluster/local") + + async def health(self) -> Any: + """Get cluster health counts.""" + return await self._json("get", "/api/v1/cluster/health") + + async def files( + self, + database: Optional[str] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + ) -> Any: + """List cluster files (admin). Supports cursor pagination.""" + params = { + k: v + for k, v in {"database": database, "cursor": cursor, "limit": limit}.items() + if v is not None + } + return await self._json("get", "/api/v1/cluster/files", params=params or None) + + async def remove_node(self, node_id: str) -> Any: + """Remove a node from the cluster via Raft (admin).""" + return await self._json("delete", f"/api/v1/cluster/nodes/{node_id}") diff --git a/src/arc_client/management/async_compaction.py b/src/arc_client/management/async_compaction.py new file mode 100644 index 0000000..040f642 --- /dev/null +++ b/src/arc_client/management/async_compaction.py @@ -0,0 +1,61 @@ +"""Asynchronous compaction-management client for Arc. + +Wraps ``/api/v1/compaction``. Reads are available to any valid token; ``trigger`` +requires an admin token. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.async_http import AsyncHTTPClient + + +class AsyncCompactionClient: + """Inspect and trigger compaction.""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = await getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Compaction request failed ({method.upper()} {path}): {e}") from e + + async def status(self) -> Any: + """Get compaction manager and scheduler status.""" + return await self._json("get", "/api/v1/compaction/status") + + async def stats(self) -> Any: + """Get raw compaction statistics.""" + return await self._json("get", "/api/v1/compaction/stats") + + async def candidates(self) -> Any: + """List partitions that are candidates for compaction.""" + return await self._json("get", "/api/v1/compaction/candidates") + + async def jobs(self) -> Any: + """List active compaction jobs.""" + return await self._json("get", "/api/v1/compaction/jobs") + + async def history(self, limit: Optional[int] = None) -> Any: + """Get recent compaction jobs (default 10).""" + params = {"limit": limit} if limit is not None else None + return await self._json("get", "/api/v1/compaction/history", params=params) + + async def trigger(self, tier: Optional[str] = None, database: Optional[str] = None) -> Any: + """Trigger a compaction cycle (admin). + + Args: + tier: Comma-separated tiers (e.g. "hourly,daily"). Default all enabled. + database: Restrict to a single database. + """ + params = {k: v for k, v in {"tier": tier, "database": database}.items() if v} + return await self._json("post", "/api/v1/compaction/trigger", params=params or None) diff --git a/src/arc_client/management/async_databases.py b/src/arc_client/management/async_databases.py new file mode 100644 index 0000000..db61aa2 --- /dev/null +++ b/src/arc_client/management/async_databases.py @@ -0,0 +1,84 @@ +"""Asynchronous databases management client for Arc.""" + +from __future__ import annotations + +from typing import Any + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError, ArcValidationError +from arc_client.http.async_http import AsyncHTTPClient +from arc_client.models.database import ( + DatabaseInfo, + DatabaseListResponse, + MeasurementListResponse, +) +from arc_client.validation import validate_name + + +class AsyncDatabaseClient: + """Async twin of :class:`arc_client.management.databases.DatabaseClient`.""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def list(self) -> DatabaseListResponse: + """List all databases.""" + try: + response = await self._http.get("/api/v1/databases") + return DatabaseListResponse.model_validate(response.json()) + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to list databases: {e}") from e + + async def get(self, name: str) -> DatabaseInfo: + """Get a single database by name.""" + validate_name(name, "database") + try: + response = await self._http.get(f"/api/v1/databases/{name}") + return DatabaseInfo.model_validate(response.json()) + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to get database {name!r}: {e}") from e + + async def list_measurements(self, name: str) -> MeasurementListResponse: + """List measurements within a database.""" + validate_name(name, "database") + try: + response = await self._http.get(f"/api/v1/databases/{name}/measurements") + return MeasurementListResponse.model_validate(response.json()) + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to list measurements for {name!r}: {e}") from e + + async def create(self, name: str) -> DatabaseInfo: + """Create a database (admin).""" + validate_name(name, "database") + try: + response = await self._http.post("/api/v1/databases", json={"name": name}) + return DatabaseInfo.model_validate(response.json()) + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to create database {name!r}: {e}") from e + + async def delete(self, name: str, confirm: bool = False) -> dict[str, Any]: + """Delete a database (admin). Requires confirm=True.""" + validate_name(name, "database") + if not confirm: + raise ArcValidationError( + "Refusing to delete database without confirm=True. This is destructive." + ) + try: + response = await self._http.delete( + f"/api/v1/databases/{name}", params={"confirm": "true"} + ) + body: dict[str, Any] = response.json() + return body + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to delete database {name!r}: {e}") from e diff --git a/src/arc_client/management/async_governance.py b/src/arc_client/management/async_governance.py new file mode 100644 index 0000000..8eff8a5 --- /dev/null +++ b/src/arc_client/management/async_governance.py @@ -0,0 +1,61 @@ +"""Asynchronous query-governance client for Arc (enterprise). + +Wraps ``/api/v1/governance`` (per-token rate limits and quotas). Requires an +admin token and the ``query_governance`` license feature. +""" + +from __future__ import annotations + +from typing import Any + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.async_http import AsyncHTTPClient + + +class AsyncGovernanceClient: + """Manage per-token query governance policies (enterprise).""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = await getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Governance request failed ({method.upper()} {path}): {e}") from e + + async def list_policies(self) -> Any: + """List all governance policies.""" + return await self._json("get", "/api/v1/governance/policies") + + async def get_policy(self, token_id: int) -> Any: + """Get the policy for a token.""" + return await self._json("get", f"/api/v1/governance/policies/{token_id}") + + async def create_policy(self, token_id: int, **limits: Any) -> Any: + """Create a policy for a token. + + Limit fields (all optional, 0 = unlimited): rate_limit_per_minute, + rate_limit_per_hour, max_queries_per_hour, max_queries_per_day, + max_rows_per_query, max_scan_duration_sec. + """ + body = {"token_id": token_id, **limits} + return await self._json("post", "/api/v1/governance/policies", json=body) + + async def update_policy(self, token_id: int, **limits: Any) -> Any: + """Update the policy for a token (same limit fields as create).""" + body = {"token_id": token_id, **limits} + return await self._json("put", f"/api/v1/governance/policies/{token_id}", json=body) + + async def delete_policy(self, token_id: int) -> Any: + """Delete the policy for a token.""" + return await self._json("delete", f"/api/v1/governance/policies/{token_id}") + + async def usage(self, token_id: int) -> Any: + """Get current usage counters and the effective policy for a token.""" + return await self._json("get", f"/api/v1/governance/usage/{token_id}") diff --git a/src/arc_client/management/async_mqtt.py b/src/arc_client/management/async_mqtt.py new file mode 100644 index 0000000..7e8b7c4 --- /dev/null +++ b/src/arc_client/management/async_mqtt.py @@ -0,0 +1,85 @@ +"""Asynchronous MQTT ingestion-management client for Arc. + +Wraps ``/api/v1/mqtt`` and ``/api/v1/mqtt/subscriptions``. Requires an admin +token. When the MQTT subsystem is disabled the server returns 503. +""" + +from __future__ import annotations + +from typing import Any + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.async_http import AsyncHTTPClient + + +class AsyncMQTTClient: + """Manage MQTT ingestion subscriptions.""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = await getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"MQTT request failed ({method.upper()} {path}): {e}") from e + + async def stats(self) -> Any: + """Get MQTT subscription statistics.""" + return await self._json("get", "/api/v1/mqtt/stats") + + async def health(self) -> Any: + """Get MQTT subsystem health.""" + return await self._json("get", "/api/v1/mqtt/health") + + # --- Subscriptions --- + async def list_subscriptions(self) -> Any: + """List MQTT subscriptions.""" + return await self._json("get", "/api/v1/mqtt/subscriptions/") + + async def get_subscription(self, subscription_id: str) -> Any: + """Get a subscription by id.""" + return await self._json("get", f"/api/v1/mqtt/subscriptions/{subscription_id}") + + async def create_subscription(self, config: dict[str, Any]) -> Any: + """Create a subscription. + + Args: + config: The subscription body (name, broker, client_id, topics, qos, + database, tls_*, auto_start, topic_mapping, etc.). See the Arc + server docs for the full field list. + """ + return await self._json("post", "/api/v1/mqtt/subscriptions/", json=config) + + async def update_subscription(self, subscription_id: str, config: dict[str, Any]) -> Any: + """Update a subscription (fields are optional; 409 if it is running).""" + return await self._json("put", f"/api/v1/mqtt/subscriptions/{subscription_id}", json=config) + + async def delete_subscription(self, subscription_id: str) -> Any: + """Delete a subscription.""" + return await self._json("delete", f"/api/v1/mqtt/subscriptions/{subscription_id}") + + async def subscription_stats(self, subscription_id: str) -> Any: + """Get statistics for a single subscription.""" + return await self._json("get", f"/api/v1/mqtt/subscriptions/{subscription_id}/stats") + + async def start_subscription(self, subscription_id: str) -> Any: + """Start a subscription.""" + return await self._json("post", f"/api/v1/mqtt/subscriptions/{subscription_id}/start") + + async def stop_subscription(self, subscription_id: str) -> Any: + """Stop a subscription.""" + return await self._json("post", f"/api/v1/mqtt/subscriptions/{subscription_id}/stop") + + async def pause_subscription(self, subscription_id: str) -> Any: + """Pause a subscription.""" + return await self._json("post", f"/api/v1/mqtt/subscriptions/{subscription_id}/pause") + + async def restart_subscription(self, subscription_id: str) -> Any: + """Restart a subscription.""" + return await self._json("post", f"/api/v1/mqtt/subscriptions/{subscription_id}/restart") diff --git a/src/arc_client/management/async_query_management.py b/src/arc_client/management/async_query_management.py new file mode 100644 index 0000000..18b4d49 --- /dev/null +++ b/src/arc_client/management/async_query_management.py @@ -0,0 +1,47 @@ +"""Asynchronous query-management client for Arc (enterprise). + +Wraps ``/api/v1/queries`` (active queries, history, inspect, kill). Requires an +admin token and the ``query_management`` license feature. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.async_http import AsyncHTTPClient + + +class AsyncQueryManagementClient: + """Inspect and cancel running queries (enterprise).""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = await getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Query-management request failed ({method.upper()} {path}): {e}") from e + + async def active(self) -> Any: + """List currently-executing queries.""" + return await self._json("get", "/api/v1/queries/active") + + async def history(self, limit: Optional[int] = None) -> Any: + """List recently-completed queries (default 50, max 1000).""" + params = {"limit": limit} if limit is not None else None + return await self._json("get", "/api/v1/queries/history", params=params) + + async def get(self, query_id: str) -> Any: + """Get details for a single query by id.""" + return await self._json("get", f"/api/v1/queries/{query_id}") + + async def kill(self, query_id: str) -> Any: + """Cancel a running query by id.""" + return await self._json("delete", f"/api/v1/queries/{query_id}") diff --git a/src/arc_client/management/async_rbac.py b/src/arc_client/management/async_rbac.py new file mode 100644 index 0000000..a074155 --- /dev/null +++ b/src/arc_client/management/async_rbac.py @@ -0,0 +1,142 @@ +"""Asynchronous RBAC management client for Arc (enterprise). + +Wraps ``/api/v1/rbac`` (organizations -> teams -> roles -> measurement +permissions) plus the token/team membership routes under ``/api/v1/auth``. + +All routes require an admin token and the ``rbac`` enterprise license feature; +a missing feature raises :class:`arc_client.exceptions.ArcLicenseError`. + +Methods return the server's parsed JSON (dicts/lists) so newer server fields are +never dropped. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.async_http import AsyncHTTPClient + + +class AsyncRBACClient: + """Manage RBAC organizations, teams, roles, and permissions (enterprise).""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + fn = getattr(self._http, method) + response = await fn(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"RBAC request failed ({method.upper()} {path}): {e}") from e + + # --- Organizations --- + async def list_organizations(self) -> Any: + """List organizations.""" + return await self._json("get", "/api/v1/rbac/organizations") + + async def create_organization(self, name: str, description: Optional[str] = None) -> Any: + """Create an organization.""" + body: dict[str, Any] = {"name": name} + if description is not None: + body["description"] = description + return await self._json("post", "/api/v1/rbac/organizations", json=body) + + async def get_organization(self, org_id: int) -> Any: + """Get an organization (includes its teams).""" + return await self._json("get", f"/api/v1/rbac/organizations/{org_id}") + + async def update_organization(self, org_id: int, **fields: Any) -> Any: + """Update an organization (name/description/enabled).""" + return await self._json("patch", f"/api/v1/rbac/organizations/{org_id}", json=fields) + + async def delete_organization(self, org_id: int) -> Any: + """Delete an organization.""" + return await self._json("delete", f"/api/v1/rbac/organizations/{org_id}") + + # --- Teams --- + async def list_teams(self, org_id: int) -> Any: + """List teams in an organization.""" + return await self._json("get", f"/api/v1/rbac/organizations/{org_id}/teams") + + async def create_team(self, org_id: int, name: str, description: Optional[str] = None) -> Any: + """Create a team in an organization.""" + body: dict[str, Any] = {"name": name} + if description is not None: + body["description"] = description + return await self._json("post", f"/api/v1/rbac/organizations/{org_id}/teams", json=body) + + async def get_team(self, team_id: int) -> Any: + """Get a team (includes its roles).""" + return await self._json("get", f"/api/v1/rbac/teams/{team_id}") + + async def update_team(self, team_id: int, **fields: Any) -> Any: + """Update a team (name/description/enabled).""" + return await self._json("patch", f"/api/v1/rbac/teams/{team_id}", json=fields) + + async def delete_team(self, team_id: int) -> Any: + """Delete a team.""" + return await self._json("delete", f"/api/v1/rbac/teams/{team_id}") + + # --- Roles --- + async def list_roles(self, team_id: int) -> Any: + """List roles in a team.""" + return await self._json("get", f"/api/v1/rbac/teams/{team_id}/roles") + + async def create_role(self, team_id: int, database_pattern: str, permissions: list[str]) -> Any: + """Create a role in a team.""" + body = {"database_pattern": database_pattern, "permissions": permissions} + return await self._json("post", f"/api/v1/rbac/teams/{team_id}/roles", json=body) + + async def get_role(self, role_id: int) -> Any: + """Get a role (includes measurement permissions).""" + return await self._json("get", f"/api/v1/rbac/roles/{role_id}") + + async def update_role(self, role_id: int, **fields: Any) -> Any: + """Update a role (database_pattern/permissions).""" + return await self._json("patch", f"/api/v1/rbac/roles/{role_id}", json=fields) + + async def delete_role(self, role_id: int) -> Any: + """Delete a role.""" + return await self._json("delete", f"/api/v1/rbac/roles/{role_id}") + + # --- Measurement permissions --- + async def list_measurement_permissions(self, role_id: int) -> Any: + """List measurement permissions for a role.""" + return await self._json("get", f"/api/v1/rbac/roles/{role_id}/measurements") + + async def create_measurement_permission( + self, role_id: int, measurement_pattern: str, permissions: list[str] + ) -> Any: + """Create a measurement permission for a role.""" + body = {"measurement_pattern": measurement_pattern, "permissions": permissions} + return await self._json("post", f"/api/v1/rbac/roles/{role_id}/measurements", json=body) + + async def delete_measurement_permission(self, permission_id: int) -> Any: + """Delete a measurement permission.""" + return await self._json("delete", f"/api/v1/rbac/measurement-permissions/{permission_id}") + + # --- Token / team membership (under /api/v1/auth) --- + async def list_token_teams(self, token_id: int) -> Any: + """List the teams a token belongs to.""" + return await self._json("get", f"/api/v1/auth/tokens/{token_id}/teams") + + async def add_token_to_team(self, token_id: int, team_id: int) -> Any: + """Add a token to a team.""" + return await self._json( + "post", f"/api/v1/auth/tokens/{token_id}/teams", json={"team_id": team_id} + ) + + async def remove_token_from_team(self, token_id: int, team_id: int) -> Any: + """Remove a token from a team.""" + return await self._json("delete", f"/api/v1/auth/tokens/{token_id}/teams/{team_id}") + + async def get_token_permissions(self, token_id: int) -> Any: + """Get the effective permissions for a token.""" + return await self._json("get", f"/api/v1/auth/tokens/{token_id}/permissions") diff --git a/src/arc_client/management/async_tiering.py b/src/arc_client/management/async_tiering.py new file mode 100644 index 0000000..408af87 --- /dev/null +++ b/src/arc_client/management/async_tiering.py @@ -0,0 +1,111 @@ +"""Asynchronous storage-tiering client for Arc (enterprise). + +Wraps ``/api/v1/tiering`` and ``/api/v1/tiering/policies``. Requires an admin +token and the ``tiering`` license feature. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.async_http import AsyncHTTPClient +from arc_client.validation import validate_name + + +class AsyncTieringClient: + """Manage hot/cold storage tiering and per-database policies (enterprise).""" + + def __init__(self, http: AsyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + async def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = await getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Tiering request failed ({method.upper()} {path}): {e}") from e + + async def status(self) -> Any: + """Get tiering status.""" + return await self._json("get", "/api/v1/tiering/status") + + async def files( + self, + tier: Optional[str] = None, + database: Optional[str] = None, + limit: Optional[int] = None, + ) -> Any: + """List tiered files, optionally filtered by tier/database.""" + params = { + k: v + for k, v in {"tier": tier, "database": database, "limit": limit}.items() + if v is not None + } + return await self._json("get", "/api/v1/tiering/files", params=params or None) + + async def migrate( + self, + from_tier: Optional[str] = None, + to_tier: Optional[str] = None, + database: Optional[str] = None, + measurement: Optional[str] = None, + dry_run: bool = False, + ) -> Any: + """Trigger a tier-migration cycle.""" + body = { + "from_tier": from_tier, + "to_tier": to_tier, + "database": database, + "measurement": measurement, + "dry_run": dry_run, + } + body = {k: v for k, v in body.items() if v is not None} + return await self._json("post", "/api/v1/tiering/migrate", json=body or None) + + async def stats(self) -> Any: + """Get tier statistics and recent migrations.""" + return await self._json("get", "/api/v1/tiering/stats") + + async def scan(self) -> Any: + """Scan the hot tier and register any unregistered files.""" + return await self._json("post", "/api/v1/tiering/scan") + + # --- Per-database policies --- + async def list_policies(self) -> Any: + """List per-database tiering policies.""" + return await self._json("get", "/api/v1/tiering/policies/") + + async def get_policy(self, database: str) -> Any: + """Get the custom tiering policy for a database (404 if none).""" + validate_name(database, "database") + return await self._json("get", f"/api/v1/tiering/policies/{database}") + + async def set_policy( + self, + database: str, + hot_only: Optional[bool] = None, + hot_max_age_days: Optional[int] = None, + ) -> Any: + """Create or update a database's tiering policy.""" + validate_name(database, "database") + body: dict[str, Any] = {} + if hot_only is not None: + body["hot_only"] = hot_only + if hot_max_age_days is not None: + body["hot_max_age_days"] = hot_max_age_days + return await self._json("put", f"/api/v1/tiering/policies/{database}", json=body) + + async def delete_policy(self, database: str) -> Any: + """Delete a database's custom policy (revert to global defaults).""" + validate_name(database, "database") + return await self._json("delete", f"/api/v1/tiering/policies/{database}") + + async def effective_policy(self, database: str) -> Any: + """Get the effective (resolved) tiering policy for a database.""" + validate_name(database, "database") + return await self._json("get", f"/api/v1/tiering/policies/{database}/effective") diff --git a/src/arc_client/management/audit.py b/src/arc_client/management/audit.py new file mode 100644 index 0000000..6d68739 --- /dev/null +++ b/src/arc_client/management/audit.py @@ -0,0 +1,64 @@ +"""Synchronous audit-log client for Arc (enterprise). + +Wraps ``/api/v1/audit``. Requires an admin token and the ``audit_logging`` +license feature. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.sync_http import SyncHTTPClient + + +class AuditClient: + """Query the audit log (enterprise).""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Audit request failed ({method.upper()} {path}): {e}") from e + + def logs( + self, + event_type: Optional[str] = None, + actor: Optional[str] = None, + database: Optional[str] = None, + since: Optional[str] = None, + until: Optional[str] = None, + limit: Optional[int] = None, + offset: Optional[int] = None, + ) -> Any: + """Query audit log entries. + + Args: + event_type, actor, database: Optional exact-match filters. + since, until: RFC3339 time bounds. + limit, offset: Pagination. + """ + params = { + "event_type": event_type, + "actor": actor, + "database": database, + "since": since, + "until": until, + "limit": limit, + "offset": offset, + } + params = {k: v for k, v in params.items() if v is not None} + return self._json("get", "/api/v1/audit/logs", params=params or None) + + def stats(self, since: Optional[str] = None) -> Any: + """Get audit statistics (optionally since an RFC3339 timestamp).""" + params = {"since": since} if since else None + return self._json("get", "/api/v1/audit/stats", params=params) diff --git a/src/arc_client/management/backup.py b/src/arc_client/management/backup.py new file mode 100644 index 0000000..119cd00 --- /dev/null +++ b/src/arc_client/management/backup.py @@ -0,0 +1,85 @@ +"""Synchronous backup/restore client for Arc (enterprise). + +Wraps ``/api/v1/backup``. Requires an admin token. Backup/restore run +asynchronously server-side; poll :meth:`status` for progress. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError, ArcValidationError +from arc_client.http.sync_http import SyncHTTPClient + + +class BackupClient: + """Create, list, inspect, delete backups and restore from them (enterprise).""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Backup request failed ({method.upper()} {path}): {e}") from e + + def create( + self, + include_metadata: Optional[bool] = None, + include_config: Optional[bool] = None, + ) -> Any: + """Start a backup (async). Returns 202 with a running status.""" + body: dict[str, Any] = {} + if include_metadata is not None: + body["include_metadata"] = include_metadata + if include_config is not None: + body["include_config"] = include_config + return self._json("post", "/api/v1/backup/", json=body or None) + + def list(self) -> Any: + """List available backups.""" + return self._json("get", "/api/v1/backup/") + + def status(self) -> Any: + """Get the current backup/restore progress (or idle).""" + return self._json("get", "/api/v1/backup/status") + + def get(self, backup_id: str) -> Any: + """Get a backup manifest by id.""" + return self._json("get", f"/api/v1/backup/{backup_id}") + + def delete(self, backup_id: str) -> Any: + """Delete a backup by id.""" + return self._json("delete", f"/api/v1/backup/{backup_id}") + + def restore( + self, + backup_id: str, + confirm: bool = False, + restore_data: Optional[bool] = None, + restore_metadata: Optional[bool] = None, + restore_config: Optional[bool] = None, + ) -> Any: + """Restore from a backup (async). Requires confirm=True. + + Raises: + ArcValidationError: If confirm is not True (restore is destructive). + """ + if not confirm: + raise ArcValidationError( + "Refusing to restore without confirm=True. This overwrites data." + ) + body: dict[str, Any] = {"backup_id": backup_id, "confirm": True} + if restore_data is not None: + body["restore_data"] = restore_data + if restore_metadata is not None: + body["restore_metadata"] = restore_metadata + if restore_config is not None: + body["restore_config"] = restore_config + return self._json("post", "/api/v1/backup/restore", json=body) diff --git a/src/arc_client/management/cluster.py b/src/arc_client/management/cluster.py new file mode 100644 index 0000000..554d173 --- /dev/null +++ b/src/arc_client/management/cluster.py @@ -0,0 +1,70 @@ +"""Synchronous cluster-management client for Arc (enterprise). + +Wraps ``/api/v1/cluster``. Reads are available to any valid token; ``files`` and +node removal require an admin token. When clustering is disabled the server +returns 200 with ``{"enabled": false, ...}`` rather than an error. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.sync_http import SyncHTTPClient + + +class ClusterClient: + """Inspect and manage the Arc cluster (enterprise).""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Cluster request failed ({method.upper()} {path}): {e}") from e + + def status(self) -> Any: + """Get overall cluster status (includes enabled/mode/license).""" + return self._json("get", "/api/v1/cluster") + + def nodes(self, role: Optional[str] = None, state: Optional[str] = None) -> Any: + """List cluster nodes, optionally filtered by role and/or state.""" + params = {k: v for k, v in {"role": role, "state": state}.items() if v} + return self._json("get", "/api/v1/cluster/nodes", params=params or None) + + def node(self, node_id: str) -> Any: + """Get a single node by id.""" + return self._json("get", f"/api/v1/cluster/nodes/{node_id}") + + def local(self) -> Any: + """Get the local node's info and capabilities.""" + return self._json("get", "/api/v1/cluster/local") + + def health(self) -> Any: + """Get cluster health counts.""" + return self._json("get", "/api/v1/cluster/health") + + def files( + self, + database: Optional[str] = None, + cursor: Optional[str] = None, + limit: Optional[int] = None, + ) -> Any: + """List cluster files (admin). Supports cursor pagination.""" + params = { + k: v + for k, v in {"database": database, "cursor": cursor, "limit": limit}.items() + if v is not None + } + return self._json("get", "/api/v1/cluster/files", params=params or None) + + def remove_node(self, node_id: str) -> Any: + """Remove a node from the cluster via Raft (admin).""" + return self._json("delete", f"/api/v1/cluster/nodes/{node_id}") diff --git a/src/arc_client/management/compaction.py b/src/arc_client/management/compaction.py new file mode 100644 index 0000000..2add4bb --- /dev/null +++ b/src/arc_client/management/compaction.py @@ -0,0 +1,61 @@ +"""Synchronous compaction-management client for Arc. + +Wraps ``/api/v1/compaction``. Reads are available to any valid token; ``trigger`` +requires an admin token. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.sync_http import SyncHTTPClient + + +class CompactionClient: + """Inspect and trigger compaction.""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Compaction request failed ({method.upper()} {path}): {e}") from e + + def status(self) -> Any: + """Get compaction manager and scheduler status.""" + return self._json("get", "/api/v1/compaction/status") + + def stats(self) -> Any: + """Get raw compaction statistics.""" + return self._json("get", "/api/v1/compaction/stats") + + def candidates(self) -> Any: + """List partitions that are candidates for compaction.""" + return self._json("get", "/api/v1/compaction/candidates") + + def jobs(self) -> Any: + """List active compaction jobs.""" + return self._json("get", "/api/v1/compaction/jobs") + + def history(self, limit: Optional[int] = None) -> Any: + """Get recent compaction jobs (default 10).""" + params = {"limit": limit} if limit is not None else None + return self._json("get", "/api/v1/compaction/history", params=params) + + def trigger(self, tier: Optional[str] = None, database: Optional[str] = None) -> Any: + """Trigger a compaction cycle (admin). + + Args: + tier: Comma-separated tiers (e.g. "hourly,daily"). Default all enabled. + database: Restrict to a single database. + """ + params = {k: v for k, v in {"tier": tier, "database": database}.items() if v} + return self._json("post", "/api/v1/compaction/trigger", params=params or None) diff --git a/src/arc_client/management/databases.py b/src/arc_client/management/databases.py new file mode 100644 index 0000000..b89ccb5 --- /dev/null +++ b/src/arc_client/management/databases.py @@ -0,0 +1,103 @@ +"""Synchronous databases management client for Arc.""" + +from __future__ import annotations + +from typing import Any + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError, ArcValidationError +from arc_client.http.sync_http import SyncHTTPClient +from arc_client.models.database import ( + DatabaseInfo, + DatabaseListResponse, + MeasurementListResponse, +) +from arc_client.validation import validate_name + + +class DatabaseClient: + """Manage Arc databases (list/get/create/delete). + + Create and delete require an admin token. Delete additionally requires the + server to have delete operations enabled and an explicit ``confirm=True``. + + Example: + >>> client.databases.create("metrics") + >>> for db in client.databases.list().databases: + ... print(db.name, db.measurement_count) + """ + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def list(self) -> DatabaseListResponse: + """List all databases.""" + try: + response = self._http.get("/api/v1/databases") + return DatabaseListResponse.model_validate(response.json()) + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to list databases: {e}") from e + + def get(self, name: str) -> DatabaseInfo: + """Get a single database by name.""" + validate_name(name, "database") + try: + response = self._http.get(f"/api/v1/databases/{name}") + return DatabaseInfo.model_validate(response.json()) + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to get database {name!r}: {e}") from e + + def list_measurements(self, name: str) -> MeasurementListResponse: + """List measurements within a database.""" + validate_name(name, "database") + try: + response = self._http.get(f"/api/v1/databases/{name}/measurements") + return MeasurementListResponse.model_validate(response.json()) + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to list measurements for {name!r}: {e}") from e + + def create(self, name: str) -> DatabaseInfo: + """Create a database (admin). + + Raises: + ArcValidationError: If the name is invalid. + ArcError: If creation fails (e.g. the database already exists, 409). + """ + validate_name(name, "database") + try: + response = self._http.post("/api/v1/databases", json={"name": name}) + return DatabaseInfo.model_validate(response.json()) + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to create database {name!r}: {e}") from e + + def delete(self, name: str, confirm: bool = False) -> dict[str, Any]: + """Delete a database (admin). + + Requires ``confirm=True`` and server-side delete operations to be + enabled (otherwise the server returns 403). + + Returns: + The server's response dict, e.g. ``{"message": ..., "files_deleted": N}``. + """ + validate_name(name, "database") + if not confirm: + raise ArcValidationError( + "Refusing to delete database without confirm=True. This is destructive." + ) + try: + response = self._http.delete(f"/api/v1/databases/{name}", params={"confirm": "true"}) + body: dict[str, Any] = response.json() + return body + except ArcError: + raise + except Exception as e: + raise ArcError(f"Failed to delete database {name!r}: {e}") from e diff --git a/src/arc_client/management/governance.py b/src/arc_client/management/governance.py new file mode 100644 index 0000000..596a4db --- /dev/null +++ b/src/arc_client/management/governance.py @@ -0,0 +1,61 @@ +"""Synchronous query-governance client for Arc (enterprise). + +Wraps ``/api/v1/governance`` (per-token rate limits and quotas). Requires an +admin token and the ``query_governance`` license feature. +""" + +from __future__ import annotations + +from typing import Any + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.sync_http import SyncHTTPClient + + +class GovernanceClient: + """Manage per-token query governance policies (enterprise).""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Governance request failed ({method.upper()} {path}): {e}") from e + + def list_policies(self) -> Any: + """List all governance policies.""" + return self._json("get", "/api/v1/governance/policies") + + def get_policy(self, token_id: int) -> Any: + """Get the policy for a token.""" + return self._json("get", f"/api/v1/governance/policies/{token_id}") + + def create_policy(self, token_id: int, **limits: Any) -> Any: + """Create a policy for a token. + + Limit fields (all optional, 0 = unlimited): rate_limit_per_minute, + rate_limit_per_hour, max_queries_per_hour, max_queries_per_day, + max_rows_per_query, max_scan_duration_sec. + """ + body = {"token_id": token_id, **limits} + return self._json("post", "/api/v1/governance/policies", json=body) + + def update_policy(self, token_id: int, **limits: Any) -> Any: + """Update the policy for a token (same limit fields as create).""" + body = {"token_id": token_id, **limits} + return self._json("put", f"/api/v1/governance/policies/{token_id}", json=body) + + def delete_policy(self, token_id: int) -> Any: + """Delete the policy for a token.""" + return self._json("delete", f"/api/v1/governance/policies/{token_id}") + + def usage(self, token_id: int) -> Any: + """Get current usage counters and the effective policy for a token.""" + return self._json("get", f"/api/v1/governance/usage/{token_id}") diff --git a/src/arc_client/management/mqtt.py b/src/arc_client/management/mqtt.py new file mode 100644 index 0000000..8c1f2ba --- /dev/null +++ b/src/arc_client/management/mqtt.py @@ -0,0 +1,85 @@ +"""Synchronous MQTT ingestion-management client for Arc. + +Wraps ``/api/v1/mqtt`` and ``/api/v1/mqtt/subscriptions``. Requires an admin +token. When the MQTT subsystem is disabled the server returns 503. +""" + +from __future__ import annotations + +from typing import Any + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.sync_http import SyncHTTPClient + + +class MQTTClient: + """Manage MQTT ingestion subscriptions.""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"MQTT request failed ({method.upper()} {path}): {e}") from e + + def stats(self) -> Any: + """Get MQTT subscription statistics.""" + return self._json("get", "/api/v1/mqtt/stats") + + def health(self) -> Any: + """Get MQTT subsystem health.""" + return self._json("get", "/api/v1/mqtt/health") + + # --- Subscriptions --- + def list_subscriptions(self) -> Any: + """List MQTT subscriptions.""" + return self._json("get", "/api/v1/mqtt/subscriptions/") + + def get_subscription(self, subscription_id: str) -> Any: + """Get a subscription by id.""" + return self._json("get", f"/api/v1/mqtt/subscriptions/{subscription_id}") + + def create_subscription(self, config: dict[str, Any]) -> Any: + """Create a subscription. + + Args: + config: The subscription body (name, broker, client_id, topics, qos, + database, tls_*, auto_start, topic_mapping, etc.). See the Arc + server docs for the full field list. + """ + return self._json("post", "/api/v1/mqtt/subscriptions/", json=config) + + def update_subscription(self, subscription_id: str, config: dict[str, Any]) -> Any: + """Update a subscription (fields are optional; 409 if it is running).""" + return self._json("put", f"/api/v1/mqtt/subscriptions/{subscription_id}", json=config) + + def delete_subscription(self, subscription_id: str) -> Any: + """Delete a subscription.""" + return self._json("delete", f"/api/v1/mqtt/subscriptions/{subscription_id}") + + def subscription_stats(self, subscription_id: str) -> Any: + """Get statistics for a single subscription.""" + return self._json("get", f"/api/v1/mqtt/subscriptions/{subscription_id}/stats") + + def start_subscription(self, subscription_id: str) -> Any: + """Start a subscription.""" + return self._json("post", f"/api/v1/mqtt/subscriptions/{subscription_id}/start") + + def stop_subscription(self, subscription_id: str) -> Any: + """Stop a subscription.""" + return self._json("post", f"/api/v1/mqtt/subscriptions/{subscription_id}/stop") + + def pause_subscription(self, subscription_id: str) -> Any: + """Pause a subscription.""" + return self._json("post", f"/api/v1/mqtt/subscriptions/{subscription_id}/pause") + + def restart_subscription(self, subscription_id: str) -> Any: + """Restart a subscription.""" + return self._json("post", f"/api/v1/mqtt/subscriptions/{subscription_id}/restart") diff --git a/src/arc_client/management/query_management.py b/src/arc_client/management/query_management.py new file mode 100644 index 0000000..1392841 --- /dev/null +++ b/src/arc_client/management/query_management.py @@ -0,0 +1,47 @@ +"""Synchronous query-management client for Arc (enterprise). + +Wraps ``/api/v1/queries`` (active queries, history, inspect, kill). Requires an +admin token and the ``query_management`` license feature. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.sync_http import SyncHTTPClient + + +class QueryManagementClient: + """Inspect and cancel running queries (enterprise).""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Query-management request failed ({method.upper()} {path}): {e}") from e + + def active(self) -> Any: + """List currently-executing queries.""" + return self._json("get", "/api/v1/queries/active") + + def history(self, limit: Optional[int] = None) -> Any: + """List recently-completed queries (default 50, max 1000).""" + params = {"limit": limit} if limit is not None else None + return self._json("get", "/api/v1/queries/history", params=params) + + def get(self, query_id: str) -> Any: + """Get details for a single query by id.""" + return self._json("get", f"/api/v1/queries/{query_id}") + + def kill(self, query_id: str) -> Any: + """Cancel a running query by id.""" + return self._json("delete", f"/api/v1/queries/{query_id}") diff --git a/src/arc_client/management/rbac.py b/src/arc_client/management/rbac.py new file mode 100644 index 0000000..a4f796a --- /dev/null +++ b/src/arc_client/management/rbac.py @@ -0,0 +1,142 @@ +"""Synchronous RBAC management client for Arc (enterprise). + +Wraps ``/api/v1/rbac`` (organizations -> teams -> roles -> measurement +permissions) plus the token/team membership routes under ``/api/v1/auth``. + +All routes require an admin token and the ``rbac`` enterprise license feature; +a missing feature raises :class:`arc_client.exceptions.ArcLicenseError`. + +Methods return the server's parsed JSON (dicts/lists) so newer server fields are +never dropped. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.sync_http import SyncHTTPClient + + +class RBACClient: + """Manage RBAC organizations, teams, roles, and permissions (enterprise).""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + fn = getattr(self._http, method) + response = fn(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"RBAC request failed ({method.upper()} {path}): {e}") from e + + # --- Organizations --- + def list_organizations(self) -> Any: + """List organizations.""" + return self._json("get", "/api/v1/rbac/organizations") + + def create_organization(self, name: str, description: Optional[str] = None) -> Any: + """Create an organization.""" + body: dict[str, Any] = {"name": name} + if description is not None: + body["description"] = description + return self._json("post", "/api/v1/rbac/organizations", json=body) + + def get_organization(self, org_id: int) -> Any: + """Get an organization (includes its teams).""" + return self._json("get", f"/api/v1/rbac/organizations/{org_id}") + + def update_organization(self, org_id: int, **fields: Any) -> Any: + """Update an organization (name/description/enabled).""" + return self._json("patch", f"/api/v1/rbac/organizations/{org_id}", json=fields) + + def delete_organization(self, org_id: int) -> Any: + """Delete an organization.""" + return self._json("delete", f"/api/v1/rbac/organizations/{org_id}") + + # --- Teams --- + def list_teams(self, org_id: int) -> Any: + """List teams in an organization.""" + return self._json("get", f"/api/v1/rbac/organizations/{org_id}/teams") + + def create_team(self, org_id: int, name: str, description: Optional[str] = None) -> Any: + """Create a team in an organization.""" + body: dict[str, Any] = {"name": name} + if description is not None: + body["description"] = description + return self._json("post", f"/api/v1/rbac/organizations/{org_id}/teams", json=body) + + def get_team(self, team_id: int) -> Any: + """Get a team (includes its roles).""" + return self._json("get", f"/api/v1/rbac/teams/{team_id}") + + def update_team(self, team_id: int, **fields: Any) -> Any: + """Update a team (name/description/enabled).""" + return self._json("patch", f"/api/v1/rbac/teams/{team_id}", json=fields) + + def delete_team(self, team_id: int) -> Any: + """Delete a team.""" + return self._json("delete", f"/api/v1/rbac/teams/{team_id}") + + # --- Roles --- + def list_roles(self, team_id: int) -> Any: + """List roles in a team.""" + return self._json("get", f"/api/v1/rbac/teams/{team_id}/roles") + + def create_role(self, team_id: int, database_pattern: str, permissions: list[str]) -> Any: + """Create a role in a team.""" + body = {"database_pattern": database_pattern, "permissions": permissions} + return self._json("post", f"/api/v1/rbac/teams/{team_id}/roles", json=body) + + def get_role(self, role_id: int) -> Any: + """Get a role (includes measurement permissions).""" + return self._json("get", f"/api/v1/rbac/roles/{role_id}") + + def update_role(self, role_id: int, **fields: Any) -> Any: + """Update a role (database_pattern/permissions).""" + return self._json("patch", f"/api/v1/rbac/roles/{role_id}", json=fields) + + def delete_role(self, role_id: int) -> Any: + """Delete a role.""" + return self._json("delete", f"/api/v1/rbac/roles/{role_id}") + + # --- Measurement permissions --- + def list_measurement_permissions(self, role_id: int) -> Any: + """List measurement permissions for a role.""" + return self._json("get", f"/api/v1/rbac/roles/{role_id}/measurements") + + def create_measurement_permission( + self, role_id: int, measurement_pattern: str, permissions: list[str] + ) -> Any: + """Create a measurement permission for a role.""" + body = {"measurement_pattern": measurement_pattern, "permissions": permissions} + return self._json("post", f"/api/v1/rbac/roles/{role_id}/measurements", json=body) + + def delete_measurement_permission(self, permission_id: int) -> Any: + """Delete a measurement permission.""" + return self._json("delete", f"/api/v1/rbac/measurement-permissions/{permission_id}") + + # --- Token / team membership (under /api/v1/auth) --- + def list_token_teams(self, token_id: int) -> Any: + """List the teams a token belongs to.""" + return self._json("get", f"/api/v1/auth/tokens/{token_id}/teams") + + def add_token_to_team(self, token_id: int, team_id: int) -> Any: + """Add a token to a team.""" + return self._json( + "post", f"/api/v1/auth/tokens/{token_id}/teams", json={"team_id": team_id} + ) + + def remove_token_from_team(self, token_id: int, team_id: int) -> Any: + """Remove a token from a team.""" + return self._json("delete", f"/api/v1/auth/tokens/{token_id}/teams/{team_id}") + + def get_token_permissions(self, token_id: int) -> Any: + """Get the effective permissions for a token.""" + return self._json("get", f"/api/v1/auth/tokens/{token_id}/permissions") diff --git a/src/arc_client/management/tiering.py b/src/arc_client/management/tiering.py new file mode 100644 index 0000000..488cd08 --- /dev/null +++ b/src/arc_client/management/tiering.py @@ -0,0 +1,111 @@ +"""Synchronous storage-tiering client for Arc (enterprise). + +Wraps ``/api/v1/tiering`` and ``/api/v1/tiering/policies``. Requires an admin +token and the ``tiering`` license feature. +""" + +from __future__ import annotations + +from typing import Any, Optional + +from arc_client.config import ClientConfig +from arc_client.exceptions import ArcError +from arc_client.http.sync_http import SyncHTTPClient +from arc_client.validation import validate_name + + +class TieringClient: + """Manage hot/cold storage tiering and per-database policies (enterprise).""" + + def __init__(self, http: SyncHTTPClient, config: ClientConfig) -> None: + self._http = http + self._config = config + + def _json(self, method: str, path: str, **kwargs: Any) -> Any: + try: + response = getattr(self._http, method)(path, **kwargs) + return response.json() + except ArcError: + raise + except Exception as e: + raise ArcError(f"Tiering request failed ({method.upper()} {path}): {e}") from e + + def status(self) -> Any: + """Get tiering status.""" + return self._json("get", "/api/v1/tiering/status") + + def files( + self, + tier: Optional[str] = None, + database: Optional[str] = None, + limit: Optional[int] = None, + ) -> Any: + """List tiered files, optionally filtered by tier/database.""" + params = { + k: v + for k, v in {"tier": tier, "database": database, "limit": limit}.items() + if v is not None + } + return self._json("get", "/api/v1/tiering/files", params=params or None) + + def migrate( + self, + from_tier: Optional[str] = None, + to_tier: Optional[str] = None, + database: Optional[str] = None, + measurement: Optional[str] = None, + dry_run: bool = False, + ) -> Any: + """Trigger a tier-migration cycle.""" + body = { + "from_tier": from_tier, + "to_tier": to_tier, + "database": database, + "measurement": measurement, + "dry_run": dry_run, + } + body = {k: v for k, v in body.items() if v is not None} + return self._json("post", "/api/v1/tiering/migrate", json=body or None) + + def stats(self) -> Any: + """Get tier statistics and recent migrations.""" + return self._json("get", "/api/v1/tiering/stats") + + def scan(self) -> Any: + """Scan the hot tier and register any unregistered files.""" + return self._json("post", "/api/v1/tiering/scan") + + # --- Per-database policies --- + def list_policies(self) -> Any: + """List per-database tiering policies.""" + return self._json("get", "/api/v1/tiering/policies/") + + def get_policy(self, database: str) -> Any: + """Get the custom tiering policy for a database (404 if none).""" + validate_name(database, "database") + return self._json("get", f"/api/v1/tiering/policies/{database}") + + def set_policy( + self, + database: str, + hot_only: Optional[bool] = None, + hot_max_age_days: Optional[int] = None, + ) -> Any: + """Create or update a database's tiering policy.""" + validate_name(database, "database") + body: dict[str, Any] = {} + if hot_only is not None: + body["hot_only"] = hot_only + if hot_max_age_days is not None: + body["hot_max_age_days"] = hot_max_age_days + return self._json("put", f"/api/v1/tiering/policies/{database}", json=body) + + def delete_policy(self, database: str) -> Any: + """Delete a database's custom policy (revert to global defaults).""" + validate_name(database, "database") + return self._json("delete", f"/api/v1/tiering/policies/{database}") + + def effective_policy(self, database: str) -> Any: + """Get the effective (resolved) tiering policy for a database.""" + validate_name(database, "database") + return self._json("get", f"/api/v1/tiering/policies/{database}/effective") diff --git a/src/arc_client/models/__init__.py b/src/arc_client/models/__init__.py index a6479b9..bd10416 100644 --- a/src/arc_client/models/__init__.py +++ b/src/arc_client/models/__init__.py @@ -13,7 +13,19 @@ CQExecution, ExecuteCQResponse, ) +from arc_client.models.database import ( + DatabaseInfo, + DatabaseListResponse, + DatabaseMeasurement, + MeasurementListResponse, +) from arc_client.models.delete import DeleteConfigResponse, DeleteResponse +from arc_client.models.import_ import ( + ImportResult, + ImportStats, + LPImportResult, + TLEImportResult, +) from arc_client.models.query import EstimateResponse, MeasurementInfo, QueryResponse from arc_client.models.retention import ( ExecuteRetentionResponse, @@ -25,6 +37,14 @@ "ContinuousQuery", "CQExecution", "CreateTokenResponse", + "DatabaseInfo", + "DatabaseListResponse", + "DatabaseMeasurement", + "MeasurementListResponse", + "ImportResult", + "ImportStats", + "LPImportResult", + "TLEImportResult", "DeleteConfigResponse", "DeleteResponse", "EstimateResponse", diff --git a/src/arc_client/models/database.py b/src/arc_client/models/database.py new file mode 100644 index 0000000..a6466a7 --- /dev/null +++ b/src/arc_client/models/database.py @@ -0,0 +1,51 @@ +"""Models for the databases management endpoints.""" + +from __future__ import annotations + +from typing import Optional + +from pydantic import BaseModel, ConfigDict, Field + + +class DatabaseInfo(BaseModel): + """Information about a database. + + Attributes: + name: Database name. + measurement_count: Number of measurements in the database. + created_at: Creation timestamp (RFC3339), when reported by the server. + """ + + model_config = ConfigDict(extra="allow") + + name: str + measurement_count: int = 0 + created_at: Optional[str] = None + + +class DatabaseListResponse(BaseModel): + """Response from ``GET /api/v1/databases``.""" + + model_config = ConfigDict(extra="allow") + + databases: list[DatabaseInfo] = Field(default_factory=list) + count: int = 0 + + +class DatabaseMeasurement(BaseModel): + """A measurement within a database.""" + + model_config = ConfigDict(extra="allow") + + name: str + file_count: int = 0 + + +class MeasurementListResponse(BaseModel): + """Response from ``GET /api/v1/databases/:name/measurements``.""" + + model_config = ConfigDict(extra="allow") + + database: str + measurements: list[DatabaseMeasurement] = Field(default_factory=list) + count: int = 0 diff --git a/src/arc_client/models/import_.py b/src/arc_client/models/import_.py new file mode 100644 index 0000000..f2e079d --- /dev/null +++ b/src/arc_client/models/import_.py @@ -0,0 +1,95 @@ +"""Models for bulk import endpoints. + +These mirror the Arc server's import result structs. Extra fields are allowed so +newer server fields don't break older clients. +""" + +from __future__ import annotations + +from typing import Optional + +from pydantic import BaseModel, ConfigDict, Field + + +class ImportResult(BaseModel): + """Result of a CSV or Parquet import. + + Attributes: + database: Target database. + measurement: Measurement (table) the rows were imported into. + rows_imported: Number of rows ingested. + partitions_created: Number of distinct hour partitions created. + time_range_min: Earliest timestamp imported (RFC3339), if any. + time_range_max: Latest timestamp imported (RFC3339), if any. + columns: Column names in the imported data (time column normalized). + duration_ms: Server-side import duration in milliseconds. + """ + + model_config = ConfigDict(extra="allow") + + database: str + measurement: str + rows_imported: int = 0 + partitions_created: int = 0 + time_range_min: Optional[str] = None + time_range_max: Optional[str] = None + columns: list[str] = Field(default_factory=list) + duration_ms: int = 0 + + +class LPImportResult(BaseModel): + """Result of a line-protocol file import. + + Attributes: + database: Target database. + measurements: Measurements that received data. + rows_imported: Number of rows ingested. + precision: Timestamp precision used to parse the file. + duration_ms: Server-side import duration in milliseconds. + """ + + model_config = ConfigDict(extra="allow") + + database: str + measurements: list[str] = Field(default_factory=list) + rows_imported: int = 0 + precision: str = "ns" + duration_ms: int = 0 + + +class TLEImportResult(BaseModel): + """Result of a TLE (Two-Line Element) file import. + + Attributes: + database: Target database. + measurement: Measurement the satellites were imported into. + satellite_count: Number of distinct satellites parsed. + rows_imported: Number of rows ingested. + parse_warnings: Non-fatal parse warnings, if any. + duration_ms: Server-side import duration in milliseconds. + """ + + model_config = ConfigDict(extra="allow") + + database: str + measurement: str + satellite_count: int = 0 + rows_imported: int = 0 + parse_warnings: list[str] = Field(default_factory=list) + duration_ms: int = 0 + + +class ImportStats(BaseModel): + """Cumulative import counters from ``GET /api/v1/import/stats``. + + Attributes: + total_requests: Total import requests handled. + total_records: Total records ingested across imports. + total_errors: Total import errors. + """ + + model_config = ConfigDict(extra="allow") + + total_requests: int = 0 + total_records: int = 0 + total_errors: int = 0 diff --git a/src/arc_client/models/query.py b/src/arc_client/models/query.py index dbf21ed..ed8f742 100644 --- a/src/arc_client/models/query.py +++ b/src/arc_client/models/query.py @@ -27,6 +27,9 @@ class QueryResponse(BaseModel): execution_time_ms: float = 0.0 timestamp: Optional[str] = None error: Optional[str] = None + # Populated only by the msgpack query path (``query_msgpack``); the JSON + # path leaves it empty. Parallel to ``columns``: Arrow type name per column. + types: list[str] = Field(default_factory=list) class EstimateResponse(BaseModel): diff --git a/src/arc_client/query/async_executor.py b/src/arc_client/query/async_executor.py index 48fbd4d..692b9b2 100644 --- a/src/arc_client/query/async_executor.py +++ b/src/arc_client/query/async_executor.py @@ -8,9 +8,11 @@ import pyarrow.ipc as ipc from arc_client.config import ClientConfig -from arc_client.exceptions import ArcQueryError, ArcValidationError +from arc_client.exceptions import ArcNotSupportedError, ArcQueryError, ArcValidationError from arc_client.http.async_http import AsyncHTTPClient from arc_client.models.query import EstimateResponse, MeasurementInfo, QueryResponse +from arc_client.query.msgpack_response import decode_msgpack_query_response +from arc_client.validation import validate_name if TYPE_CHECKING: pass @@ -82,6 +84,49 @@ async def query(self, sql: str, database: Optional[str] = None) -> QueryResponse except Exception as e: raise ArcQueryError(f"Query failed: {e}") from e + async def query_msgpack(self, sql: str, database: Optional[str] = None) -> QueryResponse: + """Execute a SQL query and decode the MessagePack response. + + **Experimental.** Uses Arc's ``/api/v1/query/msgpack`` endpoint, which + returns a compact columnar MessagePack envelope with native timestamp + encoding. The result is transposed to row-major and returned as a + :class:`QueryResponse` (same shape as :meth:`query`), with per-column + Arrow types in ``QueryResponse.types``. + + Requires the Arc server to be built with the ``duckdb_arrow`` build tag; + otherwise the server responds with HTTP 501. + + Args: + sql: SQL query to execute. + database: Target database. Uses client default if not specified. + + Returns: + QueryResponse with row-major ``data`` and populated ``types``. + + Raises: + ArcValidationError: If the SQL is empty. + ArcNotSupportedError: If the server lacks msgpack query support (501). + ArcQueryError: If the query fails. + """ + if not sql or not sql.strip(): + raise ArcValidationError("SQL query cannot be empty") + + headers = {"Accept": "application/msgpack"} + if database: + headers["x-arc-database"] = database + + try: + response = await self._http.post( + "/api/v1/query/msgpack", + json={"sql": sql}, + headers=headers, + ) + return decode_msgpack_query_response(response.content) + except (ArcQueryError, ArcNotSupportedError, ArcValidationError): + raise + except Exception as e: + raise ArcQueryError(f"Query failed: {e}") from e + async def query_pandas(self, sql: str, database: Optional[str] = None) -> Any: """Execute a SQL query and return results as a pandas DataFrame. @@ -293,5 +338,5 @@ async def show_tables(self, database: Optional[str] = None) -> QueryResponse: Returns: QueryResponse with table information. """ - db = database or "default" + db = validate_name(database or "default", "database") return await self.query(f"SHOW TABLES FROM {db}") diff --git a/src/arc_client/query/executor.py b/src/arc_client/query/executor.py index 5e5a1d1..0f260d7 100644 --- a/src/arc_client/query/executor.py +++ b/src/arc_client/query/executor.py @@ -8,9 +8,11 @@ import pyarrow.ipc as ipc from arc_client.config import ClientConfig -from arc_client.exceptions import ArcQueryError, ArcValidationError +from arc_client.exceptions import ArcNotSupportedError, ArcQueryError, ArcValidationError from arc_client.http.sync_http import SyncHTTPClient from arc_client.models.query import EstimateResponse, MeasurementInfo, QueryResponse +from arc_client.query.msgpack_response import decode_msgpack_query_response +from arc_client.validation import validate_name if TYPE_CHECKING: pass @@ -87,6 +89,49 @@ def query(self, sql: str, database: Optional[str] = None) -> QueryResponse: except Exception as e: raise ArcQueryError(f"Query failed: {e}") from e + def query_msgpack(self, sql: str, database: Optional[str] = None) -> QueryResponse: + """Execute a SQL query and decode the MessagePack response. + + **Experimental.** This uses Arc's ``/api/v1/query/msgpack`` endpoint, + which returns a compact columnar MessagePack envelope with native + timestamp encoding. The result is transposed to row-major and returned + as a :class:`QueryResponse` (same shape as :meth:`query`), with the + per-column Arrow types available in ``QueryResponse.types``. + + The endpoint requires the Arc server to be built with the + ``duckdb_arrow`` build tag; otherwise the server responds with HTTP 501. + + Args: + sql: SQL query to execute. + database: Target database. Uses client default if not specified. + + Returns: + QueryResponse with row-major ``data`` and populated ``types``. + + Raises: + ArcValidationError: If the SQL is empty. + ArcNotSupportedError: If the server lacks msgpack query support (501). + ArcQueryError: If the query fails. + """ + if not sql or not sql.strip(): + raise ArcValidationError("SQL query cannot be empty") + + headers = {"Accept": "application/msgpack"} + if database: + headers["x-arc-database"] = database + + try: + response = self._http.post( + "/api/v1/query/msgpack", + json={"sql": sql}, + headers=headers, + ) + return decode_msgpack_query_response(response.content) + except (ArcQueryError, ArcNotSupportedError, ArcValidationError): + raise + except Exception as e: + raise ArcQueryError(f"Query failed: {e}") from e + def query_pandas(self, sql: str, database: Optional[str] = None) -> Any: """Execute a SQL query and return results as a pandas DataFrame. @@ -332,5 +377,5 @@ def show_tables(self, database: Optional[str] = None) -> QueryResponse: >>> for row in result.data: ... print(row) """ - db = database or "default" + db = validate_name(database or "default", "database") return self.query(f"SHOW TABLES FROM {db}") diff --git a/src/arc_client/query/msgpack_response.py b/src/arc_client/query/msgpack_response.py new file mode 100644 index 0000000..66871f6 --- /dev/null +++ b/src/arc_client/query/msgpack_response.py @@ -0,0 +1,95 @@ +"""Decoder for the experimental MessagePack query response format. + +Arc's ``POST /api/v1/query/msgpack`` returns a **columnar** MessagePack envelope +(``application/msgpack``): a map with keys ``success, columns, types, data, +row_count, execution_time_ms, timestamp``. ``data`` is an array-of-columns +(one inner array per column), not row-major, and timestamp columns arrive as +native MessagePack timestamp extensions (decoded here to ``datetime``). Per-cell +nulls are possible. + +The error envelope is a 4-key map ``{success: false, error, execution_time_ms, +timestamp}``. + +This endpoint is experimental and requires the server's ``duckdb_arrow`` build +tag; without it the server responds 501, surfaced as ``ArcNotSupportedError`` by +the HTTP error handler before this decoder runs. +""" + +from __future__ import annotations + +from typing import Any + +import msgpack + +from arc_client.exceptions import ArcQueryError +from arc_client.models.query import QueryResponse + + +def decode_msgpack_query_response(content: bytes) -> QueryResponse: + """Decode a MessagePack query response envelope into a :class:`QueryResponse`. + + The columnar ``data`` is transposed to row-major so the result matches the + JSON query path (``QueryResponse.data`` is a list of rows). + + Args: + content: Raw MessagePack response body. + + Returns: + A :class:`QueryResponse`. For a success envelope, ``data`` is row-major + and ``types`` carries the per-column Arrow type names. + + Raises: + ArcQueryError: If the body cannot be decoded or reports ``success=false``. + """ + try: + # timestamp=3 decodes native msgpack timestamp extensions to datetime. + envelope = msgpack.unpackb(content, timestamp=3, raw=False) + except Exception as e: # pragma: no cover - malformed body is rare + raise ArcQueryError(f"Failed to decode msgpack query response: {e}") from e + + if not isinstance(envelope, dict): + raise ArcQueryError("Malformed msgpack query response: expected a map") + + if not envelope.get("success", False): + message = envelope.get("error") or "Query failed" + raise ArcQueryError(str(message)) + + columns: list[str] = list(envelope.get("columns", [])) + types: list[str] = list(envelope.get("types", [])) + column_data: list[list[Any]] = list(envelope.get("data", [])) + row_count = int(envelope.get("row_count", 0)) + + rows = _transpose_columns(column_data, row_count) + + execution_time_ms = envelope.get("execution_time_ms", 0) + timestamp = envelope.get("timestamp") + + return QueryResponse( + success=True, + columns=columns, + types=types, + data=rows, + row_count=row_count if row_count else len(rows), + execution_time_ms=float(execution_time_ms), + timestamp=str(timestamp) if timestamp is not None else None, + ) + + +def _transpose_columns(column_data: list[list[Any]], row_count: int) -> list[list[Any]]: + """Transpose an array-of-columns into row-major rows. + + Uses ``row_count`` when the columns are empty (e.g. an empty result set with + known columns) and otherwise the longest column, so ragged input never drops + trailing rows. + """ + if not column_data: + return [] + + n_rows = max((len(col) for col in column_data), default=0) + if row_count and row_count > n_rows: + n_rows = row_count + + rows: list[list[Any]] = [] + for i in range(n_rows): + rows.append([col[i] if i < len(col) else None for col in column_data]) + return rows diff --git a/src/arc_client/validation.py b/src/arc_client/validation.py new file mode 100644 index 0000000..78b3bf1 --- /dev/null +++ b/src/arc_client/validation.py @@ -0,0 +1,41 @@ +"""Shared client-side validation helpers. + +Mirrors Arc's server-side identifier rules so invalid names fail fast with a +clear error (and never reach a SQL string via interpolation). +""" + +from __future__ import annotations + +import re + +from arc_client.exceptions import ArcValidationError + +# Matches Arc's server-side rule: start with a letter, then letters/digits/ +# underscore/hyphen, max 64 characters total. +_NAME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9_-]{0,63}$") + + +def is_valid_name(name: str) -> bool: + """Return True if ``name`` is a valid Arc database/measurement identifier.""" + return bool(_NAME_RE.match(name)) + + +def validate_name(name: str, kind: str = "name") -> str: + """Validate a database/measurement identifier, returning it unchanged. + + Args: + name: The identifier to validate. + kind: A short label used in the error message (e.g. "database"). + + Returns: + The validated ``name``. + + Raises: + ArcValidationError: If ``name`` is not a valid identifier. + """ + if not isinstance(name, str) or not is_valid_name(name): + raise ArcValidationError( + f"Invalid {kind} {name!r}: must start with a letter and contain only " + "letters, digits, underscores, or hyphens (max 64 characters)." + ) + return name diff --git a/tests/integration/test_live.py b/tests/integration/test_live.py new file mode 100644 index 0000000..3beea66 --- /dev/null +++ b/tests/integration/test_live.py @@ -0,0 +1,83 @@ +"""Opt-in integration tests against a live Arc server. + +These run only when ``ARC_TEST_URL`` (and usually ``ARC_TEST_TOKEN``) are set; +otherwise every test is skipped. They exercise the new 0.2.0 surface end-to-end. + +Run with, e.g.:: + + ARC_TEST_URL=http://localhost:8000 ARC_TEST_TOKEN=... uv run pytest tests/integration -v + +The msgpack query test additionally requires the server to be built with the +``duckdb_arrow`` build tag; it is skipped (not failed) on a 501. +""" + +from __future__ import annotations + +import contextlib +import os +import time + +import pytest + +from arc_client import ArcClient +from arc_client.exceptions import ArcLicenseError, ArcNotSupportedError + +pytestmark = pytest.mark.skipif( + not os.environ.get("ARC_TEST_URL"), + reason="Set ARC_TEST_URL (and ARC_TEST_TOKEN) to run live integration tests.", +) + + +def test_health_and_ready(sync_client: ArcClient) -> None: + health = sync_client.health() + assert health.status + assert sync_client.ready() in (True, False) + + +def test_columnar_write_then_query_msgpack(sync_client: ArcClient) -> None: + base = int(time.time() * 1_000_000) + measurement = "sdk_it_cpu" + sync_client.write.write_columnar( + measurement, + {"time": [base, base + 1_000_000], "host": ["a", "b"], "usage": [1.0, 2.0]}, + ) + # Give the server a moment to flush. + time.sleep(1.0) + + try: + result = sync_client.query.query_msgpack(f"SELECT * FROM {measurement} ORDER BY time") + except ArcNotSupportedError: + pytest.skip("Server built without the duckdb_arrow build tag") + + assert result.success + assert "usage" in result.columns + + +def test_databases_lifecycle(sync_client: ArcClient) -> None: + name = "sdk_it_db" + # May already exist from a previous run; ignore and continue. + with contextlib.suppress(Exception): + sync_client.databases.create(name) + + listing = sync_client.databases.list() + assert any(db.name == name for db in listing.databases) + + info = sync_client.databases.get(name) + assert info.name == name + + +def test_import_csv(sync_client: ArcClient) -> None: + csv = b"time,value\n1633046400000000,1.5\n1633046401000000,2.5\n" + result = sync_client.import_.import_csv( + csv, measurement="sdk_it_import", time_format="epoch_us" + ) + assert result.rows_imported == 2 + + +def test_enterprise_governance_usage_or_license_error(sync_client: ArcClient) -> None: + # On a licensed server this returns usage; on OSS it should be a clean + # license/permission error rather than a crash. + try: + sync_client.governance.list_policies() + except ArcLicenseError: + pytest.skip("Server has no query_governance license feature (expected on OSS)") diff --git a/tests/unit/http/__init__.py b/tests/unit/http/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/unit/http/test_async_roundtrip.py b/tests/unit/http/test_async_roundtrip.py new file mode 100644 index 0000000..7975b4c --- /dev/null +++ b/tests/unit/http/test_async_roundtrip.py @@ -0,0 +1,137 @@ +"""HTTP round-trip tests for the async client (parity with the sync tests).""" + +from __future__ import annotations + +from datetime import datetime, timezone + +import httpx +import msgpack +import pytest +from pytest_httpx import HTTPXMock + +from arc_client import AsyncArcClient, Compression +from arc_client.compression import is_zstd +from arc_client.exceptions import ArcLicenseError, ArcNotSupportedError, ArcRateLimitError + + +@pytest.fixture(autouse=True) +def _no_sleep(monkeypatch: pytest.MonkeyPatch) -> None: + async def _noop(*_: object) -> None: + return None + + monkeypatch.setattr("arc_client.http.async_http.asyncio.sleep", _noop) + + +async def test_async_zstd_write_no_content_encoding(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=204) + client = AsyncArcClient(host="h", token="t", compression=Compression.ZSTD) + + await client.write.write_columnar("cpu", {"time": [1], "v": [1.0]}) + + request = httpx_mock.get_requests()[0] + assert is_zstd(request.read()) + assert "Content-Encoding" not in request.headers + await client.close() + + +async def test_async_query_msgpack_decodes(httpx_mock: HTTPXMock) -> None: + ts = datetime(2021, 10, 1, tzinfo=timezone.utc) + content = msgpack.packb( + { + "success": True, + "columns": ["time", "v"], + "types": ["timestamp[us, tz=UTC]", "double"], + "data": [[ts], [1.5]], + "row_count": 1, + "execution_time_ms": 2, + "timestamp": "x", + }, + datetime=True, + ) + httpx_mock.add_response(content=content, headers={"Content-Type": "application/msgpack"}) + client = AsyncArcClient(host="h", token="t") + + result = await client.query.query_msgpack("SELECT * FROM cpu") + + assert result.columns == ["time", "v"] + assert isinstance(result.data[0][0], datetime) + await client.close() + + +async def test_async_query_msgpack_501(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=501, json={"error": "no build tag"}) + client = AsyncArcClient(host="h", token="t") + + with pytest.raises(ArcNotSupportedError): + await client.query.query_msgpack("SELECT 1") + await client.close() + + +async def test_async_import_csv(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response( + json={ + "status": "ok", + "result": {"database": "db", "measurement": "cpu", "rows_imported": 7}, + } + ) + client = AsyncArcClient(host="h", token="t", database="db") + + result = await client.import_.import_csv(b"time,v\n1,2\n", measurement="cpu") + + assert result.rows_imported == 7 + await client.close() + + +async def test_async_databases_create(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=201, json={"name": "newdb", "measurement_count": 0}) + client = AsyncArcClient(host="h", token="t") + + result = await client.databases.create("newdb") + + assert result.name == "newdb" + await client.close() + + +async def test_async_retry_429(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=429) + httpx_mock.add_response(json={"status": "ok"}) + client = AsyncArcClient(host="h", token="t", max_retries=2) + + await client.health() + + assert len(httpx_mock.get_requests()) == 2 + await client.close() + + +async def test_async_retry_gives_up(httpx_mock: HTTPXMock) -> None: + for _ in range(3): + httpx_mock.add_response(status_code=429) + client = AsyncArcClient(host="h", token="t", max_retries=2) + + with pytest.raises(ArcRateLimitError): + await client.health() + assert len(httpx_mock.get_requests()) == 3 + await client.close() + + +async def test_async_connection_error_recovers(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_exception(httpx.ConnectError("refused")) + httpx_mock.add_response(json={"status": "ok"}) + client = AsyncArcClient(host="h", token="t", max_retries=2) + + await client.health() + + assert len(httpx_mock.get_requests()) == 2 + await client.close() + + +async def test_async_enterprise_license_error(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response( + status_code=403, + json={"error": "RBAC requires an enterprise license with the 'rbac' feature enabled"}, + ) + client = AsyncArcClient(host="h", token="t") + + with pytest.raises(ArcLicenseError): + await client.rbac.list_organizations() + await client.close() diff --git a/tests/unit/http/test_enterprise.py b/tests/unit/http/test_enterprise.py new file mode 100644 index 0000000..e8bde57 --- /dev/null +++ b/tests/unit/http/test_enterprise.py @@ -0,0 +1,88 @@ +"""HTTP round-trip tests: representative enterprise wrappers.""" + +from __future__ import annotations + +import pytest +from pytest_httpx import HTTPXMock + +from arc_client import ArcClient +from arc_client.exceptions import ArcLicenseError + + +@pytest.fixture +def client() -> ArcClient: + c = ArcClient(host="h", token="t") + yield c + c.close() + + +def test_rbac_create_organization(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + status_code=201, json={"success": True, "organization": {"id": 1, "name": "acme"}} + ) + + result = client.rbac.create_organization("acme", description="Acme Inc") + + assert result["organization"]["name"] == "acme" + request = httpx_mock.get_requests()[0] + assert request.url.path == "/api/v1/rbac/organizations" + import json + + assert json.loads(request.read()) == {"name": "acme", "description": "Acme Inc"} + + +def test_governance_usage(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response(json={"success": True, "usage": {"queries_this_hour": 5}}) + + result = client.governance.usage(42) + + assert result["usage"]["queries_this_hour"] == 5 + assert httpx_mock.get_requests()[0].url.path == "/api/v1/governance/usage/42" + + +def test_queries_active(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response(json={"success": True, "queries": [], "count": 0}) + + result = client.queries.active() + + assert result["count"] == 0 + assert httpx_mock.get_requests()[0].url.path == "/api/v1/queries/active" + + +def test_enterprise_license_error(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + status_code=403, + json={ + "success": False, + "error": ( + "Query governance requires an enterprise license with the " + "'query_governance' feature enabled" + ), + }, + ) + + with pytest.raises(ArcLicenseError): + client.governance.list_policies() + + +def test_tiering_set_policy(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response(json={"message": "ok", "database": "db", "policy": {}}) + + client.tiering.set_policy("db", hot_only=True, hot_max_age_days=7) + + request = httpx_mock.get_requests()[0] + assert request.method == "PUT" + assert request.url.path == "/api/v1/tiering/policies/db" + import json + + assert json.loads(request.read()) == {"hot_only": True, "hot_max_age_days": 7} + + +def test_mqtt_start_subscription(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response(json={"success": True, "message": "started"}) + + client.mqtt.start_subscription("sub-1") + + request = httpx_mock.get_requests()[0] + assert request.method == "POST" + assert request.url.path == "/api/v1/mqtt/subscriptions/sub-1/start" diff --git a/tests/unit/http/test_headers_and_errors.py b/tests/unit/http/test_headers_and_errors.py new file mode 100644 index 0000000..3d266ae --- /dev/null +++ b/tests/unit/http/test_headers_and_errors.py @@ -0,0 +1,94 @@ +"""HTTP round-trip tests: header assembly and error mapping.""" + +from __future__ import annotations + +import pytest +from pytest_httpx import HTTPXMock + +from arc_client import ArcClient +from arc_client.exceptions import ( + ArcAuthenticationError, + ArcLicenseError, + ArcNotFoundError, + ArcNotSupportedError, + ArcRateLimitError, + ArcServerError, +) +from arc_client.http.base import USER_AGENT + + +@pytest.fixture +def client() -> ArcClient: + c = ArcClient(host="testhost", port=8000, token="secret-token", database="mydb") + yield c + c.close() + + +def test_headers_include_auth_db_and_versioned_user_agent( + httpx_mock: HTTPXMock, client: ArcClient +) -> None: + httpx_mock.add_response(json={"status": "ok"}) + + client.health() + + request = httpx_mock.get_requests()[0] + assert request.headers["Authorization"] == "Bearer secret-token" + assert request.headers["x-arc-database"] == "mydb" + assert request.headers["User-Agent"] == USER_AGENT + assert USER_AGENT == "arc-client-python/0.2.0" + + +def test_user_agent_matches_package_version(httpx_mock: HTTPXMock, client: ArcClient) -> None: + import arc_client + + assert f"arc-client-python/{arc_client.__version__}" == USER_AGENT + + +@pytest.mark.parametrize( + ("status", "exc"), + [ + (401, ArcAuthenticationError), + (403, ArcAuthenticationError), + (404, ArcNotFoundError), + (429, ArcRateLimitError), + (500, ArcServerError), + (502, ArcServerError), + (501, ArcNotSupportedError), + ], +) +def test_error_status_mapping(httpx_mock: HTTPXMock, status: int, exc: type) -> None: + # Test the HTTP layer directly: sub-clients rewrap into domain errors, but + # the status->exception mapping lives in the shared HTTP error handler. + # max_retries=0 so retryable statuses (429/503) raise immediately. + c = ArcClient(host="testhost", token="t", max_retries=0) + httpx_mock.add_response(status_code=status, json={"error": "boom"}) + + with pytest.raises(exc): + c._get_http().get("/api/v1/anything") + c.close() + + +def test_license_error_mapped_from_403(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + status_code=403, + json={ + "success": False, + "error": "RBAC requires an enterprise license with the 'rbac' feature enabled", + }, + ) + + with pytest.raises(ArcLicenseError): + client.rbac.list_organizations() + + +def test_license_error_is_authentication_error_subclass() -> None: + assert issubclass(ArcLicenseError, ArcAuthenticationError) + + +def test_per_call_database_override(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response(json={"success": True, "columns": [], "data": [], "row_count": 0}) + + client.query.query("SELECT 1", database="otherdb") + + request = httpx_mock.get_requests()[0] + assert request.headers["x-arc-database"] == "otherdb" diff --git a/tests/unit/http/test_imports_and_databases.py b/tests/unit/http/test_imports_and_databases.py new file mode 100644 index 0000000..837b18e --- /dev/null +++ b/tests/unit/http/test_imports_and_databases.py @@ -0,0 +1,143 @@ +"""HTTP round-trip tests: bulk imports and databases CRUD.""" + +from __future__ import annotations + +import pytest +from pytest_httpx import HTTPXMock + +from arc_client import ArcClient +from arc_client.exceptions import ArcIngestionError, ArcValidationError + + +@pytest.fixture +def client() -> ArcClient: + c = ArcClient(host="h", token="t", database="db") + yield c + c.close() + + +# --- Imports --- + + +def test_import_csv_multipart_and_params(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + json={ + "status": "ok", + "result": { + "database": "db", + "measurement": "cpu", + "rows_imported": 42, + "partitions_created": 2, + "columns": ["time", "usage"], + "duration_ms": 10, + }, + } + ) + + result = client.import_.import_csv(b"time,usage\n1,2.0\n", measurement="cpu", delimiter=",") + + assert result.rows_imported == 42 + assert result.measurement == "cpu" + request = httpx_mock.get_requests()[0] + assert request.url.path == "/api/v1/import/csv" + assert b'name="file"' in request.read() # multipart field + assert request.url.params["measurement"] == "cpu" + assert request.headers["x-arc-database"] == "db" + + +def test_import_tle_sends_measurement_header(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + json={ + "status": "ok", + "result": { + "database": "db", + "measurement": "satellite_tle", + "satellite_count": 3, + "rows_imported": 3, + "duration_ms": 5, + }, + } + ) + + result = client.import_.import_tle(b"1 ...\n2 ...\n") + + assert result.satellite_count == 3 + request = httpx_mock.get_requests()[0] + assert request.headers["x-arc-measurement"] == "satellite_tle" + + +def test_import_lp_parses_result(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + json={ + "status": "ok", + "result": { + "database": "db", + "measurements": ["cpu", "mem"], + "rows_imported": 100, + "precision": "ns", + "duration_ms": 12, + }, + } + ) + + result = client.import_.import_lp(b"cpu v=1 1\n", precision="ns") + + assert result.measurements == ["cpu", "mem"] + assert result.rows_imported == 100 + + +def test_import_unexpected_response_raises(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response(json={"status": "ok"}) # no "result" + + with pytest.raises(ArcIngestionError): + client.import_.import_csv(b"x", measurement="cpu") + + +# --- Databases --- + + +def test_databases_list(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + json={ + "databases": [{"name": "db", "measurement_count": 3}], + "count": 1, + } + ) + + result = client.databases.list() + + assert result.count == 1 + assert result.databases[0].name == "db" + + +def test_databases_create(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + status_code=201, json={"name": "newdb", "measurement_count": 0, "created_at": "now"} + ) + + result = client.databases.create("newdb") + + assert result.name == "newdb" + request = httpx_mock.get_requests()[0] + assert request.url.path == "/api/v1/databases" + + +def test_databases_create_invalid_name(client: ArcClient) -> None: + with pytest.raises(ArcValidationError): + client.databases.create("1bad name!") + + +def test_databases_delete_requires_confirm(client: ArcClient) -> None: + with pytest.raises(ArcValidationError): + client.databases.delete("db", confirm=False) + + +def test_databases_delete_sends_confirm(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response(json={"message": "deleted", "files_deleted": 5}) + + result = client.databases.delete("db", confirm=True) + + assert result["files_deleted"] == 5 + request = httpx_mock.get_requests()[0] + assert request.method == "DELETE" + assert request.url.params["confirm"] == "true" diff --git a/tests/unit/http/test_query_msgpack.py b/tests/unit/http/test_query_msgpack.py new file mode 100644 index 0000000..1bb2dc6 --- /dev/null +++ b/tests/unit/http/test_query_msgpack.py @@ -0,0 +1,84 @@ +"""HTTP round-trip tests: msgpack query endpoint.""" + +from __future__ import annotations + +from datetime import datetime, timezone + +import msgpack +import pytest +from pytest_httpx import HTTPXMock + +from arc_client import ArcClient +from arc_client.exceptions import ArcNotSupportedError, ArcQueryError + + +@pytest.fixture +def client() -> ArcClient: + c = ArcClient(host="h", token="t", database="db") + yield c + c.close() + + +def _columnar_envelope() -> bytes: + ts = datetime(2021, 10, 1, tzinfo=timezone.utc) + return msgpack.packb( + { + "success": True, + "columns": ["time", "host", "usage"], + "types": ["timestamp[us, tz=UTC]", "utf8", "double"], + "data": [[ts, ts], ["s1", None], [1.5, 2.5]], + "row_count": 2, + "execution_time_ms": 4, + "timestamp": "2021-10-01T00:00:00Z", + }, + datetime=True, + ) + + +def test_query_msgpack_decodes_columnar_to_rows(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + content=_columnar_envelope(), + headers={"Content-Type": "application/msgpack"}, + ) + + result = client.query.query_msgpack("SELECT * FROM cpu") + + assert result.success + assert result.columns == ["time", "host", "usage"] + assert result.types == ["timestamp[us, tz=UTC]", "utf8", "double"] + # Transposed to row-major. + assert result.data[0][1] == "s1" + assert result.data[1][1] is None # null preserved + assert isinstance(result.data[0][0], datetime) # native timestamp decoded + assert result.row_count == 2 + + request = httpx_mock.get_requests()[0] + assert request.url.path == "/api/v1/query/msgpack" + assert request.headers["Accept"] == "application/msgpack" + + +def test_query_msgpack_501_raises_not_supported(httpx_mock: HTTPXMock, client: ArcClient) -> None: + httpx_mock.add_response( + status_code=501, + json={"error": "msgpack query path requires the duckdb_arrow build tag"}, + ) + + with pytest.raises(ArcNotSupportedError): + client.query.query_msgpack("SELECT 1") + + +def test_query_msgpack_error_envelope_raises(httpx_mock: HTTPXMock, client: ArcClient) -> None: + err = msgpack.packb( + {"success": False, "error": "bad sql", "execution_time_ms": 1, "timestamp": "x"} + ) + httpx_mock.add_response(content=err, headers={"Content-Type": "application/msgpack"}) + + with pytest.raises(ArcQueryError, match="bad sql"): + client.query.query_msgpack("SELECT bad") + + +def test_query_msgpack_empty_sql_raises(client: ArcClient) -> None: + from arc_client.exceptions import ArcValidationError + + with pytest.raises(ArcValidationError): + client.query.query_msgpack("") diff --git a/tests/unit/http/test_retry.py b/tests/unit/http/test_retry.py new file mode 100644 index 0000000..170540f --- /dev/null +++ b/tests/unit/http/test_retry.py @@ -0,0 +1,86 @@ +"""HTTP round-trip tests: retry/backoff behavior.""" + +from __future__ import annotations + +import httpx +import pytest +from pytest_httpx import HTTPXMock + +from arc_client import ArcClient +from arc_client.exceptions import ArcConnectionError, ArcRateLimitError, ArcServerError + + +@pytest.fixture(autouse=True) +def _no_sleep(monkeypatch: pytest.MonkeyPatch) -> None: + """Make retry backoff instant.""" + monkeypatch.setattr("arc_client.http.sync_http.time.sleep", lambda *_: None) + + +def test_retries_429_then_succeeds(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=429, headers={"Retry-After": "0"}) + httpx_mock.add_response(status_code=429) + httpx_mock.add_response(json={"status": "ok"}) + client = ArcClient(host="h", token="t", max_retries=3) + + client.health() # should succeed on the 3rd attempt + + assert len(httpx_mock.get_requests()) == 3 + client.close() + + +def test_retries_503_then_succeeds(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=503) + httpx_mock.add_response(json={"status": "ok"}) + client = ArcClient(host="h", token="t", max_retries=2) + + client.health() + + assert len(httpx_mock.get_requests()) == 2 + client.close() + + +def test_gives_up_after_max_retries_on_429(httpx_mock: HTTPXMock) -> None: + # max_retries=2 -> 3 total attempts, all 429 -> raises. + for _ in range(3): + httpx_mock.add_response(status_code=429) + client = ArcClient(host="h", token="t", max_retries=2) + + with pytest.raises(ArcRateLimitError): + client.health() + + assert len(httpx_mock.get_requests()) == 3 + client.close() + + +def test_no_retry_when_disabled(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=503) + client = ArcClient(host="h", token="t", max_retries=0) + + with pytest.raises(ArcServerError): + client.health() + + assert len(httpx_mock.get_requests()) == 1 + client.close() + + +def test_retries_connection_error_then_raises(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_exception(httpx.ConnectError("refused")) + httpx_mock.add_exception(httpx.ConnectError("refused")) + client = ArcClient(host="h", token="t", max_retries=1) + + with pytest.raises(ArcConnectionError): + client.health() + + assert len(httpx_mock.get_requests()) == 2 + client.close() + + +def test_connection_error_recovers_on_retry(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_exception(httpx.ConnectError("refused")) + httpx_mock.add_response(json={"status": "ok"}) + client = ArcClient(host="h", token="t", max_retries=2) + + client.health() + + assert len(httpx_mock.get_requests()) == 2 + client.close() diff --git a/tests/unit/http/test_writes_compression.py b/tests/unit/http/test_writes_compression.py new file mode 100644 index 0000000..6eb72d0 --- /dev/null +++ b/tests/unit/http/test_writes_compression.py @@ -0,0 +1,101 @@ +"""HTTP round-trip tests: write compression and Content-Encoding behavior.""" + +from __future__ import annotations + +from pytest_httpx import HTTPXMock + +from arc_client import ArcClient, Compression +from arc_client.compression import decompress_gzip, decompress_zstd, is_gzipped, is_zstd + + +def _make_client(compression: object) -> ArcClient: + return ArcClient(host="testhost", token="t", compression=compression) + + +def test_zstd_write_sends_zstd_magic_and_no_content_encoding(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=204) + client = _make_client(Compression.ZSTD) + + client.write.write_columnar("cpu", {"time": [1, 2], "v": [1.0, 2.0]}) + + request = httpx_mock.get_requests()[0] + body = request.read() + assert is_zstd(body), "body should carry the zstd magic bytes" + assert "Content-Encoding" not in request.headers + assert request.headers["Content-Type"] == "application/msgpack" + # Round-trips back to a valid msgpack payload. + assert decompress_zstd(body) + client.close() + + +def test_gzip_write_sends_gzip_magic_and_no_content_encoding(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=204) + client = _make_client("gzip") + + client.write.write_columnar("cpu", {"time": [1], "v": [1.0]}) + + request = httpx_mock.get_requests()[0] + body = request.read() + assert is_gzipped(body) + assert "Content-Encoding" not in request.headers + assert decompress_gzip(body) + client.close() + + +def test_no_compression_sends_raw_msgpack(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=204) + client = _make_client("none") + + client.write.write_columnar("cpu", {"time": [1], "v": [1.0]}) + + body = httpx_mock.get_requests()[0].read() + assert not is_gzipped(body) + assert not is_zstd(body) + client.close() + + +def test_per_call_compress_override(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=204) + # Client default zstd, but override to gzip for this call. + client = _make_client(Compression.ZSTD) + + client.write.write_columnar("cpu", {"time": [1], "v": [1.0]}, compress="gzip") + + body = httpx_mock.get_requests()[0].read() + assert is_gzipped(body) + client.close() + + +def test_bool_true_compress_is_gzip_backcompat(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=204) + client = _make_client(True) + + client.write.write_columnar("cpu", {"time": [1], "v": [1.0]}) + + assert is_gzipped(httpx_mock.get_requests()[0].read()) + client.close() + + +def test_line_protocol_write_targets_correct_endpoint(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=204) + client = _make_client("zstd") + + client.write.write_line_protocol("cpu,host=s1 usage=1.0") + + request = httpx_mock.get_requests()[0] + assert request.url.path == "/api/v1/write/line-protocol" + assert request.headers["Content-Type"] == "text/plain" + assert is_zstd(request.read()) + client.close() + + +def test_write_tle_sends_measurement_header(httpx_mock: HTTPXMock) -> None: + httpx_mock.add_response(status_code=204) + client = _make_client("none") + + client.write.write_tle("1 25544U ...\n2 25544 ...", measurement="iss_tle") + + request = httpx_mock.get_requests()[0] + assert request.url.path == "/api/v1/write/tle" + assert request.headers["x-arc-measurement"] == "iss_tle" + client.close() diff --git a/tests/unit/test_bugfixes.py b/tests/unit/test_bugfixes.py new file mode 100644 index 0000000..fc38782 --- /dev/null +++ b/tests/unit/test_bugfixes.py @@ -0,0 +1,75 @@ +"""Regression tests for bugs fixed in 0.2.0.""" + +from __future__ import annotations + +import time + +import pytest + +from arc_client.ingestion.buffered import BufferedWriter + + +class _FakeWriteClient: + def __init__(self) -> None: + self.writes: list[tuple[str, dict]] = [] + + def write_columnar(self, measurement: str, columns: dict) -> None: + self.writes.append((measurement, columns)) + + +def test_buffered_background_flush_fires_when_idle() -> None: + """The buffered writer flushes on flush_interval even without new writes.""" + fake = _FakeWriteClient() + with BufferedWriter(fake, batch_size=10_000, flush_interval=0.2) as buf: + buf.write_columnar("cpu", {"time": [1], "v": [1.0]}) + # Stay idle past the interval; the background timer must flush. + deadline = time.monotonic() + 2.0 + while not fake.writes and time.monotonic() < deadline: + time.sleep(0.05) + assert fake.writes, "background timer should have flushed the idle buffer" + + +def test_buffered_flushes_remaining_on_close() -> None: + fake = _FakeWriteClient() + with BufferedWriter(fake, batch_size=10_000, flush_interval=60.0) as buf: + buf.write_columnar("cpu", {"time": [1], "v": [1.0]}) + # After exit, the buffer must be flushed exactly once. + assert len(fake.writes) == 1 + + +EXPECTED_MICROS = 1633046400000000 # 2021-10-01T00:00:00Z in microseconds + + +@pytest.mark.parametrize("unit", ["ns", "us", "ms", "s"]) +@pytest.mark.parametrize("tz", [None, "UTC"]) +def test_pandas_datetime_all_resolutions(unit: str, tz: str | None) -> None: + """Every pandas datetime resolution (tz-naive and tz-aware) scales to micros. + + Regression for a bug where tz-naive datetime64[us]/[ms]/[s] columns were + mis-scaled because numpy datetime64 has no ``.unit`` attribute (only pandas' + tz-aware DatetimeTZDtype does), so the code fell back to a nanosecond + assumption and produced timestamps 1000x+ too small (landing in ~1970). + """ + pd = pytest.importorskip("pandas") + from arc_client.ingestion.msgpack import dataframe_to_columnar + + ts = pd.to_datetime(["2021-10-01T00:00:00"]) + if tz is not None: + ts = ts.tz_localize(tz) + ts = ts.as_unit(unit) + df = pd.DataFrame({"time": ts, "v": [1.0]}) + + assert dataframe_to_columnar(df, "m")["time"] == [EXPECTED_MICROS] + + +def test_polars_datetime_conversion() -> None: + pl = pytest.importorskip("polars") + from datetime import datetime, timezone + + from arc_client.ingestion.msgpack import dataframe_to_columnar + + df = pl.DataFrame({"time": [datetime(2021, 10, 1, tzinfo=timezone.utc)], "v": [1.0]}) + + micros = dataframe_to_columnar(df, "m")["time"] + + assert micros == [1633046400000000] diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py index b8ae69f..d7dac7a 100644 --- a/tests/unit/test_client.py +++ b/tests/unit/test_client.py @@ -4,7 +4,7 @@ import pytest -from arc_client import ArcClient, AsyncArcClient, ClientConfig +from arc_client import ArcClient, AsyncArcClient, ClientConfig, Compression class TestClientConfig: @@ -18,9 +18,18 @@ def test_default_config(self) -> None: assert config.token is None assert config.database == "default" assert config.timeout == 30.0 - assert config.compression is True + assert config.compression is Compression.ZSTD assert config.ssl is False assert config.verify_ssl is True + assert config.max_retries == 3 + + def test_compression_coercion(self) -> None: + """Compression accepts enum, string, and bool (back-compat).""" + assert ClientConfig(compression=True).compression is Compression.GZIP + assert ClientConfig(compression=False).compression is Compression.NONE + assert ClientConfig(compression="gzip").compression is Compression.GZIP + assert ClientConfig(compression="zstd").compression is Compression.ZSTD + assert ClientConfig(compression=Compression.NONE).compression is Compression.NONE def test_base_url_http(self) -> None: """Test base URL generation for HTTP.""" diff --git a/uv.lock b/uv.lock index fed2efb..464c4c8 100644 --- a/uv.lock +++ b/uv.lock @@ -58,8 +58,7 @@ wheels = [ ] [[package]] -name = "arc-client" -version = "0.1.0" +name = "arc-tsdb-client" source = { editable = "." } dependencies = [ { name = "httpx" }, @@ -67,6 +66,7 @@ dependencies = [ { name = "pyarrow", version = "21.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, { name = "pyarrow", version = "22.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, { name = "pydantic" }, + { name = "zstandard" }, ] [package.optional-dependencies] @@ -126,8 +126,9 @@ requires-dist = [ { name = "sphinx", marker = "extra == 'docs'", specifier = ">=7.2.0" }, { name = "sphinx-autodoc-typehints", marker = "extra == 'docs'", specifier = ">=2.0.0" }, { name = "sphinx-rtd-theme", marker = "extra == 'docs'", specifier = ">=2.0.0" }, + { name = "zstandard", specifier = ">=0.22.0" }, ] -provides-extras = ["pandas", "polars", "all", "dev", "docs"] +provides-extras = ["all", "dev", "docs", "pandas", "polars"] [[package]] name = "attrs" @@ -3231,3 +3232,109 @@ sdist = { url = "https://files.pythonhosted.org/packages/e3/02/0f2892c661036d50e wheels = [ { url = "https://files.pythonhosted.org/packages/2e/54/647ade08bf0db230bfea292f893923872fd20be6ac6f53b2b936ba839d75/zipp-3.23.0-py3-none-any.whl", hash = "sha256:071652d6115ed432f5ce1d34c336c0adfd6a884660d1e9712a256d3d3bd4b14e", size = 10276, upload-time = "2025-06-08T17:06:38.034Z" }, ] + +[[package]] +name = "zstandard" +version = "0.25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fd/aa/3e0508d5a5dd96529cdc5a97011299056e14c6505b678fd58938792794b1/zstandard-0.25.0.tar.gz", hash = "sha256:7713e1179d162cf5c7906da876ec2ccb9c3a9dcbdffef0cc7f70c3667a205f0b", size = 711513, upload-time = "2025-09-14T22:15:54.002Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/7a/28efd1d371f1acd037ac64ed1c5e2b41514a6cc937dd6ab6a13ab9f0702f/zstandard-0.25.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e59fdc271772f6686e01e1b3b74537259800f57e24280be3f29c8a0deb1904dd", size = 795256, upload-time = "2025-09-14T22:15:56.415Z" }, + { url = "https://files.pythonhosted.org/packages/96/34/ef34ef77f1ee38fc8e4f9775217a613b452916e633c4f1d98f31db52c4a5/zstandard-0.25.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:4d441506e9b372386a5271c64125f72d5df6d2a8e8a2a45a0ae09b03cb781ef7", size = 640565, upload-time = "2025-09-14T22:15:58.177Z" }, + { url = "https://files.pythonhosted.org/packages/9d/1b/4fdb2c12eb58f31f28c4d28e8dc36611dd7205df8452e63f52fb6261d13e/zstandard-0.25.0-cp310-cp310-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:ab85470ab54c2cb96e176f40342d9ed41e58ca5733be6a893b730e7af9c40550", size = 5345306, upload-time = "2025-09-14T22:16:00.165Z" }, + { url = "https://files.pythonhosted.org/packages/73/28/a44bdece01bca027b079f0e00be3b6bd89a4df180071da59a3dd7381665b/zstandard-0.25.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e05ab82ea7753354bb054b92e2f288afb750e6b439ff6ca78af52939ebbc476d", size = 5055561, upload-time = "2025-09-14T22:16:02.22Z" }, + { url = "https://files.pythonhosted.org/packages/e9/74/68341185a4f32b274e0fc3410d5ad0750497e1acc20bd0f5b5f64ce17785/zstandard-0.25.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:78228d8a6a1c177a96b94f7e2e8d012c55f9c760761980da16ae7546a15a8e9b", size = 5402214, upload-time = "2025-09-14T22:16:04.109Z" }, + { url = "https://files.pythonhosted.org/packages/8b/67/f92e64e748fd6aaffe01e2b75a083c0c4fd27abe1c8747fee4555fcee7dd/zstandard-0.25.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:2b6bd67528ee8b5c5f10255735abc21aa106931f0dbaf297c7be0c886353c3d0", size = 5449703, upload-time = "2025-09-14T22:16:06.312Z" }, + { url = "https://files.pythonhosted.org/packages/fd/e5/6d36f92a197c3c17729a2125e29c169f460538a7d939a27eaaa6dcfcba8e/zstandard-0.25.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4b6d83057e713ff235a12e73916b6d356e3084fd3d14ced499d84240f3eecee0", size = 5556583, upload-time = "2025-09-14T22:16:08.457Z" }, + { url = "https://files.pythonhosted.org/packages/d7/83/41939e60d8d7ebfe2b747be022d0806953799140a702b90ffe214d557638/zstandard-0.25.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:9174f4ed06f790a6869b41cba05b43eeb9a35f8993c4422ab853b705e8112bbd", size = 5045332, upload-time = "2025-09-14T22:16:10.444Z" }, + { url = "https://files.pythonhosted.org/packages/b3/87/d3ee185e3d1aa0133399893697ae91f221fda79deb61adbe998a7235c43f/zstandard-0.25.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:25f8f3cd45087d089aef5ba3848cd9efe3ad41163d3400862fb42f81a3a46701", size = 5572283, upload-time = "2025-09-14T22:16:12.128Z" }, + { url = "https://files.pythonhosted.org/packages/0a/1d/58635ae6104df96671076ac7d4ae7816838ce7debd94aecf83e30b7121b0/zstandard-0.25.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:3756b3e9da9b83da1796f8809dd57cb024f838b9eeafde28f3cb472012797ac1", size = 4959754, upload-time = "2025-09-14T22:16:14.225Z" }, + { url = "https://files.pythonhosted.org/packages/75/d6/57e9cb0a9983e9a229dd8fd2e6e96593ef2aa82a3907188436f22b111ccd/zstandard-0.25.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:81dad8d145d8fd981b2962b686b2241d3a1ea07733e76a2f15435dfb7fb60150", size = 5266477, upload-time = "2025-09-14T22:16:16.343Z" }, + { url = "https://files.pythonhosted.org/packages/d1/a9/ee891e5edf33a6ebce0a028726f0bbd8567effe20fe3d5808c42323e8542/zstandard-0.25.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:a5a419712cf88862a45a23def0ae063686db3d324cec7edbe40509d1a79a0aab", size = 5440914, upload-time = "2025-09-14T22:16:18.453Z" }, + { url = "https://files.pythonhosted.org/packages/58/08/a8522c28c08031a9521f27abc6f78dbdee7312a7463dd2cfc658b813323b/zstandard-0.25.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:e7360eae90809efd19b886e59a09dad07da4ca9ba096752e61a2e03c8aca188e", size = 5819847, upload-time = "2025-09-14T22:16:20.559Z" }, + { url = "https://files.pythonhosted.org/packages/6f/11/4c91411805c3f7b6f31c60e78ce347ca48f6f16d552fc659af6ec3b73202/zstandard-0.25.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:75ffc32a569fb049499e63ce68c743155477610532da1eb38e7f24bf7cd29e74", size = 5363131, upload-time = "2025-09-14T22:16:22.206Z" }, + { url = "https://files.pythonhosted.org/packages/ef/d6/8c4bd38a3b24c4c7676a7a3d8de85d6ee7a983602a734b9f9cdefb04a5d6/zstandard-0.25.0-cp310-cp310-win32.whl", hash = "sha256:106281ae350e494f4ac8a80470e66d1fe27e497052c8d9c3b95dc4cf1ade81aa", size = 436469, upload-time = "2025-09-14T22:16:25.002Z" }, + { url = "https://files.pythonhosted.org/packages/93/90/96d50ad417a8ace5f841b3228e93d1bb13e6ad356737f42e2dde30d8bd68/zstandard-0.25.0-cp310-cp310-win_amd64.whl", hash = "sha256:ea9d54cc3d8064260114a0bbf3479fc4a98b21dffc89b3459edd506b69262f6e", size = 506100, upload-time = "2025-09-14T22:16:23.569Z" }, + { url = "https://files.pythonhosted.org/packages/2a/83/c3ca27c363d104980f1c9cee1101cc8ba724ac8c28a033ede6aab89585b1/zstandard-0.25.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:933b65d7680ea337180733cf9e87293cc5500cc0eb3fc8769f4d3c88d724ec5c", size = 795254, upload-time = "2025-09-14T22:16:26.137Z" }, + { url = "https://files.pythonhosted.org/packages/ac/4d/e66465c5411a7cf4866aeadc7d108081d8ceba9bc7abe6b14aa21c671ec3/zstandard-0.25.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a3f79487c687b1fc69f19e487cd949bf3aae653d181dfb5fde3bf6d18894706f", size = 640559, upload-time = "2025-09-14T22:16:27.973Z" }, + { url = "https://files.pythonhosted.org/packages/12/56/354fe655905f290d3b147b33fe946b0f27e791e4b50a5f004c802cb3eb7b/zstandard-0.25.0-cp311-cp311-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:0bbc9a0c65ce0eea3c34a691e3c4b6889f5f3909ba4822ab385fab9057099431", size = 5348020, upload-time = "2025-09-14T22:16:29.523Z" }, + { url = "https://files.pythonhosted.org/packages/3b/13/2b7ed68bd85e69a2069bcc72141d378f22cae5a0f3b353a2c8f50ef30c1b/zstandard-0.25.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:01582723b3ccd6939ab7b3a78622c573799d5d8737b534b86d0e06ac18dbde4a", size = 5058126, upload-time = "2025-09-14T22:16:31.811Z" }, + { url = "https://files.pythonhosted.org/packages/c9/dd/fdaf0674f4b10d92cb120ccff58bbb6626bf8368f00ebfd2a41ba4a0dc99/zstandard-0.25.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:5f1ad7bf88535edcf30038f6919abe087f606f62c00a87d7e33e7fc57cb69fcc", size = 5405390, upload-time = "2025-09-14T22:16:33.486Z" }, + { url = "https://files.pythonhosted.org/packages/0f/67/354d1555575bc2490435f90d67ca4dd65238ff2f119f30f72d5cde09c2ad/zstandard-0.25.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:06acb75eebeedb77b69048031282737717a63e71e4ae3f77cc0c3b9508320df6", size = 5452914, upload-time = "2025-09-14T22:16:35.277Z" }, + { url = "https://files.pythonhosted.org/packages/bb/1f/e9cfd801a3f9190bf3e759c422bbfd2247db9d7f3d54a56ecde70137791a/zstandard-0.25.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9300d02ea7c6506f00e627e287e0492a5eb0371ec1670ae852fefffa6164b072", size = 5559635, upload-time = "2025-09-14T22:16:37.141Z" }, + { url = "https://files.pythonhosted.org/packages/21/88/5ba550f797ca953a52d708c8e4f380959e7e3280af029e38fbf47b55916e/zstandard-0.25.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:bfd06b1c5584b657a2892a6014c2f4c20e0db0208c159148fa78c65f7e0b0277", size = 5048277, upload-time = "2025-09-14T22:16:38.807Z" }, + { url = "https://files.pythonhosted.org/packages/46/c0/ca3e533b4fa03112facbe7fbe7779cb1ebec215688e5df576fe5429172e0/zstandard-0.25.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:f373da2c1757bb7f1acaf09369cdc1d51d84131e50d5fa9863982fd626466313", size = 5574377, upload-time = "2025-09-14T22:16:40.523Z" }, + { url = "https://files.pythonhosted.org/packages/12/9b/3fb626390113f272abd0799fd677ea33d5fc3ec185e62e6be534493c4b60/zstandard-0.25.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6c0e5a65158a7946e7a7affa6418878ef97ab66636f13353b8502d7ea03c8097", size = 4961493, upload-time = "2025-09-14T22:16:43.3Z" }, + { url = "https://files.pythonhosted.org/packages/cb/d3/23094a6b6a4b1343b27ae68249daa17ae0651fcfec9ed4de09d14b940285/zstandard-0.25.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:c8e167d5adf59476fa3e37bee730890e389410c354771a62e3c076c86f9f7778", size = 5269018, upload-time = "2025-09-14T22:16:45.292Z" }, + { url = "https://files.pythonhosted.org/packages/8c/a7/bb5a0c1c0f3f4b5e9d5b55198e39de91e04ba7c205cc46fcb0f95f0383c1/zstandard-0.25.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:98750a309eb2f020da61e727de7d7ba3c57c97cf6213f6f6277bb7fb42a8e065", size = 5443672, upload-time = "2025-09-14T22:16:47.076Z" }, + { url = "https://files.pythonhosted.org/packages/27/22/503347aa08d073993f25109c36c8d9f029c7d5949198050962cb568dfa5e/zstandard-0.25.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:22a086cff1b6ceca18a8dd6096ec631e430e93a8e70a9ca5efa7561a00f826fa", size = 5822753, upload-time = "2025-09-14T22:16:49.316Z" }, + { url = "https://files.pythonhosted.org/packages/e2/be/94267dc6ee64f0f8ba2b2ae7c7a2df934a816baaa7291db9e1aa77394c3c/zstandard-0.25.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:72d35d7aa0bba323965da807a462b0966c91608ef3a48ba761678cb20ce5d8b7", size = 5366047, upload-time = "2025-09-14T22:16:51.328Z" }, + { url = "https://files.pythonhosted.org/packages/7b/a3/732893eab0a3a7aecff8b99052fecf9f605cf0fb5fb6d0290e36beee47a4/zstandard-0.25.0-cp311-cp311-win32.whl", hash = "sha256:f5aeea11ded7320a84dcdd62a3d95b5186834224a9e55b92ccae35d21a8b63d4", size = 436484, upload-time = "2025-09-14T22:16:55.005Z" }, + { url = "https://files.pythonhosted.org/packages/43/a3/c6155f5c1cce691cb80dfd38627046e50af3ee9ddc5d0b45b9b063bfb8c9/zstandard-0.25.0-cp311-cp311-win_amd64.whl", hash = "sha256:daab68faadb847063d0c56f361a289c4f268706b598afbf9ad113cbe5c38b6b2", size = 506183, upload-time = "2025-09-14T22:16:52.753Z" }, + { url = "https://files.pythonhosted.org/packages/8c/3e/8945ab86a0820cc0e0cdbf38086a92868a9172020fdab8a03ac19662b0e5/zstandard-0.25.0-cp311-cp311-win_arm64.whl", hash = "sha256:22a06c5df3751bb7dc67406f5374734ccee8ed37fc5981bf1ad7041831fa1137", size = 462533, upload-time = "2025-09-14T22:16:53.878Z" }, + { url = "https://files.pythonhosted.org/packages/82/fc/f26eb6ef91ae723a03e16eddb198abcfce2bc5a42e224d44cc8b6765e57e/zstandard-0.25.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7b3c3a3ab9daa3eed242d6ecceead93aebbb8f5f84318d82cee643e019c4b73b", size = 795738, upload-time = "2025-09-14T22:16:56.237Z" }, + { url = "https://files.pythonhosted.org/packages/aa/1c/d920d64b22f8dd028a8b90e2d756e431a5d86194caa78e3819c7bf53b4b3/zstandard-0.25.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:913cbd31a400febff93b564a23e17c3ed2d56c064006f54efec210d586171c00", size = 640436, upload-time = "2025-09-14T22:16:57.774Z" }, + { url = "https://files.pythonhosted.org/packages/53/6c/288c3f0bd9fcfe9ca41e2c2fbfd17b2097f6af57b62a81161941f09afa76/zstandard-0.25.0-cp312-cp312-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:011d388c76b11a0c165374ce660ce2c8efa8e5d87f34996aa80f9c0816698b64", size = 5343019, upload-time = "2025-09-14T22:16:59.302Z" }, + { url = "https://files.pythonhosted.org/packages/1e/15/efef5a2f204a64bdb5571e6161d49f7ef0fffdbca953a615efbec045f60f/zstandard-0.25.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dffecc361d079bb48d7caef5d673c88c8988d3d33fb74ab95b7ee6da42652ea", size = 5063012, upload-time = "2025-09-14T22:17:01.156Z" }, + { url = "https://files.pythonhosted.org/packages/b7/37/a6ce629ffdb43959e92e87ebdaeebb5ac81c944b6a75c9c47e300f85abdf/zstandard-0.25.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:7149623bba7fdf7e7f24312953bcf73cae103db8cae49f8154dd1eadc8a29ecb", size = 5394148, upload-time = "2025-09-14T22:17:03.091Z" }, + { url = "https://files.pythonhosted.org/packages/e3/79/2bf870b3abeb5c070fe2d670a5a8d1057a8270f125ef7676d29ea900f496/zstandard-0.25.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:6a573a35693e03cf1d67799fd01b50ff578515a8aeadd4595d2a7fa9f3ec002a", size = 5451652, upload-time = "2025-09-14T22:17:04.979Z" }, + { url = "https://files.pythonhosted.org/packages/53/60/7be26e610767316c028a2cbedb9a3beabdbe33e2182c373f71a1c0b88f36/zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5a56ba0db2d244117ed744dfa8f6f5b366e14148e00de44723413b2f3938a902", size = 5546993, upload-time = "2025-09-14T22:17:06.781Z" }, + { url = "https://files.pythonhosted.org/packages/85/c7/3483ad9ff0662623f3648479b0380d2de5510abf00990468c286c6b04017/zstandard-0.25.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:10ef2a79ab8e2974e2075fb984e5b9806c64134810fac21576f0668e7ea19f8f", size = 5046806, upload-time = "2025-09-14T22:17:08.415Z" }, + { url = "https://files.pythonhosted.org/packages/08/b3/206883dd25b8d1591a1caa44b54c2aad84badccf2f1de9e2d60a446f9a25/zstandard-0.25.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:aaf21ba8fb76d102b696781bddaa0954b782536446083ae3fdaa6f16b25a1c4b", size = 5576659, upload-time = "2025-09-14T22:17:10.164Z" }, + { url = "https://files.pythonhosted.org/packages/9d/31/76c0779101453e6c117b0ff22565865c54f48f8bd807df2b00c2c404b8e0/zstandard-0.25.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1869da9571d5e94a85a5e8d57e4e8807b175c9e4a6294e3b66fa4efb074d90f6", size = 4953933, upload-time = "2025-09-14T22:17:11.857Z" }, + { url = "https://files.pythonhosted.org/packages/18/e1/97680c664a1bf9a247a280a053d98e251424af51f1b196c6d52f117c9720/zstandard-0.25.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:809c5bcb2c67cd0ed81e9229d227d4ca28f82d0f778fc5fea624a9def3963f91", size = 5268008, upload-time = "2025-09-14T22:17:13.627Z" }, + { url = "https://files.pythonhosted.org/packages/1e/73/316e4010de585ac798e154e88fd81bb16afc5c5cb1a72eeb16dd37e8024a/zstandard-0.25.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f27662e4f7dbf9f9c12391cb37b4c4c3cb90ffbd3b1fb9284dadbbb8935fa708", size = 5433517, upload-time = "2025-09-14T22:17:16.103Z" }, + { url = "https://files.pythonhosted.org/packages/5b/60/dd0f8cfa8129c5a0ce3ea6b7f70be5b33d2618013a161e1ff26c2b39787c/zstandard-0.25.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:99c0c846e6e61718715a3c9437ccc625de26593fea60189567f0118dc9db7512", size = 5814292, upload-time = "2025-09-14T22:17:17.827Z" }, + { url = "https://files.pythonhosted.org/packages/fc/5f/75aafd4b9d11b5407b641b8e41a57864097663699f23e9ad4dbb91dc6bfe/zstandard-0.25.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:474d2596a2dbc241a556e965fb76002c1ce655445e4e3bf38e5477d413165ffa", size = 5360237, upload-time = "2025-09-14T22:17:19.954Z" }, + { url = "https://files.pythonhosted.org/packages/ff/8d/0309daffea4fcac7981021dbf21cdb2e3427a9e76bafbcdbdf5392ff99a4/zstandard-0.25.0-cp312-cp312-win32.whl", hash = "sha256:23ebc8f17a03133b4426bcc04aabd68f8236eb78c3760f12783385171b0fd8bd", size = 436922, upload-time = "2025-09-14T22:17:24.398Z" }, + { url = "https://files.pythonhosted.org/packages/79/3b/fa54d9015f945330510cb5d0b0501e8253c127cca7ebe8ba46a965df18c5/zstandard-0.25.0-cp312-cp312-win_amd64.whl", hash = "sha256:ffef5a74088f1e09947aecf91011136665152e0b4b359c42be3373897fb39b01", size = 506276, upload-time = "2025-09-14T22:17:21.429Z" }, + { url = "https://files.pythonhosted.org/packages/ea/6b/8b51697e5319b1f9ac71087b0af9a40d8a6288ff8025c36486e0c12abcc4/zstandard-0.25.0-cp312-cp312-win_arm64.whl", hash = "sha256:181eb40e0b6a29b3cd2849f825e0fa34397f649170673d385f3598ae17cca2e9", size = 462679, upload-time = "2025-09-14T22:17:23.147Z" }, + { url = "https://files.pythonhosted.org/packages/35/0b/8df9c4ad06af91d39e94fa96cc010a24ac4ef1378d3efab9223cc8593d40/zstandard-0.25.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ec996f12524f88e151c339688c3897194821d7f03081ab35d31d1e12ec975e94", size = 795735, upload-time = "2025-09-14T22:17:26.042Z" }, + { url = "https://files.pythonhosted.org/packages/3f/06/9ae96a3e5dcfd119377ba33d4c42a7d89da1efabd5cb3e366b156c45ff4d/zstandard-0.25.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a1a4ae2dec3993a32247995bdfe367fc3266da832d82f8438c8570f989753de1", size = 640440, upload-time = "2025-09-14T22:17:27.366Z" }, + { url = "https://files.pythonhosted.org/packages/d9/14/933d27204c2bd404229c69f445862454dcc101cd69ef8c6068f15aaec12c/zstandard-0.25.0-cp313-cp313-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:e96594a5537722fdfb79951672a2a63aec5ebfb823e7560586f7484819f2a08f", size = 5343070, upload-time = "2025-09-14T22:17:28.896Z" }, + { url = "https://files.pythonhosted.org/packages/6d/db/ddb11011826ed7db9d0e485d13df79b58586bfdec56e5c84a928a9a78c1c/zstandard-0.25.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bfc4e20784722098822e3eee42b8e576b379ed72cca4a7cb856ae733e62192ea", size = 5063001, upload-time = "2025-09-14T22:17:31.044Z" }, + { url = "https://files.pythonhosted.org/packages/db/00/87466ea3f99599d02a5238498b87bf84a6348290c19571051839ca943777/zstandard-0.25.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:457ed498fc58cdc12fc48f7950e02740d4f7ae9493dd4ab2168a47c93c31298e", size = 5394120, upload-time = "2025-09-14T22:17:32.711Z" }, + { url = "https://files.pythonhosted.org/packages/2b/95/fc5531d9c618a679a20ff6c29e2b3ef1d1f4ad66c5e161ae6ff847d102a9/zstandard-0.25.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:fd7a5004eb1980d3cefe26b2685bcb0b17989901a70a1040d1ac86f1d898c551", size = 5451230, upload-time = "2025-09-14T22:17:34.41Z" }, + { url = "https://files.pythonhosted.org/packages/63/4b/e3678b4e776db00f9f7b2fe58e547e8928ef32727d7a1ff01dea010f3f13/zstandard-0.25.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8e735494da3db08694d26480f1493ad2cf86e99bdd53e8e9771b2752a5c0246a", size = 5547173, upload-time = "2025-09-14T22:17:36.084Z" }, + { url = "https://files.pythonhosted.org/packages/4e/d5/ba05ed95c6b8ec30bd468dfeab20589f2cf709b5c940483e31d991f2ca58/zstandard-0.25.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:3a39c94ad7866160a4a46d772e43311a743c316942037671beb264e395bdd611", size = 5046736, upload-time = "2025-09-14T22:17:37.891Z" }, + { url = "https://files.pythonhosted.org/packages/50/d5/870aa06b3a76c73eced65c044b92286a3c4e00554005ff51962deef28e28/zstandard-0.25.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:172de1f06947577d3a3005416977cce6168f2261284c02080e7ad0185faeced3", size = 5576368, upload-time = "2025-09-14T22:17:40.206Z" }, + { url = "https://files.pythonhosted.org/packages/5d/35/398dc2ffc89d304d59bc12f0fdd931b4ce455bddf7038a0a67733a25f550/zstandard-0.25.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3c83b0188c852a47cd13ef3bf9209fb0a77fa5374958b8c53aaa699398c6bd7b", size = 4954022, upload-time = "2025-09-14T22:17:41.879Z" }, + { url = "https://files.pythonhosted.org/packages/9a/5c/36ba1e5507d56d2213202ec2b05e8541734af5f2ce378c5d1ceaf4d88dc4/zstandard-0.25.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:1673b7199bbe763365b81a4f3252b8e80f44c9e323fc42940dc8843bfeaf9851", size = 5267889, upload-time = "2025-09-14T22:17:43.577Z" }, + { url = "https://files.pythonhosted.org/packages/70/e8/2ec6b6fb7358b2ec0113ae202647ca7c0e9d15b61c005ae5225ad0995df5/zstandard-0.25.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:0be7622c37c183406f3dbf0cba104118eb16a4ea7359eeb5752f0794882fc250", size = 5433952, upload-time = "2025-09-14T22:17:45.271Z" }, + { url = "https://files.pythonhosted.org/packages/7b/01/b5f4d4dbc59ef193e870495c6f1275f5b2928e01ff5a81fecb22a06e22fb/zstandard-0.25.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:5f5e4c2a23ca271c218ac025bd7d635597048b366d6f31f420aaeb715239fc98", size = 5814054, upload-time = "2025-09-14T22:17:47.08Z" }, + { url = "https://files.pythonhosted.org/packages/b2/e5/fbd822d5c6f427cf158316d012c5a12f233473c2f9c5fe5ab1ae5d21f3d8/zstandard-0.25.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf", size = 5360113, upload-time = "2025-09-14T22:17:48.893Z" }, + { url = "https://files.pythonhosted.org/packages/8e/e0/69a553d2047f9a2c7347caa225bb3a63b6d7704ad74610cb7823baa08ed7/zstandard-0.25.0-cp313-cp313-win32.whl", hash = "sha256:7030defa83eef3e51ff26f0b7bfb229f0204b66fe18e04359ce3474ac33cbc09", size = 436936, upload-time = "2025-09-14T22:17:52.658Z" }, + { url = "https://files.pythonhosted.org/packages/d9/82/b9c06c870f3bd8767c201f1edbdf9e8dc34be5b0fbc5682c4f80fe948475/zstandard-0.25.0-cp313-cp313-win_amd64.whl", hash = "sha256:1f830a0dac88719af0ae43b8b2d6aef487d437036468ef3c2ea59c51f9d55fd5", size = 506232, upload-time = "2025-09-14T22:17:50.402Z" }, + { url = "https://files.pythonhosted.org/packages/d4/57/60c3c01243bb81d381c9916e2a6d9e149ab8627c0c7d7abb2d73384b3c0c/zstandard-0.25.0-cp313-cp313-win_arm64.whl", hash = "sha256:85304a43f4d513f5464ceb938aa02c1e78c2943b29f44a750b48b25ac999a049", size = 462671, upload-time = "2025-09-14T22:17:51.533Z" }, + { url = "https://files.pythonhosted.org/packages/3d/5c/f8923b595b55fe49e30612987ad8bf053aef555c14f05bb659dd5dbe3e8a/zstandard-0.25.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:e29f0cf06974c899b2c188ef7f783607dbef36da4c242eb6c82dcd8b512855e3", size = 795887, upload-time = "2025-09-14T22:17:54.198Z" }, + { url = "https://files.pythonhosted.org/packages/8d/09/d0a2a14fc3439c5f874042dca72a79c70a532090b7ba0003be73fee37ae2/zstandard-0.25.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:05df5136bc5a011f33cd25bc9f506e7426c0c9b3f9954f056831ce68f3b6689f", size = 640658, upload-time = "2025-09-14T22:17:55.423Z" }, + { url = "https://files.pythonhosted.org/packages/5d/7c/8b6b71b1ddd517f68ffb55e10834388d4f793c49c6b83effaaa05785b0b4/zstandard-0.25.0-cp314-cp314-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:f604efd28f239cc21b3adb53eb061e2a205dc164be408e553b41ba2ffe0ca15c", size = 5379849, upload-time = "2025-09-14T22:17:57.372Z" }, + { url = "https://files.pythonhosted.org/packages/a4/86/a48e56320d0a17189ab7a42645387334fba2200e904ee47fc5a26c1fd8ca/zstandard-0.25.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:223415140608d0f0da010499eaa8ccdb9af210a543fac54bce15babbcfc78439", size = 5058095, upload-time = "2025-09-14T22:17:59.498Z" }, + { url = "https://files.pythonhosted.org/packages/f8/ad/eb659984ee2c0a779f9d06dbfe45e2dc39d99ff40a319895df2d3d9a48e5/zstandard-0.25.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e54296a283f3ab5a26fc9b8b5d4978ea0532f37b231644f367aa588930aa043", size = 5551751, upload-time = "2025-09-14T22:18:01.618Z" }, + { url = "https://files.pythonhosted.org/packages/61/b3/b637faea43677eb7bd42ab204dfb7053bd5c4582bfe6b1baefa80ac0c47b/zstandard-0.25.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ca54090275939dc8ec5dea2d2afb400e0f83444b2fc24e07df7fdef677110859", size = 6364818, upload-time = "2025-09-14T22:18:03.769Z" }, + { url = "https://files.pythonhosted.org/packages/31/dc/cc50210e11e465c975462439a492516a73300ab8caa8f5e0902544fd748b/zstandard-0.25.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e09bb6252b6476d8d56100e8147b803befa9a12cea144bbe629dd508800d1ad0", size = 5560402, upload-time = "2025-09-14T22:18:05.954Z" }, + { url = "https://files.pythonhosted.org/packages/c9/ae/56523ae9c142f0c08efd5e868a6da613ae76614eca1305259c3bf6a0ed43/zstandard-0.25.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a9ec8c642d1ec73287ae3e726792dd86c96f5681eb8df274a757bf62b750eae7", size = 4955108, upload-time = "2025-09-14T22:18:07.68Z" }, + { url = "https://files.pythonhosted.org/packages/98/cf/c899f2d6df0840d5e384cf4c4121458c72802e8bda19691f3b16619f51e9/zstandard-0.25.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4089a10e598eae6393756b036e0f419e8c1d60f44a831520f9af41c14216cf2", size = 5269248, upload-time = "2025-09-14T22:18:09.753Z" }, + { url = "https://files.pythonhosted.org/packages/1b/c0/59e912a531d91e1c192d3085fc0f6fb2852753c301a812d856d857ea03c6/zstandard-0.25.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:f67e8f1a324a900e75b5e28ffb152bcac9fbed1cc7b43f99cd90f395c4375344", size = 5430330, upload-time = "2025-09-14T22:18:11.966Z" }, + { url = "https://files.pythonhosted.org/packages/a0/1d/7e31db1240de2df22a58e2ea9a93fc6e38cc29353e660c0272b6735d6669/zstandard-0.25.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:9654dbc012d8b06fc3d19cc825af3f7bf8ae242226df5f83936cb39f5fdc846c", size = 5811123, upload-time = "2025-09-14T22:18:13.907Z" }, + { url = "https://files.pythonhosted.org/packages/f6/49/fac46df5ad353d50535e118d6983069df68ca5908d4d65b8c466150a4ff1/zstandard-0.25.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:4203ce3b31aec23012d3a4cf4a2ed64d12fea5269c49aed5e4c3611b938e4088", size = 5359591, upload-time = "2025-09-14T22:18:16.465Z" }, + { url = "https://files.pythonhosted.org/packages/c2/38/f249a2050ad1eea0bb364046153942e34abba95dd5520af199aed86fbb49/zstandard-0.25.0-cp314-cp314-win32.whl", hash = "sha256:da469dc041701583e34de852d8634703550348d5822e66a0c827d39b05365b12", size = 444513, upload-time = "2025-09-14T22:18:20.61Z" }, + { url = "https://files.pythonhosted.org/packages/3a/43/241f9615bcf8ba8903b3f0432da069e857fc4fd1783bd26183db53c4804b/zstandard-0.25.0-cp314-cp314-win_amd64.whl", hash = "sha256:c19bcdd826e95671065f8692b5a4aa95c52dc7a02a4c5a0cac46deb879a017a2", size = 516118, upload-time = "2025-09-14T22:18:17.849Z" }, + { url = "https://files.pythonhosted.org/packages/f0/ef/da163ce2450ed4febf6467d77ccb4cd52c4c30ab45624bad26ca0a27260c/zstandard-0.25.0-cp314-cp314-win_arm64.whl", hash = "sha256:d7541afd73985c630bafcd6338d2518ae96060075f9463d7dc14cfb33514383d", size = 476940, upload-time = "2025-09-14T22:18:19.088Z" }, + { url = "https://files.pythonhosted.org/packages/14/0d/d0a405dad6ab6f9f759c26d866cca66cb209bff6f8db656074d662a953dd/zstandard-0.25.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:b9af1fe743828123e12b41dd8091eca1074d0c1569cc42e6e1eee98027f2bbd0", size = 795263, upload-time = "2025-09-14T22:18:21.683Z" }, + { url = "https://files.pythonhosted.org/packages/ca/aa/ceb8d79cbad6dabd4cb1178ca853f6a4374d791c5e0241a0988173e2a341/zstandard-0.25.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:4b14abacf83dfb5c25eb4e4a79520de9e7e205f72c9ee7702f91233ae57d33a2", size = 640560, upload-time = "2025-09-14T22:18:22.867Z" }, + { url = "https://files.pythonhosted.org/packages/88/cd/2cf6d476131b509cc122d25d3416a2d0aa17687ddbada7599149f9da620e/zstandard-0.25.0-cp39-cp39-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:a51ff14f8017338e2f2e5dab738ce1ec3b5a851f23b18c1ae1359b1eecbee6df", size = 5344244, upload-time = "2025-09-14T22:18:24.724Z" }, + { url = "https://files.pythonhosted.org/packages/5c/71/e14820b61a1c137966b7667b400b72fa4a45c836257e443f3d77607db268/zstandard-0.25.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3b870ce5a02d4b22286cf4944c628e0f0881b11b3f14667c1d62185a99e04f53", size = 5054550, upload-time = "2025-09-14T22:18:26.445Z" }, + { url = "https://files.pythonhosted.org/packages/f9/ce/26dc5a6fa956be41d0e984909224ed196ee6f91d607f0b3fd84577741a77/zstandard-0.25.0-cp39-cp39-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:05353cef599a7b0b98baca9b068dd36810c3ef0f42bf282583f438caf6ddcee3", size = 5401150, upload-time = "2025-09-14T22:18:28.745Z" }, + { url = "https://files.pythonhosted.org/packages/f2/1b/402cab5edcfe867465daf869d5ac2a94930931c0989633bc01d6a7d8bd68/zstandard-0.25.0-cp39-cp39-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:19796b39075201d51d5f5f790bf849221e58b48a39a5fc74837675d8bafc7362", size = 5448595, upload-time = "2025-09-14T22:18:30.475Z" }, + { url = "https://files.pythonhosted.org/packages/86/b2/fc50c58271a1ead0e5a0a0e6311f4b221f35954dce438ce62751b3af9b68/zstandard-0.25.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:53e08b2445a6bc241261fea89d065536f00a581f02535f8122eba42db9375530", size = 5555290, upload-time = "2025-09-14T22:18:32.336Z" }, + { url = "https://files.pythonhosted.org/packages/d2/20/5f72d6ba970690df90fdd37195c5caa992e70cb6f203f74cc2bcc0b8cf30/zstandard-0.25.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:1f3689581a72eaba9131b1d9bdbfe520ccd169999219b41000ede2fca5c1bfdb", size = 5043898, upload-time = "2025-09-14T22:18:34.215Z" }, + { url = "https://files.pythonhosted.org/packages/e4/f1/131a0382b8b8d11e84690574645f528f5c5b9343e06cefd77f5fd730cd2b/zstandard-0.25.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:d8c56bb4e6c795fc77d74d8e8b80846e1fb8292fc0b5060cd8131d522974b751", size = 5571173, upload-time = "2025-09-14T22:18:36.117Z" }, + { url = "https://files.pythonhosted.org/packages/53/f6/2a37931023f737fd849c5c28def57442bbafadb626da60cf9ed58461fe24/zstandard-0.25.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:53f94448fe5b10ee75d246497168e5825135d54325458c4bfffbaafabcc0a577", size = 4958261, upload-time = "2025-09-14T22:18:38.098Z" }, + { url = "https://files.pythonhosted.org/packages/b5/52/ca76ed6dbfd8845a5563d3af4e972da3b9da8a9308ca6b56b0b929d93e23/zstandard-0.25.0-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:c2ba942c94e0691467ab901fc51b6f2085ff48f2eea77b1a48240f011e8247c7", size = 5265680, upload-time = "2025-09-14T22:18:39.834Z" }, + { url = "https://files.pythonhosted.org/packages/7a/59/edd117dedb97a768578b49fb2f1156defb839d1aa5b06200a62be943667f/zstandard-0.25.0-cp39-cp39-musllinux_1_2_ppc64le.whl", hash = "sha256:07b527a69c1e1c8b5ab1ab14e2afe0675614a09182213f21a0717b62027b5936", size = 5439747, upload-time = "2025-09-14T22:18:41.647Z" }, + { url = "https://files.pythonhosted.org/packages/75/71/c2e9234643dcfbd6c5e975e9a2b0050e1b2afffda6c3a959e1b87997bc80/zstandard-0.25.0-cp39-cp39-musllinux_1_2_s390x.whl", hash = "sha256:51526324f1b23229001eb3735bc8c94f9c578b1bd9e867a0a646a3b17109f388", size = 5818805, upload-time = "2025-09-14T22:18:43.602Z" }, + { url = "https://files.pythonhosted.org/packages/f5/93/8ebc19f0a31c44ea0e7348f9b0d4b326ed413b6575a3c6ff4ed50222abb6/zstandard-0.25.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:89c4b48479a43f820b749df49cd7ba2dbc2b1b78560ecb5ab52985574fd40b27", size = 5362280, upload-time = "2025-09-14T22:18:45.625Z" }, + { url = "https://files.pythonhosted.org/packages/b8/e9/29cc59d4a9d51b3fd8b477d858d0bd7ab627f700908bf1517f46ddd470ae/zstandard-0.25.0-cp39-cp39-win32.whl", hash = "sha256:1cd5da4d8e8ee0e88be976c294db744773459d51bb32f707a0f166e5ad5c8649", size = 436460, upload-time = "2025-09-14T22:18:49.077Z" }, + { url = "https://files.pythonhosted.org/packages/41/b5/bc7a92c116e2ef32dc8061c209d71e97ff6df37487d7d39adb51a343ee89/zstandard-0.25.0-cp39-cp39-win_amd64.whl", hash = "sha256:37daddd452c0ffb65da00620afb8e17abd4adaae6ce6310702841760c2c26860", size = 506097, upload-time = "2025-09-14T22:18:47.342Z" }, +]