feat(server-rust): import the Rust Durable Streams server#4652
Merged
Conversation
New Rust implementation of the Durable Streams protocol following the zero-copy research in notes/rust-server-research.md: - Contiguous wire-byte storage: data files hold exactly the response bytes (JSON stored as `value,` runs), so reads are pure byte ranges - Coalesced group-commit barrier-fsync (matches Node libuv durability) - Per-stream serialization for producer validation; watch-channel wakeups for long-poll/SSE - Conformance: 247 passed, all failures are fork tests (501, out of scope for now); forks, __ds control plane, compression not implemented - Benchmarks vs Node server: 35x lower latency overhead, 19.6x small-message throughput, ~9x read throughput (bench/RESULTS.md) Includes conformance harness (conformance/) and benchmark tooling (bench/: official suite runner, Node server launcher, multi-process scale-out load generator). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements forks per PROTOCOL.md: offset + sub-offset divergence (JSON message counting / binary bytes), inherited reads through the fork parent chain via stitched file segments, content-type and TTL/expiry inheritance, idempotent fork re-creation, soft-delete refcount lifecycle (410 on direct ops, 409 on re-create/fork-from, cascade GC through chains when the last fork is deleted). Reads also no longer take the appender lock (shared Arc<File> on StreamState), removing reader/writer contention. Conformance: 326 passed / 6 skipped / 0 failed (skips: subscriptions, disabled). Benchmarks re-verified, no regressions (RESULTS.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each stream gets a .meta sidecar: written durably (fsync) on create/close/delete and refcount changes; producer state and TTL access times flush debounced (100ms, no fsync) off the hot path. On boot the store rebuilds every stream from data file + sidecar — tail is just base_offset + file size thanks to the contiguous wire-byte layout (no log scan) — and re-links fork parent chains recursively, including soft-deleted sources. Crash window (documented, allowed by PROTOCOL.md): producer dedup state may lag the data file; producers should bump their epoch after a server crash. Verified: 8-scenario manual restart test (data, producers, closed, TTL, fork chains, soft-delete, append continuity, post-restart cascade GC), full conformance still 326/0, benchmarks unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--host (default 127.0.0.1) so the server can bind 0.0.0.0 in containers. Full conformance suite passes on Linux (326/0, rust:1-slim container), exercising the fdatasync durability path; scale-out append sanity run sustained 19k msg/s through the Docker VM. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Handlers now speak api::{Req, Resp, Body} instead of hyper types. The
key piece is Body::FileRange: read responses are described as data-file
segments plus optional JSON framing bytes, and the HTTP engine decides
how to serve them — the hyper engine (engine_hyper.rs) copies through
userspace buffers as before; a future engine can sendfile the same
description. No behavior change: conformance 326/0.
Prepares for a pluggable --http-engine flag (hyper | raw).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds engine_raw.rs: a minimal HTTP/1.1 server loop (tokio + httparse) selectable with --http-engine raw (default remains hyper). Supports keep-alive, content-length and chunked request bodies, Expect: 100-continue, sized responses, and chunked streaming for SSE. Owning the socket directly is what permits a kernel zero-copy read path (sendfile), which lands separately. Both engines pass the full conformance suite (326/0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Body::FileRange segments are now served with sendfile on Linux — the kernel copies page cache → socket directly with no userspace buffer, driven by write readiness on the nonblocking socket (tokio try_io). Other platforms keep the positioned-read fallback. Validated in a Linux container: full conformance 326/0 with --http-engine raw, including fork-chain stitched reads (multiple sendfile segments per response with JSON framing interleaved). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements the reserved subscription APIs per PROTOCOL.md:
- PUT/GET/DELETE /__ds/subscriptions/:id with idempotent re-confirm,
glob patterns (* one segment, ** any), explicit stream membership
(POST/DELETE .../streams), SSRF validation of webhook URLs (https
only except loopback; RFC1918/link-local rejected)
- Ed25519-signed webhook delivery (Webhook-Signature: t=..,kid=..,
ed25519=<base64url over "t.body">) with JWKS publication at
/__ds/jwks.json; synchronous {done:true} auto-acks the wake snapshot
- Async callbacks and pull-wake claim/ack/release with
subscription-scoped leases, generation fencing (409 FENCED), and
ALREADY_CLAIMED arbitration; wake events append to the configured
wake stream through the normal append path
- Pattern subscriptions backfill existing streams at their tail; new
matching streams link from offset 0 and trigger wakes on activity
Registry is in-memory (not persisted across restarts); webhook
delivery client is plain HTTP — front TLS with a proxy.
Conformance now runs with subscriptions enabled: 332 passed / 0
skipped / 0 failed, on BOTH HTTP engines (hyper and raw).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three-way comparison on macOS + Linux. Headline: on Linux the raw engine serves equal read throughput (~1,580 MB/s) at 11.8% CPU vs hyper's 113.9% — ~10x less server CPU from sendfile(2). Append path remains fsync-bound; read path remains load-generator-bound, so the CPU measurement is what exposes the real server-side gap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From three review passes (correctness / cleanup / performance): Bugs / hardening: - engine_raw sendfile: treat a 0-byte return with bytes still pending as an error instead of spinning forever (e.g. file truncated mid-read); skip zero-length segments up front. - engine_raw chunked decode: use checked_add for the chunk size + CRLF and the running-total bound, rejecting hostile chunk-size lines that would otherwise overflow. Cleanup / dedup (no behavior change, both engines still 332/332): - Shared base64 in api.rs (charset + pad params) replaces the two near- identical encoders in handlers.rs (standard+pad) and subs.rs (url, no pad). - store::materialize_segments() consolidates the buffered segment-read used by the hyper engine, SSE batches, and inline reads (was duplicated 3×). - Segment::file_end() replaces repeated file_start+len arithmetic. - subs::stream_list_json() consolidates the wake/claim payload builder. - collapse a nested if (clippy); clippy now clean. Net -127/+123 lines. Verified: hyper 332/332 and raw 332/332 on macOS; raw on Linux passes all read/fork/sendfile coverage (the only 2 misses are webhook-delivery tests, which need the containerized server to reach the host's loopback — environmental, not code). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…llocs Second correctness pass confirmed async-safety is clean (no std Mutex held across .await anywhere in the control plane) and the fsync/fencing/lease logic is sound. It found one real bug, now fixed: - subs::decode_chunked could panic on `&raw[size+2..]` when a webhook endpoint returns a chunk without the trailing CRLF. With panic=abort that crashes the server — a webhook-response-triggered DoS. Now bounds- checked with checked_add (matches the request-side decoder). Performance: the two constant security headers (nosniff, CORP) are now emitted directly by each engine's response writer instead of allocating two Strings per response in the handler hot path. Post-change scale-out (raw, macOS): 33,020 appends/s and 3,999 MB/s reads — no regression. Both engines remain 332/332. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reorient the README from a design doc to a quickstart: lead with what Durable Streams is and what this implementation is, build/run + a copy-paste curl walkthrough, flags table, scope (now including the __ds control plane), and a preliminary benchmarks section inlined from bench/RESULTS.md for review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ESULTS.md Inline the benchmark numbers in the README as the canonical copy and add reproduce steps for the official suite (Rust on each engine + Node baseline). Remove the scale-out load generator (bench/scale-out.ts) and the separate bench/RESULTS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop notes/rust-server-research.md (design scratchpad) and the dangling reference to it in store.rs now that the implementation and README cover the on-disk layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Rust server had a full conformance config but it was never run in CI — only as a manual RUST_SERVER_URL command. Wire it in like the Caddy job: - The conformance harness now spawns the release binary itself (with --long-poll-timeout-ms matched to the suite), falling back to RUST_SERVER_URL for manual runs against an external server. - Register a `server-rust` vitest project (mirrors `caddy`). - Add a `conformance (rust)` CI job: cargo build --release, then run the full suite against the binary. Same suite Caddy/Node run — 332/332. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…SSE close Two wire-level differences from the reference Node server made the TypeScript client classify Rust responses differently: - Stream-Seq regression returned a 409 body of "Stream-Seq regression". The client distinguishes a sequence conflict from a generic conflict by the word "sequence" in the message, so it fell back to CONFLICT instead of SEQUENCE_CONFLICT. Use "Sequence conflict" to match the Node server. - When a stream is closed atomically with a final append, the SSE reader emitted `data` then a plain up-to-date `control` and only signalled the close in a later, separate control event. A reader taking one chunk saw streamClosed=false. Fold streamClosed:true into the control event that immediately follows the final data, matching the Node server. Verified: server conformance still 332/332 on both engines; the full TS client conformance suite against the Rust server now passes every test except the 23 that require the Node test server's fault-injection endpoint (retry/parse/SSE-resilience), which a real server cannot provide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wrap a long line flagged by `prettier --check` (autofix.ci). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a tag-triggered (server-rust-v*) workflow that builds the durable-streams-server binary for linux + macOS (x86_64 and arm64) and attaches the tarballs + SHA-256 checksums to a GitHub Release, mirroring the Caddy release model. Add repository/homepage/readme to Cargo.toml and document the release process in the README. (crates.io is intentionally not wired up: the obvious crate name is taken by an unrelated third party, so that channel needs a naming decision.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…urability Address review findings (verified: hyper + raw conformance both 332/332): - subs.rs SSRF: webhook_url_allowed now resolves the host and classifies the resolved IP(s) against private/link-local/ULA/metadata ranges (incl. IPv4-mapped IPv6), closing the encoded-IP and IPv6 bypasses of the old textual prefix check. Loopback http delivery (the documented dev exception the suite relies on) is preserved. Residual DNS-rebind-at-delivery risk is documented. Also cap the webhook response read at 1 MiB (was unbounded). - engine_raw.rs: bound the chunked-body *trailer* scan by MAX_HEADER_BYTES — a `0\r\n` followed by an endless non-terminating stream could otherwise grow the buffer without limit (OOM/DoS). The size-line scan was already bounded. - engines: unify the request body-size cap. The raw engine capped at 1 GiB (413); the hyper engine was unbounded, so the same request got different status by --http-engine. Move MAX_BODY_BYTES to api.rs and enforce 413 in both (hyper checks the size hint, then the collected length). - handlers.rs parse_rfc3339: reject impossible calendar dates (per-month day limits + leap February) and seconds >= 60, instead of silently rolling them into a different expiry instant. Matches the reference server's strictness. - handlers.rs close durability: notify long-poll/SSE readers of closure only AFTER the closure is made durable (fsync), matching the Node ordering (data durable -> close durable -> notify). Previously readers could observe EOF before the close was durable; a crash in that window recovered the stream OPEN, violating monotonic closure (PROTOCOL.md §4.1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…set MSRV - release workflow: boot the freshly-built binary and hit /health on each target before packaging, so a miscompiled or wrong-arch artifact can't ship silently (each target builds on a native runner, so the binary runs there). - gitignore benchmark-results.json (written to cwd by the bench harness). - set rust-version = "1.74" in Cargo.toml (documents the supported minimum). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The __ds control plane (subscriptions, pull-wake leases/cursors, and the Ed25519 webhook-signing key) is not persisted across restarts. Surface this as a "known limitation" note in the README and a startup log line, so operators know subscriptions and the signing kid/JWKS reset on restart. Persisting the control plane is a planned follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The raw engine ran sendfile inline on the async worker. On a page-cache
miss that blocks the worker on disk, stalling every connection on its run
queue (head-of-line blocking). A live tail feed is always resident, but a
catch-up read replaying old offsets can be cold.
Serve catch-up reads with sendfile on the blocking pool (dup'd fd +
spawn_blocking) so a disk fault parks a pool thread, not a worker, while
keeping the zero-copy page-cache -> socket transfer. Live tail reads (a
caught-up long-poll wake, flagged via Body::FileRange.hot) stay inline.
Strategy is selectable via --read-offload {inline|tail|always} (default
tail) for benchmarking. Benchmarks (Linux): inline wins small reads (~2x
at 1 KB), pooling wins large reads (16 MB: better throughput and ~3x
better p99); tail keeps the hot path inline and pools backfills.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a --read-offload section to the README: hot-path read throughput per mode (inline best for small reads, pooling a net win at 16 MB) and the cold-isolation result (tail caps worst-case hot-read latency ~10x under concurrent cold backfills). Documents why tail is the default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… module - New `--http-engine uring` (Linux): thread-per-core tokio-uring runtime; io_uring socket I/O and streamed io_uring file reads (bounded memory). Reuses the shared HTTP/1.1 logic and async handlers; conformance-green (the 2 webhook-delivery tests only fail in-container for loopback reasons). - Resident tail cache (StreamState.last_chunk, <=256 KiB): caught-up live readers and immediate catch-up reads share one copy instead of a per-subscriber file read; populated before waking subscribers. - Extract src/http1.rs: request-head parser, response-head serializer, and chunked framing shared by the raw and uring engines (was duplicated). - Release profile: panic = "unwind" (was "abort") so a connection-task panic drops one connection instead of aborting the whole process. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ocument engine choice - Add http1::ByteSource trait (buffered/consume/fill) and move read_head, read_sized, and decode_chunked into http1 over it. engine_raw (RawSource, tokio BytesMut) and engine_uring (UringSource, tokio-uring Vec) each provide a small adapter; the duplicated chunked decoder, sized-body, and head-read loops are removed (one copy of the protocol logic). - README: add "Choosing an engine" with the io_uring availability caveat (uring is experimental — many container/seccomp/hardened hosts block io_uring; raw is the portable Linux fast path that works under restrictive seccomp). - MSRV 1.74 -> 1.75 (async fn in traits); CI builds on stable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…EADME markdown - ARCHITECTURE.md: write-path / read-path diagram + an ingestion-to-fan-out data-flow diagram, with the performance techniques (wire-byte storage, group-commit fsync, lock-free reads, watch wakeups, resident tail cache, zero-copy egress). Reference for a performance write-up. - README: fix io_uring markdown mangled by the formatter (underscores). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(feature-gated)
Add OpenTelemetry instrumentation behind a new `telemetry` Cargo feature that is
OFF BY DEFAULT: zero extra dependencies and zero runtime overhead in a default
build (the no-op `record_*` shims inline away, and `tracing` compiles out
sub-info records via `release_max_level_info`). With `--features telemetry` the
server exports spans + metrics over OTLP/gRPC, configured entirely via the
standard OTEL_* env vars.
Spans (tracing macros, always compiled, exported only when the feature is on):
- ds.request at the handlers dispatcher — attrs: http.method, route bucket
(/<stream> | /health | /__ds/*), status_class. Never the stream path/id.
The SSE producer task carries the span via .instrument (no Entered across await).
Metrics (typed OTel instruments, explicit-bucket views):
- ds.http.requests Counter {method, status_class}
- ds.append.fsync.duration Histogram(s)
- ds.append.fsync.batch_size Histogram({appends coalesced per fsync})
- ds.append.lock_wait.duration Histogram(s)
- ds.append.duration Histogram(s) {outcome: accept|dup|conflict|closed, is_json}
- ds.read.duration Histogram(s) {live: catchup|long-poll|sse, cache: hit|miss}
- ds.read.tail_cache Counter {result: hit|miss, live}
- ds.read.offload.wait Histogram(s) (Linux blocking-pool queue wait)
The two pivot signals are ds.append.fsync.batch_size (group-commit health — how
many appends fold into one barrier-fsync) and ds.read.offload.wait (cold-read /
blocking-pool futex pressure).
Label sets are strictly bounded (engine, live, cache, outcome, is_json, method,
status_class) — never a stream path/id, producer id, offset, or etag.
Resolved crate versions: opentelemetry 0.32.0, opentelemetry_sdk 0.32.1,
opentelemetry-otlp 0.32.0 (grpc-tonic), opentelemetry-semantic-conventions 0.32,
tracing-opentelemetry 0.33.0.
Off by default; verified zero behavior change — conformance 332/332 on both the
hyper and raw engines; clippy clean with the feature OFF and ON.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first telemetry pass left always-on cost on the append hot path even with the feature off: an AtomicUsize pending-appender counter (2 contended RMWs per append) in SyncCoalescer, and unconditional Instant::now() timing calls. An A/B (same VM, pre-OTel vs OTel) showed a ~5-13% append-throughput regression. Gate them behind the feature: the pending counter + PendingGuard are now cfg(feature="telemetry"), and all hot-path timing goes through a new telemetry::Timer that is a ZST (start/elapsed_secs compile away) in a default build — so no clock is read and no atomic is touched when telemetry is off. A/B after the fix: parity with pre-OTel (within noise). Conformance 332/332 on hyper + raw; clippy clean with the feature on and off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…; --splice-appends) Binary-mode appends store the request body verbatim (wire == body), so the body bytes on the socket equal the bytes on disk — splice-eligible. The raw engine owns the request socket fd, so for a binary (non-JSON) stream it can move the body socket -> pipe -> file entirely in the kernel, skipping the socket -> userspace -> file copy. JSON streams transform the body (flatten + `,` delimiter) so on-disk bytes differ from socket bytes; they fall back. Design (begin/commit fast path, engine/handler separation preserved): - handlers::try_binary_append_splice() owns the whole critical section. It re-uses the normal append precedence (deleted -> content-type -> closed -> producer dedup -> seq), acquires the per-stream appender lock, then invokes an engine callback that splices the body, then commits: advances the tail, invalidates the resident tail-chunk cache for the spliced range (the bytes never entered userspace), publishes the tail watch, and runs the group-commit fsync. Producer idempotency, closed-stream 409, TTL, and exact offset/response semantics are identical to the normal path. Returns Done / Fallback (engine reads the body normally, body untouched) / Reject. - engine_raw splices the body INLINE on the async worker via tokio readiness (stream.readable().await + try_io): the worker parks whenever the socket has no data, so a large upload never monopolizes it, with no per-request blocking-pool handoff or fd dup. The data fd is O_APPEND (splice refuses an O_APPEND target with EINVAL), so a fresh O_WRONLY fd is opened and the file leg uses an explicit offset (safe under the appender lock). Any head-read over-read into the body is written first via a positioned write; exactly content-length body bytes are consumed so keep-alive / pipelining stay framed. Off by default via --splice-appends (an AtomicBool, like --read-offload), so default behaviour is byte-identical to today. Linux-only (#[cfg]); on non-Linux the path is compiled out and binary appends take the normal read+write path. Measured (Docker Linux, wrk, large binary appends, splice OFF vs ON): - Appends are fsync-bound, so throughput is roughly unchanged (within VM noise; 1 MiB / 32 conn, 3 runs: OFF ~1090-1293 req/s vs ON ~971-1072 req/s). - The real win is CPU: ON sustains the same append rate at roughly HALF to a THIRD of the server CPU (1 MiB: ~78% vs ~180%; 64 KiB: ~103% vs ~160%) — the eliminated copy shows up as CPU headroom, not higher throughput. Honest read: worth it for CPU-bound / high-fan-in binary-ingest deployments; not a throughput lever, hence off by default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
11 #[tokio::test] cases for http1::decode_chunked (+ read_sized) via an in-memory ByteSource that varies fill granularity (1..=len bytes per fill) so bodies split mid-size-line / mid-chunk are exercised: single/multi chunk, empty body, hex sizes, chunk extensions, trailer consumption, malformed size, early-EOF, the size+2 overflow guard, and the MAX_BODY_BYTES cap. Inspired by stratovolt's buffer-utils boundary tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
balegas
requested review from
KyleAMathews,
alco,
icehaunter,
kevin-dp,
msfstef,
paulharter and
robacourt
as code owners
Contributor
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4652 +/- ##
=======================================
Coverage 59.99% 60.00%
=======================================
Files 395 395
Lines 43747 43747
Branches 12583 12581 -2
=======================================
+ Hits 26248 26249 +1
Misses 17420 17420
+ Partials 79 78 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Electric Agents Mobile BuildLocal mobile checks ran for commit The EAS Android preview build was skipped because the |
thruflo
approved these changes
Jun 25, 2026
Rename the npm main + 4 platform packages from @durable-streams/server-rust* to @electric-ax/durable-streams-server-rust* (the electric-ax product scope), keeping the durable-streams brand. Crate name (`durable-streams`) and binary (`durable-streams-server`) unchanged. assemble.mjs/launcher.cjs derive names from targets.json, so no logic change. npm tooling tests pass (4/4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379
…conformance) - Add packages/server-rust/package.json as the changeset version anchor (@electric-ax/durable-streams-server-rust, private: true — changesets bumps the version but our custom CI publishes the assembled npm/crate/Docker artifacts). - Decouple the conformance suite: depend on the published @durable-streams/server-conformance-tests@^0.3.5 and import it by package name instead of a relative path into a sibling package. - Add scripts/sync-cargo-version.mjs to propagate the changeset version from package.json into Cargo.toml before cargo publish / Docker build. - Make the conformance vitest config package-relative so `pnpm test:conformance` runs from the package dir. - Remove the webhook / `__ds` subscription-control-plane mentions from the README and neutralize the harness comment (the 6 subscription tests stay skipped via `subscriptions: false`). Verified in electric: cargo build + `pnpm test:conformance` → 326 passed, 6 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379
…matrix) Add .github/workflows/server_rust_tests.yml (path-filtered to packages/server-rust, mirroring electric's per-product *_tests.yml workflows on blacksmith runners): - `cargo` job: cargo build --release, cargo test --release, clippy -D warnings, then uploads the server binary as an artifact. - `conformance` job: the 4-config run matrix (wal-default, wal-resident-cache, wal-read-offload-always, memory) — downloads the binary and runs `pnpm test:conformance` with RUST_SERVER_ARGS per config. Suppress the one pre-existing clippy `too_many_arguments` on encode_header_into (the args are the wire header fields) so `clippy -D warnings` is green. Verified locally: clippy clean, cargo test passes, conformance 326 passed / 6 skipped across the matrix command. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379
The repo-root .gitignore has a bare `wal` rule (runtime WAL data dirs) that also matched this crate's src/wal/ source module, blocking edits to it (the files are tracked via the import, but `git add` refused changes). Re-include src/wal/ in the package .gitignore so the WAL source is editable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379
…flow - packages/server-rust/Dockerfile: multistage rust:1-bookworm build → gcr.io/distroless/cc-debian12 runtime (glibc, no shell). Default features only (minimal; comments note how to add --features tier). CMD defaults --host 0.0.0.0 so the container serves externally (the binary defaults to 127.0.0.1). - .github/workflows/server_rust_dockerhub_image.yml: mirrors agent_server_dockerhub_image.yml — push(main)/workflow_call/workflow_dispatch, derives the version from package.json (the anchor is private → no git tag), tags latest+version on release / canary otherwise, and calls the reusable docker_multiarch_image.yml (amd64+arm64) for electricax/durable-streams-server-rust. Verified locally: image builds, runs in distroless, serves the protocol (PUT → 201, HEAD → 200). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379
- server_rust_publish.yml (reusable): build the 4 target binaries (native runners) with a /health smoke test, then OIDC-publish the crate (cargo publish, version synced from package.json) and the npm packages (assemble + npm publish --access public --provenance). - changesets_release.yml: since the @electric-ax/durable-streams-server-rust anchor is private (not in publishedPackages), detect a release idempotently (publish only if its package.json version isn't already on npm), then fan out to publish-server-rust (npm + crate) and publish-server-rust-to-dockerhub. - README: rewrite Releasing for the Changesets flow (add a changeset → merge the Version Packages PR → CI publishes npm + crates.io + Docker Hub); swap the prebuilt-tarball install for the Docker image. Verified: YAML valid; sync-cargo-version + npm assemble/dry-run + the Docker image all checked earlier. The OIDC publish + changeset fan-out can only be exercised by a real release (needs the one-time crates.io/npm trusted-publishing bootstrap). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379
…ield Audit follow-ups for electric's conventions: - Add a changeset for @electric-ax/durable-streams-server-rust so the required Check Changeset CI passes (the script does not exempt private packages, and the release design relies on Changesets bumping this anchor). - Add the `repository` field (with directory) to package.json, matching siblings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379
Not publishing the npm packages / crate / Docker image yet. Gate the release off while keeping CI builds and the conformance matrix: - changesets_release.yml: `false &&`-guard the publish-server-rust and publish-server-rust-to-dockerhub jobs (easy to re-enable). - server_rust_dockerhub_image.yml: comment out the merge `push` trigger (the canary-image publish); keep workflow_call/workflow_dispatch for later. - README: note that publishing is currently disabled. server_rust_tests.yml (build/test/clippy + conformance) is unaffected. The manual workflow_dispatch escape hatches remain for when we're ready to publish.
The crate (crates.io) and the Docker image are independent of the npm packages, so publish those two and keep npm off: - changesets_release.yml: re-enable publish-server-rust (crate) and publish-server-rust-to-dockerhub (release image). - server_rust_publish.yml: gate off the `npm-publish` job (`if: false`); `build` (smoke test) + `cargo-publish` still run. - README: note that only npm is disabled now. The crate publish requires crates.io Trusted Publishing to be configured for `durable-streams` first (the name is already reserved).
Drop the "configure trusted publishing first" caveats now that crates.io OIDC trusted publishing is set up for the `durable-streams` crate. npm publishers are still pending (npm publishing remains disabled).
…ippy) `clippy -D warnings` failed on Linux at src/engine_raw.rs:630 and :666: `(SPLICE_F_MOVE | SPLICE_F_MORE) as u32` is an unnecessary cast — the flags are already `c_uint` (u32) and `libc::splice`'s flags param is `c_uint`. macOS doesn't compile this Linux-only splice path, so local clippy didn't catch it. Verified the Linux compile still builds (Docker).
The TS formatting check (`prettier --check .`) failed on 3 imported npm files I hadn't edited (so lint-staged never reformatted them) — they kept the durable-streams style: npm/assemble.mjs, npm/bin/launcher.cjs, npm/test/launcher.test.mjs. Formatting only; npm tooling tests still pass (4/4).
balegas
force-pushed
the
vbalegas/durable-streams-server-rust
branch
from
0c2f42a to
05d072b
Compare
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The server-rust release detection checked npm for the anchor version to decide whether to publish. With npm publishing disabled, that version never lands on npm, so the check always missed and re-fired the crate + Docker publish on every unrelated release, failing cargo publish on the duplicate version. Check crates.io (the artifact that actually publishes) instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Imports the high-performance Rust Durable Streams server into the monorepo as
packages/server-rust(@electric-ax/durable-streams-server-rust), wired into electric's tooling and release flow.What's here
fdatasync, amemoryzero-copy mode, optional S3 tiering, a hand-rolled HTTP/1.1 engine withsendfile/splice). Imported with its git history (viagit subtree, sogit blamefollows the original commits).@electric-ax/durable-streams-server-rust(+ 4 platform packages).package.jsonanchor isprivate(Changesets bumps the version; CI publishes the real artifacts). A release currently publishes thedurable-streamscrate (crates.io) and a multi-archelectricax/durable-streams-server-rustDocker image (distroless). npm publishing is intentionally disabled for now (gated off inserver_rust_publish.yml).server_rust_tests.yml:cargo build/test/clippy -D warnings+ the conformance matrix (wal-default, wal-resident-cache, wal-read-offload-always, memory). Conformance consumes the published@durable-streams/server-conformance-tests.Notes for review
durable-streamscrate name is reserved on crates.io (yanked0.0.0, owned byelectric-sql:core) and its crates.io Trusted Publishing is configured.Verified locally
cargo build(default/tier/telemetry) +clippy -D warningsclean.cargo test(release + debug, ±tier): 87 passed.memory(Linux-only) run inside the Docker image./health200,PUT201,HEAD200).Publishing status
electricaxcreds viasecrets: inherit).npm-publishjob (searchDISABLED:) and configure npm trusted publishers when ready.🤖 Generated with Claude Code
https://claude.ai/code/session_019zZcfPyDkSmxpkcZJvi379