Streaming phase 3#611
Open
sroussey wants to merge 10 commits into
Open
Conversation
… backings Filesystem/in-memory streaming backings resolve a cache-miss synchronously via an existence probe (openSync / Map lookup). Async stores (IndexedDB, SQL) cannot, so widen getOutputStreamByRef to optionally return a Promise and await it in the shared streamRefViaBacking helper. A dangling ref now resolves to undefined (a miss) for async backings instead of a truthy Promise mistaken for a live stream. Synchronous backings are unaffected (awaiting a non-Promise is identity). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
Persists a ref's bytes as ordered chunk rows keyed by [refKey, seq] with a manifest row (existence witness + size + createdAt). Streaming writes never accumulate; reads page the chunk store so memory stays bounded and each page runs in its own short-lived transaction. A sibling instance over the same dbName resolves refs (cross-instance read contract). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
A TaskOutputRepository whose JSON rows persist via IndexedDbTabularStorage and whose streamed port payloads persist as chunked blobs (IdbBlobChunkStore), referenced by CacheRef. Implements the full streaming surface (saveOutputStream/saveOutputStreamPort/getOutputByRef/getOutputStreamByRef/ deleteOutputByRef) with async reads; supportsStreaming/Reads/Ports all true and isDurable true. Round-trips every codec mode, keeps distinct ports at distinct refs, prunes blobs with rows, resolves refs cross-instance, and converts a dangling ref into a cache miss through streamRefViaBacking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
The streaming-read contract now permits an async getOutputStreamByRef, so the base-typed contract suites (deterministic CacheStreamOut and run-private RunPrivateFsFolderStream) must await the reader before consuming it. Runtime is identical for the synchronous FsFolder/in-memory backings (awaiting a non-Promise is identity); the suites still assert the same bytes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
The column definitions and the ON CONFLICT clause already quote identifiers, but the CREATE TABLE PRIMARY KEY clause did not — so a schema with a camelCase primary-key column (e.g. TaskOutputSchema's taskType) created a quoted "taskType" column while the PK clause referenced the folded tasktype, failing with 'column "tasktype" named in key does not exist'. Quote the PK list consistently; snake_case schemas are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
TabularBlobChunkStore persists a ref's bytes as ordered (refKey, seq, bytes)
chunk rows plus a manifest row over any ITabularStorage, with keyset-paged
bounded-memory reads. TabularStreamingTaskOutputRepository is the shared base
(rows + blob store + <scheme>://<refKey> refs); Streaming{Postgres,Sqlite}
TaskOutputRepository are thin subclasses over bytea / BLOB chunk tables. Tested
with InMemory (store logic), in-process PGlite (Postgres), and better-sqlite3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
Parity with the PostgresTabularStorage fix: the exec_sql bootstrap DDL quoted the column definitions but not the PRIMARY KEY clause, so a camelCase-PK schema created a quoted column while the PK referenced the folded lowercase name. Only the local-dev/test exec_sql bootstrap path is affected (production tables are owned by Supabase migrations); snake_case schemas are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
StreamingSupabaseTaskOutputRepository is a thin TabularStreamingTaskOutputRepository subclass over SupabaseTabularStorage (bytea chunk tables via TabularBlobChunkStore), tested through the PGlite-backed Supabase mock. Two mock fixes make a camelCase schema faithful to real PostgREST (which quotes identifiers): quote the ON CONFLICT target in upsert, and map delete-chain neq(col, null) to IS NOT NULL so deleteAll actually clears. No external Supabase required. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QehfSG3zCtk6WPebaCuo8R
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.