Skip to content

parser: prefix-validated incremental parse for Claude JSONL appends#744

Draft
avs-io wants to merge 2 commits into
getagentseal:mainfrom
avs-io:fix/703-incremental-claude-parse
Draft

parser: prefix-validated incremental parse for Claude JSONL appends#744
avs-io wants to merge 2 commits into
getagentseal:mainfrom
avs-io:fix/703-incremental-claude-parse

Conversation

@avs-io

@avs-io avs-io commented Jul 18, 2026

Copy link
Copy Markdown
Member

Draft, stacked on #743 (its base commit is that PR's branch — the diff shows both commits until #743 merges, at which point this rebases clean). Same deal as #743: stays a draft until you give the go-ahead.

Implements the revised item-5 design (latest comment on #703) — capability-gated v1, Claude only.

What changed

  • Claude CachedFile entries gain an optional checkpoint { parsedBytes, prefixSha256, version }. parsedBytes sits at the boundary before the current still-open turn (a turn only closes when the next non-empty user record arrives, and streaming-ID updates may still replace assistant messages inside it), so resume re-reads everything that can still change. The safety argument is written on the type.
  • Warm refresh of a changed Claude file: if size grew AND sha256 of bytes [0, parsedBytes) matches, parse only from the checkpoint and splice; anything else — shrunk, same-size change, hash mismatch, no checkpoint, read error — full re-parse of that file, exactly as today. If the suffix supersedes a streaming ID from a retained turn (rare cross-boundary shape), it also falls back to full parse rather than guessing.
  • No serialized parser state anywhere: dedup seeding reconstructs from cached turns as it already does; the overlap re-parse re-derives the open turn.
  • All other providers keep byte-identical behavior (snapshot/journal/SQLite/aggregate-cost sources per the design's provider audit). Cache schema v5→v6: one full parse to re-checkpoint, then incremental.
  • Runs entirely inside cache: strict cross-process gate for the warm session-cache refresh transaction #743's warm transaction gate, closing the shared-checkpoint concurrent-writer hole.

How tested

  • Parity suite (tests/parser-claude-incremental-parity.test.ts, 9 tests): incremental result must deep-equal a from-scratch parse across appended complete turns; append while a turn was open; streaming-ID last-wins updates; a later turn superseding a retained-turn ID (→ full parse); same-length prefix mutation (→ full parse); truncation; truncate-then-append-to-larger with matching size but changed bytes (proves the prefix hash is load-bearing — passes only because of the hash check); mtime-only touch; mcpInventory equality.
  • Full suite: 2171 passed with dist built.

Measured effect (synthetic, honest caveats)

52.4MB Claude JSONL + 100KB appended turn: warm incremental refresh 102ms vs 207ms full parse of the same bytes. This synthetic ratio is a floor, and it is sha256-bound (hashing the 52MB prefix is most of the 102ms): real Claude session files parse far slower per byte than they hash (object construction, turn grouping, dedup), and per-file today-corpora are typically well under this fixture's size, so the real-world win per tick is larger. Happy to attach real-corpus numbers if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant