parser: prefix-validated incremental parse for Claude JSONL appends#744
Draft
avs-io wants to merge 2 commits into
Draft
parser: prefix-validated incremental parse for Claude JSONL appends#744avs-io wants to merge 2 commits into
avs-io wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, stacked on #743 (its base commit is that PR's branch — the diff shows both commits until #743 merges, at which point this rebases clean). Same deal as #743: stays a draft until you give the go-ahead.
Implements the revised item-5 design (latest comment on #703) — capability-gated v1, Claude only.
What changed
CachedFileentries gain an optionalcheckpoint { parsedBytes, prefixSha256, version }.parsedBytessits at the boundary before the current still-open turn (a turn only closes when the next non-empty user record arrives, and streaming-ID updates may still replace assistant messages inside it), so resume re-reads everything that can still change. The safety argument is written on the type.[0, parsedBytes)matches, parse only from the checkpoint and splice; anything else — shrunk, same-size change, hash mismatch, no checkpoint, read error — full re-parse of that file, exactly as today. If the suffix supersedes a streaming ID from a retained turn (rare cross-boundary shape), it also falls back to full parse rather than guessing.How tested
tests/parser-claude-incremental-parity.test.ts, 9 tests): incremental result must deep-equal a from-scratch parse across appended complete turns; append while a turn was open; streaming-ID last-wins updates; a later turn superseding a retained-turn ID (→ full parse); same-length prefix mutation (→ full parse); truncation; truncate-then-append-to-larger with matching size but changed bytes (proves the prefix hash is load-bearing — passes only because of the hash check); mtime-only touch; mcpInventory equality.distbuilt.Measured effect (synthetic, honest caveats)
52.4MB Claude JSONL + 100KB appended turn: warm incremental refresh 102ms vs 207ms full parse of the same bytes. This synthetic ratio is a floor, and it is sha256-bound (hashing the 52MB prefix is most of the 102ms): real Claude session files parse far slower per byte than they hash (object construction, turn grouping, dedup), and per-file today-corpora are typically well under this fixture's size, so the real-world win per tick is larger. Happy to attach real-corpus numbers if useful.