Skip to content

perf(update): skip redundant basis re-hash across sparse chain steps - #265

Merged
peters merged 2 commits into
mainfrom
autoresearch/verified-hash-carry
Sep 1, 2026
Merged

perf(update): skip redundant basis re-hash across sparse chain steps#265
peters merged 2 commits into
mainfrom
autoresearch/verified-hash-carry

Conversation

@peters

@peters peters commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Purpose

Autoresearch round 8 (update surface). Each chain step verifies the
patched file's target SHA-256; the next step's basis check then
re-reads and re-hashes the very same content. For the canonical sdk
step (one 4 KiB page in a 1.06 GB file) that is a ~0.55 s full-file
read + SHA-256 per step.

The chain walker now carries a VerifiedFileHashes map (path →
verified sha256) through apply_target_deltas:

  • PatchFile: basis verification skipped when the cache records the
    expected hash; first step and any mismatch still verify fully
  • WriteFile: target hash recorded after verification
  • Delete: entry dropped

Behavior impact

  • No change to patch formats, wire bytes, or the installed payload.
  • Verification semantics preserved: every file state is still
    hash-verified before it is used as a basis (just not twice); the
    post-patch target hash and the per-step full-archive SHA-256 are
    unchanged. A tamper-between-steps regression test corrupts the file
    after step 1 (bypassing the cache) and asserts step 2 still fails
    with a hash mismatch — verified the skip path is actually taken
    (marker in the skip branch fired in the test).
  • Single-shot apply passes a fresh empty cache (no cross-step state).
  • Doc note: also restores the identity-chunk (format v2) block in
    docs/performance/update-chains.md, which was lost when a
    multi-block edit was rejected wholesale in the perf(diff): identity-chunk markers in chunked patches (format v2) #256 round, and
    records the corrected per-step breakdown.

Test evidence

  • New regression test
    test_in_place_chain_with_verified_cache_detects_external_modification;
    existing carried-chain equivalence test updated to use a shared
    cache (asserts byte-identical output with caching on).

  • cargo test --workspace green; clippy all-targets + strict
    (-D clippy::unwrap_used -D clippy::expect_used) clean; fmt clean;
    maintainability guardrail clean.

  • Same-session A/B, 100-delta chain, scale 1.0, sdk-only, seed 42
    (baseline = main bba9f4f in a separate worktree, interleaved run):

    apply baseline with carry
    run 216,777 ms 156,599 / 158,057 ms

    −27.4% (0.9% spread); per step ~2.2 s → ~1.6 s; download bytes
    (503,524 B) and install tree (1,214,024,029 B) identical across
    all runs.

  • Full-chain history on this benchmark: 657 s → 489 s (perf(update): carry the extracted tree across sparse chain steps #252) →
    221 s (perf(diff): identity-chunk markers in chunked patches (format v2) #256) → ~157 s (this change) — cumulative −76%, payload
    byte-identical at every stage.

The chain walker verifies every patched file's target SHA-256 at the
end of each step; the next step's basis check then re-reads and
re-hashes the very same content. Carry a VerifiedFileHashes map
(path -> verified sha256) through apply_target_deltas:

- PatchFile: basis verification is skipped when the cache records the
  expected hash (set by this walk's own target verification);
  otherwise the full basis check runs as before
- WriteFile: target hash recorded after verification
- Delete: entry dropped

Safety: entries only ever come from verifications performed in the
same walk, a missing/mismatching entry falls back to the full basis
verification, and the post-patch target hash + per-step full-archive
SHA-256 are unchanged, so external modification between steps is
still caught (regression test corrupts the file between steps and
expects the mismatch).

Single-shot apply passes a fresh empty cache (no cross-step state).
…doc block

The identity-chunk doc block from the #256 change was dropped when a
multi-block edit was rejected wholesale; restore it and record the
verified-hash carry numbers (100-delta apply 216.8 s -> 157 s,
cumulative 657 s -> 157 s since the original baseline), the corrected
per-step breakdown, and the updated open items.
@peters
peters merged commit f4b93d6 into main Sep 1, 2026
15 checks passed
@peters
peters deleted the autoresearch/verified-hash-carry branch September 1, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant