perf(update): stream the target SHA-256 into the bspatch write - #270
Merged
Conversation
The per-step sparse-delta target hash was a separate full read + hash pass after the patched file was written. chunked_bspatch_file now accepts an optional Sha256 and chunked_bspatch_file_with_progress_and_ sha256 returns the reconstructed file's hash computed while writing, removing that second full read per step (additive API; existing functions unchanged). Same-session evidence (scale 1.0, seed 42, 48-core shared host): - phase-instrumented 10-step A/B, 4 interleaved pairs: the 540-563 ms/step target-hash phase is eliminated; net per step ~2,270 -> ~1,980-2,060 ms in 3 of 4 pairs (4th pair was a load-spike window hitting both sides) - 100-delta wall clock: new 186,122/167,182/157,514 ms vs base 178,113/166,808 ms — the ~0.2-0.3 s/step gain is near the shared host's ~10% wall noise floor, documented as such Verification semantics unchanged: the hash covers exactly the bytes written, mismatch errors are identical, and the tamper-between-steps regression test still fails closed.
Merge of #267 (format-1-by-default, format field on ChunkedDiffOptions) left the round-trip test added by this branch with the old two-field initializer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Autoresearch round 9 on the
auto/update/surface (continuing thechain-apply optimization series #252 → #256 → #265). Phase
instrumentation in #256/#265 showed the per-step target SHA-256 is a
separate full read + hash pass after the bspatch write (~0.55 s
per step at the 1 GB benchmark scale). This change removes that
second read: the chunked bspatch write path now computes the
reconstructed file's hash while writing and returns it.
Behavior impact
the streamed hash covers exactly the bytes written; mismatch errors
keep the same message; the tamper-between-steps regression test
still fails closed; per-step repack + full-archive SHA-256
unchanged.
chunked_bspatch_file_with_progress_and_sha256(+
ChunkedBspatchResult); the existingchunked_bspatch_file/chunked_bspatch_file_with_progressareuntouched, so the archive strategy and all other callers are
unaffected.
hash CPU is now paid inside the write loop, so the net win is the
saved read (~0.2-0.3 s/step at benchmark scale — larger on
slow/flash storage where the re-read dominates).
Test evidence
RUSTFLAGS="-D warnings" cargo test --workspace— all green(includes the new round-trip test asserting the returned hash
equals the rebuilt file's SHA-256).
cargo clippy --workspace --all-targets --all-featuresclean;strict panic-path clippy clean;
cargo fmt --all -- --checkclean;scripts/check-maintainability.shpass.host), interleaved base/new runs:
target-hash phase is eliminated; per step ~2,270 →
~1,980-2,060 ms in 3 of 4 pairs (the 4th pair was a load-spike
window that hit both sides' bspatch and repack).
base 178,113 / 166,808 ms. The ~0.2-0.3 s/step gain sits near
this shared host's ~10% wall noise floor, so the claim rests on
the phase-level evidence; documented in
docs/performance/update-chains.mdwith the raw numbers.(bench asserts).