perf(update): carry the extracted tree across sparse chain steps - #252
Merged
Conversation
The chain walker re-extracted the full current archive for every sparse delta before applying the per-file ops. The 100-delta per-step profile (BENCH_STEP_TIMING) showed the per-step cost was ops ~4.3 s + extract ~1.0 s + repack ~0.6 s + hash ~0.07 s, so the re-extract was removable without touching the per-step repack and full SHA-256 verification: - apply_target_deltas now keeps a working dir across consecutive sparse deltas: the starting archive is extracted once, each step applies its ops in place, and the step's repack + per-step full SHA-256 check are unchanged - a non-sparse hop rebuilds from archive bytes and drops the carried tree; the next sparse hop re-extracts once - sparse_ops split into shared ops+repack segments with a new apply_sparse_step_in_place entry; the single-shot path reuses the same code - unit test: a two-step carried chain produces byte-identical archives to the single-shot path at every step Measured (48-core/251 GB host, seed 42, same session A/B): - 100-delta scale 1.0: apply 656,990 -> 488,689 ms (-25.6%) - 20-delta scale 1.0: 128,699 -> 100,689 ms (-21.8%), repeat 101,076 ms - 20-delta scale 0.25: 35,125-36,690 -> 28,170 ms - download bytes and the installed payload are byte-identical (install-tree bench assertion; unit equivalence test)
There was a problem hiding this comment.
Pull request overview
Optimizes sparse delta-chain application by reusing an extracted working tree while preserving per-step archive verification.
Changes:
- Adds in-place sparse delta application and repacking.
- Carries working trees across consecutive sparse updates.
- Adds equivalence coverage and records benchmark improvements.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
crates/surge-core/src/update/manager/apply/delta.rs |
Implements carried-tree chain application. |
crates/surge-core/src/releases/delta/sparse_ops.rs |
Extracts shared in-place operations and repacking. |
crates/surge-core/src/releases/delta/mod.rs |
Exposes internal sparse-step helpers. |
crates/surge-core/src/releases/delta/tests.rs |
Tests byte-identical consecutive sparse steps. |
docs/performance/update-chains.md |
Documents measured performance and remaining costs. |
auto/update/program.md |
Records the successful optimization experiment. |
auto/update/results.tsv |
Adds benchmark results. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- run the sparse arm inside a Result-returning closure so every failure flows through the shared per-delta error handler (filename context + VerifyFailureBudget) instead of early-returning raw - drop the carried working dir on non-sparse hops: a sparse -> archive -> sparse chain previously applied the final sparse delta to the stale pre-hop tree (caught by a new mixed-chain regression test: per-file source hash mismatch on the final hop) - normalize (trim) patch_format/algorithm in is_sparse_file_ops_delta to match the dispatch normalization, so padded descriptors take the carry path like the dispatcher classifies them Regression tests: mixed sparse/archive/sparse chain applies to the correct tree; a corrupted sparse delta reports the filename through the shared handler; classifier normalization unit test.
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Autoresearch update-surface idea 1, measured to a keep: the chain walker was re-extracting the full current archive for every sparse delta. The per-step profile (via the
BENCH_STEP_TIMINGknob from #251) showed the step cost was ops ~4.3 s + extract ~1.0 s + repack ~0.6 s + hash ~0.07 s — the extract is removable without touching verification.Change
apply_target_deltascarries an extracted working dir across consecutive sparse deltas: the starting archive is extracted once, each step applies its ops in place, and the per-step repack + full SHA-256 check are unchanged (verification semantics intact).sparse_opssplit into shared ops+repack segments with a newapply_sparse_step_in_placeentry; the single-shot path reuses the same code.Measured (48-core/251 GB host, seed 42, same-session A/B)
Download bytes and the installed payload are byte-identical (the update bench asserts the install tree matches the last release; publish side untouched).
Behavior impact
Same product behavior: per-step full SHA-256 verification preserved, payload byte-identical, no format/API change. Client-side apply only — publisher cost unchanged.
Test evidence
RUSTFLAGS="-D warnings" cargo test --workspace— green (incl. new equivalence test)auto/update/results.tsv;docs/performance/update-chains.mdupdated with the per-step breakdown and the new open item (the ops phase itself now dominates; the remaining ~0.7 s per-step repack+hash is kept deliberately — it is what enables per-step verification)