Add support for --from-delta for bootc switch and update by alexlarsson · Pull Request #2449 · bootc-dev/bootc

alexlarsson · 2026-09-10T11:50:23Z

This adds native support for oci-delta files to bootc, for both the native composefs backend and the ostree backend.

The composefs backend just uses the existing support in composefs-rs, whereas the ostree backend has a bit more custom code. This code uses the generic delta code in composefs-rs to avoid having to reimplement delta stuff as we're applying deltas directly. This depends on composefs/composefs-rs#394 and means we have a WIP commit that points to this MR. The intent is to get some release with that out before merging this PR.

Note: I have not really tested the native composefs codepath, because I didn't manage to get such a setup going. I will try to do this.

alexlarsson · 2026-09-10T14:54:45Z

New version that is not depending on the new composefs-rs, instead using the separated out delta parsing from the rust code in containers/oci-delta.

cgwalters

Just an initial skim overall looks sane

cgwalters · 2026-09-22T14:07:15Z

+}
+
+/// Reject `--from-delta` for an image that also has to be in containers-storage.
+pub(crate) fn reject_unified_storage(delta: &Delta, use_unified: bool) -> Result<()> {


So...we are going to need a plan to make this work though...

I think the most viable thing will be extending podman-container-tools/container-libs#651 to support pushes?

I guess we can merge this as is and just add "fix deltas" to the unified storage tracker TODO (can you do that as part of this PR?)

I'll add to the todo.

That said, I must say I'm not convinced by this rube goldberg contraption that is unified storage. It seems to me that a more robust approach would be to use something like https://github.com/oras-project/rust-oci-client to implement rust-based pulling, including zstd::chunked into the composefs repo, and then just have some rust code to maintain an additional storage directory via reflinks.

In fact, such a system would perform better, disk-usage wise, because it would deduplicate within layers in the container store in a way that containers/storage isn't atm (unless you're using the special composefs mode).

I added this to the docs/src/bootc-experimental-unified-storage.7.md file. Is that what you meant?

cgwalters · 2026-09-22T14:08:02Z

+}
+
+/// Check the delta's internal consistency, to fail early
+fn validate(p: &ParsedDelta) -> Result<()> {


Old but good https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/

(Also this should be part of impl ParsedDelta right)

Moved this to ParsedDelta and updated the oci-delta dependency.

cgwalters · 2026-09-22T14:11:02Z

+    let config = ImageConfiguration::from_reader(&p.target_config_raw[..])
+        .context("Parsing embedded target config")?;
+    let layers = p.target_manifest.layers();
+    ensure!(!layers.is_empty(), "Delta target image has no layers");


Like on this topic there is https://docs.rs/nonempty/0.12.0/nonempty/ for example (and arguably we should use that in oci-spec by default for images, or at least have a standard falliable accessor)

cgwalters · 2026-09-22T14:23:19Z

+}
+
+/// Verify that `data` hashes to `expected`.
+fn verify_digest(what: &str, data: &[u8], expected: &Digest) -> Result<()> {


I think what should be an impl Read in the general case for this, I'd be surprised if we don't have a variant of this somewhere

I converted this to a reading verifier, and I agree there should be a generic one for this, but I didn't find one.

cgwalters · 2026-09-22T14:30:04Z

+            delta.validate_image_reference(spec_imgref)?;
+            crate::delta::reject_unified_storage(delta, use_unified)?;
+            if !quiet {
+                println!("Applying delta {}", delta.describe());


This needs to use the progress API.

We may need a blanket ban on (e)println! in crates/lib

I dropped all new of these in the commit. However, there is a bunch of existing ones.

println!() panics if stdout is closed (e.g. a broken pipe when piping into head), whereas writeln!() lets us propagate the error normally. These call sites are the actual output of a subcommand (digests, JSON, tables, GC stats), so an explicit stdout writer is the right thing anyway. Prep for denying clippy::print_stdout in crates/lib, see bootc-dev#2449 (comment) Generated-by: AI

fsck() already takes an explicit writer for its errors, but the "ok: <check>" lines bypassed it and went straight to stdout. The only caller passes stdout, so the output is unchanged; this just makes the function honor its own API. Prep for denying clippy::print_stdout in crates/lib, see bootc-dev#2449 (comment) Generated-by: AI

Subcommand implementations such as install, upgrade, switch and rollback print status lines for a human at a terminal ("Installing bootloader via bootupd", "Installation complete!", ...). We want to deny direct println!/eprintln! in the library, but exempting each of these functions with #[expect] would also exempt everything else in them, and several are very large. Instead, put the one exemption on a pair of small helpers and use them at these call sites, so a new stray println!() anywhere else, including inside those functions, still trips the lint. Output is unchanged: cli_status!() writes a line to stdout and cli_warn!() to stderr, just as the calls they replace did. Prep for denying clippy::print_stdout in crates/lib, see bootc-dev#2449 (comment) Generated-by: AI

Library code writing directly to stdout/stderr is easy to add and hard to notice in review, and it corrupts output for callers that expect something machine-readable (or that want progress routed through ProgressWriter). This came up in bootc-dev#2449 (comment) where a new println!() slipped into the composefs pull path. Deny both lints at the crate root so new call sites fail clippy. They are restriction lints, so the -A clippy::all in the Makefile validate target doesn't touch them and CI enforces the deny as is. Human-facing status output already goes through cli_status!() and cli_warn!(). The few remaining direct prints get a targeted #[expect] with a reason: global_init() runs before tracing is set up, journal_send() reports that the journal itself is failing, and the spinner has a plain-text fallback when there is no tty. Using expect rather than allow means the attribute gets flagged once it is no longer needed. Tests are exempted via allow-print-in-tests in clippy.toml. Generated-by: AI

println!() panics if stdout is closed (e.g. a broken pipe when piping into head), whereas writeln!() lets us propagate the error normally. These call sites are the actual output of a subcommand (digests, JSON, tables, GC stats), so an explicit stdout writer is the right thing anyway. Prep for denying clippy::print_stdout in crates/lib, see bootc-dev#2449 (comment) Generated-by: AI

fsck() already takes an explicit writer for its errors, but the "ok: <check>" lines bypassed it and went straight to stdout. The only caller passes stdout, so the output is unchanged; this just makes the function honor its own API. Prep for denying clippy::print_stdout in crates/lib, see bootc-dev#2449 (comment) Generated-by: AI

Subcommand implementations such as install, upgrade, switch and rollback print status lines for a human at a terminal ("Installing bootloader via bootupd", "Installation complete!", ...). We want to deny direct println!/eprintln! in the library, but exempting each of these functions with #[expect] would also exempt everything else in them, and several are very large. Instead, put the one exemption on a pair of small helpers and use them at these call sites, so a new stray println!() anywhere else, including inside those functions, still trips the lint. Output is unchanged: cli_status!() writes a line to stdout and cli_warn!() to stderr, just as the calls they replace did. Prep for denying clippy::print_stdout in crates/lib, see bootc-dev#2449 (comment) Generated-by: AI

Library code writing directly to stdout/stderr is easy to add and hard to notice in review, and it corrupts output for callers that expect something machine-readable (or that want progress routed through ProgressWriter). This came up in bootc-dev#2449 (comment) where a new println!() slipped into the composefs pull path. Deny both lints at the crate root so new call sites fail clippy. They are restriction lints, so the -A clippy::all in the Makefile validate target doesn't touch them and CI enforces the deny as is. Human-facing status output already goes through cli_status!() and cli_warn!(). The few remaining direct prints get a targeted #[expect] with a reason: global_init() runs before tracing is set up, journal_send() reports that the journal itself is failing, and the spinner has a plain-text fallback when there is no tty. Using expect rather than allow means the attribute gets flagged once it is no longer needed. Tests are exempted via allow-print-in-tests in clippy.toml. Generated-by: AI

bootc prints a fair amount of human-readable status ("Queued for next boot: ...", "Installing bootloader via bootupd", ...) directly to stdout, which is invisible to anything consuming --progress-fd, and it's the kind of library output we want to get away from (see bootc-dev#2449 (comment)). Add a generic Message event with a level and the text, and a ProgressWriter::message() helper that prints the text to the terminal exactly as before and, if a progress fd is attached, also sends the event. So output without --progress-fd is unchanged, and the default ProgressWriter can be used anywhere as a plain printer. A new event type can trip up consumers that match strictly on "type", so bump the protocol to 0.2.0 and document that unknown event types must be ignored. Start with the summary printed at the end of staging; converting the rest of the call sites follows. Generated-by: AI

Library code writing directly to stdout/stderr is easy to add and hard to notice in review, and it bypasses --progress-fd. This came up in bootc-dev#2449 (comment) where a new println!() slipped into the composefs pull path. Now that status output goes through ProgressWriter::message() and command results through explicit writers, deny both lints at the crate root so new call sites fail clippy. These cover print!(), println!(), eprint!() and eprintln!(), and unlike disallowed-macros in clippy.toml they can be scoped to this crate alone. They are restriction lints, so the -A clippy::all in the Makefile validate target doesn't touch them and CI enforces the deny as is. The few remaining direct prints get a targeted #[expect] with a reason: global_init() runs before tracing is set up, journal_send() and the progress writer report that they themselves are failing, and the spinner has a plain-text fallback when there is no tty. Tests are exempted via allow-print-in-tests in clippy.toml. Generated-by: AI

bootc prints a fair amount of human-readable status ("Queued for next boot: ...", "Installing bootloader via bootupd", ...) directly to stdout, which is invisible to anything consuming --progress-fd, and it's the kind of library output we want to get away from (see bootc-dev#2449 (comment)). Add a generic Message event with a level and the text, and a ProgressWriter::message() helper that prints the text to the terminal exactly as before and, if a progress fd is attached, also sends the event. So output without --progress-fd is unchanged, and the default ProgressWriter can be used anywhere as a plain printer. A new event type can trip up consumers that match strictly on "type", so bump the protocol to 0.2.0 and document that unknown event types must be ignored. The booted pushpull-upgrade test now checks that a switch with --progress-fd emits Message events, and skips them where it checks the progress events by position. Start with the summary printed at the end of staging; converting the rest of the call sites follows. Generated-by: AI

Library code writing directly to stdout/stderr is easy to add and hard to notice in review, and it bypasses --progress-fd. This came up in bootc-dev#2449 (comment) where a new println!() slipped into the composefs pull path. Now that status output goes through ProgressWriter::message() and command results through explicit writers, deny both lints at the crate root so new call sites fail clippy. They cover print!(), println!(), eprint!() and eprintln!(). A crate attribute is the simplest way to scope this to bootc-lib: clippy.toml is shared by the whole workspace, and a crates/lib/clippy.toml would replace the root config rather than extend it. These are restriction lints, and in-source attributes take precedence over the -A clippy::all in the Makefile validate target, so CI enforces the deny as is. anstream's print macros aren't covered by those lints, so also list them in disallowed-macros in clippy.toml and deny that lint here. Only this crate denies the lint, and it is the only user of those macros. The few remaining direct prints get a targeted #[expect] with a reason: global_init() runs before tracing is set up, journal_send() and the progress writer report that they themselves are failing, the spinner has a plain-text fallback when there is no tty, and medium_visibility_warning() colors its warning for the terminal (and also sends it as a Message event). Tests are exempted via allow-print-in-tests in clippy.toml. Generated-by: AI

alexlarsson · 2026-09-29T14:27:54Z

So, I rebased this and started testing with the bootc composefs backend. I run into issues reconstructing layers from the delta that reference files in /boot, because those files are not in the boot-transformed image...

alexlarsson · 2026-09-29T14:59:45Z

No, this seems to be a delta issue, lemme fix it.

cgwalters · 2026-09-29T15:44:41Z

those files are not in the boot-transformed image...

composefs/composefs-rs#373 (comment)

alexlarsson · 2026-09-29T15:55:40Z

@cgwalters That wasn't the issue, and I got further by fixing the delta file.

However now I run into:

[root@fedora mnt]# bootc switch --from-delta uki-v1-to-v2.oci-delta foo
Applying delta u[ki-v1-to-v2.oci-delta: target manifest sha256:9bd728a2bcc28a39ef97f17fb8b50b1cd3aa0e072d432d1e50a1c036ed91ff22, 3 layers (2 patched, 1 reused from source config sha256:42eee39c5ca3b6e0e74169526e305bc603d87223ad68cc38ee034211f3bdd5bf)
Detected delta artifact...
Looking up source image...
Building data source from base image...
[eta 0s] ###########################-------------       2/3       objects apply-delta                                                                  
Storing config and manifest... 
error: Switching: Composefs Switching: Performing Upgrade Operation: Setting up UKI boot: Writing Grub menuentry: opening boot/grub2: No such file or directory (os error 2)

I seem to get this with bootc switch --transport oci-archive v2.oci-archive as well though:

[root@fedora mnt]# bootc switch --transport oci-archive v2.oci-archive
Fetching config sha256:40da47d6bf8eee20c482ca20ede6bd5459efacf6f41ac0614d4df3e09a39f79e                                                                Storing manifest sha256:81372097659478fa2f76dd28b9ce89bb320804c2f846ec7fa17a80a8fcec171e                                                               error: Switching: Composefs Switching: Performing Upgrade Operation: Setting up UKI boot: Writing Grub menuentry: opening boot/grub2: No such file or directory (os error 2)

Is this a known issue?

cgwalters · 2026-09-29T16:10:06Z

Is this a known issue?

Yes I've queued cgwalters-forge#29 to be pushed here.

(The cgwalters-bot midstream is where I pre-review things to ensure they're not total garbage before they appear here, there's a growing backlog in https://github.com/cgwalters-forge/bootc/pulls that I'm kind of brainstorming how to manage better)

println!() panics if stdout is closed (e.g. a broken pipe when piping into head), whereas writeln!() lets us propagate the error normally. These call sites are the actual output of a subcommand (digests, JSON, tables, GC stats), so an explicit stdout writer is the right thing anyway. Prep for denying clippy::print_stdout in crates/lib, see bootc-dev#2449 (comment) Generated-by: AI

fsck() already takes an explicit writer for its errors, but the "ok: <check>" lines bypassed it and went straight to stdout. The only caller passes stdout, so the output is unchanged; this just makes the function honor its own API. Prep for denying clippy::print_stdout in crates/lib, see bootc-dev#2449 (comment) Generated-by: AI

bootc prints a fair amount of human-readable status ("Queued for next boot: ...", "Installing bootloader via bootupd", ...) directly to stdout, which is invisible to anything consuming --progress-fd, and it's the kind of library output we want to get away from (see bootc-dev#2449 (comment)). Add a generic Message event with a level and the text, and a ProgressWriter::message() helper that prints the text to the terminal exactly as before and, if a progress fd is attached, also sends the event. So output without --progress-fd is unchanged, and the default ProgressWriter can be used anywhere as a plain printer. A new event type can trip up consumers that match strictly on "type", so bump the protocol to 0.2.0 and document that unknown event types must be ignored. The booted pushpull-upgrade test now checks that a switch with --progress-fd emits Message events, and skips them where it checks the progress events by position. Start with the summary printed at the end of staging; converting the rest of the call sites follows. Generated-by: AI

Library code writing directly to stdout/stderr is easy to add and hard to notice in review, and it bypasses --progress-fd. This came up in bootc-dev#2449 (comment) where a new println!() slipped into the composefs pull path. Now that status output goes through ProgressWriter::message() and command results through explicit writers, deny both lints at the crate root so new call sites fail clippy. They cover print!(), println!(), eprint!() and eprintln!(). A crate attribute is the simplest way to scope this to bootc-lib: clippy.toml is shared by the whole workspace, and a crates/lib/clippy.toml would replace the root config rather than extend it. These are restriction lints, and in-source attributes take precedence over the -A clippy::all in the Makefile validate target, so CI enforces the deny as is. anstream's print macros aren't covered by those lints, so also list them in disallowed-macros in clippy.toml and deny that lint here. Only this crate denies the lint, and it is the only user of those macros. The few remaining direct prints get a targeted #[expect] with a reason: global_init() runs before tracing is set up, journal_send() and the progress writer report that they themselves are failing, the spinner has a plain-text fallback when there is no tty, and medium_visibility_warning() colors its warning for the terminal (and also sends it as a Message event). Tests are exempted via allow-print-in-tests in clippy.toml. Generated-by: AI

The diff_id appended for the generated layer was the bare hex encoding rather than the `sha256:...` form the spec requires. Our own importer does not look at it, so nothing noticed, but any other consumer of an image built by this helper trips over it. Assisted-by: AI Signed-off-by: Alexander Larsson <alexl@redhat.com>

This adds a `--from-delta PATH` flag to `bootc upgrade` and `bootc switch` that applies an oci-delta in one step. This commit add support only for the composefs backend, and uses the native delta pull support in composefs-rs. Later commits add ostree backend support. Basic parsing of the delta file is done by the oci-delta rust bindings. See https://github.com/containers/oci-delta for details of the format. Unified storage cannot apply a delta yet and will fail. Real support for this would require delta support in containers/storage. Generated-by: AI Signed-off-by: Alexander Larsson <alexl@redhat.com>

The importer has always fetched both the image metadata and the layer bytes over the same containers-image-proxy connection. To apply an OCI delta we need to split those apart: the manifest and config come out of the delta file, and each layer is either already in the repository or reconstructed locally from a binary patch. No network access at all. Introduce a `LayerSource` trait, which is just the existing `fetch_layer` signature behind a trait object, and a `ProxyLayerSource` implementing today's behaviour. `PreparedImport` now carries a LayerSource instead of a bare `OpenedImage`, and `ImageImporter::prepare_from_manifest` builds a `PreparedImport` from a manifest and config the caller already has, with the layers coming from whatever source it passes in. `ImageImporter::new_without_proxy` skips spawning skopeo entirely, so an importer that will never touch the network does not need it installed. Along the way, `prepare_internal` splits into `check_sigverify` and `diff_previous_state` so both prepare paths share the "is this already imported?" logic, and `cache_pending` moves into `create_prepared_import` where both paths reach it. `ProxyLayerSource` also fetches the proxy's layer info once and caches it, where before both `unencapsulate_base()` and `import()` asked for it separately. Assisted-by: AI Signed-off-by: Alexander Larsson <alexl@redhat.com>

We will need this later. We're reusing most of the old list_container_deployment_manifests() for this. Signed-off-by: Alexander Larsson <alexl@redhat.com>

alexlarsson · 2026-09-30T11:19:32Z

Rebased on main

`bootc upgrade --from-delta` and `bootc switch --from-delta` worked only with the composefs backend. Wire them up for ostree too. A tar-diff patch reads the source image by *path*, not from the original tar byte stream, so the source does not have to be a container layer: an ostree commit holding the same filesystem works just as well. `OstreeDataSource` serves file content out of a commit, and `DeltaLayerSource` plugs that into the container importer in place of the image proxy, so nothing is fetched. ostree does not store an image's root filesystem verbatim - the tar importer moves `/etc` to `/usr/etc`, and `/var` to `/usr/share/factory/var` on ostree older than v2024.3 - so a source path is tried at each of the locations that importer could have put it. Each reconstructed layer is streamed to the importer over a pipe, with the reconstruction itself running as the driver future that `join_fetch` already polls concurrently, so a multi-GB base layer never has to be spooled to disk. The source image is looked up by config digest among the images in the repository, and an absent source is fatal: there is no guarantee of a network connection at the point a delta is applied, so falling back to the registry would defeat the purpose. Byte-level progress is not reported for delta layers: the reconstructed bytes are uncompressed, so counting them against the descriptor's compressed size would overshoot. Per-layer start/completion still is. A layer's ref is written as the layer is unpacked, i.e. before the driver future has had the chance to report a diff_id mismatch, so a failed apply can leave a ref to unverified content that a retry would reuse. Prune the unreferenced layers after a failed delta apply. Assisted-by: AI Signed-off-by: Alexander Larsson <alexl@redhat.com>

Three tests covering the whole path from a delta on disk to an imported image, at increasing degrees of realism: `test_apply_delta_whole_layers` builds a delta whose patches are the target layers carried verbatim. The format allows that, and the layer reconstruction dispatches on media type, so this exercises parsing, validation, source lookup and the import while needing no external tooling - it runs everywhere. `test_apply_delta_chunked` and `test_apply_delta_derived` use the real `oci-delta` to build the delta, and skip themselves when it is not installed. The first covers an ostree-native image, whose layers are made of repo objects under `sysroot/ostree/` that do not exist in the commit's file tree - it passes only because oci-delta is told never to ask for one. The second covers a derived layer, an ordinary root filesystem tar, which is the case that actually drives `OstreeDataSource`; the file it patches lives at `/etc/bigconf`, i.e. at a path the importer relocates, and the test asserts both that it lands at `/usr/etc/bigconf` and that the patch is far smaller than the layer, which it can only be if the content really was read back out of the source commit. `oci-delta` goes in a new optional package list, because it is not available on every distribution we build on. Assisted-by: AI Signed-off-by: Alexander Larsson <alexl@redhat.com>

alexlarsson · 2026-09-30T12:23:30Z

Ok, I retried an update with the fixes from #2516 applied to this, and it works fine now.

alexlarsson · 2026-09-30T12:50:42Z

I tried both backends, and they work.

cgwalters · 2026-10-02T18:35:03Z

It'd be really helpful to have some integration tests for this - I think we could add one that basically does what we do in a lot of the tests, build a derived image on the target, switch to it via containers-storage transport etc.

(Of course the other large gap we have is around registries, that was discussed prior)

github-actions Bot added area/install Issues related to `bootc install` area/ostree Issues related to ostree area/documentation Updates to the documentation labels Sep 10, 2026

bootc-bot Bot requested a review from cgwalters September 10, 2026 11:50

alexlarsson mentioned this pull request Sep 10, 2026

oci: Make delta parsing and applying public composefs/composefs-rs#394

Closed

alexlarsson force-pushed the from-delta branch 2 times, most recently from e99dc64 to 033dc0d Compare September 10, 2026 14:53

alexlarsson force-pushed the from-delta branch from 033dc0d to eb615f0 Compare September 11, 2026 08:11

cgwalters reviewed Sep 22, 2026

View reviewed changes

cgwalters-bot mentioned this pull request Sep 23, 2026

lib: Deny println!/eprintln! in bootc-lib via clippy cgwalters-bot/bootc#1

Closed

cgwalters-bot mentioned this pull request Sep 23, 2026

lib: Route status output through the progress API, deny (e)println! cgwalters-forge/bootc#1

Draft

cgwalters-bot mentioned this pull request Sep 26, 2026

bootc: Disallow (e)println! in crates/lib via clippy cgwalters-forge/tracker#37

Open

alexlarsson force-pushed the from-delta branch from eb615f0 to ed83dcb Compare September 29, 2026 12:53

cgwalters added this to the 1.18 milestone Sep 29, 2026

alexlarsson added 4 commits September 30, 2026 12:53

ostree-ext: Add list_container_deployment_commits()

09f0417

We will need this later. We're reusing most of the old list_container_deployment_manifests() for this. Signed-off-by: Alexander Larsson <alexl@redhat.com>

alexlarsson force-pushed the from-delta branch from ed83dcb to 4af1da1 Compare September 30, 2026 11:11

alexlarsson added 2 commits September 30, 2026 13:39

alexlarsson force-pushed the from-delta branch from 4af1da1 to b69fb90 Compare September 30, 2026 11:39

Conversation

alexlarsson commented Sep 10, 2026

Uh oh!

alexlarsson commented Sep 10, 2026

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexlarsson commented Sep 29, 2026

Uh oh!

alexlarsson commented Sep 29, 2026

Uh oh!

cgwalters commented Sep 29, 2026

Uh oh!

alexlarsson commented Sep 29, 2026

Uh oh!

cgwalters commented Sep 29, 2026

Uh oh!

alexlarsson commented Sep 30, 2026

Uh oh!

alexlarsson commented Sep 30, 2026

Uh oh!

alexlarsson commented Sep 30, 2026

Uh oh!

cgwalters commented Oct 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants