Conversation
|
@cgwalters thoughts? I see in #7 you mention trying out a different approach. |
|
The tradeoff here is "link bcvk statically". Perhaps we could just have a little helper binary that's statically linked? Though I guess mechanically the "bcvk is a single binary" thing would need to be solved by actually bundling the inner statically linked binary as an ELF segment in the outer binary or so...this came up in #259 too. There's some corner cases here, like I am not totally sure this will work if we're doing e.g. cross-arch qemu emulation, as we'd be mixing a native arch binary into a foreign arch root. But...yeah on the positive side it's really nice to move what's currently in shell script into Rust. |
There was a problem hiding this comment.
Pull request overview
Moves ephemeral container namespace setup from bubblewrap/bash into bcvk, addressing AppArmor compatibility and reducing target-image dependencies.
Changes:
- Adds Rust-based hybrid-root and namespace initialization.
- Reworks container launch and status monitoring around the static bcvk executable.
- Removes the shell/bubblewrap entrypoint.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile |
Builds and installs a statically linked binary. |
docs/src/installation.md |
Updates target-image requirements. |
crates/kit/src/sandbox.rs |
Implements namespace and root setup. |
crates/kit/src/run_ephemeral.rs |
Launches bcvk directly as container entrypoint. |
crates/kit/src/run_ephemeral_ssh.rs |
Uses bcvk for status monitoring. |
crates/kit/src/main.rs |
Runs sandbox setup before Tokio initialization. |
crates/kit/scripts/entrypoint.sh |
Removes the former bubblewrap entrypoint. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| RUSTFLAGS="-C target-feature=+crt-static" cargo build --release --target $(CARGO_BUILD_TARGET) | ||
| install -D -m755 target/$(CARGO_BUILD_TARGET)/release/bcvk target/release/bcvk |
| For `bcvk ephemeral` operations, the bootc container images you run must contain: | ||
| - systemctl (systemd) | ||
| - objcopy (binutils) | ||
| - bwrap (bubblewrap) | ||
| - ssh, ssh-keygen (openssh-clients) |
| /// Assemble the hybrid root and make it this process's root. | ||
| /// | ||
| /// Only the VM supervisor calls this. Anything else arrives later through | ||
| /// `podman exec`, which joins the supervisor's namespaces and so is already in | ||
| /// the hybrid root. |
|
|
||
| all: bin manpages | ||
|
|
||
| # bcvk is bind-mounted and ran inside the container it starts. We build it statically |
Yes would be nice if this can be made to support qemu emulation too as i'd also like to be able to do this with bcvk in the future. Let me investigate a bit more how hard it would be to make this work more generally. |
| .unwrap_or_default() | ||
| } | ||
|
|
||
| /// Populate /etc/passwd in the hybrid root, which ssh-keygen wants to exist. |
There was a problem hiding this comment.
Hmm I think this is some pretty ugly technical debt that ideally we could drop...needs some consideration
| # bcvk is bind-mounted and ran inside the container it starts. We build it statically | ||
| # so that it does not depend on the image's loader or its libc. An explicit --target | ||
| # is required because proc-macros cannot be built statically. | ||
| CARGO_BUILD_TARGET := $(shell rustc -vV | sed -n 's/^host: //p') |
There was a problem hiding this comment.
This looks pretty hacky...I think we can express our desire to statically link via build.rs probably?
|
@cgwalters apologies for the radio silence. I'll pick this up again this week :) |
|
Thanks, I would like to get something in this form in, but it's a big change and needs some careful consideration. |
Requiring bubblewrap in the target image is not obvious to users, and several distributions' images do not carry it. bwrap was providing an unshare, a handful of bind mounts, a pivot_root and a PID namespace. Podman already gives the container a PID namespace with bcvk as its init, so what remains is a short piece of rustix. The entrypoint script moves under /run, because `podman exec` joins the namespace bcvk now runs in, where /var/lib/bcvk is not visible. Closes: bootc-dev#7 Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
The script assembled /run/tmproot, read the image's systemd version and exec'd bcvk. Doing this in Rust allows us to stop depending on bash in a target image. Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
…lation readme. Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
bcvk is bind-mounted into the container it starts and runs there before the entrypoint switches to the host's /usr. A dynamic build must load against the image's glibc, which fails when the image ships an older one than the build host, such as centos-bootc:stream9. A static build has no such dependency. Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
systemd-sysusers was executed from the image's PATH before the root change, which required the target image to ship it and to be executable on the host architecture. The host's systemd-sysusers cannot simply be called by path before the pivot either, as its loader and libc would then be resolved from the image. Run it after pivot_root instead, where / is the hybrid root: the host's binary, its libc and its sysusers.d then produce the /etc/passwd that the host's ssh-keygen consumes. systemctl --version remains the only image binary executed, as it has to report the image's systemd. Signed-off-by: Peter Siegel <psiegel2000@icloud.com> Assisted-by: AI
c880e1a to
abf170c
Compare
|
Ok looks like this also fixes a separate issue in latest bcvk where the backgrounded bwrap was cutting off stdin: bcvk/crates/kit/scripts/entrypoint.sh Line 70 in f5d5f5f |
cgwalters
left a comment
cgwalters
left a comment
There was a problem hiding this comment.
I'm overall good to merge this, but I think this won't be the final form.
I am still not happy with static linking; the way more complex option of generating a tiny statically linked binary or getting the required code into something else we expect to be in the image (util-linux, systemd, bootc, crun/runc) would probably make sense.
Otherwise mostly just needs a rebase now
| // SAFETY: unshare is unsafe only for UnshareFlags::FILES, where one thread | ||
| // can be left unable to use another's file descriptors. | ||
| #[allow(unsafe_code)] | ||
| unsafe { unshare_unsafe(UnshareFlags::NEWNS) }.context("Unsharing mount namespace")?; |
There was a problem hiding this comment.
BTW something to investigate...how about requiring util-linux? The thing about bwrap is we don't actually need security here just access to the Linux containerization syscalls which modern util-linux also is generally exposing?
I bet we could do this with unshare -m + mount --bind?
|
@cgwalters static linking also messes with some bcvk rpm packaging linting as well, no? I guess i'd have to disable some lints. One alternative idea: instead of using the image as the container root and pivoting into the hybrid root, we could try |
Probably... The thing I really don't like statically linking is e.g. openssl.so - also statically linking glibc breaks NSS and such, and we've had historical problems with Go in that regard.
This path is really A It would clearly make sense to have But it'd be a breaking change to require the host to be bootc of course... It'd be a good research spike though. I guess an interesting topic in this is do we use the host kernel or the target bootc kernel? I strongly believe it really needs to be the target by default - otherwise lots of "cross OS/distro" scenarios around the installation will break. That's why A model that skews the host userspace with the guest kernel is very likely to break in potentially obscure ways. I mean it'd often work but currently e.g. we don't require the host kernel to have So yes in a nutshell that's the reason for the current hacky "shell script+bwrap" to get the virt stack from the host, but otherwise use the target image. |
Thanks for the explanation. Ok I think I'll close this for now. It was an interesting experiment and for now I can work around/live with the apparmor issues on 26.04 and adding the dependencies we shell out to in the entrypoint while we converge to the right long term fix. |
|
Hmm you did a lot of work here though...I guess one thing we could do in theory is make this a build-time option? That said I'd like to do a spike on #329 (comment) @cgwalters-bot take a quick crack at that |
|
@cgwalters util-linux spike re #329 (comment): it works. Draft: cgwalters-forge#12 It keeps
Generated-by: https://github.com/cgwalters/#llms |
I completely forgot to associate this, but I also ran face-first into that issue; me and my agents ended up doing https://github.com/bootc-dev/actions/blob/f77e18eca58220a5f8e82a115d65ee5973fcec83/bootc-host-setup/ci/workarounds.sh#L16 I am honestly a bit skeptical of the security properties the AA profile for bwrap provides in general I think it would literally work to bypass it by doing e.g. It would also likely make sense to patch the AA profile if possible to differentate between "in init userns" vs not. Anyways it's worth noting that CI here actually runs on 26.04 but doesn't fail precisely because we have that workaround. |
Closes: #7
Target images currently have to ship bubblewrap and bash.
entrypoint.shbuilds the hybrid root and then execs bwrap, which unshares, binds the API filesystems, callspivot_rootand becomes PID 1.bcvk can do this itself: podman already provides the PID namespace with bcvk as its init, so the rest is replaced by a lightweight namespace setup in the container entrypoint. The dependency on both bubblewrap and bash go away with removing
entrypoint.sh.bcvk now runs in the image's userspace so it needs to be linked statically to work on older distros like stream9 with an older libc. Thoughts on this change?
This also fixes running Ubuntu 26.04 hosts. This was actually the original reason I looked into removing bubblewrap. On 26.04 the bwrap-userns-restrict AppArmor profile denies capabilities to bwrap's children. virtiofsd exits at startup with "can't apply the child capabilities" and bcvk polls for SSH until it times out. This might also be related to #306.
Tested by running all integration tests (
just test-integration) on an ubuntu 26.04 host. Note thattest_to_base_disk_integration_with_listfails with:FAIL [ 0.326s] integration-tests::integration-tests test_to_base_disk_integration_with_list stdout ─── running 1 test Testing to-base-disk integration with base-disks list Initial base disk count: 12 to-base-disk output: Created base disk: /home/psiegel/.local/share/libvirt/images/bootc-base-edda9015d6f14aa4.qcow2 Final base-disks list: ┌───────────────────────────────────┬──────────┬──────┬──────────────────┬──────────────────────────────────────────────────────────┐ │ NAME ┆ SIZE ┆ REFS ┆ CREATED ┆ IMAGE DIGEST │ ╞═══════════════════════════════════╪══════════╪══════╪══════════════════╪══════════════════════════════════════════════════════════╡ │ bootc-base-7b23c3ceb66bf50c.qcow2 ┆ 1.69 GiB ┆ 0 ┆ 2026-08-23 01:12 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ bootc-base-edda9015d6f14aa4.qcow2 ┆ 1.55 GiB ┆ 0 ┆ 2026-08-23 01:20 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ bootc-base-92414e97c3abaecc.qcow2 ┆ 1.55 GiB ┆ 0 ┆ 2026-08-23 01:22 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ bootc-base-487758b3396b22bb.qcow2 ┆ 1.69 GiB ┆ 0 ┆ 2026-08-23 01:10 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ bootc-base-7ac29c1405a4366c.qcow2 ┆ 1.69 GiB ┆ 0 ┆ 2026-08-23 01:08 ┆ sha256:6673fbbd49b11314700b0e842ed75149a673eb9faf7ec4... │ └───────────────────────────────────┴──────────┴──────┴──────────────────┴──────────────────────────────────────────────────────────┘ Found 5 base disks test test_to_base_disk_integration_with_list ... FAILED failures: ---- test_to_base_disk_integration_with_list ---- test panicked: Base disk count should increase after creation failures: test_to_base_disk_integration_with_list test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 89 filtered out; finished in 0.32s stderr ─── thread '<unnamed>' (3280798) panicked at crates/integration-tests/src/tests/libvirt_to_base_disk.rs:205:9: Base disk count should increase after creation note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceBut this looks unrelated.
Also tested together with https://github.com/yeetypete/bootc-ubuntu where this fixes a previous hang due to the AppArmor issue.
Assisted-by: AI