feat(coreutils): ship POSIX utilities in the binary by raphaelvigee · Pull Request #438 · hephbuild/heph

raphaelvigee · 2026-08-29T17:01:12Z

heph ships ~40 POSIX utilities inside its own binary, so a recipe that runs cp, install or sha256sum behaves identically on Linux and macOS. This is the foundation: the crate, the entry points, and the plumbing. Off by default — #445 flips it.

Why

The divergences are not exotic; they are the first things anyone writes in a build recipe.

install -D is GNU-only, so the standard "make the parent dirs and copy" idiom simply fails on macOS.
sed -i takes an optional suffix on GNU and requires one on BSD, so sed -i 's/a/b/' f edits in place on Linux and eats the next argument as a filename on macOS.
wc -l < f pads with leading spaces on BSD, so comparing against "3" is Linux-only.
sort collates by locale, so the same input orders differently on two machines — and that one silently changes build outputs, not just exit codes.

A build system whose contract is same inputs, same outputs cannot leave the tools that produce those outputs host-defined.

What's here

crates/coreutils — the applet table over MIT-licensed uu_* crates (0.10), and COREUTILS_VERSION.
Multicall dispatch as the first statement of main() — heph __coreutils <applet> and argv[0] basename, ahead of logging, clap, self-update and tokio. Tests pin that it declines __supervisor and __runner-exec.
The shim directory — one symlink per applet pointing at the heph binary, materialized once per (version, binary path) under the heph home. Running cp execs heph with argv[0] == "cp": one process, no wrapper script, no host /bin/sh dependency.
PathPolicy::suffix carries it — the builtins compose behind everything the environment provides, so they fill a gap rather than shadow a tool the environment deliberately ships. A target's own declared tools still lead.
COREUTILS_VERSION in the exec def hash, and byte-identical keys when the toolbox is off — so this PR invalidates nothing.
heph tool coreutils list | which | run — because silently shadowing cp is exactly what produces an unanswerable bug report.

Cost, measured

+7.55 MiB stripped (+19.3%), built under the real [profile.release] on aarch64-apple-darwin. Trimming barely helps: dropping the eight lowest-value applets saves only 1.37 MiB, because the cost is a shared uucore+clap floor rather than the applet count. Close to all-or-nothing.

Startup is unchanged. heph __coreutils true is 5.63 ms against 5.91 ms for today's heph --version — the ~6 ms is pre-existing and lands before main, in 372 starlark #[starlark_module] constructors. Adding the applets moved it by nothing.

Note on the dependency shape

The exec driver takes a closure and a version rather than the coreutils crate, so plugin-exec — which most of the workspace links — does not pull in forty utility crates to know it has a directory to put on PATH. Caught because engine had started linking 45 uu_* crates and inflating every test binary; it now links zero.

The stack

Merge bottom-up, and gh stack sync after each one lands — master is squash-only, so the rebase will conflict and the resolution rule in CLAUDE.md applies.

	PR	What
6	#446	drop the host directories from a target's `PATH` — breaking
5	#445	the toolbox on by default
4	#440	the `template` rule and the `tmpl` applet
3	#453	grep, find, xargs, sed, tar, gzip, zstd
2	#438	the crate, the entry points, the shim directory
1	#451	the runner `PATH` seam ← base, targets `master`

Only #451 builds automatically: since #449 a stacked PR is skipped unless it carries ci/force-ci. Every layer was checked locally on its own — cargo build --workspace --all-targets, cargo clippy --workspace --all-targets, and its unit tests — not just at the top of the stack.

🤖 Generated with Claude Code

https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q

A heph target runs its recipe with a sandbox PATH of the host's directories, so `cp` means GNU coreutils on Linux and a BSD userland on macOS. The sandbox isolates files; it does nothing about the two hosts disagreeing. `install -D` does not exist on macOS at all, `wc` pads its output so `[ "$(wc -l < f)" = 3 ]` is Linux-only, and `sort` collates by locale — which silently changes build *outputs*, not just exit codes. A build system whose contract is "same inputs, same outputs" cannot leave the tools that produce those outputs undeclared and host-defined. So heph ships its own. `crates/coreutils` compiles 40 MIT-licensed uutils/coreutils applets into the binary and reaches them by re-exec — `heph __coreutils <applet>`, or a symlink named after the applet (argv[0] dispatch, busybox style). Dispatch is the first thing in `main()`, before logging, clap, the self-update check or any runtime: a build may invoke `cp` thousands of times and each one is a fresh process. It runs ahead of the `__supervisor` and `__runner-exec` branches, so it is tested against those argv shapes — eating one would kill the sidecar with a broken pipe rather than an error anyone could read. The shims are one directory under the heph home, materialized once per (toolbox version, binary path) and contributed to `hexecrunner`'s `PathPolicy` as a tier directly behind the target's own tools. A recipe that provisions its own `sed` still gets that `sed` — the builtins only displace the host's. Per-sandbox cost is one extra PATH entry: nothing written per target, nothing staged, nothing to tear down. `plugin-exec` takes a `CoreutilsShims` closure and a version rather than depending on `crates/coreutils`. The driver needs a directory to put on PATH and a number to hash, not knowledge of what an applet is — and most of the workspace links `plugin-exec`, so the dependency would have dragged forty utility crates into `engine`, `e2e` and `plugingo-e2e` builds and test binaries (`cargo tree -p engine | grep -c uu_`: 45 before, 0 after). The closure also keeps materialization lazy, so a `heph query` never touches the filesystem for it. `coreutils: true` with no supplied shims is a hard error, not a shrug: running against the host's utilities while the cache key claims heph's is the silently-wrong-build case. Off by default (`coreutils: true` on the exec/bash driver). Turning it on changes what every recipe's `cp` resolves to, and it moves every exec target's cache key. Cache correctness. The utilities are on a target's PATH without being declared, and nothing can tell which of them a shell command will invoke without parsing it, so `COREUTILS_VERSION` goes into the def hash whole or not at all — bumping it invalidates every exec target in every workspace, which is release-gated, not routine. Nothing is hashed while the toolbox is off, so a workspace that never opts in keeps today's keys. Cost, measured on this tree (aarch64-apple-darwin, rustc 1.96, the real release profile): +7.55 MiB stripped, +19.3%. Trimming does not help — dropping the eight lowest-value applets saves 1.37 MiB of that, because the cost is a shared uucore+clap floor, not the applet count. Startup is unchanged: `heph --version` already costs ~6 ms, essentially all of it before `main`, and the applets add nothing to it. Verified end to end on darwin/arm64: with the toolbox on, a bash target resolves `cp` to `.heph3/coreutils/v1-<hash>/bin/cp` and reports `cp (uutils coreutils) 0.10.0`; with it off the same target gets `/bin/cp`, which rejects `--version`. `heph tool coreutils list | which <name> | run <name> …` is the diagnostic surface — shadowing `cp` silently is exactly the kind of magic that produces an unanswerable bug report. Not in this change: sed, grep, find, xargs, tar, gzip and the template renderer; removing the host directories from the sandbox PATH; and making the toolbox the default. Design and measurements in docs/COREUTILS.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q

raphaelvigee force-pushed the raphaelvigee/core-utils-templating-engine branch 4 times, most recently from a2abb37 to cfb9516 Compare September 3, 2026 16:26

raphaelvigee mentioned this pull request Sep 3, 2026

fix(execrunner): PATH across the runner seam — carry the prefix out of band, and a tier for what heph supplies #451

Open

raphaelvigee force-pushed the raphaelvigee/core-utils-templating-engine branch from cfb9516 to 962ce7c Compare September 3, 2026 16:56

raphaelvigee force-pushed the raphaelvigee/core-utils-templating-engine branch from 962ce7c to 11d00de Compare September 3, 2026 17:30

raphaelvigee changed the base branch from master to raphaelvigee/execrunner-path-prefix-out-of-band September 3, 2026 17:31

Navigation Menu

Uh oh!

feat(coreutils): ship POSIX utilities in the binary - #438

feat(coreutils): ship POSIX utilities in the binary#438
raphaelvigee wants to merge 1 commit into
raphaelvigee/execrunner-path-prefix-out-of-bandfrom
raphaelvigee/core-utils-templating-engine

raphaelvigee commented Aug 29, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

raphaelvigee commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What's here

Cost, measured

Note on the dependency shape

The stack

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

raphaelvigee commented Aug 29, 2026 •

edited

Loading