COD-3218 / COD-3440: remove the LD_PRELOAD hack and port exec-harness to musl - #531
Draft
moha-bekh wants to merge 11 commits into
Draft
COD-3218 / COD-3440: remove the LD_PRELOAD hack and port exec-harness to musl#531moha-bekh wants to merge 11 commits into
moha-bekh wants to merge 11 commits into
Conversation
`libbpf-sys` vendors elfutils, whose `configure` aborts on a musl target
because `argp_parse` is a glibc extension that musl does not implement:
checking for library containing argp_parse... no
configure: error: failed to find argp_parse
libelf does not actually need those symbols -- they are used by the
elfutils CLI tools for argument parsing, but `configure.ac` checks for
them unconditionally, even when only the library is being built. Seeding
autoconf's cache (`ac_cv_search_argp_parse="none required"`) skips the
check, but the elfutils sources still `#include <argp.h>`, so the header
has to exist. Nothing that gets compiled calls into it, hence
declarations only.
Lives inside the crate, next to `wrapper.h` and `src/ebpf/c`, rather than
at the repo root: it is a memtrack build input, and it needs its own
directory because the path goes on the include path via `-I`.
Refs: libbpf/libbpf-sys#137
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the one gap the local COD-3440 spike could not: whether the musl
build of memtrack actually loads its BPF programs on a real x86_64
kernel. The spike was done on an aarch64 host, where an x86_64 build can
only be cross-compiled -- its skeleton targets the x86_64 ABI and cannot
load against an aarch64 kernel.
`workflow_dispatch` only, so it never runs on its own, and it touches
nothing in `release.yml`, `dist-workspace.toml` or any `Cargo.toml`.
Beyond the autoconf cache seeds and the argp.h stub, Debian needs one
thing the spike host did not: `LIBBPF_SYS_EXTRA_CFLAGS` with
`-idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include`. Its
musl-gcc runs with `-nostdinc` and only the musl include directory, so
libbpf cannot find the kernel UAPI headers it includes directly:
bpf.c:28:10: fatal error: asm/unistd.h: No such file or directory
../include/linux/types.h:12:10: fatal error: asm/types.h: No such file
**This must not reach `main`.** Delete it once the question is answered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The harness used to inject a `libcodspeed_preload.so` into the benchmark process so the callgrind client requests were issued from inside it. That was only necessary because instrumentation state did not propagate across `fork`, which COD-2349 has since fixed. Instrumentation is now toggled in exec-harness itself, around the spawn of each benchmark command. The benchmarked child inherits the live state across `fork`/`exec`, callgrind records the spawn edge on the dump part live at fork time, and `set_executed_benchmark` names that same part with the benchmark URI, so the backend can attribute the child's whole trace to the benchmark. Dropping the preload removes the "CPU Simulation mode does not support statically linked binaries" limitation, since nothing has to be injected into the benchmarked executable any more. It also unblocks building exec-harness for musl (COD-3440), which a preloaded `.so` made impossible. `--instr-atstart=inherit` becomes unconditional: it is what makes the benchmark measurable at all now, so it can no longer hang off the opt-in `--simulation-track-subprocess`, which keeps its name but from now on only selects `--separate-threads`. Since every way this can go wrong is silent -- the harness runs, the benchmark completes, and the measurement is empty -- the harness now fails loudly when it finds itself uninstrumented. BREAKING CHANGE: the measured region of an exec-harness benchmark now begins in exec-harness before the fork rather than in the child's ELF constructor, so it also covers the fork/exec/wait path and the child's pre-main startup. Absolute numbers shift in a step and history is not comparable across this change. A post-preload exec-harness also requires a runner that passes `--instr-atstart=inherit`, valgrind-codspeed >= iteration 6, and a backend with spawn-chain attribution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When cc-rs fails to compile the native library, the build script printed a `cargo:warning` and compiled the noop `InstrumentHooks` instead, in which every hook returns `Ok(())`. A build that landed there ran benchmarks and measured nothing, at exit code 0. That is reachable by accident: building for a musl target without a musl C compiler on PATH is enough, which the exec-harness musl port makes a routine thing to do. Make it a build failure on Linux, where we actually measure, and point at the missing toolchain. Other platforms keep the warning so macOS dev builds are unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the preload moves the callgrind client requests out of the benchmark child and up into exec-harness, so the measurement now rests on valgrind propagating instrumentation state across fork/exec and on the spawn edges valgrind-codspeed records. None of that is observable on the dev host (aarch64 Arch, no valgrind, and the CodSpeed .deb is Ubuntu-only), so this runs it on a real x86_64 runner. Asserts on the content of the .out files rather than the exit code, since the failure mode being guarded against is a run that completes happily and measures nothing: a part must carry the benchmark URI, that part must list the spawn edge, and every process in the chain must have its own .out with non-zero cost. The benchmark deliberately nests spawns (exec-harness -> sh -> seq/wc) so the chain is walked, not just one edge. Two guards against the check passing vacuously: the run is bracketed by a hash of the exec-harness on PATH, because the runner silently downloads the released preload build when the local one is missing, and the musl leg asserts the installed binary really is static. A baseline job runs the same benchmark on main to quantify the step change in reported cost that dropping the preload causes. Manual trigger only, and it must not reach main -- the header says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`gh workflow run` answers "HTTP 404: workflow not found on the default branch" for a workflow_dispatch-only file that has never existed on main, and it stays that way: GitHub does not register such a file on its own, so waiting and retrying the dispatch gets nowhere. A push trigger is what forces registration -- GitHub runs the file on push and assigns it an id, after which --ref dispatch works too. The COD-3440 workflow next door was registered exactly this way; its first run is a `push` one from a commit that temporarily added the same trigger. Recorded in the `on:` block so the next person does not rediscover it. Scoped to the spike branch, and it goes away with the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first run found all three; `instrumentation (gnu)` passed and met the verification bar, so the change itself is fine. 1. The musl leg asserted `file` says "statically linked". rustc emits a static-PIE for x86_64 musl, which `file` calls "static-pie linked"; only aarch64 gets the non-PIE spelling, which is why this passed locally and failed on CI. The binary was static all along -- the `build-musl` job's readelf check confirms no NEEDED/RPATH. Assert through readelf instead: no DT_NEEDED and no interpreter, which is the property we mean. 2. Cost was summed over `summary:` AND `totals:`. Child dumps carry both with near-equal values, so their cost was counted twice -- and only for some files, which inflated the branch total to 11508827 against main's 4819368 and made the comparison meaningless. On `totals:` alone it is 5924410 vs 4688549. Same fix in the baseline job. 3. The URI and the spawn edge were only required to be in the same FILE. The real dumps put both on the same PART, which is the invariant the backend walks: attribution starts at the URI-bearing part and follows its edges, so an edge on a neighbouring part would not attribute anything. Extract the spawn pids from the URI-bearing part itself. Also prints a per-file cost breakdown, so the comparison can be read without digging through the headers. Re-tested against synthetic dumps in the shape the run actually produced: the happy path passes and six failure modes each fail with the right diagnosis, including the new same-file-different-part case that 3. adds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The single fixed-size probe reports +26.4%, which is dominated by fixed per-process startup and so says nothing useful about a real benchmark. Extrapolating ~+7% from the one process that did real work (`seq`) is a guess, not a measurement. This holds the process shape identical across sizes (exec-harness -> sh -> seq) and varies only the work, which makes the model falsifiable: if the shift really is a fixed per-process cost, `branch - main` stays roughly CONSTANT in absolute Ir as N grows while the ratio collapses towards 1. If the delta instead grows with N, the cost is proportional and the "only matters for tiny benchmarks" reading is wrong. Both variants run the same sizes through the same script, at pinned commits (github.sha rather than the branch name, which may move), so the pairs are directly comparable. Carries the same exec-harness tamper guard as the instrumentation job. Throwaway, like the rest of this workflow -- delete once the number is recorded on the ticket. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keeps the change component-local. The goal is to unblock the musl build by removing the preload; forcing `--instr-atstart=inherit` on every simulation run was a bigger behavioural change than that needs, and it reached runs it had no business touching: Entrypoint runs (`cargo codspeed run`, pytest-codspeed) got `inherit` too. Harmless for a benchmark that never forks -- top-level `inherit` starts instrumentation off, same as `no` -- but an entrypoint benchmark that DOES fork would suddenly have its children instrumented and counted, silently changing its numbers. That is presumably why the flag was opt-in to begin with. It may also be unnecessary. Per COD-2349 the instrumentation state crosses `exec` by injecting `--instr-atstart=yes|no` into the child valgrind's argv (`VG_(needs_child_exec_args)`), not via the top-level flag; `inherit` covers the fork-only case. exec-harness spawns with `Command::status()`, i.e. fork + exec, so the child should pick the state up through the argv channel whatever the top level says. The check on this branch will confirm or refute that -- and with `src/` now identical to main, it isolates the exec-harness change on its own. If it turns out the runner does need a nudge, the shape to use is deriving it from `uses_exec_harness` (already threaded to `executor_config_for_command`) rather than hardcoding it here, so entrypoint runs keep their current behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the preload moved the instrumentation toggles out of the benchmark child and into exec-harness, which forks it. Measured on CI: with `--instr-atstart=no` the child dumps a single zero-cost `Trigger: Program termination` part, so the benchmark reports nothing at all -- the parent's live instrumentation state does not reach the child on its own. `--instr-atstart=inherit` is what enables that propagation; the argv-injection channel COD-2349 added for `exec` is not sufficient by itself. Derive it from `uses_exec_harness`, which the orchestrator already threads down to `executor_config_for_command`, rather than making `--instr-atstart=inherit` unconditional in `get_valgrind_args`. That leaves `measure.rs` untouched and keeps entrypoint runs on exactly their current behaviour, which matters: an entrypoint benchmark that forks would otherwise start having its children instrumented and counted, silently changing its numbers. The parameter was already `!uses_exec_harness` at the call site, for `enable_introspection`; it now passes the positive form and both derived values are computed inside. `ExecutorConfig::test()` passes `false`, which reproduces its previous field values exactly -- its target is an entrypoint one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ck-musl # Conflicts: # src/executor/config.rs
Merging this PR will not alter performance
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Covers COD-3218 (remove the LD_PRELOAD hack) and the exec-harness half of
COD-3440 (port the components to musl). They are the same problem: the preload was the
only thing keeping exec-harness on gnu.
Why
COD-3440 wants one multi-call binary, which requires the runner,
exec-harnessandmemtrackto target the same libc. The runner is musl; the other two were gnu. A staticallylinked musl binary can't be preloaded into a glibc process, so the musl port was blocked on
removing the
LD_PRELOADmechanism.What changed
exec-harness no longer injects
libcodspeed_preload.so. It toggles instrumentation inits own process, around the spawn of each benchmark command. Callgrind records the spawn edge
on the dump part live at fork time, and
set_executed_benchmarknames that same part withthe benchmark URI, so COD-2349's spawn-chain walk attributes the child's whole trace to the
benchmark. Confirmed on real dumps — the URI and the spawn edge land on the same part:
The whole preload inventory is gone: the C source, the
include_bytes!+ temp-fileextraction, the
LD_PRELOADcompatibility check, the shared-library build inbuild.rs,and the
cc/objectdependencies.perform_with_valgrindcollapses intoperform.Removes a user-facing limitation. Deleting the
LD_PRELOADcheck removes the "CPUSimulation mode does not support statically linked binaries" error — nothing has to be
injected into the benchmarked executable any more.
Runner side, four substantive lines.
--instr-atstart=inheritis required: measured,with
nothe child dumps a single zero-cost part and the benchmark reports nothing at all,so COD-2349's argv-injection channel is not sufficient on its own. Rather than making it
unconditional in
measure.rs(left untouched), it is derived inexecutor_config_for_command, which already receiveduses_exec_harness:So entrypoint runs (
cargo codspeed run, pytest-codspeed) keep exactly their currentbehaviour — an entrypoint benchmark that forks does not suddenly start counting its
children — and
--simulation-track-subprocesskeeps its original meaning.Two guards against silent failure. Every failure mode on this path is mute: the harness
runs, the benchmark completes, exit code is 0, and the measurement is empty.
is_instrumented()is false.instrument-hooks-bindings/build.rsno longer falls back to the noop implementation onLinux. It used to emit a
cargo:warningand compile hooks that all returnOk(())—reachable by accident, since building for a musl target with no musl C compiler on
PATHis enough. Now a build failure, with the
musl-toolshint. Non-Linux keeps the warning somacOS dev builds are unaffected.
Breaking change: the measurement baseline shifts
The measured region used to start in the child at the
.so's ELF constructor — after fork,after exec, after most of
ld.so. It now starts in exec-harness before the fork, so it alsocovers the fork/exec/wait path and the child's full pre-
mainstartup.Measured on x86_64 with the process shape held constant and only the work varying:
seq 1 N)It is a constant ~940k Ir per benchmark, not a percentage — stable to 0.22% across a
1000× range of benchmark size. Decomposes as ~340k for exec-harness's own fork/exec/wait
plus ~290–320k per spawned process (its pre-
mainstartup); predicted vs measured matchesexactly on a 4-process probe. So it is under 1% for a benchmark above ~94M Ir and a large
relative shift for very short ones.
Heads-up for platform
Children no longer self-label with the URI.
LD_PRELOADused to be inherited by everydescendant, so each one issued its own
DUMP_STATS_ATwith the same URI and attributionnever needed the spawn chain. Now only exec-harness's dump carries it, so attribution rests
entirely on the spawn-chain walk — if that regresses, cost goes silently unattributed
rather than degrading. (It also fixes a gap:
shwas previously unattributed.)musl
With the
.sogone, exec-harness builds for musl with no further change. Verified on CI(x86_64):
static-pie linked, noDT_NEEDED, noRPATH/RUNPATH, no interpreter — and thestatic musl binary drives valgrind instrumentation correctly end to end, so callgrind client
requests work fine from a static musl binary.
musl costs ~100k Ir more than glibc in exec-harness's own fork/exec/wait region (+1.7% on
this probe); the children are identical to within noise, as they must be.
crates/memtrack/musl/argp.his the declarations-only stub from the memtrack musl spike.The rest of that recipe (autoconf cache seeds,
LIBBPF_SYS_EXTRA_CFLAGS,-lgcconaarch64) is not in this branch — where those env vars should live so a plain
cargo build --target …-muslworks both locally and in CI is still an open question, and.cargo/config.toml's[env]would leak into the gnu build.Verification
Exit code 0 proves nothing here, so the checks assert on
.outcontent: a part carryingthe benchmark URI, a
Spawned pid:edge on that same part, and non-zero cost for everyprocess in the chain, walked recursively. The probe nests spawns
(
exec-harness → sh → seq/wc) so intermediate forwarding is exercised.Two precautions keep it from passing vacuously: the runner silently downloads the released
exec-harness when the local one isn't on
PATH, so the run is bracketed by a hash of thebinary; and the musl leg asserts the installed binary really has no dynamic dependency.
Last run: all six jobs green, on both gnu and static musl.
Local:
fmtandclippy --all-targets -D warningsclean,cargo test -p exec-harness25/25. Runner unit tests are identical to
main(same pass/fail counts, same failure set).The two workflows in this diff
.github/workflows/cod-3218-exec-harness-check.ymlandcod-3440-musl-check.ymlarethrowaway,
workflow_dispatchplus a branch-scopedpushtrigger. They exist becausethis work can't be validated on the dev machine (aarch64 Arch, no valgrind; the CodSpeed
valgrind
.debis Ubuntu-only). Both must be deleted before anything merges tomain—each says so in its header.