Skip to content

COD-3218 / COD-3440: remove the LD_PRELOAD hack and port exec-harness to musl - #531

Draft
moha-bekh wants to merge 11 commits into
mainfrom
spike/cod-3440-memtrack-musl
Draft

COD-3218 / COD-3440: remove the LD_PRELOAD hack and port exec-harness to musl#531
moha-bekh wants to merge 11 commits into
mainfrom
spike/cod-3440-memtrack-musl

Conversation

@moha-bekh

Copy link
Copy Markdown
Member

Review only — not for merging. Opened as a draft to collect feedback. Two throwaway
CI workflows are in the diff on purpose (see the last section) and must be removed before
anything lands on main.

Covers COD-3218 (remove the LD_PRELOAD hack) and the exec-harness half of
COD-3440 (port the components to musl). They are the same problem: the preload was the
only thing keeping exec-harness on gnu.

Why

COD-3440 wants one multi-call binary, which requires the runner, exec-harness and
memtrack to target the same libc. The runner is musl; the other two were gnu. A statically
linked musl binary can't be preloaded into a glibc process, so the musl port was blocked on
removing the LD_PRELOAD mechanism.

What changed

exec-harness no longer injects libcodspeed_preload.so. It toggles instrumentation in
its own process, around the spawn of each benchmark command. Callgrind records the spawn edge
on the dump part live at fork time, and set_executed_benchmark names that same part with
the benchmark URI, so COD-2349's spawn-chain walk attributes the child's whole trace to the
benchmark. Confirmed on real dumps — the URI and the spawn edge land on the same part:

part: 2
desc: Spawned pid: 12570
desc: Trigger: Client Request: exec_harness::cod3218_probe
totals: 339979

The whole preload inventory is gone: the C source, the include_bytes! + temp-file
extraction, the LD_PRELOAD compatibility check, the shared-library build in build.rs,
and the cc / object dependencies. perform_with_valgrind collapses into perform.

Removes a user-facing limitation. Deleting the LD_PRELOAD check removes the "CPU
Simulation mode does not support statically linked binaries"
error — nothing has to be
injected into the benchmarked executable any more.

Runner side, four substantive lines. --instr-atstart=inherit is required: measured,
with no the child dumps a single zero-cost part and the benchmark reports nothing at all,
so COD-2349's argv-injection channel is not sufficient on its own. Rather than making it
unconditional in measure.rs (left untouched), it is derived in
executor_config_for_command, which already received uses_exec_harness:

enable_introspection: !uses_exec_harness,
simulation_track_subprocess: self.simulation_track_subprocess || uses_exec_harness,

So entrypoint runs (cargo codspeed run, pytest-codspeed) keep exactly their current
behaviour — an entrypoint benchmark that forks does not suddenly start counting its
children — and --simulation-track-subprocess keeps its original meaning.

Two guards against silent failure. Every failure mode on this path is mute: the harness
runs, the benchmark completes, exit code is 0, and the measurement is empty.

  • exec-harness now hard-fails when is_instrumented() is false.
  • instrument-hooks-bindings/build.rs no longer falls back to the noop implementation on
    Linux. It used to emit a cargo:warning and compile hooks that all return Ok(())
    reachable by accident, since building for a musl target with no musl C compiler on PATH
    is enough. Now a build failure, with the musl-tools hint. Non-Linux keeps the warning so
    macOS dev builds are unaffected.

Breaking change: the measurement baseline shifts

The measured region used to start in the child at the .so's ELF constructor — after fork,
after exec, after most of ld.so. It now starts in exec-harness before the fork, so it also
covers the fork/exec/wait path and the child's full pre-main startup.

Measured on x86_64 with the process shape held constant and only the work varying:

N (seq 1 N) before after delta ratio
1 000 407 704 1 350 987 943 283 ×3.31
10 000 877 820 1 821 219 943 399 ×2.07
100 000 5 592 827 6 535 673 942 846 ×1.17
1 000 000 52 849 099 53 790 416 941 317 ×1.018

It is a constant ~940k Ir per benchmark, not a percentage — stable to 0.22% across a
1000× range of benchmark size. Decomposes as ~340k for exec-harness's own fork/exec/wait
plus ~290–320k per spawned process (its pre-main startup); predicted vs measured matches
exactly on a 4-process probe. So it is under 1% for a benchmark above ~94M Ir and a large
relative shift for very short ones.

Heads-up for platform

Children no longer self-label with the URI. LD_PRELOAD used to be inherited by every
descendant, so each one issued its own DUMP_STATS_AT with the same URI and attribution
never needed the spawn chain. Now only exec-harness's dump carries it, so attribution rests
entirely on the spawn-chain walk — if that regresses, cost goes silently unattributed
rather than degrading. (It also fixes a gap: sh was previously unattributed.)

musl

With the .so gone, exec-harness builds for musl with no further change. Verified on CI
(x86_64): static-pie linked, no DT_NEEDED, no RPATH/RUNPATH, no interpreter — and the
static musl binary drives valgrind instrumentation correctly end to end, so callgrind client
requests work fine from a static musl binary.

musl costs ~100k Ir more than glibc in exec-harness's own fork/exec/wait region (+1.7% on
this probe); the children are identical to within noise, as they must be.

crates/memtrack/musl/argp.h is the declarations-only stub from the memtrack musl spike.
The rest of that recipe (autoconf cache seeds, LIBBPF_SYS_EXTRA_CFLAGS, -lgcc on
aarch64) is not in this branch — where those env vars should live so a plain
cargo build --target …-musl works both locally and in CI is still an open question, and
.cargo/config.toml's [env] would leak into the gnu build.

Verification

Exit code 0 proves nothing here, so the checks assert on .out content: a part carrying
the benchmark URI, a Spawned pid: edge on that same part, and non-zero cost for every
process in the chain, walked recursively. The probe nests spawns
(exec-harness → sh → seq/wc) so intermediate forwarding is exercised.

Two precautions keep it from passing vacuously: the runner silently downloads the released
exec-harness when the local one isn't on PATH, so the run is bracketed by a hash of the
binary; and the musl leg asserts the installed binary really has no dynamic dependency.

Last run: all six jobs green, on both gnu and static musl.

Local: fmt and clippy --all-targets -D warnings clean, cargo test -p exec-harness
25/25. Runner unit tests are identical to main (same pass/fail counts, same failure set).

The two workflows in this diff

.github/workflows/cod-3218-exec-harness-check.yml and cod-3440-musl-check.yml are
throwaway, workflow_dispatch plus a branch-scoped push trigger. They exist because
this work can't be validated on the dev machine (aarch64 Arch, no valgrind; the CodSpeed
valgrind .deb is Ubuntu-only). Both must be deleted before anything merges to main
each says so in its header.

moha-bekh and others added 11 commits September 4, 2026 17:48
`libbpf-sys` vendors elfutils, whose `configure` aborts on a musl target
because `argp_parse` is a glibc extension that musl does not implement:

    checking for library containing argp_parse... no
    configure: error: failed to find argp_parse

libelf does not actually need those symbols -- they are used by the
elfutils CLI tools for argument parsing, but `configure.ac` checks for
them unconditionally, even when only the library is being built. Seeding
autoconf's cache (`ac_cv_search_argp_parse="none required"`) skips the
check, but the elfutils sources still `#include <argp.h>`, so the header
has to exist. Nothing that gets compiled calls into it, hence
declarations only.

Lives inside the crate, next to `wrapper.h` and `src/ebpf/c`, rather than
at the repo root: it is a memtrack build input, and it needs its own
directory because the path goes on the include path via `-I`.

Refs: libbpf/libbpf-sys#137

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the one gap the local COD-3440 spike could not: whether the musl
build of memtrack actually loads its BPF programs on a real x86_64
kernel. The spike was done on an aarch64 host, where an x86_64 build can
only be cross-compiled -- its skeleton targets the x86_64 ABI and cannot
load against an aarch64 kernel.

`workflow_dispatch` only, so it never runs on its own, and it touches
nothing in `release.yml`, `dist-workspace.toml` or any `Cargo.toml`.

Beyond the autoconf cache seeds and the argp.h stub, Debian needs one
thing the spike host did not: `LIBBPF_SYS_EXTRA_CFLAGS` with
`-idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include`. Its
musl-gcc runs with `-nostdinc` and only the musl include directory, so
libbpf cannot find the kernel UAPI headers it includes directly:

    bpf.c:28:10: fatal error: asm/unistd.h: No such file or directory
    ../include/linux/types.h:12:10: fatal error: asm/types.h: No such file

**This must not reach `main`.** Delete it once the question is answered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The harness used to inject a `libcodspeed_preload.so` into the benchmark
process so the callgrind client requests were issued from inside it. That
was only necessary because instrumentation state did not propagate across
`fork`, which COD-2349 has since fixed.

Instrumentation is now toggled in exec-harness itself, around the spawn of
each benchmark command. The benchmarked child inherits the live state
across `fork`/`exec`, callgrind records the spawn edge on the dump part
live at fork time, and `set_executed_benchmark` names that same part with
the benchmark URI, so the backend can attribute the child's whole trace to
the benchmark.

Dropping the preload removes the "CPU Simulation mode does not support
statically linked binaries" limitation, since nothing has to be injected
into the benchmarked executable any more. It also unblocks building
exec-harness for musl (COD-3440), which a preloaded `.so` made impossible.

`--instr-atstart=inherit` becomes unconditional: it is what makes the
benchmark measurable at all now, so it can no longer hang off the opt-in
`--simulation-track-subprocess`, which keeps its name but from now on only
selects `--separate-threads`.

Since every way this can go wrong is silent -- the harness runs, the
benchmark completes, and the measurement is empty -- the harness now fails
loudly when it finds itself uninstrumented.

BREAKING CHANGE: the measured region of an exec-harness benchmark now
begins in exec-harness before the fork rather than in the child's ELF
constructor, so it also covers the fork/exec/wait path and the child's
pre-main startup. Absolute numbers shift in a step and history is not
comparable across this change. A post-preload exec-harness also requires a
runner that passes `--instr-atstart=inherit`, valgrind-codspeed >= iteration
6, and a backend with spawn-chain attribution.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When cc-rs fails to compile the native library, the build script printed a
`cargo:warning` and compiled the noop `InstrumentHooks` instead, in which
every hook returns `Ok(())`. A build that landed there ran benchmarks and
measured nothing, at exit code 0.

That is reachable by accident: building for a musl target without a musl C
compiler on PATH is enough, which the exec-harness musl port makes a
routine thing to do. Make it a build failure on Linux, where we actually
measure, and point at the missing toolchain. Other platforms keep the
warning so macOS dev builds are unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the preload moves the callgrind client requests out of the
benchmark child and up into exec-harness, so the measurement now rests on
valgrind propagating instrumentation state across fork/exec and on the
spawn edges valgrind-codspeed records. None of that is observable on the
dev host (aarch64 Arch, no valgrind, and the CodSpeed .deb is Ubuntu-only),
so this runs it on a real x86_64 runner.

Asserts on the content of the .out files rather than the exit code, since
the failure mode being guarded against is a run that completes happily and
measures nothing: a part must carry the benchmark URI, that part must list
the spawn edge, and every process in the chain must have its own .out with
non-zero cost. The benchmark deliberately nests spawns
(exec-harness -> sh -> seq/wc) so the chain is walked, not just one edge.

Two guards against the check passing vacuously: the run is bracketed by a
hash of the exec-harness on PATH, because the runner silently downloads the
released preload build when the local one is missing, and the musl leg
asserts the installed binary really is static.

A baseline job runs the same benchmark on main to quantify the step change
in reported cost that dropping the preload causes.

Manual trigger only, and it must not reach main -- the header says so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`gh workflow run` answers "HTTP 404: workflow not found on the default
branch" for a workflow_dispatch-only file that has never existed on main,
and it stays that way: GitHub does not register such a file on its own, so
waiting and retrying the dispatch gets nowhere.

A push trigger is what forces registration -- GitHub runs the file on push
and assigns it an id, after which --ref dispatch works too. The COD-3440
workflow next door was registered exactly this way; its first run is a
`push` one from a commit that temporarily added the same trigger. Recorded
in the `on:` block so the next person does not rediscover it.

Scoped to the spike branch, and it goes away with the file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first run found all three; `instrumentation (gnu)` passed and met the
verification bar, so the change itself is fine.

1. The musl leg asserted `file` says "statically linked". rustc emits a
   static-PIE for x86_64 musl, which `file` calls "static-pie linked"; only
   aarch64 gets the non-PIE spelling, which is why this passed locally and
   failed on CI. The binary was static all along -- the `build-musl` job's
   readelf check confirms no NEEDED/RPATH. Assert through readelf instead:
   no DT_NEEDED and no interpreter, which is the property we mean.

2. Cost was summed over `summary:` AND `totals:`. Child dumps carry both
   with near-equal values, so their cost was counted twice -- and only for
   some files, which inflated the branch total to 11508827 against main's
   4819368 and made the comparison meaningless. On `totals:` alone it is
   5924410 vs 4688549. Same fix in the baseline job.

3. The URI and the spawn edge were only required to be in the same FILE.
   The real dumps put both on the same PART, which is the invariant the
   backend walks: attribution starts at the URI-bearing part and follows
   its edges, so an edge on a neighbouring part would not attribute
   anything. Extract the spawn pids from the URI-bearing part itself.

Also prints a per-file cost breakdown, so the comparison can be read
without digging through the headers.

Re-tested against synthetic dumps in the shape the run actually produced:
the happy path passes and six failure modes each fail with the right
diagnosis, including the new same-file-different-part case that 3. adds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The single fixed-size probe reports +26.4%, which is dominated by fixed
per-process startup and so says nothing useful about a real benchmark.
Extrapolating ~+7% from the one process that did real work (`seq`) is a
guess, not a measurement.

This holds the process shape identical across sizes
(exec-harness -> sh -> seq) and varies only the work, which makes the model
falsifiable: if the shift really is a fixed per-process cost, `branch -
main` stays roughly CONSTANT in absolute Ir as N grows while the ratio
collapses towards 1. If the delta instead grows with N, the cost is
proportional and the "only matters for tiny benchmarks" reading is wrong.

Both variants run the same sizes through the same script, at pinned commits
(github.sha rather than the branch name, which may move), so the pairs are
directly comparable. Carries the same exec-harness tamper guard as the
instrumentation job.

Throwaway, like the rest of this workflow -- delete once the number is
recorded on the ticket.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keeps the change component-local. The goal is to unblock the musl build by
removing the preload; forcing `--instr-atstart=inherit` on every simulation
run was a bigger behavioural change than that needs, and it reached runs it
had no business touching:

Entrypoint runs (`cargo codspeed run`, pytest-codspeed) got `inherit` too.
Harmless for a benchmark that never forks -- top-level `inherit` starts
instrumentation off, same as `no` -- but an entrypoint benchmark that DOES
fork would suddenly have its children instrumented and counted, silently
changing its numbers. That is presumably why the flag was opt-in to begin
with.

It may also be unnecessary. Per COD-2349 the instrumentation state crosses
`exec` by injecting `--instr-atstart=yes|no` into the child valgrind's argv
(`VG_(needs_child_exec_args)`), not via the top-level flag; `inherit`
covers the fork-only case. exec-harness spawns with `Command::status()`,
i.e. fork + exec, so the child should pick the state up through the argv
channel whatever the top level says. The check on this branch will confirm
or refute that -- and with `src/` now identical to main, it isolates the
exec-harness change on its own.

If it turns out the runner does need a nudge, the shape to use is deriving
it from `uses_exec_harness` (already threaded to
`executor_config_for_command`) rather than hardcoding it here, so entrypoint
runs keep their current behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the preload moved the instrumentation toggles out of the benchmark
child and into exec-harness, which forks it. Measured on CI: with
`--instr-atstart=no` the child dumps a single zero-cost
`Trigger: Program termination` part, so the benchmark reports nothing at
all -- the parent's live instrumentation state does not reach the child on
its own. `--instr-atstart=inherit` is what enables that propagation; the
argv-injection channel COD-2349 added for `exec` is not sufficient by
itself.

Derive it from `uses_exec_harness`, which the orchestrator already threads
down to `executor_config_for_command`, rather than making
`--instr-atstart=inherit` unconditional in `get_valgrind_args`. That leaves
`measure.rs` untouched and keeps entrypoint runs on exactly their current
behaviour, which matters: an entrypoint benchmark that forks would
otherwise start having its children instrumented and counted, silently
changing its numbers.

The parameter was already `!uses_exec_harness` at the call site, for
`enable_introspection`; it now passes the positive form and both derived
values are computed inside. `ExecutorConfig::test()` passes `false`, which
reproduces its previous field values exactly -- its target is an entrypoint
one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ck-musl

# Conflicts:
#	src/executor/config.rs
@codspeed-hq

codspeed-hq Bot commented Sep 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks


Comparing spike/cod-3440-memtrack-musl (c83b2a5) with main (37eba9e)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant