feat!: ship one binary — bundle exec-harness and memtrack, drop the LD_PRELOAD hack by moha-bekh · Pull Request #531 · CodSpeedHQ/codspeed

moha-bekh · 2026-09-07T15:37:43Z

exec-harness and memtrack are compiled into codspeed and reached as hidden subcommands, and
exec-harness no longer injects libcodspeed_preload.so. The two are one change: a statically
linked musl binary cannot be preloaded into a glibc process, so the single binary was blocked on
removing the preload.

One release artifact instead of three, 11.0 MB compressed against 13.5 MB today. The download
machinery and both installer pins are gone.

Three things worth a reviewer's attention:

The measurement baseline shifts. The measured region now starts in exec-harness before the
fork, so it also covers fork/exec/wait and the child's pre-main startup. It is a constant
~940k Ir per exec-harness benchmark, not a percentage — stable to 0.22% across a 1000× range of
benchmark size. Shipping as is: no forced baseline, no history surgery.
Attribution rests entirely on the spawn-chain walk. Children no longer self-label with the
benchmark URI, since LD_PRELOAD is what used to be inherited by every descendant.
setcap now lands on codspeed itself, so memtrack's five capabilities sit on the CLI.
They are +ep with no inheritable set, so a spawned benchmark does not receive them.

Also removes the user-facing "CPU Simulation mode does not support statically linked binaries"
error: nothing is injected into the benchmarked executable any more.

Closes COD-3218
Closes COD-3440

codspeed · 2026-09-07T15:44:50Z

Merging this PR will degrade performance by 73.48%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 22 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`sleep 1`	245.7 µs	926.6 µs	-73.48%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing spike/cod-3440-memtrack-musl (ac9a23c) with main (59cfc90)}

greptile-apps · 2026-09-17T14:41:21Z

Confidence Score: 3/5

The PR is not yet safe to merge because unsupported-platform setup can install an unpinned valgrind-codspeed HEAD, and the new prompt violates an explicit repository output rule.

Summary

This PR bundles exec-harness and memtrack into the main executable, removes preload-based simulation, adds static musl build coverage, batches memtrack events, and introduces source-building of valgrind on systems without published packages.

Internal subcommands now re-execute and dispatch through the main binary.
Memory tracking capabilities are granted to the main executable.
Recent changes add batched eBPF event delivery and unsupported-platform valgrind setup.
The source-build fallback currently installs an unpinned repository HEAD.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CLI[codspeed] --> EH[Bundled exec-harness]
  CLI --> MT[Bundled memtrack]
  EH --> VG[valgrind-codspeed]
  MT --> BPF[eBPF tracker]
  VG --> BENCH[Benchmark process]
  BPF --> BENCH
  CLI --> SETUP[Setup]
  SETUP --> PKG[Pinned valgrind package]
  SETUP --> SRC[Source-build fallback]
  SRC --> HEAD[Unpinned default-branch HEAD]

_{Reviews (2) · Last reviewed commit: "test(executor): grant capabilities to th..."}

`self_exe()` resolves the binary that internal subcommands are re-invoked through, and the memory executor hands that same path to `sudo setcap <caps>+ep` so the capabilities land on the binary that is actually exec'd. Reading an environment variable there means anyone able to set one variable chooses which file receives CAP_SYS_ADMIN and CAP_BPF. The override exists for the tests, where `current_exe()` is the test harness and cannot dispatch a subcommand. Nothing in production sets it -- the doc comment justified it with a launcher scenario that has no caller. Putting it behind `cfg(test)`, constant included, removes the escalation path outright while keeping the tests working; a release build now always resolves `current_exe()`. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`codspeed memtrack` and `codspeed exec-harness` are a re-exec of this binary and share nothing with the runner, but they were dispatched at the bottom of `run()` -- after the profile config is loaded, after the API client is built, and after `DiscoveredProjectConfig::discover_and_load` walks the filesystem. That last one is the problem: the re-exec runs in the benchmark's working directory, which is the user's project. A malformed `codspeed.yaml` there aborts the subcommand, so a measurement fails for a reason that has nothing to do with the measurement, and a `--config` given to the outer run is not forwarded to the inner one to override it. Move them into `run_internal`, called right after `Cli::parse()`. The logger match loses its internal arms for the same reason it had them. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The released Linux artifacts are `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`, and nothing in CI built either: a break in the argp stub, in the kernel-header paths or in the aarch64 `-lgcc` link flag would have surfaced for the first time during a tag-triggered release. The throwaway spike workflow used to cover this and was deleted with the spike. Both legs build on a native runner, with no environment variables, which is also what keeps `.cargo/config.toml` honest -- it has to carry the whole recipe on its own. The assertions are `readelf`-based rather than a `file` string, since rustc emits a static-PIE for x86_64 musl and spells it differently from aarch64, and `codspeed exec-harness --version` / `codspeed memtrack --version` answer only if both CLIs really are linked in. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`self_exe()` resolves the binary that internal subcommands are re-invoked through, and the memory executor hands that same path to `sudo setcap <caps>+ep` so the capabilities land on the binary that is actually exec'd. Reading an environment variable there means anyone able to set one variable chooses which file receives CAP_SYS_ADMIN and CAP_BPF. The override exists for the tests, where `current_exe()` is the test harness and cannot dispatch a subcommand. Nothing in production sets it -- the doc comment justified it with a launcher scenario that has no caller. Putting it behind `cfg(test)`, constant included, removes the escalation path outright while keeping the tests working; a release build now always resolves `current_exe()`. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`codspeed memtrack` and `codspeed exec-harness` are a re-exec of this binary and share nothing with the runner, but they were dispatched at the bottom of `run()` -- after the profile config is loaded, after the API client is built, and after `DiscoveredProjectConfig::discover_and_load` walks the filesystem. That last one is the problem: the re-exec runs in the benchmark's working directory, which is the user's project. A malformed `codspeed.yaml` there aborts the subcommand, so a measurement fails for a reason that has nothing to do with the measurement, and a `--config` given to the outer run is not forwarded to the inner one to override it. Move them into `run_internal`, called right after `Cli::parse()`. The logger match loses its internal arms for the same reason it had them. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The released Linux artifacts are `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`, and nothing in CI built either: a break in the argp stub, in the kernel-header paths or in the aarch64 `-lgcc` link flag would have surfaced for the first time during a tag-triggered release. The throwaway spike workflow used to cover this and was deleted with the spike. Both legs build on a native runner, with no environment variables, which is also what keeps `.cargo/config.toml` honest -- it has to carry the whole recipe on its own. The assertions are `readelf`-based rather than a `file` string, since rustc emits a static-PIE for x86_64 musl and spells it differently from aarch64, and `codspeed exec-harness --version` / `codspeed memtrack --version` answer only if both CLIs really are linked in. Reported by Greptile on #531. Refs COD-3440 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

GuillaumeLagrange

@not-matthias can you do a first round of review 🙏 ? Let's ignore the LD_PRELOAD removal overhead for the exec-harness simulation, I'll spec this in a dedicated ticket and we'll do it before we merge and release this.

not-matthias

Overall pretty good, just a few comments on how to better structure the code/comments

And a few notes on how to best structure the PRs for reviewers:

Try to keep the changes as minimal as possible
Do not modify unrelated comments/code -> should be done in separate PRs for easier review
Remove obvious/LLM-written/bloated comments (using the deslop skill)
you should also review the PR before putting it into review

GuillaumeLagrange

See comments for details.

I also see that there are unresolved comments.

When you handle a review, all comments should either be

resolved and handled properly (very important to not silently resolve comments)
answered with either counter arguments or questions so the reviewer can either accept your arguments or open a discussion

Finally, there are 35 commits in the PR, you may want to re-do commits before the next round of reviews

GuillaumeLagrange · 2026-09-21T09:48:59Z

+    // exec-harness toggles the instrumentation in its own process and then forks
+    // the benchmark, so the child is measured only if valgrind propagates that
+    // state across `fork`/`exec`. With `no` the run dumps a single zero-cost part.
+    if config.simulation_track_subprocess || config.uses_exec_harness {


I think this should be resolved earlier in executor_config_for_command, and set the value directly in simulation_track_subprocess, and then we can remove the uses_exec_harness from the executor config, cause ideally it should be transparent for the executor

GuillaumeLagrange · 2026-09-21T09:49:00Z

+    if config.simulation_track_subprocess {
+        args.push("--separate-threads=yes".to_string());
+    } else {
        args.push("--separate-threads=no".to_string());
    }


Let's keep everything bundled together to avoid making the logic too complex for something that will become default behavior anyway next major (the subprocess tracking + thread costs)

I'll do the refactor from the other three comments. The bundling I'd like
confirmed first: it undoes f1110dd, which @not-matthias asked for when he flagged
forcing simulation_track_subprocess on as a breaking change (off by default
today, slower CI, results can move).

That's why the flags are split: exec-harness forces --instr-atstart=inherit
only, which it can't run without, and --separate-threads stays where it was.
Bundling them back turns it on for exec-harness too. Your call about the next
major is a fair argument — settle it with him and I'll land all four in one
commit.

As discussed, in order to remove LD_PRELOAD, let's enable simulation_track_subprocess when exec harness is used.

GuillaumeLagrange · 2026-09-21T09:49:00Z

+    // exec-harness toggles the instrumentation in its own process and then forks
+    // the benchmark, so the child is measured only if valgrind propagates that
+    // state across `fork`/`exec`. With `no` the run dumps a single zero-cost part.


This comment could be shortened, and put in executor_config_for_command where we force simulation_track_subprocess to true

On the threads I left open on purpose: four are answered and waiting on you two
rather than on me — the walltime PYTHONPERFSUPPORT, self_exe(), the exit
codes, and the --separate-threads bundling. The three on measure.rs and
executor/config.rs are the same change, and it hangs on that last one: I'll
land them in one commit once you've settled it.

GuillaumeLagrange · 2026-09-21T09:49:01Z

+    // memtrack's exit code is the tracked command's own, and the runner reads
+    // it to decide whether the benchmark failed, so it has to become ours.
+    let code = ::memtrack::cli::run_cli(argv)?;
+    std::process::exit(code);


A bit weird that we have different behaviors between this and exec-harness regarding the exit code, any way we could align them?

The difference is between the two tools, not between the two wrappers:
memtrack::cli::run_cli returns Result<i32> — the tracked command's own exit
code — while exec_harness::cli::run_cli returns Result<()> and bails when the
benchmark exits non-zero. Each bundled subcommand does what its standalone binary
already did on main: memtrack's main() ended on process::exit(code),
exec-harness's returned the Result.

Aligning them means changing one of the two contracts with the runner: either
exec-harness starts forwarding the child's exit code instead of erroring, or
memtrack stops forwarding it and the code is lost. Both are behavior changes I'd
rather not smuggle into this PR — happy to do it in a follow-up if you have a
preference on which way.

GuillaumeLagrange · 2026-09-21T09:49:02Z

+    /// Whether this execution is driven by exec-harness rather than by a plain
+    /// entrypoint command.
+    pub uses_exec_harness: bool,


As told in other comments, I do not think that's necessary, we should just override the simulation_track_subprocess where relevant

GuillaumeLagrange

See comments for details.

I also see that there are unresolved comments.

When you handle a review, all comments should either be

resolved and handled properly (very important to not silently resolve comments)
answered with either counter arguments or questions so the reviewer can either accept your arguments or open a discussion

Finally, there are 35 commits in the PR, you may want to re-do commits before the next round of reviews

moha-bekh · 2026-09-21T09:51:24Z

Yeah I didn't see them since I needed to click on the see more button, I will check If they are already resolved, and try to reduce the amount of commits by squashing where it's coherent

exec-harness injected a shared library into every benchmark to drive valgrind's instrumentation from inside the child. That only works on a dynamically linked executable, so statically linked benchmarks were silently unmeasurable, and it forced the harness to ship a `.so` next to its binary. The instrumentation is now toggled in exec-harness's own process, around the spawn: valgrind propagates the state across `fork`/`exec`, so the child is measured without anything being injected into it. The preload library, its compatibility check and the build script that produced it all go away, and the integration constants become plain consts. BREAKING CHANGE: exec-harness no longer ships a preload library. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Without the preload library the benchmark no longer switches instrumentation on for itself: exec-harness toggles it in its own process and then forks. With `--instr-atstart=no` valgrind starts the child uninstrumented and the run dumps a single zero-cost part, so the measurement comes back empty. Pass `--instr-atstart=inherit` for those runs. Entrypoint runs keep the previous default, and `--separate-threads` stays on `simulation_track_subprocess` alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both binaries kept their argument parsing and dispatch in `main.rs`, where nothing else can reach it. Move each into a `cli` module of its own crate and leave `main.rs` as a wrapper that installs a logger and calls `run_cli`. Nothing changes for the standalone binaries, but the runner can now link either CLI and dispatch it in-process. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The runner downloaded `exec-harness` and `memtrack` from GitHub releases at the start of a run, pinned by version, and `setup --mode memory` installed memtrack with `cargo install`. That is a network round-trip on every run, a version matrix to keep in sync, and two more artifacts to release. Link both crates instead and expose them as hidden `codspeed exec-harness` and `codspeed memtrack` subcommands, re-executing the current binary where the runner used to invoke the downloaded tool. They are dispatched before any runner setup: the re-exec happens in the benchmark's working directory, where an unrelated `codspeed.yaml` would otherwise abort the measurement. The memory executor grants the eBPF capabilities to this binary, since memtrack is now a subcommand of it, and the whole binary installer goes away. BREAKING CHANGE: `exec-harness` and `memtrack` are no longer downloaded or installed separately; the runner binary carries them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The released Linux artifacts are musl, and memtrack could not be built for them: `libbpf-sys` vendors elfutils, whose `configure` looks for `argp`, `obstack` and `fts`, none of which musl ships, and Debian's `musl-gcc` runs with `-nostdinc`, so the kernel UAPI headers libbpf needs are out of reach. Put the recipe in the cargo config so a plain `cargo build --target <arch>-unknown-linux-musl` works with no environment set up by hand: seed the autoconf cache for the three checks, add a declarations-only `argp.h` stub on `CPATH`, and add the UAPI header paths back through the per-target `CFLAGS`. aarch64 also needs `-lgcc` for libbpf's outline-atomic helpers. The libc-resolving test reads the Ubuntu multiarch path and only runs in CI: a static musl build has no libc of its own to look at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`exec-harness` and `memtrack` were released as their own artifacts, each with its own dist targets and, for memtrack, its own apt build dependencies. Nothing downloads them any more, so they stop being release units: the apt dependencies move to the runner, which is what now builds the vendored libbpf and elfutils, and memtrack is depended on with its default features so the bundled subcommand carries the tracker and not just the IPC types. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The shipped Linux artifacts are musl and nothing built them before a release tag, so a regression in the musl recipe surfaced at release time. Build both distribution targets on native runners, with no environment set by hand, which doubles as the check that the cargo config carries the whole recipe. The artifact is asserted static with `readelf` rather than a `file` string, since rustc emits a static-PIE for x86_64 musl and `file` spells it differently from aarch64, and both bundled CLIs are asked for their version, which only answers if they are really linked in. The memtrack benchmarks stop installing memtrack separately and call it as a subcommand of the runner, which is also the binary `setup --mode memory` now grants the eBPF capabilities to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The CI used to `cargo install` memtrack and run `setup --mode memory` before the test suite, which gave the installed binary its eBPF capabilities. memtrack is now a subcommand of the binary under test, so the tests have to setcap it themselves, through the self-exe override since `current_exe()` under `cargo test` is the test harness. They also have to refuse to do it without a cached sudo ticket: `cargo test` captures the output, so the password prompt is invisible and the run blocks forever. Capabilities are an xattr, so the grant is lost on every relink. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

not-matthias

we're getting there!

not-matthias · 2026-09-21T16:22:13Z

    ) -> Result<()> {
-        install_memtrack().await
+        // memtrack ships inside this binary, nothing to install.
+        Ok(())


no need to even impl this, we can remove it entirely and use the default trait impl

not-matthias · 2026-09-21T16:22:13Z


-pub const MEMTRACK_COMMAND: &str = "codspeed-memtrack";
-pub const MEMTRACK_CODSPEED_VERSION: &str = binary_pins::MEMTRACK_VERSION;
+pub const MEMTRACK_COMMAND: &str = "memtrack";


i think we can inline this const, as it's just used in a single place

not-matthias · 2026-09-21T16:22:14Z

-            message: "could not parse version".to_string(),
+        tool_name: MEMTRACK_COMMAND.to_string(),
+        status: ToolInstallStatus::Installed {
+            version: env!("CARGO_PKG_VERSION").to_string(),


I think we should still expose the memtrack --version , not the codspeed version.
wdyt @GuillaumeLagrange?

I think this should go altogether, memtrack version has absolutely no interest here now that we have one binary

not-matthias

we're getting there!

moha-bekh changed the title ~~COD-3218 / COD-3440: remove the LD_PRELOAD hack and port exec-harness to musl~~ feat!: ship one binary — bundle exec-harness and memtrack, drop the LD_PRELOAD hack Sep 17, 2026

moha-bekh force-pushed the spike/cod-3440-memtrack-musl branch from ad8eca1 to 2d8d983 Compare September 17, 2026 14:33

moha-bekh marked this pull request as ready for review September 17, 2026 14:33

greptile-apps Bot reviewed Sep 17, 2026

View reviewed changes

Comment thread src/executor/memory/setup.rs

Comment thread src/cli/mod.rs Outdated

Comment thread crates/exec-harness/build.rs Outdated

Comment thread .github/workflows/ci.yml

moha-bekh force-pushed the spike/cod-3440-memtrack-musl branch from 3294b83 to 8e2d7e3 Compare September 17, 2026 15:34

moha-bekh requested a review from GuillaumeLagrange September 17, 2026 15:49

GuillaumeLagrange requested a review from not-matthias September 17, 2026 16:07

GuillaumeLagrange reviewed Sep 17, 2026

View reviewed changes

not-matthias requested changes Sep 17, 2026

View reviewed changes

moha-bekh requested a review from not-matthias September 18, 2026 09:58

art049 requested a review from GuillaumeLagrange September 21, 2026 08:24

GuillaumeLagrange requested changes Sep 21, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Outdated

moha-bekh and others added 8 commits September 21, 2026 08:00

moha-bekh force-pushed the spike/cod-3440-memtrack-musl branch from 96d9261 to ac9a23c Compare September 21, 2026 13:05

moha-bekh requested a review from GuillaumeLagrange September 21, 2026 13:10

not-matthias reviewed Sep 21, 2026

View reviewed changes

Conversation

moha-bekh commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 73.48%

Performance Changes

Uh oh!

greptile-apps Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

not-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

moha-bekh commented Sep 21, 2026

Uh oh!

not-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

not-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

moha-bekh commented Sep 7, 2026 •

edited

Loading

codspeed Bot commented Sep 7, 2026 •

edited

Loading

greptile-apps Bot commented Sep 17, 2026 •

edited

Loading