Skip to content

fix(launch_manager): enable ASSERT_DBG in integration tests - #591

Draft
hskang-amelia wants to merge 3 commits into
eclipse-score:mainfrom
hskang-amelia:fix/enable-assert-dbg-in-integration-tests
Draft

fix(launch_manager): enable ASSERT_DBG in integration tests#591
hskang-amelia wants to merge 3 commits into
eclipse-score:mainfrom
hskang-amelia:fix/enable-assert-dbg-in-integration-tests

Conversation

@hskang-amelia

@hskang-amelia hskang-amelia commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #579 (comment: #579 (comment)), which wired SCORE_LANGUAGE_FUTURECPP_ASSERT_LEVEL_DEBUG into unit tests via lm_cc_test but left integration tests untouched.

Integration tests link the production launch_manager daemon binary directly, so enabling the define there would mean enabling it on the same target that ships in production. Instead:

  • Split the daemon's cc_binary into a launch_manager_binary macro (score/launch_manager/src/daemon/daemon.bzl) so a second target, launch_manager_debug_asserts, can share the same srcs/deps but add the debug-assert define, leaving the production launch_manager target untouched.
  • Pointed the binaries list in each tests/integration/*/BUILD at the debug variant instead.
  • Added a binary_renames option to integration_test() (backed by pkg_files's renames) so the packaged binary still shows up as launch_manager in the test environment — the test scripts hardcode that filename, and the debug-variant target has a different Bazel target name.

While verifying this locally in host mode, also found and fixed a pre-existing bug in tests/utils/plugins/localhost.py: LocalAsyncProcess.stop() sent SIGTERM to the whole process group so children launched under fakeroot would receive it, but fakeroot's /bin/sh wrapper traps EXIT/INT, not TERM — so the broadcast killed the wrapper itself before it could wait() for its child and relay the real exit code, making every host-mode integration test report LCM did not exit cleanly, it died with code -15 regardless of whether the daemon actually shut down cleanly. Fixed by signaling only the process(es) fakeroot launched.

Test plan

  • bazel build --lockfile_mode=error --config=x86_64-linux //score/launch_manager/src/daemon/...
  • bazel test --lockfile_mode=error --config=x86_64-linux //score/launch_manager/... (16/16 executed unit tests pass)
  • bazel run //:format.check_Starlark_with_buildifier
  • Ran the integration suite with --//config:integration_mode=host: 17/19 pass in an ad-hoc sandbox, and confirmed the remaining 2 are environment limitations rather than caused by this change:
    • complex_monitoring failed to build in the ad-hoc sandbox (Ubuntu, GLIBC 2.31): its component_complex_monitoring helper depends on //score/health_monitor's Rust FFI, which needs a newer GLIBC than that sandbox's rustc/ferrocene toolchain supports. Confirmed fixed by building the same target inside the repo's own ghcr.io/eclipse-score/devcontainer:v1.10.0 image (Ubuntu 24.04, GLIBC 2.39) — it builds and runs cleanly there.
    • sandbox_options fails in its own require_realtime_scheduling fixture (before the daemon binary even runs): chrt: failed to set pid 0's policy: Operation not permitted. This one is not resolved by the devcontainer alone: even inside ghcr.io/eclipse-score/devcontainer:v1.10.0 with the --cap-add=SYS_NICE from .devcontainer/devcontainer.json's runArgs, it still fails on a Docker host whose daemon hasn't been separately configured with real-time bandwidth — the container's cpu.rt_runtime_us cgroup value is 0 unless the host's /etc/docker/daemon.json sets {"cpu-rt-period": 1000000, "cpu-rt-runtime": 950000} and the daemon is restarted (exactly what the test's own failure message says), which is a host-level setting outside this repo's control.
  • CI (opening as draft per the request on fix(launch_manager): enable ASSERT_DBG in unit tests #579, in case these — or other — failures look different under CI's Docker-mode integration tests)

Follow-up to eclipse-score#579, which wired SCORE_LANGUAGE_FUTURECPP_ASSERT_LEVEL_DEBUG
into unit tests via lm_cc_test but left integration tests untouched.
Integration tests link the production launch_manager daemon binary
directly, so enabling the define there would mean enabling it on the
target that ships in production.

Instead, split the daemon's cc_binary into a launch_manager_binary
macro (score/launch_manager/src/daemon/daemon.bzl) so a second target,
launch_manager_debug_asserts, can share the same srcs/deps but add the
debug-assert define, leaving the production launch_manager target
untouched. Each tests/integration/*/BUILD now points its `binaries`
list at the debug variant.

integration_test() gains a binary_renames option (backed by pkg_files's
renames) so the packaged binary still shows up as `launch_manager` in
the test environment — the test scripts hardcode that filename, and the
debug-variant target has a different Bazel target name.

Test plan:
- bazel build --lockfile_mode=error --config=x86_64-linux //score/launch_manager/src/daemon/...
- bazel test --lockfile_mode=error --config=x86_64-linux //score/launch_manager/... (16/16 executed unit tests pass)
- bazel run //:format.check_Starlark_with_buildifier
- Ran the 19 integration tests locally in host mode with the debug-assert
  variant; failures observed in this environment reproduce identically
  against the unmodified production binary (signal-handling and a
  Rust-toolchain GLIBC mismatch, both pre-existing and unrelated to this
  change).
LocalAsyncProcess.stop() sent SIGTERM to the whole process group so
that the actual test binary (launched under fakeroot) would receive
it and run its cleanup code. But fakeroot's /bin/sh wrapper traps
EXIT/INT, not TERM, so the broadcast killed the wrapper itself before
it could wait() for its child and relay the real exit code -- the
tracked returncode then reported the wrapper's own signal death (-15)
regardless of whether the wrapped binary actually shut down cleanly.

Signal only the process(es) fakeroot launched instead, and let the
wrapper's shell script finish normally and propagate the real exit
status.

Test plan:
- Ran the launch_manager integration suite locally in host mode
  (--//config:integration_mode=host): all 17 tests unaffected by
  pre-existing local environment limitations (a missing SCHED_FIFO
  capability, and a GLIBC/rustc mismatch pulled in by an unrelated
  Rust dependency) now pass and correctly report the wrapped binary's
  own exit status.
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Documentation preview for this pull request is available at:
pr-591: https://eclipse-score.github.io/lifecycle/pr-591/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant