Repository navigation
Restore static linking and fix --version for release binaries - #397
Merged
Merged
Conversation
The OCaml 5 / dune migration (#386) dropped the old ocamlbuild `-static` flag, so release binaries have been dynamically linked against whatever GSL/zlib/sqlite3 happen to be on the build image (e.g. libgsl.so.27 on Ubuntu 22.04) instead of shipping the fully static binary pplacer used to ship. Separately, `--version` read `git describe` at runtime, which always falls back to "dev" for a distributed binary with no .git dir. - dune: add link_flags restoring `-ccopt -static`, plus an explicit repeat of `-lgsl -lgslcblas` (dune links our own GSL-calling C stubs after the library-derived link flags, so the first occurrence must be repeated afterward for the linker to resolve them). - common_src/dune: new rule generates version.ml at build time from $PPLACER_VERSION (if set) or `git describe`, falling back to "dev". Replaces the old runtime-git-subprocess version.ml. - Dockerfile: drop the now-redundant duplicate dune heredoc (root dune already builds statically); wire through PPLACER_VERSION. - scripts/build-docker.sh, build-release.yml: compute/pass PPLACER_VERSION (git describe on the host, or the release tag) since the Docker build context has no .git dir. - scripts/build-common.sh: fix create_static_dune_config, which put the static flag in the wrong dune stanza field (flags instead of link_flags) and so never actually produced a static binary. Closes #396. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified no CI workflow, build script, Dockerfile, or documented build instruction (BUILD.md points at `dune build`) invokes any Makefile target or ocamlbuild directly; the project builds exclusively through dune. The Makefile's own `docs` target was already broken (referenced a nonexistent gen_docs.native). Its `setup_git_version` in myocamlbuild.ml also targeted common_src/version.ml, the same path the new dune rule generates, which would have collided with it. Also drop the now-meaningless ocamlbuild-artifact entries from .gitignore (*.native, *.byte, *.mltop, TAGS, bin, etc.); docs/Makefile (Sphinx) and mcl's own generated build files are unrelated and untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gcc on Ubuntu 22.04's aarch64 target still defaults to PIE even with -static, and static+PIE isn't properly supported there without special glibc scrt1/rcrt1 support. That produced a link failure specific to the arm64 CI leg: relocation truncated to fit: R_AARCH64_LD64_GOTPAGE_LO15 against symbol `__stack_chk_guard' ... too many GOT entries for -fpic -no-pie is the standard fix. Verified it doesn't affect the x86_64 build (still static, --version and all 224 tests still pass). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI confirmed -no-pie doesn't fix the arm64 link failure either (same byte-for-byte error). The actual cause: Ubuntu's prebuilt libc.a crt startup code (libc-start.o) accesses __stack_chk_guard through a short-range GOT-relative relocation (R_AARCH64_LD64_GOTPAGE_LO15, a ~1MB range) that overflows once a fully static binary this large is linked. That's baked into glibc's own prebuilt object; no link flag on our side can fix it (confirmed by searching prior reports of the same error in other large aarch64 static builds: bpftrace, zeek, etc.). Going further (musl-based static libc, alternate linker) is out of scope for #396, whose "libgsl.so.28" mention is an x86_64-style package name anyway. Detect the architecture at build time (uname -m, which correctly reports the emulated arch under the CI's QEMU cross-build) and skip -static/-no-pie only there, via a dune `(:include ...)` rule so the flag list itself is generated rather than hand-duplicated per platform. arm64 keeps its pre-existing dynamic linking; x86_64 and macOS are unaffected. Also applied the same aarch64 guard to build-linux.sh's native build path for consistency, since the same glibc limitation would apply there too. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Pplacer still requires GSL 2.8 in the current Bioconda macOS ARM64 package. The recipe (https://github.com/bioconda/bioconda-recipes/blob/master/recipes/pplacer/meta.yaml) downloads the older v1.1.alpha22 binary and pins gsl=2.8. Its build script That conflicts with r-base=4.5.2, which requires GSL 2.7. Thanks for helping me (and my robots) troubleshoot the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#396 reports two regressions from the OCaml 5 / dune migration (#386):
be on the CI build image (
libgsl.so.27on Ubuntu 22.04), where they usedto ship fully static (the old ocamlbuild config passed
-ccopt -static;that flag was dropped in the dune migration and never replaced).
--versionreportsdevinstead of the actual tag, becauseversion.mlran
git describeat runtime — which always fails for a distributedbinary with no
.gitdirectory.Both are fixed for x86_64 and macOS, confirmed via the actual
build-release.ymlCI workflow (triggered manually viaworkflow_dispatchon this branch) as well as locally on Ubuntu 24.04:
arm64 keeps its current dynamic linking (see the dedicated bullet below for
why full static isn't achievable there) — no regression, just not the
improvement the other platforms get.
dune: addlink_flagsrestoring-ccopt -static, plus an explicitrepeat of
-lgsl -lgslcblas. Dune links our own GSL-calling C stubs(
pam.c) after the library-derived link flags in the final linkcommand, so the first
-lgsloccurrence (from thegsllibrarydependency) has already resolved dynamically by the time our stubs'
references appear; repeating it statically afterward is what actually
removes the runtime GSL dependency.
common_src/dune: new rule generatesversion.mlat build time from$PPLACER_VERSIONif set, elsegit describe, else"dev". Replacesthe runtime-git-subprocess
version.ml.Dockerfile: the release build had its own copy of the dune stanza,hand-duplicated via a
RUN echo ... >> duneheredoc, which wasfunctionally identical to the root
dunefile it was overwriting (theswap did nothing). Removed it — the checked-in
dunefile alreadybuilds statically — and wired through
PPLACER_VERSION.scripts/build-docker.sh/build-release.yml: compute/passPPLACER_VERSION(viagit describeon the host, or the release tagfor a
releaseevent) since the Docker build context has no.gitdir.scripts/build-common.sh: fixedcreate_static_dune_config, which putthe static flag in the wrong dune stanza field (
flagsinstead oflink_flags) — it would never have produced a static binary. This pathis exercised by
scripts/build-linux.sh, which isn't used by CI but isa user-facing native-Linux build option.
myocamlbuild.ml,root
Makefile): confirmed no CI workflow, build script, Dockerfile, ordocumented instruction (BUILD.md points at
dune build) invokes it —the project builds exclusively through dune. Its
docstarget wasalready broken (referenced a nonexistent
gen_docs.native), and its ownsetup_git_versiontargeted the samecommon_src/version.mlpath thenew dune rule now generates, which would have collided with it. Also
dropped the now-meaningless ocamlbuild-artifact entries from
.gitignore(*.native,*.byte,*.mltop,TAGS,bin, etc.);docs/Makefile(Sphinx) and mcl's own generated build files areunrelated and untouched.
run failed on arm64 with
ld: relocation truncated to fit: R_AARCH64_LD64_GOTPAGE_LO15 against symbol '__stack_chk_guard' ... too many GOT entries for -fpic. Tried-no-pie(a plausible first guess,since gcc on Ubuntu 22.04's aarch64 target defaults to PIE even under
-static) — it made no difference, byte-for-byte identical error.Researched the actual cause: Ubuntu's prebuilt
libc.acrt startup code(
libc-start.o) accesses__stack_chk_guardthrough a short-rangeGOT-relative relocation (~1MB range) that overflows once a fully static
binary this large (the whole OCaml runtime plus every module) is linked.
That's baked into glibc's own prebuilt object — not something a link flag
on our side can fix — and matches reports of the same error in other
large aarch64 static builds (bpftrace, zeek). Real fixes (musl-based
static libc, a different linker) are out of scope for version number / dynamic libs #396, whose
libgsl.so.28mention is an x86_64-style package name anyway.dunenowdetects the architecture at build time (
uname -m, which correctlyreports the emulated arch under the CI's QEMU cross-build) via a
(:include ...)rule and skips-static/-no-pieonly onaarch64/arm64; applied the same guard to
build-linux.sh's native buildpath for consistency.
Closes #396.
Test plan
dune buildproducespplacer.exe/guppy.exe/rppr.exewithlddreporting "not a dynamic executable" (x86_64)--versionreports the git tag (v1.1.alpha22-dirty) instead ofdevPPLACER_VERSION=x dune buildoverrides the baked-in version (thepath the Docker build actually uses)
./_build/default/tests.exe→ 224/224 OK(re-verified after every change in this PR)
build-release.ymlon this branch (workflow_dispatch): linux/amd64and macOS succeeded across all runs; linux/arm64 failed twice on
static linking (
-staticalone, then-static -no-pie) beforelanding on "skip static linking on arm64" — re-run in progress to
confirm arm64 now builds (dynamically, as before) without error
🤖 Generated with Claude Code