Refactor Boost.Stacktrace detection

[d-torrance]

Every time I went to use the cmake build on my Ubuntu 24.04 machine, I had to go comment out that CMP0167 line from #4146 because my cmake was too old.

Not a great workflow, so today I finally asked, "Hey Claude, what's up with this?", and it took me down a long rabbit hole about the various ways that we can use Boost.Stacktrace. So what started out as a cmake-only branch turned into a cmake-and-autotools branch so that we end up doing the same thing in both builds.

I'll have Claude comment on the details

🤖 AI Disclosure 🤖

Claude wrote all the code, but I asked a bunch of questions and we ended up trimming down what started out as a bunch of spaghetti code into a pretty nice little patch.

[d-torrance]

Claude here (Claude Code) — Doug asked me to write up the details, so this comment is mine, not his. Everything below I verified by running it on his Ubuntu 24.04 box rather than reasoning from the docs; where a claim was only inference I've said so.

The actual bug turned out to be worse than a stale policy line

The CMP0167 line was just the thing that hurt. Chasing it turned up something bigger.

Since 8c248be, check-libraries.cmake has had this shape:

if(UNIX)
  cmake_policy(SET CMP0167 OLD)
  find_package(Boost REQUIRED QUIET COMPONENTS regex OPTIONAL_COMPONENTS stacktrace_addr2line)
else()
  find_package(Boost REQUIRED QUIET COMPONENTS regex OPTIONAL_COMPONENTS stacktrace_backtrace)
endif()
if(Boost_STACKTRACE_BACKTRACE_FOUND)
  ...

UNIX is true on Linux, macOS, BSD and Cygwin; it's false only for native MSVC or MinGW-w64, and the cmake tree has no support for either (no WIN32/MINGW/CYGWIN conditionals anywhere, no Windows CI). So the else() was dead, only stacktrace_addr2line was ever requested, and Boost_STACKTRACE_BACKTRACE_FOUND could never be true — the preference chain immediately below it had no reachable first case. Every platform got addr2line.

That matters because addr2line resolves nothing in a PIE executable, and Ubuntu's gcc is --enable-default-pie while M2 sets no PIE-related flags. Same program, same backend, only -no-pie differing:

PIE:      0# 0x0000599E270E93F5 in ./pie_a
-no-pie:  0# main at .../pie.cpp:4

backtrace() returns absolute runtime addresses; ASLR slides them relative to the DWARF in the file; Boost's addr2line backend forks addr2line without correcting for the load base, so it looks up an address the file doesn't describe and falls back to printing hex. libbacktrace doesn't have this problem because it works in-process and knows the slide.

So on the most common Linux platform, a cmake-built M2 was printing bare addresses out of its SIGSEGV/flint-abort handlers, while the autotools build — which preferred backtrace all along — printed file:line. Worth noting M2 already knew this once: the comment 01bf7b8 deleted from configure.cmake read "addr2line is more readily available, but does not work well with -fPIE".

What the patch does

Both builds now prefer backtrace, then fall back to the header-only addr2line implementation, then to Boost's default backend.

Two findings shaped that and are worth stating because neither is guessable from the diff.

1. BOOST_STACKTRACE_USE_* is inert whenever BOOST_STACKTRACE_LINK is defined. frame.hpp:73 wraps all backend dispatch in #ifndef BOOST_STACKTRACE_LINK, and every read of USE_BACKTRACE/USE_ADDR2LINE lives in frame_unwind.ipp, which is only included there. Running all four library×macro combinations confirms the linked library always wins. So main.cpp's unconditional #define BOOST_STACKTRACE_USE_ADDR2LINE was selecting nothing, and the mismatch between it and the linked library was cosmetic rather than a bug. It's now set by the build system in the one case it has an effect — header-only mode — alongside BOOST_STACKTRACE_ADDR2LINE_LOCATION.

2. Linking libboost_stacktrace_addr2line is never worth it. In link mode the addr2line path is compiled into that prebuilt library, defaulting to the literal /usr/bin/addr2line, which macOS doesn't have — and our define can't override it, per (1). Header-only produces byte-for-byte the same output where both work and honors a path we choose. I confirmed the override really takes effect by copying addr2line somewhere odd (worked), then replacing that copy with exit 1 and watching the same binary degrade to hex. So the addr2line library branch is strictly dominated; it's gone.

3. libbacktrace is checked as well as the Boost component. Boost's own cmake files do this:

set_property(TARGET Boost::stacktrace_backtrace APPEND PROPERTY INTERFACE_LINK_LIBRARIES
  backtrace dl)

Unconditionally — nothing verifies libbacktrace is installed. So a missing libbacktrace was a hard link failure rather than a graceful fallback. And in true FindBoost module mode (no BoostConfig.cmake, i.e. Boost < 1.70 or hand-built) the imported target carries nothing at all, so the patch also puts backtrace on the link line explicitly. Reproduced with -DBoost_NO_BOOST_CMAKE=ON: INTERFACE_LINK_LIBRARIES=[_i-NOTFOUND], then undefined reference to backtrace_create_state. Where Boost does supply it, cmake dedupes — -lbacktrace appears exactly once on M2-binary's link line.

The autotools side never needed this because it link-tests (the test is the check) and because plain -lboost_stacktrace_backtrace resolves to the shared library, which Debian builds self-contained (it defines backtrace_create_state itself, with no NEEDED entry). STATIC_BOOST=ON is what exposes the static .a, so this was a cmake-only hazard. I added the matching AC_CHECK_LIB anyway so the two builds can't diverge in that corner; it's free when unused, since libbacktrace ships as a static archive only — binaries built with and without a redundant -lbacktrace are byte-identical.

On CMP0167 itself

Removing it is a straight fix, and it was never doing anything: cmake_policy(VERSION 3.24) on CMakeLists.txt:19 already leaves CMP0167 unset (I checked with CMP0149, added in 3.28, which reads back empty in exactly that configuration), so FindBoost was being loaded either way — and FindBoost hands straight off to BoostConfig.cmake at line 606 when it finds it, which the Boost_DIR/boost_regex_DIR entries in the cache confirm. The upstream bug it was meant to work around, CMake #26824, is a 4.0.0 regression fixed in 4.0.1 that fires under OLD, so the workaround selected the broken path. Its only real effect was making configure fatal on every cmake < 3.30.

Setting CMP0167 NEW (going to BoostConfig.cmake directly) is deliberately not in this PR — it changes no variable we read, no target we link and no macro we define (verified side by side), so it's pure future-proofing that would raise the cmake build's Boost floor to 1.70. Boost_INCLUDE_DIR → Boost_INCLUDE_DIRS is here, though, since only the plural is set in config mode.

Verification

• cmake reconfigure of an existing tree: exit 0, selects backtrace, links libboost_stacktrace_backtrace.a + -lbacktrace, defines BOOST_STACKTRACE_LINK only; main.cpp and boostmath-tmp.cc compile.
• autoconf clean, sh -n clean, no unexpanded macros in the generated script, and a full ./configure run to exit 0 reporting boost_stacktrace_backtrace.
• Every branch of both chains exercised by forcing the conditions, including the header-only and default-backend fallbacks.
• Runtime check with the exact flags selected: demangled frames with file and line.

Two reviews were run over the result. One caught a real bug I'd introduced — I was checking for libbacktrace without linking it — and demolished my original premise that a backend needs both a library and a macro, which is what let the patch shrink to roughly half its first size. Reviewers were wrong about two things I'd flag if they come up again: Boost::stacktrace_backtrace does carry -lbacktrace transitively in config mode, and cmake_minimum_required(VERSION 3.24...3.30) does not leave CMP0167 set to NEW on cmake ≥ 3.30, because line 19 unsets it.

Things a reviewer might reasonably push back on

• Homebrew/macOS gets no backtrace backend. The formula's dependencies don't include libbacktrace and there's no libbacktrace formula in homebrew-core at all, so Boost almost certainly doesn't build that variant there (that part is inference — I have no Mac to check). addr2line on macOS comes only from keg-only binutils, and macOS is PIE regardless, so Homebrew builds realistically land on Boost's default backend: symbol names for shared-library frames, hex for M2's own. That's the honest best available, not a regression, but it's worth knowing.
• -rdynamic would help that case a lot and isn't set anywhere. It's what turns 0x00005B55918D4696 into inner() for the default backend. Out of scope here since it affects the whole link.
• target_compile_definitions(M2-binary PUBLIC ...) should probably be PRIVATE — exporting compile definitions from an executable target into Macaulay2Config.cmake is meaningless. Pre-existing, left alone.
• The second commit (check_include_files(... LANGUAGE CXX)) is an unrelated drive-by: that check ran under the C compiler and always failed on algorithm: No such file or directory, so HAVE_BOOST_MATH_TOOLS_ATOMIC_HPP was never defined in cmake builds and boostmath.dd never saw BOOST_MATH_NO_ATOMIC_INT. It's split out so it can be dropped independently. It does switch on a code path that has never run in a cmake build, so it deserves its own look.