snap: rebase onto core26 (resolute)

[UtkarshBhatthere]

Summary

Rebases the snap from core24 onto core26. Resolute ships ceph 20.2.0
(Tentacle) in the Ubuntu archive, so the interim
lmlogiudice/ceph-tentacle-rc PPA is dropped entirely.

Five changes, one per commit:

1. base: core26 — PPA removed, python path references updated
   3.12 -> 3.14 (resolute default).
2. dqlite v1.17.0 -> v1.18.7 — v1.17.0's float_t typedef clashes
   with glibc math.h via newer libuv, and v1.18.4 still trips
   dqlite's -Werror default under gcc 15 (C23 const-generic
   memchr). v1.18.7 compiles clean.
3. Bundle python3.14 — core26 ships the python3.14 stdlib in the
   base but no interpreter binary (core24 shipped both), so bin/ceph
   and the other python scripts need an in-snap python3 on PATH.
4. microceph-orch via pip --target — the python plugin's venv
   finishes by symlinking a base interpreter that no longer exists (and
   the ceph part's organize moves the staged one out of the plugin's
   search paths). The venv was only a build vehicle; pip installs the
   same site-packages content directly.
5. rust-coreutils for truncate — resolute's coreutils is an
   empty transitional package after the uutils migration; truncate
   (used by loop-file OSD creation) now lives in the rust-coreutils
   multicall binary, which snapcraft's essential-package filtering
   skips unless listed explicitly.

Testing

Built with snapcraft 9.0.0.post69 (latest/edge), installed
--dangerous in a fresh noble LXD VM with the usual manual plug
connections:

• microceph cluster bootstrap -> mon/mgr/mds up
• microceph disk add loop,1G,3 -> 3 OSDs up/in, HEALTH_OK
  (exercises the bundled uutils truncate)
• microceph.ceph -s works (proves the env python3 shebang path)
• ceph versions: 20.2.0 tentacle across all 6 daemons
• ceph mgr module enable microceph + ceph orch set backend microceph -> Backend: microceph, Available: Yes (proves the
  pip-installed deps import)

Open questions (why draft)

• Staged python is 3.14.4 vs base stdlib 3.14.3: works, but we could
  instead rely on the base stdlib and drop the staged copy (~30M).
• The patches/ loop in the ceph part is not idempotent across
  incremental rebuilds (patch creates files in $CRAFT_STAGE that
  craft-parts does not track); any re-stage without a full
  snapcraft clean collides.
• rust-coreutils vs gnu-coreutils: resolute also packages
  gnu-coreutils if uutils truncate behavior is a concern.
• CI runners need a snapcraft with core26 support.

🤖 Generated with Claude Code

[UtkarshBhatthere]

CI triage: all four matrix failures reviewed

Suite	Verdict
NFS multinode (both runs)	Real bugs, see below
Messenger v2 (one run)	Runner infra flake: LXD forkfile.sock dial failure pushing the test script into the VM; same-SHA twin passed
Deferred-Ceph multi (one run)	Flake-shaped: microcluster forward context canceled during ceph-only bootstrap; twin and all other multi-node suites passed

NFS multinode

Two independent bugs:

1. ganesha 6.x rejects the quoted RecoveryBackend token. 4.x accepted
   RecoveryBackend = 'rados_cluster'; 6.5 logs Unknown token ('rados_cluster') and silently falls back to filesystem v4 recovery,
   breaking clustered grace. Fixed in fd84435 (unquoted; verified on a
   live 3-node core26 cluster: rados_cluster backend loads and reads the
   RADOS recovery db).

2. resolute's ceph 20.2.0 libcephfs segfaults in MDSMap::decode.
   Reproduced with a ~20-line C libcephfs client (ceph_create +
   ceph_conf_read_file + ceph_mount) run against the snap's own
   cluster, no ganesha involved:

   #0 MDSMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)  libceph-common.so.2
   #1 libcephfs.so.2
   #3 Messenger::ms_deliver_dispatch(...)
   

   The crash needs a filesystem in the FSMap and appears tied to standby
   MDS entries, which is why single-node NFS (1 MDS, no standby) passes
   while multinode (3 MDS) crash-loops ganesha the moment the CephFS
   export loads. Multinode NFS will stay red on this PR until the ceph
   side is fixed.

   The tentacle-rc PPA's 20.2.1 (resolute) fixes the decode crash — the
   same reproducer mounts cleanly and the full multinode NFS flow
   (enable, export, cross-node mount, write) passes — but its ceph-mgr
   breaks every cross-module remote() call (nfs export *,
   orch status, pg_autoscaler → HEALTH_ERR, all failing with an empty
   "Remote method threw exception"), which would break more suites than
   it fixes. So this PR stays on the archive 20.2.0 and the path forward
   is getting a fixed 20.2.1 into the resolute archive (or fixing the
   PPA mgr build).

Also in fd84435: the robot teardown diagnostics now tail
ganesha.log — it logs to a file, so the journal-only dump made these
failures undiagnosable from CI output.

🤖 Generated with Claude Code

[UtkarshBhatthere]

Round 3 results + the multinode NFS crash explained

The ganesha.log teardown diagnostics paid off. CI's actual fatal on the
multinode inner nodes:

MAIN :FATAL :Failed to PR_SET_IO_FLUSHER with EPERM.
Take a look at config option allow_set_io_flusher_fail

ganesha 6.x makes a prctl(PR_SET_IO_FLUSHER) call at startup (4.x did
not) that needs CAP_SYS_RESOURCE, i.e. the process-control interface.
The single-node suite connects the full interface set; the multinode
inner nodes only connect the minimal one - hence single-node green,
multinode crash-looping. 2827de2 fixes both sides:

• Allow_Set_Io_Flusher_Fail = true in the generated ganesha.conf, so
  ganesha degrades gracefully wherever the capability is absent
  (verified: with process-control deliberately disconnected the old
  build dies FATAL, the new build logs WARN ... ignoring and the
  service stays active);
• process-control added to the tests' minimal interface set.

Round 3 scorecard (fd84435): everything green except NFS, including
messenger-v2 and deferred-ceph-multi (confirming those were flakes) -
one DSL WAL-DB consistency failure was an osd-purge-vs-osd-down timing
race whose same-SHA twin passed.

The NFS suites this round also confirmed the ceph 20.2.0 client-stack
instability is broader than MDSMap::decode: single-node NFS failed with
a ganesha-rados-grace segfault during the grace-db add (a plain
librados tool, no ganesha involved; prior rounds passed with identical
binaries). And a virgin single-node cluster on the PPA's 20.2.1
confirmed its mgr cross-module remote breakage is inherent, not
cluster-state: fs volume create, nfs export create and
pg_autoscaler fail from first boot.

So the ceph-side matrix stands: archive 20.2.0 = crashy client libs /
healthy mgr; PPA 20.2.1 = healthy client libs / broken mgr remote.
NFS needs both halves, so the suites stay at the mercy of 20.2.0's
nondeterministic client crashes until a fixed tentacle build lands in
resolute. Expectation for this round: multinode NFS gets past enable
now, and either passes or dies at export-load/mount on the libcephfs
decode bug.

🤖 Generated with Claude Code

[UtkarshBhatthere]

Root cause of the 20.2.1 (tentacle-rc PPA) mgr remote() breakage

Pinpointed to a mangled hunk in the PPA packaging, not upstream code.

Upstream src/mgr is essentially unchanged between v20.2.0 and
v20.2.1. The difference is that the PPA's 20.2.1 packaging carries
debian/patches/cephadm-pyo3-fix.patch (a backport of Samuel Just's
main-interpreter rework, needed because PyO3 modules broke cephadm on
resolute - LP: #2150762): mgr modules now load into the main
interpreter by default and dispatch_remote passes results between
modules as pickled bytes. The resolute archive's 20.2.0 does not carry
this patch, which is why its mgr works.

The backported hunk in ActivePyModule::dispatch_remote inverted the
success/error branches:

  if (ret == nullptr) {
    std::string caller = "ActivePyModule::dispatch_remote "s + method;
    *err = handle_pyerror(true, get_name(), caller);
  } else {
    return std::nullopt;   // <- success path returns the error
  }                        //    sentinel with *err still empty
  dout(20) << "Success calling '" << method << "'" << dendl;
  // ... pickle.dumps(ret) - only reachable when ret == nullptr

Upstream main has the correct flow (if (ret == nullptr) { *err = ...; return std::nullopt; }, then the success path pickles the result). So
on the PPA build every successful cross-module remote() call returns
the error sentinel with an empty message - exactly the observed
RuntimeError: Remote method threw exception: from nfs export *,
orch status, fs volume create, and pg_autoscaler, and exactly the
debug-20 log signature (the Calling <module>.<method>... line
appears, the Success calling line never does).

Fix: correct that hunk to match upstream and respin the PPA. With that,
20.2.1 should be healthy on both axes (its client libs already fix the
MDSMap::decode segfaults), and this PR can switch the core26 snap to
it, unblocking NFS entirely.

🤖 Generated with Claude Code