Add authoritative KV snapshot companion for sub-3s LB recovery

Problem

A long-lived node06 vLLM publisher now retains roughly 9,500 scheduler steps. r33 proved that reconstructing exact state through bare vLLM replay is dominated by publisher delivery, not Rust work:

A received 5,500 batches / 254,487,816 payload bytes before timing out
decode: 2.12s
exact-index fold: 0.16s
receive wait: 177.70s
maximum receive gap: 177.52s
total: 180.04s, fail closed

B's 69-batch / 3.79MB replay completed in 137ms. Both engines remained restart-zero and serving health stayed 2/2. Increasing replay timeout again is explicitly out of scope.

Bare vLLM exposes only PUB live events plus a bounded ROUTER replay from one starting sequence; it has no authoritative current-state snapshot, epoch, or too-old fallback. NVIDIA Dynamo's LocalKvIndexer provides the right recovery shape (TreeDump + real-event watermark + transactional replacement), but its dump contains block hashes rather than the raw token slices mini-dynamo's current radix tree requires.

Proposed first slice

Add a long-lived mini-dynamo-indexer companion, one service owning independent per-engine states:

Continuously consume the already-qualified r34 live/replay stream independently of LB lifecycle.
Maintain a bounded compact block-digest radix index while retaining opaque vLLM hashes only for parent/removal identity.
Expose a versioned container-local MessagePack snapshot containing:
- schema and digest algorithm/key identity
- attested engine incarnation (model/version/image/process start)
- real vLLM watermark and reset scope
- group geometry/filter metadata
- BFS block records
- record count/byte bounds and checksum
LB recovery order: subscribe live first -> request snapshot -> build scratch -> validate schema/incarnation/checksum/capacity -> drain strictly-after-watermark tail -> atomically swap/trust.
Fail closed on every gap, sequence regression, incarnation mismatch, unsupported group, malformed record, capacity breach, cancellation, or checksum mismatch.
Keep the snapshot memory-only; never log token IDs, block hashes, digest keys, prompts, or response content.

Acceptance gates

Pure mock producer/consumer unit tests cover atomic replacement and visibility.
Reject schema/digest/incarnation/checksum/capacity mismatches without exposing partial state.
Cover sparse vLLM sequence semantics, tail before/after watermark, reset, cancellation, and reconnect.
Golden compare compact digest lookup with the current raw-token index across branching/removal/variable block geometries.
Captured-shape benchmark at 36,612 resident blocks / 9.37M source token IDs completes snapshot encode + transfer + rebuild in under 3s with bounded RSS.
Shadow deploy on node06 without restarting either engine; compare exact decisions against the current index.
LB-only restart recovers both authoritative inventories in under 3s at p95.
Companion restart/bootstrap fails closed and does not impair ordinary serving.
Rollback removes only the companion/snapshot path and returns exact routing to approximate fallback.

Upstream references

vLLM ZMQ publisher/replay: https://github.com/vllm-project/vllm/blob/98f86b9c02329200a0390aecfe598e27928cbf40/vllm/distributed/kv_events.py#L290-L506
Dynamo replay comparison: https://docs.nvidia.com/dynamo/latest/components/router/kv-event-replay-dynamo-vs-v-llm
Dynamo request/response types: https://github.com/ai-dynamo/dynamo/blob/ac6b90d2c1bfbb9587dc0e4b55e827dd4ff1210e/lib/kv-router/src/indexer/types.rs#L70-L127
Dynamo local indexer/tree dump: https://github.com/ai-dynamo/dynamo/blob/ac6b90d2c1bfbb9587dc0e4b55e827dd4ff1210e/lib/kv-router/src/indexer/local.rs#L263-L676

Related: #32 and PR #40.

Foundation merged in #44 (merge commit 6b146f2).

What is now in-tree:

versioned named-field MessagePack snapshot envelope with corruption checksum, exact engine incarnation, watermark, reset scope, digest specification/key ID, group geometry, per-block digest records, capacity bounds, cancellation, and privacy-safe errors;
an arena-backed compact digest-index prototype with branch/removal/reinsert behavior, variable block geometry, bounded lookup, and fail-closed forced-collision tests;
captured-shape benchmark tooling and the companion lifecycle/recovery design.

Measured on this dev host:

36,612 records / 9,372,672 logical tokens;
5,710,914-byte frame;
encode 10.34 ms;
decode + validation 10.79 ms (8.29 ms repeated);
standalone snapshot benchmark RSS 27,672 KiB.
digest index: 2,048 blocks / 524,288 tokens, 65,536 commitment bytes vs 2,097,152 raw token bytes (32x smaller), 1.96 ms build, ~1.36 ms average full 524K-token lookup, 7,468 KiB standalone RSS.

The adversarial review found and the PR fixed a fail-open collision/removal path: poisoned child slots now retain the original arena node, pruning preserves poison, and collision -> remove -> reinsert remains fail closed.

No node06 rollout was performed; this PR is GPU-free and not wired into serving. Next slice is a production DigestKvIndex module with golden parity against ExactKvIndex, authenticated key/session ownership, snapshot import, and captured-shape snapshot-to-index timing before any live shadow deployment.

PR #45 implements and qualifies the production compact digest index: #45

Key result: the captured 36,612-block snapshot imports in 13.1ms; 80,896-token lookup is 235us (4.66x raw exact and inside the 250us/5x gate); 524K lookup is 1.53ms; matched 15,168-block RSS is 8,124KiB versus raw exact 21,548KiB (37.7%). The differential oracle ran 32 x 1,000 mutations plus explicit geometry/hash/tombstone cases with zero overclaims or mismatches.

The adversarial review also changed the contract before transport work: schema v2 carries present tombstones; v1 import semantics accept FullEngine and exactly one indexed group only; collision poison survives remove/reinsert; snapshot import is bounded, cancellable, and atomic. Whole-session authentication, incarnation/watermark freshness, and live-tail gap fencing remain hard blockers, so this is not deployed or wired to placement.

PR #46 adds the corrected authenticated bootstrap and tail-lifecycle foundation: #46

The adversarial review materially changed the first draft before publication: fixed binary framing now authenticates the client hello before snapshot work and verifies bounded borrowed response bytes before MessagePack allocation/copy; exact response values became monotonic watermark/generation floors; authenticated snapshots are opaque; and strict continuity moved to a companion delivery sequence distinct from vLLM sparse scheduler-step watermarks.

Local gates: 164 Rust unit tests, 8 exchange adversarial tests, 11 lifecycle tests, integration through private digest-index build/CatchingUp, 2 parity tests, 15 snapshot-wire tests, strict Clippy/release, Go gates, and 52 Python tests. Still unwired: UDS SO_PEERCRED, UID/GID/socket hardening, secret files, authenticated tail/control decoder, deadlines/queues/metrics and atomic-swap races remain required. No node06 state changed.

PR #47 merged the authenticated Unix exchange and tail wire: #47

The one-shot snapshot transport now verifies Linux peer UIDs on both ends before protocol bytes, uses one absolute connect/production/I/O/decode deadline, bounds response reads with max+1 detection, and immediately cancels the pending producer when the client disconnects. Authenticated tail/control frames bind the exact payload, incarnation, digest key identity, companion generation, dense delivery sequence, sparse real-event watermark, session, and direction. Tail keys are ephemeral derivatives of the separate session secret; replay/tamper/mixups fail closed, and downstream decode/index failure has an explicit terminal fence.

Local gate: 180 Rust unit tests + 18 Rust integration tests, strict all-target Clippy/release, Go parity/vet/format, and 52 Python tests. The focused transport/tail loop took 3.4s; the full local gate took 29.2s with Go/Python parallel to the one Rust lane. GitHub and both Drone pipelines passed. No node06 state changed; live health remained 2/2 with zero inflight at the final read-only check.

Still deliberately unwired: safe companion-owned socket bind/rename/inode cleanup, distinct UID/GID/secret-file deployment, bounded actor/client/tail queues, payload-to-private-index application, atomic caught-up publication, metrics, and offline fault/load qualification. Those are the next slice before a shadow-only node06 rollout.

PR #48 merged the secure filesystem boundary and atomic snapshot publication actor: #48

What is now in-tree:

the sole production path from an authenticated response to state is an opaque verifier-created generation that cross-binds full-engine scope, inner/outer watermark, incarnation, digest identity, private digest index, and CatchingUp lifecycle;
a deterministic hard-two-session actor bounds private tail queues, preserves same-identity publication during replacement catch-up, revokes on identity/key/generation changes, applies deltas only to actor-owned state, swaps atomically only on authenticated caught-up, and ignores stale-epoch disconnects;
session secrets load as exactly 32 raw bytes with trusted-ancestor, symlink, owner/mode/link, opened-inode, and bounded-read checks;
Unix sockets publish mode 0660 from a unique private bind into a companion-owned non-writable directory with atomic no-clobber semantics and same-inode-only cleanup.

Validation: 206 Rust unit tests + 20 integration tests, strict Clippy/release, Go parity/vet/format, 52 Python tests, and Compose render/mirror checks. Full local gate was 29.5s. GitHub passed in 1m19s and Drone passed after fixing one root-container-sensitive test assertion: the wrong-owner case correctly failed closed as UnexpectedOwner under UID 0 rather than the non-root host's earlier UntrustedParent. Drone now avoids duplicate feature-push + PR cold builds while retaining PR and post-merge-main coverage.

No node06 state changed. Remaining before shadow deployment: runtime accept/tail tasks, bounded CPU cancellation, KV-delta decode/application adapter, metrics, hardened companion Compose service, and adversarial two-client/slow-reader/overflow/late-completion tests.

Merged the next offline companion-runtime slice in #49 (ef0aaba).

What is now on main:

hard two-client Unix session supervision with immediate excess rejection
one absolute per-session deadline, shutdown cancellation, isolated task failure, and reliable slot reuse
bounded KV tail decoding into private digest-index store/remove/clear mutations
exact DP-rank/group/local-GPU/main-attention filtering
fail-closed clearing on any decode or index error, including a late failure after earlier valid records

Validation: 217 Rust unit tests, 20 Rust integration tests, 52 Python protocol tests, strict Clippy, release build, and the Go parity suite. Warm local full gate was ~4s; GitHub Actions passed in 1m24s and Drone passed.

Still intentionally offline: the authenticated session handler that joins transport + snapshot build + tail + publication, content-free metrics/config, compose sandboxing, and node06 shadow validation. No production state changed.

Merged bounded companion config/observability in #50 (a958bc6).

The runtime is off by default and now validates the fixed two-engine/two-client shape plus queue, deadline, frame, decoded-batch/event, path/topic, and endpoint bounds. Config/source Debug redacts path, UID, topic, and endpoint identity. Prometheus uses only fixed engine/client slots and closed enums, covering session terminal/rejection/failure outcomes, prepare/decode/apply/caught-up phases, tail batches/events, fences/discards, identity changes, and published generation/block/token counts; arbitrary protocol/error/content values cannot create labels.

Independent review caught and fixed missing frame/batch limits, terminal/discard/token metrics, and sensitive derived Debug output before merge.

Validation: 229 Rust unit tests, 20 Rust integrations, 52 Python tests, strict Clippy, release build, Go parity. GitHub passed in 1m38s and Drone passed. No production state changed.

Merged the remaining r40 offline boundaries:

Add offline snapshot companion security harness #51 (bce13a8): profile-disabled snapshot security harness. Default render has zero services; reserved .invalid images and explicit fixture-only commands prevent accidental start. The harness enforces no network/ports/GPU/host IPC/Docker socket, read-only roots, dropped capabilities, no-new-privileges, fixed distinct UIDs/shared GID, resource bounds, a companion-writable host-tmpfs-backed socket bind exposed read-only to the LB fixture, and an exact read-only 32-byte secret. CI semantically validates the contract. Canonical production compose and node06 were untouched.
Consume authenticated snapshot tail sessions #52 (007ed77): LB-side authenticated snapshot/tail consumer. It verifies SO_PEERCRED before protocol, builds private digest state on a cancellation-aware blocking worker while buffering authenticated tail, checks dense delivery independently of sparse real watermarks, atomically publishes on caught-up, and synchronously fences/revokes on timeout, task abort, EOF, MAC/protocol failure, gap, or disconnect.

Six real Unix tests cover happy/sparse publication, authenticated disconnect, bad MAC, gap, stale generation, EOF revocation, and task-abort revocation. Full gate: 235 Rust unit + 20 integration tests, 52 Python tests, Go parity, strict Clippy, release build. GitHub and Drone passed.

Remaining before node06 shadow: outbound reconnect/fresh-challenge reuse prevention, companion-side snapshot/tail producer, and the documented offline fault-matrix gaps. No production state changed.

r41 is merged on main through #55, #56, and #57 (head 6aba4fb).

Produce authenticated snapshot tail sessions #55: companion/server producer behind the two-client supervisor. Auth gates source work; live capture begins before snapshot build; length-framed snapshot plus authenticated dense-sequenced/sparse-watermark tail; bounded backpressure; client-drop/deadline/rollover cancellation. 7 real Unix tests.
Add adversarial snapshot consumer matrix #56: 9 adversarial LB consumer integration tests for deadlines, oversized/truncated framing, 60k-record build cancellation, two-session atomic replacement/stale disconnect, generation rollover, real store/remove deltas, and content-safe errors. No runtime bug found.
Own snapshot consumer reconnect lifecycle #57: LB outbound reconnect owner with trusted-parent validation on every connect, OS-random 256-bit challenge nonreuse, bounded jittered backoff, one absolute deadline, prompt shutdown, serial reconnects, and explicit capacity-one rolling handoff. 7 focused tests.

The local producer full gate passed 242 Rust unit + 20 integrations; with the adversarial matrix the Rust total is 264. Go parity, 52 Python tests, strict Clippy, and release builds pass. GitHub and Drone passed all PRs.

The protocol/runtime ownership boundary is now complete offline. Remaining critical path: a concrete long-lived vLLM-backed index source, runtime command/config/metrics wiring, then full captured-shape end-to-end and node06 shadow. No production state changed.

r42 is merged on main through #58, #59, and #60 (head 1ad4a862).

Compose snapshot companion runtime #58: off-by-default companion coordinator with hardened secret loading, bind-last socket publication, bounded two-client supervision, cleanup, readiness, and closed-label metrics. It remains library-only and accepts exactly one injected source because the current hello has no engine selector.
Add full snapshot stack lifecycle gate #59: true authenticated Unix-stack E2E across publication, live store/remove, rolling handoff, LB restart, companion shutdown/restart, identity rollover, capacity rejection, and leak-free teardown. Captured 36,612-block / 9.37M-token handoff: 0.15s process wall, 22.355ms private rebuild, about 58MiB peak RSS.
Add bounded long-lived companion index source #60: concrete long-lived per-engine digest source. Replay is private and exact-terminal before atomic publication; live watermarks are correctly sparse; replay/index/generation failures fence closed; export/encode is cancellation-aware blocking work. Tail payloads are shared Bytes, queues have both count and aggregate byte bounds (16MiB default/64MiB max), and prioritized source revocation prevents stale FIFO drain. Real Unix tests prove overflow/rebuild close within 250ms without sending queued stale frames.

Independent parallel review caught and fixed six material defects before #60 merged: dense interpretation of sparse vLLM watermarks, reusable invalid replay staging, synchronous reactor-blocking export, fail-open generation exhaustion, stale queued-tail drain during fencing, and an item-only queue that could retain about 16GiB under defaults. The final full gate passed 300 Rust tests (269 unit + 31 integration/E2E), strict Clippy/fmt, Go tests/vet, GitHub Actions, and Drone.

Measured remaining caveat: synchronous atomic index clone under the source lock is 7.66ms for one captured-shape start and 22.99ms for two serialized starts; at 131,072 records it is 28.45/82.27ms. It clears the 3s recovery gate but needs clone-duration telemetry and immutable/COW ownership if live ingestion p99 must stay below 10ms.

Still offline/no node06 change. Next critical path is the process-level owner: hardened digest-secret and attestation inputs, ZmqKvEventSource, subscribe-before-replay, KvEventFence, reconnect/live feeding, source readiness metrics, authenticated engine selection or per-engine sockets, and phase-specific snapshot vs resettable tail-idle deadlines. Only after that should the 100,000-decision revision-stable shadow begin.

r43 is merged through #62, #63, and #64 (main 8f213d25; subsequent #65 is Infernal-only tooling).

Drive companion readiness from exact source state #62: operational readiness is now listening && source_ready, with separate fixed-cardinality listening/source phase/watermark/block/session diagnostics. Listener exit cannot leave the legacy ready gauge green.
Split companion snapshot and tail deadlines #63: one absolute authenticated snapshot/bootstrap/write budget plus resettable bounded tail wait/write budgets. Healthy progressing tails are no longer killed by a bootstrap lifetime, while idle/blocked clients, disconnect, source revocation, and shutdown remain bounded and prioritized.
Own companion KV lifecycle and gate Infernal C128A #64: the process-level per-engine ZMQ owner. It subscribes before replay, streams full sparse replay into generation-guarded private state, folds the replay ROUTER's bounded post-watermark tail before publication, fences on disconnect/authority/replay/apply failure, and reconnects with bounded backoff.

Independent integration review fixed three initially green correctness/resource defects before #64 merged: (1) reuse of the last pre-disconnect watermark could miss events emitted before the new SUB existed, so reconnect now remains fenced until a fresh live watermark; (2) incremental Vec<SequencedBatch> replay had a replay_limit * max_payload_bytes retention product, so gaps now stream a full private rebuild; (3) the libzmq drain interval did not poll async SUB, so the ROUTER tail is now folded into the same private generation and later SUB frames deduplicate. Repeated connect failures/clean rebuild stages are generation-idempotent, and abandoned blocking workers are generation-fenced.

Integrated validation: 315 Rust tests, strict Clippy/fmt, Go parity, 61 Python tests, agent corpus, real TCP ZMQ, and 25.72s warm release build. Node06 remains unchanged and healthy 2/2. Remaining production boundary: one authenticated owner/socket per engine in an off-by-default executable plus the two-engine offline Compose/security harness, then revision-stable node06 shadow.

The standalone one-engine companion foundation is now merged in #70 (main 340e203).

It composes the authenticated attestation watcher, one ZMQ owner, digest/session secret preflight, bind-last UDS publication, bounded two-client supervision, closed-cardinality telemetry, and clean shutdown in a separate binary/image so the LB release path does not relink it. Review fixes landed before merge: serve mode rejects DS4_SNAPSHOT_MAX_CLIENTS != 2 before any side effect; root CI uses explicit non-root test identities without weakening production UID checks; and healthcheck <socket> passively verifies the safe parent plus socket ownership/type/mode/link count without consuming an authenticated session. The companion image publishes no misleading TCP metrics port.

Drone #207 passed the full 334-test Rust surface, strict Clippy, Go parity, 101 Python tests, and Compose validation in 59s. This remains offline and off by default; node06 was not changed and is healthy 2/2. Remaining production boundary is the host attestation provisioner, real dual companion Compose wiring, and a separate permission-isolated metrics UDS/Caddy path before revision-stable shadow enablement.

Upstream refresh (2026-08-13) adds one GPU-free admission gate before node06 shadow:

replay the captured 2,442-removal / 192-batch eviction burst through the compact companion index
record p50/p95/max apply latency and source-lock hold time
require p99 <=10ms, otherwise move generation ownership toward immutable/COW state before shadow

Rationale: current Dynamo main optimized batched compressed-edge removals, but mini-dynamo uses a different digest representation, so we should measure our actual captured shape before copying that optimization: ai-dynamo/dynamo@c0c8974

This does not alter the critical path: host attestation provisioner + dual production-shaped Compose + permission-isolated metrics UDS, then 100k revision-stable shadow comparisons and <3s LB recovery.

Captured eviction-burst gate is now implemented GPU-free in examples/companion_eviction_bench.rs. The 20-iteration run replayed 3,840 source calls covering the observed 192-batch shape each time: 2,442 removal events (882 main MLA, 1,560 filtered non-main), with exact final residency 3,456 - 882 = 2,574 blocks.

Measured uncontended apply_live elapsed time (a conservative upper bound on source-lock hold with no subscribers): p50 0.38us, p95 0.95us, p99 1.55us, max 45.45us. This clears the 10ms admission threshold by over two orders of magnitude, so the captured eviction shape does not justify immutable/COW generation ownership.

Command: cargo run --release --locked --example companion_eviction_bench. This was local only; node06 remained untouched.

Production admission foundation is now merged through #89, #90, #91, and #92 (main fa6cd2f).

Newly complete:

fixed-cardinality LB snapshot readiness/reconnect metrics tied to authoritative publication, with balanced attempt/connection gauges;
hot authenticated engine-incarnation refresh: loss/rotation/channel closure synchronously revokes exact state and cancels stale sessions; recovery uses a fresh challenge without LB restart;
production-shaped deploy/dspark_0731 overlay with one isolated companion per engine, separate provisioner profile, distinct session/metrics groups, immutable image digests, Caddy metrics-only UDS routes, semantic Compose validation, and host preflight;
fail-closed off-mode startup: raw events, exact router, and snapshot authority are disabled together; shadow uses the same single control so the gates cannot diverge;
explicit Drone dependency image, reducing real runner LB publication 135s -> 90s and companion 129s -> 85s; deployment-only merges publish no images.

Independent review caught and fixed two rollout defects before #90 merged: the inherited exact-shadow default made off-mode startup invalid, and the first Caddy validator allowed an extra session-socket upstream. Both now have negative regression tests. Current deployment tests: 118 Python total, including 10 focused production-Compose cases.

No node06 mutation yet. The remaining rollout sequence is deliberately operational:

provision documented UIDs/GIDs and distinct /run tmpfs authorities/secrets;
pass host validator in pre-provision mode;
generate fresh engine metadata/attestations and pass full mode;
start both companions and the pinned LB with snapshot/exact mode still off;
verify 2/2 ordinary serving and metrics-only Caddy access;
enable shadow, measure LB-only recovery p95 <3s and captured-shape state;
collect >=100,000 revision-stable exact-vs-approximate comparisons before placement can consume snapshot state.

Node06 currently lacks the required identities/authorities, so profile start remains gated on the documented host provisioning and preflight.

Production admission advanced through #103 and #104, followed by the first node06 off-mode qualification.

Hardened host setup created the fixed non-login identities, six isolated tmpfs parents, and four independent secrets; --check and the full host validator pass.
Fresh metadata and authenticated attestations were produced for both unchanged r34 engine processes.
Both v0.1.0 companion containers are running healthy with zero restarts and permission-isolated UDS metrics.
The restricted LB overlay was started with snapshot/exact modes explicitly off. LB-only admission took 2.06s, /health stayed 2/2, engines retained exact container IDs/start times and zero restarts, and a real c1/max16 stream passed at 131.4ms TTFT / 364.9 decode tok/s.
The normal raw-event-shadow LB was then restored in 2.28s so the off-mode admission test would not replace ordinary observation telemetry. Approximate serving remained authoritative throughout.

The important negative result: both new companion sources connected but fenced with zero indexed blocks. Their vLLM publishers have already aged past a reconstructable sequence-zero generation, so a new long-lived consumer cannot honestly synthesize current state. Shadow was therefore not enabled. The companions remain running to retain the next natural engine generation; neither engine was restarted to manufacture a pass.

Next #41 gate is now operational rather than implementation work: after each engine next rolls naturally, require companion source ready=1 with an attested matching incarnation, then test an LB-only snapshot-shadow recovery and measure p95 against the <3s target. Until both sources are authoritative, exact placement remains disabled and ordinary approximate routing is unaffected.

Production-shaped companion rollout update:

Stop invalid companion replay churn #116 (4203b06) stops completed structurally invalid full replays from reconnecting/repeating against the same engine generation. The private state is discarded and the existing SUB connection remains fenced observe-only until an authoritative clear or attested incarnation change; transport failures remain retryable. A regression proves exactly one replay, no reconnect on ordinary events, and clear-boundary recovery.
Repin snapshot deployment images #117 pins exact immutable LB/companion GHCR digests. Host authority, production host, semantic Compose, and full-profile render gates passed on node06.
Only snapshot-companion-a/b were recreated under the common lock with --no-deps; both are healthy/restart-zero. LB and both r34 engine start identities were unchanged, /health remains 2/2, and parallel direct probes returned HTTP 200 in 0.369/0.371s.
Each companion has one connect attempt/connection and no replay churn. Both remain source_ready=0 because they joined engine generations already beyond the bounded 10,000-step bootstrap window. This is an admission failure, not a recovery result; no engine/cache was restarted or cleared.

Evidence and iteration timings are in #118. Next accepted gate is readiness on the next independently justified engine generation, then snapshot shadow + LB-only restart recovery timing; do not manufacture readiness by restarting a healthy engine.

Diagnostic companion rollout update:

Classify companion replay failures #119 added bounded replay-invalid phase telemetry (apply, boundary, tail, commit) and stable fencing for private commit failures.
Pin snapshot replay diagnostic images #120 pinned the immutable 123dd9d companion/LB artifacts; Drone chore: pin v0.8.0 release images #316 passed in 27s.
The complete node06 host-authority, production-host, semantic-Compose, and dual-profile render gates passed before mutation.
Under the common deployment lock, only snapshot-companion-a/b were recreated (--no-deps). Both became healthy with zero restarts; the locked recreate/verification took 8.3s.
The LB and both r34 vLLM engines retained their original start timestamps and zero restart counts. LB/direct-A/direct-B probes returned HTTP 200 in 0.131–0.139s.
Each companion has one stable connect attempt/connection, authenticated authority, and a listening UDS. Both remain source_ready=0 because the current engine generations are already beyond bounded bootstrap. There is no replay/reconnect churn, and snapshot routing remains off.

Evidence is merged in #121. The remaining qualification is unchanged: keep the companions resident through the next independently justified fresh engine generation, require both sources authoritative, enable shadow mode, then measure LB-only authenticated recovery (p95 <3s) before the 52/64-app capacity boundary. We did not restart or clear a healthy engine to manufacture readiness.

The fail-closed operational gate for the remaining #41 acceptance work is merged in #122 (ac0c288).

What it now guarantees:

default audit mode is read-only and returns exit 3 before mutation unless both companions are authoritative, watermarked, ready, listening, and stable across two UDS metric samples;
current LB config must match a freshly rendered base-service hash before apply mode is admitted, so the exact rollback is reproducible;
apply mode reserves a new mode-0600 content-free journal first, then holds the common deployment lock across five LB-only snapshot-shadow recreates and mandatory rollback;
each sample requires both LB inventories authoritative, 2/2 health, and unchanged engine container/image/start/restart identity;
nearest-rank five-sample p95 must be <=3s; success or failure restores and verifies the original image/config hash before unlocking;
there is no engine or companion start/stop/restart/clear path.

Validation: 11 focused gate tests; 334 Rust tests plus integrations; 199 Python tests; strict Clippy; release build; agent corpus; production Compose validators/render. Drone #320 passed in 59s without an image build.

The merged script is installed at /home/luke/inference/dspark_0731/snapshot_recovery_gate.py on node06 (SHA-256 dd60f82b5be20f4075f785a2ebdb6a3fb9ed96a24fc533214b2332e269ba6d9a). Its post-merge audit again returned the expected companion_source_not_ready: both sources have one stable connection but zero watermark/indexed blocks. All LB/engine/companion IDs, start timestamps, and restart counts were unchanged; /health remained 2/2. No Compose mutation occurred.

The next accepted action remains the same, but is now one command after the next independently justified fresh engine generation: audit must return ready, then run --apply. Do not restart or clear an engine solely to manufacture this result.

Fresh-generation evidence from the issue #12 rollback narrows the remaining blocker. Engine B was restored to canonical r34 K5/standard, authenticated readiness passed, and a new mode-0600 metadata capture plus atomic mode-0440 attestation were accepted by the deployed 123dd9d companion.

The companion reached the fresh publisher, briefly reported source-ready transitions, but its one bounded replay failed closed at the compact-index apply boundary. It published zero blocks and then remained stable observe-only: attempts/connections/replay-invalid counters were unchanged across repeated samples. The read-only recovery gate returned companion_source_not_ready in 2.66s and performed no Compose mutation. A remains fenced on its old generation.

Both engines are healthy 2/2 with zero restarts after rollback; snapshot/exact routing remain off and ordinary approximate serving is unaffected. This is no longer waiting for a natural B generation—the next task is to reproduce and diagnose B replay apply incompatibility from bounded synthetic/captured shape evidence before another engine candidate. Details are merged in #123 / EXPERIMENTS.md.

Issue #41 recovery acceptance update (node06, 2026-08-13):

Preserve compact replay across orphaned r34 partial blocks #124 fixed compact replay parity for r34's unsupported partial/orphan MLA stores; Pin the qualified r97 snapshot companion #125 pinned companion-rust-979ff7b@sha256:5cc99e…476f2.
Independently justified canonical A/B rolls supplied fresh attestations. Both companions are authoritative and stable at 36/173 blocks (9,216/44,288 logical tokens), with zero invalid replay events and zero container restarts.
Report selected exact inventory health #127 made /health report the exact backend actually selected by routing and separated snapshot publication time from ordinary upstream probe readiness. Pin qualified r99 load balancer #128 pinned the qualified LB as rust-81f648c@sha256:531f34…526be.
The first five-cycle gate attempt failed safely because the gate expected attempts_active=0; a healthy long-lived reconnect attempt owns its consumer until disconnect and correctly reports one active attempt plus one active connection. It rolled back in 2.03s with both engines unchanged. Fix snapshot recovery active attempt gate #129 corrected that fail-closed invariant and added zero/multiple-owner regression coverage.
The corrected production gate passed 5/5 LB-only recreates. Publication samples were 0.862s, 0.884s, 1.117s, 1.138s, and 0.880s; nearest-rank p95 was 1.138s against the 3.0s SLO. Every iteration recovered the same 36/173 blocks and 9,216/44,288 tokens. Mandatory rollback passed in 2.025s.
Both r34 engines retained their original container/image/start/restart identities (restart count zero). Final ordinary serving is 2/2 healthy on the immutable baseline, the deployment lock is free, and snapshot placement is off.

This completes the LB-only sub-3s recovery acceptance item. Remaining before snapshot state can influence placement: run at least 100,000 revision-stable exact-vs-approximate shadow comparisons, analyze disagreement/overlap/load boundaries, and separately exercise companion restart/bootstrap fail-closed behavior.

Post-recovery qualification update (PR #130): the authenticated 52/64-app snapshot-shadow capacity boundary passed with exact zero-spread counter reconciliation, both compact inventories trusted throughout, no engine/LB restart, and ordinary serving 2/2 afterward.

52 apps / 26.86MiB: 104/104, reuse hit 99.83%, return TTFT p95 1.493s.
64 apps / 33.06MiB: 128/128, reuse hit 43.68%; 36/64 returns cold, cold TTFT p95 51.240s.

The remaining 100k comparison gate cannot honestly use route_replay.py: its privacy contract omits token IDs and exact inventory revisions. The roadmap now specifies an off-by-default bounded in-process soak that captures real attested token vectors plus actual approximate decisions from these source cells, then runs 100k marker-before/lookup/marker-after compact comparisons without upstream inference. It will report unique source count separately, cap bytes/attempts, require zero trust/lookup/attestation failures and unchanged decisions, and keep soak counters separate from organic metrics.

This is a revision/concurrency qualification, not a claim of 100k independent workloads. Snapshot placement remains disabled.

r106 shadow-soak admission and recovery tooling is merged in PR #131 (main merge 4d18b0e).

Only the two proxy-authenticated pre-dispatch conditions, tokenizer unavailable and attestation changed, are retryable. Generic, unhealthy, and upstream 503s fail immediately; each logical request has one bounded deadline, and a pass requires client retry reasons to equal the proxy source-attempt counters.
The detached gate binds immutable image, Compose project/config/file identities, protected environment and helper digests; proves private frozen baseline/candidate renders before mutation; owns the common lock; preserves engine/companion process identities; and forces a frozen second rollback attempt if canonical recovery fails.
Local qualification passed 348 Rust tests plus integrations, 230 Python tests, strict Clippy, agent corpus, both production Compose profiles, and release/image builds. Two blocker-only recovery reviews found no remaining code-level rollout issue.
Drone PR build #338 passed in 62s. Main build #339 passed in 185s and published ghcr.io/helixml/mini-dynamo:rust-4d18b0e at digest sha256:e3b16a547b0d6e51159cb025cdead41fa036a8675164923e9b01211b4edf8794.

The live qualification remains open. node06 disappeared from Tailscale during the rejected r105 attempt and currently reports offline, last seen 2026-08-14 00:40 UTC. Before any new mutation, the gate still requires an r105 rollback/boot/OOM audit and unchanged engine identities. Then one fresh detached 104-source/100,000-comparison journal must pass with mandatory baseline restoration. Exact placement remains off.

Navigation Menu

Repository navigation

Add authoritative KV snapshot companion for sub-3s LB recovery #41

Problem

Proposed first slice

Acceptance gates

Upstream references

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add authoritative KV snapshot companion for sub-3s LB recovery #41

Description

Problem

Proposed first slice

Acceptance gates

Upstream references

Activity

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 13, 2026

nessie993 commented on Aug 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions