Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 25 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,11 @@ bench/build_transfer.sh rust-rNN-description-$(git rev-parse --short HEAD) --nod

The cold first build still downloads the Rust base and compiles dependencies;
judge the workflow by its warm edit/rebuild time. A dependency change is a
legitimate one-time cold build. GitHub Actions uses `Swatinem/rust-cache` for
dependency artifacts (including failed runs), and Drone fans Rust, Go, and the
GPU-free protocol suite out in parallel. If a no-dependency CI change still
does a cold compile, inspect the cache action before accepting the delay.
legitimate one-time cold build. Drone fetches dependencies once, then fans
Clippy and Rust tests into isolated target directories while Go, Python, and
Compose run beside them. Build #205 completed the full PR gate in 58s versus
69s for the prior serial Rust lane. If a no-dependency change regresses that
loop, inspect the step timings and cache/export path before accepting it.

`Cargo.toml` deliberately limits the crate package to Rust sources, examples,
compatibility fixtures, and Cargo manifests. This keeps edits under `bench/`
Expand Down Expand Up @@ -211,6 +212,19 @@ fenced observe-only mode. Do not reconnect on each subsequent event: only an
authoritative all-blocks clear or an attested incarnation change may restore
trust without a complete replay from zero.

`mini-dynamo-snapshot-companion` is the standalone one-engine composition. It
must remain a separate binary and `Dockerfile.companion`; build the ordinary LB
with `--bin mini-dynamo` so companion changes do not add a second release relink
to the serving-image loop. Serve mode must preflight the session secret, digest
secret, and authenticated incarnation file before any TCP bind, ZMQ connect, or
UDS publication. Invalid attestation refreshes immediately remove authority;
logs and metrics expose only closed reason labels. The container publishes no
metrics port: loopback inside a bridge container is not host-scrapable. Add a
separate, permission-isolated metrics UDS and Caddy/Prometheus wiring before a
node06 rollout; never put Caddy in the snapshot/session authority group. The
executable is not ready for node06 until the host-side attestation provisioner
and production Compose wiring exist and pass the dual-domain validator.

`snapshot_reconnect` is the LB-side owner around the consumer. Normal attempts
are serial; only an explicit bounded replacement may overlap a second session.
Validate the trusted socket parent on every connect, use a fresh OS-random
Expand Down Expand Up @@ -314,13 +328,13 @@ export BENCH_TOKEN=$(grep -o 'Bearer [A-Za-z0-9_-]*' /etc/caddy/Caddyfile | head
- Drone intentionally runs once per PR (and once after merge on `main`), not
again for every feature-branch push. Its Rust lane omits a redundant release
build: the local pre-push gate builds release, and Drone's post-merge main
pipeline builds and publishes the release container to GHCR (the Actions
GITHUB_TOKEN is denied write on that package — issue #39; the `publish-image`
step reads `ghcr_username`/`ghcr_token` Drone secrets instead). GitHub
Actions keeps format, Clippy, test, and deploy-config validation only. The
Drone image build runs in a fresh dind daemon, so it pays a cold cargo
release build on every publish; node06 deploys should keep using the warm
local `bench/build_transfer.sh` path.
pipeline builds and publishes the release container to GHCR. The
`publish-image` step reads `ghcr_username`/`ghcr_token` only on a main push;
pull-request containers never receive those secrets. GitHub Actions is not a
second CI system for this repository. Drone's PR quality budget is currently
about 58s; inspect and record regressions instead of adding duplicate gates.
Node06 deploys should keep using the warm local `bench/build_transfer.sh`
path because it reuses the development BuildKit cache.
- Build the LB locally and stream it to node06 when the local amd64 Docker
cache is warm. A typical warm LB transfer is seconds and avoids consuming
node06's scarce 8-9GiB available host memory:
Expand Down
3 changes: 2 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ WORKDIR /src
COPY . .
RUN --mount=type=cache,id=mini-dynamo-cargo-registry,target=/usr/local/cargo/registry \
--mount=type=cache,id=mini-dynamo-target,target=/src/target \
cargo build --release --locked && cp target/release/mini-dynamo /mini-dynamo
cargo build --release --locked --bin mini-dynamo \
&& cp target/release/mini-dynamo /mini-dynamo

FROM gcr.io/distroless/cc-debian12
LABEL org.opencontainers.image.source="https://github.com/helixml/mini-dynamo"
Expand Down
13 changes: 13 additions & 0 deletions Dockerfile.companion
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# syntax=docker/dockerfile:1.7
FROM rust:1.95-bookworm AS build
WORKDIR /src
COPY . .
RUN --mount=type=cache,id=mini-dynamo-cargo-registry,target=/usr/local/cargo/registry \
--mount=type=cache,id=mini-dynamo-target,target=/src/target \
cargo build --release --locked --bin mini-dynamo-snapshot-companion \
&& cp target/release/mini-dynamo-snapshot-companion /mini-dynamo-snapshot-companion

FROM gcr.io/distroless/cc-debian12
LABEL org.opencontainers.image.source="https://github.com/helixml/mini-dynamo"
COPY --from=build /mini-dynamo-snapshot-companion /mini-dynamo-snapshot-companion
ENTRYPOINT ["/mini-dynamo-snapshot-companion"]
30 changes: 30 additions & 0 deletions EXPERIMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3606,3 +3606,33 @@ were identical before and after; mini-dynamo remained 2/2 healthy with both
non-writable under `/tmp` on node06, but no tool container, CUDA context, LB
recreate, or benchmark ran. The next gate remains the separately acknowledged
1MiB/two-GPU scout during a low-traffic window.

## 2026-08-13 — r46 standalone single-engine snapshot companion

The previously separate owner, source, producer, supervisor, and metrics pieces
now compose into the off-by-default `mini-dynamo-snapshot-companion` executable.
Serve mode accepts exactly one live/replay endpoint pair and one group geometry.
Before binding metrics, connecting ZMQ, or publishing its Unix socket, it
hardens and loads three distinct protected files: the session secret, digest
secret, and an HMAC-authenticated engine-incarnation envelope. The refresh
watch removes authority immediately on any filesystem, schema, field, or MAC
failure and restores it only after a valid changed envelope. Owner telemetry is
closed-label and content-free, including the stable `replay_too_large`
observe-only state required by long-lived node06 engine A.

One absolute shutdown deadline covers the owner, snapshot runtime, metrics
server, and attestation watcher; tasks that miss it are aborted. The ordinary
LB Dockerfile now names only `--bin mini-dynamo`, while the companion has a
dedicated `Dockerfile.companion`. With one shared BuildKit target cache, the
measured source builds were 42.3s for the LB followed by 9.7s for the companion;
no-op warm builds were 2.16s and 3.38s. Final images were 14.27MB and 11.50MB.
This preserves the fast LB-only loop instead of paying the observed extra
roughly 41-second companion relink for every router image.

The rebased gate passed 298 library tests, 31 integration/E2E tests, strict
Clippy, formatting, release builds, off-mode executable/container smoke, and
the dual-domain offline Compose validator. No production Compose, image push,
or node06 state changed. The executable is intentionally not deployment-ready
yet: a bounded host provisioner must derive and atomically write the authenticated
incarnation from current engine metadata before either per-engine service can be
enabled.
19 changes: 12 additions & 7 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -603,10 +603,13 @@ node06). The design rationale for each lives in DESIGN.md.
two isolated processes and authority domains: distinct companions, UIDs,
tmpfs directories, sockets, secrets, clients, and readiness checks per engine.
Its default profile renders zero services, and static fault projection proves
one failed pair cannot publish or substitute through its healthy peer. Next
wire one authenticated owner/socket per engine into an executable and run at
least 100,000 revision-stable shadow comparisons before placement can consume
it.
one failed pair cannot publish or substitute through its healthy peer. A
dedicated off-by-default executable now composes exactly one authenticated
incarnation watch, ZMQ owner, long-lived source, snapshot socket, and bounded
metrics endpoint. It is built separately from the LB so ordinary router edits
do not relink both binaries. Next add the host-side authenticated-attestation
provisioner and production-shaped Compose wiring, then run at least 100,000
revision-stable shadow comparisons before placement can consume it.

The off-by-default library runtime now composes typed config, hardened secret
loading, bind-last safe socket publication, the bounded supervisor, producer,
Expand All @@ -618,9 +621,11 @@ node06). The design rationale for each lives in DESIGN.md.
The coordinator delegates one absolute
snapshot-phase timeout and a resettable tail-idle/write timeout to the
producer; the supervisor retains the two-client cap and immediate shutdown
cancellation without imposing a total lifetime on healthy progress. Next
inject the completed process owner and its authenticated engine-incarnation
watch into each isolated executable instance.
cancellation without imposing a total lifetime on healthy progress. The
standalone process now injects the completed owner and a refreshable,
HMAC-authenticated engine-incarnation watch; any invalid refresh fences
authority immediately. Deployment remains blocked on generating that manifest
from current engine metadata without exposing either digest key or identity.

A true offline public-stack harness now proves initial publication, live
store/remove, rolling handoff, LB owner restart, companion shutdown/socket
Expand Down
91 changes: 91 additions & 0 deletions src/bin/mini-dynamo-snapshot-companion.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
use std::{ffi::OsString, path::PathBuf};

use anyhow::{Context, bail};
use mini_dynamo::companion_service::{SingleEngineCompanionConfig, run_single_engine_companion};
use mini_dynamo::snapshot_socket_path::validate_published_socket;
use tokio::sync::watch;
use tracing_subscriber::{EnvFilter, layer::SubscriberExt, util::SubscriberInitExt};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
match command(std::env::args_os().skip(1))? {
Command::Run => run().await,
Command::Healthcheck(path) => {
validate_published_socket(&path).context("snapshot companion healthcheck failed")?;
Ok(())
}
}
}

enum Command {
Run,
Healthcheck(PathBuf),
}

fn command(mut arguments: impl Iterator<Item = OsString>) -> anyhow::Result<Command> {
let Some(first) = arguments.next() else {
return Ok(Command::Run);
};
if first != "healthcheck" {
bail!("invalid snapshot companion command");
}
let Some(path) = arguments.next() else {
bail!("snapshot companion healthcheck requires one socket path");
};
if arguments.next().is_some() {
bail!("snapshot companion healthcheck accepts exactly one socket path");
}
Ok(Command::Healthcheck(PathBuf::from(path)))
}

async fn run() -> anyhow::Result<()> {
tracing_subscriber::registry()
.with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
.with(tracing_subscriber::fmt::layer().json())
.init();

let config = SingleEngineCompanionConfig::from_env()
.context("invalid standalone snapshot companion configuration")?;
tracing::info!(enabled = config.enabled(), "snapshot companion starting");
let (shutdown_tx, shutdown_rx) = watch::channel(false);
let service = run_single_engine_companion(config, shutdown_rx);
tokio::pin!(service);
let result = tokio::select! {
biased;
result = &mut service => result,
() = shutdown_signal() => {
let _ = shutdown_tx.send(true);
service.await
}
};
match result {
Ok(report) => {
tracing::info!(report = ?report, "snapshot companion stopped");
Ok(())
}
Err(error) => {
tracing::error!(reason = error.reason(), "snapshot companion failed");
Err(anyhow::anyhow!(
"snapshot companion failed: {}",
error.reason()
))
}
}
}

async fn shutdown_signal() {
#[cfg(unix)]
{
let mut terminate =
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
.expect("install SIGTERM handler");
tokio::select! {
result = tokio::signal::ctrl_c() => { let _ = result; },
_ = terminate.recv() => {},
}
}
#[cfg(not(unix))]
{
let _ = tokio::signal::ctrl_c().await;
}
}
Loading