Skip to content

Commit 96b1d67

Browse files
ericksoagithub-actions[bot]cv
authored
feat(runtime): implement durable Docker state mutation (#8658)
<!-- markdownlint-disable MD041 --> ## Summary <!-- 1-3 plain sentences: what changes and why. Describe before-and-after behavior when it applies. Follow the NemoClaw Writing Guide: https://github.com/NVIDIA/NemoClaw/blob/main/WRITING.md. Do not add unrelated prose cleanup. --> Implements the B4-C2 slice of #7744 by replacing Docker's unsupported `stateMutation` surface with a durable, provider-fenced implementation. Hermes Shields is the named managed-image consumer, with fencing retained across publication, validation, rollback, activation, and controller restart; Podman remains unregistered. ## Related Issue <!-- Fixes #NNN or Closes #NNN. Remove this section if none. --> Refs #7744 ## Changes <!-- List concrete changes. If this adds an abstraction, configuration, fallback, migration, or compatibility path, name its current requirement and consumer, explain why a direct change is insufficient, and identify the test that protects it. --> - Extends the runtime-provider `stateMutation` contract to plan schema v2 and registers a real Docker implementation. Hermes Shields requires a canonical AgentDefinition protection projection, exact serialized plan digest, phase-bearing fence, rollback posture, activation proof, and explicit release. A direct filesystem mutation cannot prove that it applies to the current sandbox lifecycle, runtime, mount namespace, state root, or configuration projection. This behavior is covered by `state-mutation.test.ts`, `runtime-provider-contract.test.ts`, and `docker-state-mutation.test.ts`. - Adds durable persisted-engine lifecycle claims, exact request/receipt validation, Docker operation authority, and exclusion of ordinary privileged container execution for the full SSH or OpenShell/fallback operation lifetime. Process-local locking or a check immediately before spawning is insufficient because provider publication, subprocess execution, fallback, cleanup, and controller recovery can cross process boundaries. This behavior is covered by `persisted-engine-lifecycle.test.ts`, `docker-operation-authority.test.ts`, `command-transport.test.ts`, `privileged-exec.test.ts`, and `privileged-exec-exclusion.test.ts`. - Packages fixed, root-owned mutation control, startup-gate, and Hermes publisher helpers plus an exact capability descriptor in the managed image. The image-side controller validates the durable request, publishes or rolls back the protection posture, records activation evidence, and gates gateway startup and recovery. Host-only mutation is insufficient because publication and restart recovery must be enforced inside the image that owns the state mount. This behavior is covered by `runtime-state-mutation-control.test.ts`, `runtime-state-mutation-startup-gate.test.ts`, `runtime-state-mutation-hermes-publisher.test.ts`, `hermes-final-image-layout.test.ts`, and `hermes-gateway-supervisor-recovery.test.ts`. - Wires Hermes Shields as the named production consumer, including retained-fence recovery, rollback publication, activation proof handling, timer process identity, and transition-lock coordination. Only a current managed Hermes Docker image with the exact root-owned capability uses the provider protocol; images without it retain the existing compatibility path. A global switch is insufficient while older images remain valid deployments. This behavior is covered by `hermes-runtime-state-mutation.test.ts`, `flow.test.ts`, `legacy-hermes-compat.test.ts`, `openclaw-transition.test.ts`, `policy-transition.test.ts`, `timer-bound-lock.test.ts`, and `timer-process.test.ts`. - Hardens Docker operation authority by binding the qualified executable, interpreter chain, credential and SSH helpers, endpoint, PATH semantics, and privileged-execution lease through every capture and spawn. The installer now preserves already-present user-local OpenShell and npm PATH entries so persisted authority survives status and cleanup without weakening real helper or endpoint drift checks. - Restricts qualification-only Docker loopback publication to a freshly queried live Docker Engine >= 28.3.3 authority. Registry and model-server publications each consume a separate single-use authority; ordinary managed llama.cpp onboarding retains the private bridge, disables contradictory inherited image healthchecks, and performs no Docker port publication. - Documents provider selection, durable fencing and phases, restart hold and recovery, rollback, ledger state, host files, troubleshooting, and the runtime trust boundary. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent Codex Desktop code and security reviews found no actionable findings in the B4-C2 change set; final stabilization reviews accepted and fixed timer identity caching plus guardian stop polling, dismissed two false-positive seam/import findings with evidence, and verified analyzer-only Python cleanups preserve fail-closed behavior. The post-review code and security reviews found no actionable findings after all five valid final CodeRabbit findings were fixed; focused suites plus npm run validate:pr passed at exact head 2cd934e. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review <!-- Required for code and documentation changes after the changes and applicable validation are complete. Keep one review checkbox and one instance of each visible or hidden field. For Evidence, list changed documentation paths. For documentation-only changes, also state that the writing rules and documentation style were reviewed. For other results, explain why no documentation change is needed or why the review is blocked. For Agent, use a consistent product and surface name, such as Codex Desktop, Codex CLI, Claude Code, or Cursor. After committing all review changes, put `git rev-parse --short HEAD` and `git rev-parse --short HEAD:AGENTS.md` in the hidden metadata below. Rerun the review and refresh that metadata after any new commit. This receipt is advisory during the data-collection pilot. --> - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: docs/inference/set-up-llama-cpp.mdx; docs/manage-sandboxes/runtime-controls.mdx; docs/reference/commands.mdx; docs/reference/host-files-and-state.mdx; docs/reference/troubleshooting.mdx; docs/security/tcb-boundary.mdx - Agent: Codex Desktop <!-- docs-review-head-sha: 2cd934e --> <!-- docs-review-agents-blob-sha: c4923a3 --> ## DGX Station Hardware Evidence <!-- Required only when scripts/prepare-dgx-station-host.sh changes. Maintainers must review the linked evidence before approving or merging. This is human-reviewed evidence, not authenticated hardware provenance. Exceptional bypasses use existing repository governance and must be documented on the PR. --> - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run validate:pr` passed after refreshing `origin/main` when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: - Latest-base installer, qualification, runtime-provider, Hermes, and lifecycle matrix: 8 files, 231 passed. Earlier full affected matrix: 20 files, 391 passed, 1 intentionally skipped. - Docker authority, health, and publication regressions: 80 of 80 passed in the final focused rerun; forged/reused publication tokens, npm PATH idempotence, inherited healthcheck suppression, and create-boundary propagation are covered. - Independent code, security, and documentation reviews passed at exact final head 2cd934e. Security validation covered 279 passed tests with one expected skip; the compiler probe confirms the sync-or-Promise recovery signature matches runtime behavior. - Post-integration validation passed: 526/526 changed CLI tests, 247/247 integration tests, and 156/156 protected composition tests. Exact final head 2cd934e passed 279 focused tests with one expected skip, npm run typecheck:cli, npm run typecheck, and npm run validate:pr. The local installer/package-contract rerun passed 96/98; its two installer-preflight failures reproduce on untouched base under the ambient Homebrew trust state and are outside this diff. - `npm run validate:pr`, timer tests (22/22), mutation controller/startup-gate/publisher tests (16/16), and protected-readiness support tests (49/49) passed through exact head 2cd934e against runtime base e677fcb; pre-push hooks also passed. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: Exact-head canonical CI run 31421039363 and managed-image all-agent activation run 31421040652 passed at prior head d4271fa. Current exact-head canonical CI run 31447285073, managed-image run 31447285090, and trusted protected E2E run 31447308972 are running at candidate 2cd934e against runtime base e677fcb and trusted workflow 2afba61. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] npm run docs builds without warnings (doc changes only) - `npm run docs` completed with 0 errors. Fern reported two non-PR warnings: the unauthenticated redirects check was skipped, and the existing light-mode accent contrast is 2.41:1. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Aaron Erickson <aerickson@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added managed Hermes runtime-state mutation support with protected transitions, rollback, recovery, and startup gating. * Added durable lifecycle tracking and fail-closed validation for runtime changes. * Added coordinated execution leases for privileged sandbox and transport operations. * Added Docker authority checks and secure loopback publishing for supported inference workflows. * Added safer Docker launch configuration for host-local inference. * **Bug Fixes** * Improved timer authorization, process identity validation, recovery handling, and installer PATH preservation. * **Documentation** * Added guidance for runtime controls, recovery, lifecycle state, troubleshooting, and inference setup. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Aaron Erickson <aerickson@nvidia.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
1 parent 9257fe0 commit 96b1d67

94 files changed

Lines changed: 22319 additions & 1279 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

agents/hermes/Dockerfile

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,11 @@ COPY agents/hermes/mcp-config-transaction.py /usr/local/lib/nemoclaw/hermes-mcp-
120120
COPY agents/hermes/cron-restore-control.py /usr/local/lib/nemoclaw/hermes-cron-restore-control.py
121121
COPY src/lib/actions/sandbox/openshell-child-visible-credentials.v0.0.101.json /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.101.json
122122
COPY scripts/state-dir-guard.py /usr/local/lib/nemoclaw/state-dir-guard.py
123+
COPY scripts/runtime-state-mutation-control.py /usr/local/lib/nemoclaw/runtime-state-mutation-control.py
124+
COPY scripts/runtime-state-mutation-startup-gate.py /usr/local/lib/nemoclaw/runtime-state-mutation-startup-gate.py
125+
COPY scripts/runtime_state_mutation_hermes_publisher.py /usr/local/lib/nemoclaw/runtime_state_mutation_hermes_publisher.py
123126
COPY agents/hermes/state-lock-plan.json /usr/local/share/nemoclaw/state-lock-plan.json
127+
COPY agents/hermes/runtime-state-mutation-publisher-v1.json /usr/local/share/nemoclaw/runtime-state-mutation-publisher-v1.json
124128
COPY nemoclaw-blueprint/scripts/*.js /usr/local/lib/nemoclaw/preloads/
125129

126130
FROM scratch AS hermes-wrapper-payload
@@ -400,7 +404,10 @@ RUN managed_runtime_assertion_failed() { \
400404
&& test -f /usr/local/lib/nemoclaw/managed-bootstrap-trampoline.sh \
401405
&& test ! -L /usr/local/lib/nemoclaw/managed-bootstrap-trampoline.sh \
402406
&& test "$(stat -c '%u:%g:%a' /usr/local/lib/nemoclaw/managed-bootstrap-trampoline.sh)" = '0:0:444' \
403-
&& install -d -o root -g root -m 0755 /run/nemoclaw
407+
&& install -d -o root -g root -m 0755 /run/nemoclaw /var/lib/nemoclaw \
408+
&& install -d -o root -g root -m 0711 \
409+
/run/nemoclaw/runtime-state-mutation-startup \
410+
/var/lib/nemoclaw/runtime-state-mutation
404411

405412
# Ensure sandbox user can read blueprint files copied as root
406413
RUN chmod -R a+rX /opt/nemoclaw-blueprint/
@@ -415,12 +422,14 @@ RUN chmod -R a+rX /opt/nemoclaw-blueprint/
415422
# minimum supported Hermes sandbox base tag guarantees those artifacts and
416423
# test/sandbox-rlimit-hooks.test.ts covers that base.
417424
RUN chmod 755 /usr/local/bin/nemoclaw-start /usr/local/bin/nemoclaw-managed-startup-hold /usr/local/bin/nemoclaw-managed-bootstrap /usr/local/lib/nemoclaw/sandbox-init.sh /usr/local/lib/nemoclaw/validate-hermes-env-secret-boundary.py /usr/local/lib/nemoclaw/patch-hermes-session-list-preview.py /usr/local/lib/nemoclaw/patch-hermes-sqlite-temp-store.py /usr/local/lib/nemoclaw/patch-hermes-discord-recovery-permissions.py /usr/local/lib/nemoclaw/patch-hermes-profile-policy-defaults.py /usr/local/lib/nemoclaw/seed-hermes-dashboard-config.py /usr/local/lib/nemoclaw/hermes-runtime-config-guard.py /usr/local/lib/nemoclaw/finalize-tirith-marker.py /usr/local/lib/nemoclaw/hermes-mcp-config-transaction.py \
418-
&& chown root:root /usr/local/bin/nemoclaw-gateway-control /usr/local/lib/nemoclaw/gateway-supervisor.sh /usr/local/lib/nemoclaw/state-dir-guard.py /usr/local/share/nemoclaw/state-lock-plan.json /usr/local/lib/nemoclaw/managed-gateway-control.py /usr/local/lib/nemoclaw/build-hermes-mcp-digest.py /usr/local/lib/nemoclaw/hermes-cron-restore-control.py /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.101.json \
425+
&& chown root:root /usr/local/bin/nemoclaw-gateway-control /usr/local/lib/nemoclaw/gateway-supervisor.sh /usr/local/lib/nemoclaw/state-dir-guard.py /usr/local/lib/nemoclaw/runtime-state-mutation-control.py /usr/local/lib/nemoclaw/runtime-state-mutation-startup-gate.py /usr/local/lib/nemoclaw/runtime_state_mutation_hermes_publisher.py /usr/local/share/nemoclaw/state-lock-plan.json /usr/local/share/nemoclaw/runtime-state-mutation-publisher-v1.json /usr/local/lib/nemoclaw/managed-gateway-control.py /usr/local/lib/nemoclaw/build-hermes-mcp-digest.py /usr/local/lib/nemoclaw/hermes-cron-restore-control.py /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.101.json \
419426
&& chmod 700 /usr/local/bin/nemoclaw-gateway-control /usr/local/lib/nemoclaw/hermes-cron-restore-control.py \
420-
&& chmod 500 /usr/local/lib/nemoclaw/state-dir-guard.py /usr/local/lib/nemoclaw/managed-gateway-control.py \
421-
&& chmod 444 /usr/local/lib/nemoclaw/entrypoint-env-wrapper.sh /usr/local/share/nemoclaw/state-lock-plan.json /usr/local/lib/nemoclaw/gateway-supervisor.sh /usr/local/lib/nemoclaw/build-hermes-mcp-digest.py /usr/local/lib/nemoclaw/managed_policy.py \
427+
&& chmod 500 /usr/local/lib/nemoclaw/state-dir-guard.py /usr/local/lib/nemoclaw/runtime-state-mutation-control.py /usr/local/lib/nemoclaw/runtime_state_mutation_hermes_publisher.py /usr/local/lib/nemoclaw/managed-gateway-control.py \
428+
&& chmod 555 /usr/local/lib/nemoclaw/runtime-state-mutation-startup-gate.py \
429+
&& chmod 444 /usr/local/lib/nemoclaw/entrypoint-env-wrapper.sh /usr/local/share/nemoclaw/state-lock-plan.json /usr/local/share/nemoclaw/runtime-state-mutation-publisher-v1.json /usr/local/lib/nemoclaw/gateway-supervisor.sh /usr/local/lib/nemoclaw/build-hermes-mcp-digest.py /usr/local/lib/nemoclaw/managed_policy.py \
422430
&& chmod 444 /usr/local/lib/nemoclaw/patch-hermes-langfuse-credentials.mts \
423431
&& chmod 444 /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.101.json \
432+
&& /opt/hermes/.venv/bin/python3 -I -c 'import runpy, yaml; assert yaml.safe_load("ready: true")["ready"] is True; runpy.run_path("/usr/local/lib/nemoclaw/runtime-state-mutation-control.py", run_name="nemoclaw_runtime_state_mutation_control_probe"); runpy.run_path("/usr/local/lib/nemoclaw/runtime_state_mutation_hermes_publisher.py", run_name="nemoclaw_runtime_state_mutation_publisher_probe"); runpy.run_path("/usr/local/lib/nemoclaw/runtime-state-mutation-startup-gate.py", run_name="nemoclaw_runtime_state_mutation_gate_probe"); runpy.run_path("/usr/local/lib/nemoclaw/hermes-runtime-config-guard.py", run_name="nemoclaw_runtime_config_guard_probe")' \
424433
&& if [ -d /usr/local/lib/nemoclaw/preloads ]; then \
425434
chown -R 0:0 /usr/local/lib/nemoclaw/preloads \
426435
&& find /usr/local/lib/nemoclaw/preloads -type f -exec chmod 444 {} + \
@@ -1243,6 +1252,12 @@ RUN check_metadata() { \
12431252
&& check_metadata /usr/local/lib/nemoclaw/managed-bootstrap-trampoline.sh 'root:root 444' \
12441253
&& check_metadata /usr/local/bin/nemoclaw-gateway-control 'root:root 700' \
12451254
&& check_metadata /usr/local/share/nemoclaw/state-lock-plan.json 'root:root 444' \
1255+
&& check_metadata /usr/local/lib/nemoclaw/runtime-state-mutation-control.py 'root:root 500' \
1256+
&& check_metadata /usr/local/lib/nemoclaw/runtime-state-mutation-startup-gate.py 'root:root 555' \
1257+
&& check_metadata /usr/local/lib/nemoclaw/runtime_state_mutation_hermes_publisher.py 'root:root 500' \
1258+
&& check_metadata /var/lib/nemoclaw/runtime-state-mutation 'root:root 711' \
1259+
&& check_metadata /run/nemoclaw/runtime-state-mutation-startup 'root:root 711' \
1260+
&& check_metadata /usr/local/share/nemoclaw/runtime-state-mutation-publisher-v1.json 'root:root 444' \
12461261
&& check_metadata /usr/local/lib/nemoclaw/hermes-cron-restore-control.py 'root:root 700' \
12471262
&& check_metadata /usr/local/lib/nemoclaw/preloads/sandbox-safety-net.js 'root:root 444' \
12481263
&& check_metadata /usr/local/lib/nemoclaw/hermes-wrapper.py 'root:root 755' \
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"schemaVersion":1,"protocol":"nemoclaw-runtime-state-mutation-publisher-v1","agent":"hermes","providerId":"docker","stateRoot":"/sandbox/.hermes","planSchemaVersion":2,"entrypoint":"/usr/local/lib/nemoclaw/runtime_state_mutation_hermes_publisher.py"}

agents/hermes/start.sh

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,105 @@ set -euo pipefail
1919

2020
# SECURITY: Lock down PATH before resolving or sourcing root startup helpers.
2121
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
22+
NEMOCLAW_RUNTIME_STATE_MUTATION_RETRY_ARGV=("$@")
23+
24+
# The provider gate is a fixed root-owned file beneath a search-only directory
25+
# outside /sandbox, so the sandbox identity cannot rename either the gate or
26+
# its parent. Run the immutable validator as this shell's direct child before
27+
# sourcing helpers or reading mutable state; its permit is bound to this exact
28+
# process identity. Invalid or uninspectable gate state is a hold, never an
29+
# availability-to-integrity downgrade.
30+
readonly NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_PYTHON="/opt/hermes/.venv/bin/python3"
31+
readonly NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_HELPER="/usr/local/lib/nemoclaw/runtime-state-mutation-startup-gate.py"
32+
readonly NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_SETPRIV="/usr/bin/setpriv"
33+
34+
if [ ! -x "$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_PYTHON" ] \
35+
|| [ ! -f "$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_HELPER" ] \
36+
|| [ -L "$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_HELPER" ] \
37+
|| { [ "$EUID" -eq 0 ] && [ ! -x "$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_SETPRIV" ]; }; then
38+
printf '%s\n' '[SECURITY] Required runtime state mutation startup gate is unavailable.' >&2
39+
exit 1
40+
fi
41+
42+
nemoclaw_runtime_state_mutation_gate() {
43+
local action="$1"
44+
if [ "$EUID" -eq 0 ]; then
45+
"$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_SETPRIV" \
46+
--reuid=sandbox --regid=sandbox --init-groups -- \
47+
"$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_PYTHON" -I \
48+
"$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_HELPER" "$action" >/dev/null
49+
return
50+
fi
51+
"$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_PYTHON" -I \
52+
"$NEMOCLAW_RUNTIME_STATE_MUTATION_GATE_HELPER" "$action" >/dev/null
53+
}
54+
55+
nemoclaw_runtime_state_mutation_retry_exec() {
56+
local status
57+
if nemoclaw_runtime_state_mutation_gate restart; then
58+
status=0
59+
else
60+
status=$?
61+
fi
62+
if [ "$status" -eq 12 ]; then
63+
exec /usr/local/bin/nemoclaw-start \
64+
"${NEMOCLAW_RUNTIME_STATE_MUTATION_RETRY_ARGV[@]}"
65+
fi
66+
printf '%s\n' '[SECURITY] Runtime state mutation retry was not authenticated; holding startup.' >&2
67+
kill -STOP "$$"
68+
}
69+
trap nemoclaw_runtime_state_mutation_retry_exec USR2
70+
71+
while :; do
72+
if nemoclaw_runtime_state_mutation_gate admit; then
73+
break
74+
else
75+
_nemoclaw_runtime_state_mutation_gate_status=$?
76+
fi
77+
case "$_nemoclaw_runtime_state_mutation_gate_status" in
78+
10) break ;;
79+
75)
80+
printf '%s\n' '[SECURITY] Hermes startup held by an active runtime state mutation.' >&2
81+
/bin/sleep 1 || true
82+
;;
83+
*)
84+
printf '%s\n' '[SECURITY] Runtime state mutation startup gate failed.' >&2
85+
exit 1
86+
;;
87+
esac
88+
done
89+
unset _nemoclaw_runtime_state_mutation_gate_status
90+
91+
# Publish a candidate only after the complete gateway topology is healthy.
92+
# The shell then stops itself until the root controller has independently
93+
# authenticated the candidate, frozen the exact process tree, and published a
94+
# release receipt. Calling this with no active mutation is a cheap no-op.
95+
nemoclaw_runtime_state_mutation_checkpoint() {
96+
local status
97+
if nemoclaw_runtime_state_mutation_gate checkpoint; then
98+
return 0
99+
else
100+
status=$?
101+
fi
102+
if [ "$status" -ne 11 ]; then
103+
printf '%s\n' '[SECURITY] Runtime state mutation startup checkpoint was refused; holding startup.' >&2
104+
kill -STOP "$$"
105+
return 1
106+
fi
107+
kill -STOP "$$"
108+
if nemoclaw_runtime_state_mutation_gate resume; then
109+
return 0
110+
else
111+
status=$?
112+
fi
113+
if [ "$status" -eq 12 ]; then
114+
exec /usr/local/bin/nemoclaw-start \
115+
"${NEMOCLAW_RUNTIME_STATE_MUTATION_RETRY_ARGV[@]}"
116+
fi
117+
printf '%s\n' '[SECURITY] Runtime state mutation release receipt was not authenticated; holding startup.' >&2
118+
kill -STOP "$$"
119+
return 1
120+
}
22121

23122
# managed-entrypoint-env-wrapper begin
24123
_NEMOCLAW_ENTRYPOINT_ENV_WRAPPER="/usr/local/lib/nemoclaw/entrypoint-env-wrapper.sh"
@@ -127,6 +226,7 @@ exec > >(tee -a "$_START_LOG") 2> >(tee -a "$_START_LOG" >&2)
127226
drop_capabilities /usr/local/bin/nemoclaw-start "$@"
128227

129228
NEMOCLAW_CMD=("$@")
229+
NEMOCLAW_RUNTIME_STATE_MUTATION_RETRY_ARGV=("${NEMOCLAW_CMD[@]}")
130230

131231
_chat_ui_url_port() {
132232
[ -n "${CHAT_UI_URL:-}" ] || return 1
@@ -2908,6 +3008,11 @@ handle_hermes_gateway_control_request() {
29083008
return 1
29093009
fi
29103010
refresh_hermes_supervised_child_pids
3011+
nemoclaw_runtime_state_mutation_checkpoint || {
3012+
stop_hermes_gateway_fail_closed
3013+
gateway_control_fail internal "$old_pid"
3014+
return 1
3015+
}
29113016
gateway_control_complete ok "$old_pid" "$GATEWAY_PID"
29123017
}
29133018

@@ -3136,6 +3241,7 @@ recover_hermes_gateway_current_user() {
31363241
return 1
31373242
fi
31383243
refresh_hermes_supervised_child_pids
3244+
nemoclaw_runtime_state_mutation_checkpoint || return 1
31393245
return 0
31403246
fi
31413247
echo "[gateway] Hermes auxiliary repair failed; retrying while the exact gateway remains healthy" >&2
@@ -3228,6 +3334,7 @@ bootstrap_hermes_gateway_current_user() {
32283334
return 1
32293335
fi
32303336
refresh_hermes_supervised_child_pids
3337+
nemoclaw_runtime_state_mutation_checkpoint || return 1
32313338
return 0
32323339
fi
32333340

@@ -3243,6 +3350,7 @@ bootstrap_hermes_gateway_current_user() {
32433350
sleep 2 || true
32443351
recover_hermes_gateway_current_user || return 1
32453352
refresh_hermes_supervised_child_pids
3353+
nemoclaw_runtime_state_mutation_checkpoint || return 1
32463354
}
32473355

32483356
# ── Main ─────────────────────────────────────────────────────────
@@ -3371,6 +3479,7 @@ if ! "$_HERMES_PYTHON" -I "$_HERMES_RUNTIME_CONFIG_GUARD" publish-startup-ready
33713479
echo "[gateway-control] failed to publish Hermes startup readiness" >&2
33723480
exit 1
33733481
fi
3482+
nemoclaw_runtime_state_mutation_checkpoint || exit 1
33743483
print_dashboard_urls
33753484

33763485
# PID 1 remains alive even when Hermes stops its gateway. Host recovery uses

docs/inference/set-up-llama-cpp.mdx

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,9 @@ The default recipe serves NVIDIA Nemotron 3 Nano 30B-A3B through the OpenAI Chat
8383
<Note>
8484
This path is an experimental implementation.
8585
It does not establish a supported agent, model, and runtime tuple until the protected qualification and activation gates pass.
86+
The protected qualification runner requires Docker Engine `28.3.3` or newer and the trusted daemon's default protected NAT and firewall behavior; it re-queries the live daemon and consumes a fresh single-use authority immediately before each temporary loopback publication.
87+
That requirement supersedes older Docker 27 qualification evidence.
88+
Ordinary managed onboarding does not publish a Docker port: it retains its no-publication container contract and uses the host-owned private bridge, so this runner-specific version floor does not apply to onboarding.
8689
</Note>
8790

8891
Before you start, confirm these prerequisites:
@@ -135,7 +138,7 @@ Onboarding performs these actions:
135138
- Reuses a verified GGUF from the shared `~/.cache/huggingface/` cache, or acquires the exact file through the existing Hugging Face mechanism when it is absent or invalid.
136139
- Verifies the GGUF size and SHA-256 digest before it starts the runtime.
137140
- Runs the authenticated container as the current non-root user with one NVIDIA GPU and no CPU fallback.
138-
- Starts a private host bridge on `127.0.0.1:8081` and connects the container to a Docker internal network without publishing a container port.
141+
- Starts a host-owned private bridge on `127.0.0.1:8081` and connects the container to a Docker internal network without a Docker-published port.
139142
- Registers `llama-cpp-local` and routes agent traffic through `https://inference.local/v1`.
140143

141144
The runtime cannot download a model and has no egress on its internal Docker network.
@@ -147,14 +150,17 @@ Selecting either runtime fails before model acquisition or runtime mutation and
147150

148151
## Verify the Managed Runtime
149152

150-
Run the managed runtime and route checks:
153+
Run the managed runtime, route, and host-port checks:
151154

152155
```bash
153156
$$nemoclaw my-assistant status
154157
$$nemoclaw my-assistant doctor
158+
docker inspect --format '{{json .HostConfig.PortBindings}}' nemoclaw-llama-cpp
155159
```
156160

157161
Accept the result when `status` reports `Managed llama.cpp: running`, the inference route is `reachable`, and `doctor` exits with status `0`.
162+
The Docker inspection output must be `{}`, confirming that the container has no Docker-published port.
163+
The host-owned private bridge still listens on loopback.
158164
These checks do not establish agent and model qualification.
159165

160166
`status` reports the recipe ID, model digest, image reference, endpoint, and lifecycle state without exposing the API key.

docs/manage-sandboxes/runtime-controls.mdx

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,33 @@ NemoClaw binds the write to the SHA-256 digest of the matching read, temporarily
9393
`shields up` also publishes fresh config, environment, and compatibility-hash inodes so a descriptor opened before lockdown cannot retain write authority.
9494
Shields up keeps `/sandbox/.hermes/profiles/dashboard-home/` sandbox-owned at mode `0700` so the dashboard can update its isolated profile.
9595
Other Hermes profiles remain read-only during lockdown.
96-
The same root-only mutation lock stays held through Hermes config writes, the full `shields up` or `shields down` filesystem transition and verification, and lifecycle recovery that needs to seal those paths.
96+
The root-only mutation lock stays held through every Hermes host-side config write.
97+
On the sealed-plan and compatibility Shields paths, it also stays held through the full `shields up` or `shields down` filesystem transition and verification, and lifecycle recovery that needs to seal those paths.
98+
99+
Current NemoClaw-managed Hermes images on the Docker driver use the `provider-state-mutation-v2` runtime provider state mutation contract for Shields filesystem transitions.
100+
NemoClaw selects this contract only when all these conditions apply:
101+
102+
- The registered sandbox is a managed Hermes image.
103+
- The sandbox registry records its lifecycle generation.
104+
- The image exposes the exact root-owned capability installed by NemoClaw.
105+
106+
An older managed image uses the sealed-plan transition only after NemoClaw proves that the capability is absent.
107+
A custom image, including a legacy Dockerfile workflow, remains on its existing sealed-plan or compatibility contract and cannot opt in by adding a similarly named file.
108+
An invalid capability that is present fails closed and requires a rebuild with a current managed image.
109+
110+
The Docker provider binds each transition to the registered lifecycle generation, exact container, qualified Docker engine authority, mount namespace, `/sandbox/.hermes` inode, AgentDefinition-derived plan and projection, target posture, rollback posture, and fresh nonce.
111+
Before the provider fence becomes active, NemoClaw drains any earlier privileged sandbox execution lease.
112+
While the fence is active, new direct-container, SSH, and OpenShell command transports are rejected before a sandbox command starts.
113+
The installed root-owned controller and Hermes publisher apply and verify the exact target posture, then the provider starts only the bound Hermes entrypoint and proves the replacement gateway and startup checkpoint before it releases the fence.
114+
115+
The owner-only ledger under `~/.nemoclaw/state/runtime-provider-lifecycle/` survives a host controller restart and remains authoritative for recovery of the exact target.
116+
The next Shields command, including `shields status`, must recover a retained fence before it can report or change posture.
117+
Hermes startup also checks the image-owned startup gate before it reads mutable state and remains held when the runtime provider state mutation is active or cannot be authenticated.
118+
While recovery is incomplete, do not:
119+
120+
- Delete the ledger.
121+
- Kill a held entrypoint.
122+
- Use a manual container command to bypass the fence.
97123

98124
If another host mutation is active, the command reports `Hermes config mutation is already in progress`.
99125
If another lifecycle request owns the supervisor, it reports `SUPERVISOR_BUSY`.

0 commit comments

Comments
 (0)