Commit 96b1d67
feat(runtime): implement durable Docker state mutation (#8658)
<!-- markdownlint-disable MD041 -->
## Summary
<!-- 1-3 plain sentences: what changes and why. Describe
before-and-after behavior when it applies. Follow the NemoClaw Writing
Guide: https://github.com/NVIDIA/NemoClaw/blob/main/WRITING.md. Do not
add unrelated prose cleanup. -->
Implements the B4-C2 slice of #7744 by replacing Docker's unsupported
`stateMutation` surface with a durable, provider-fenced implementation.
Hermes Shields is the named managed-image consumer, with fencing
retained across publication, validation, rollback, activation, and
controller restart; Podman remains unregistered.
## Related Issue
<!-- Fixes #NNN or Closes #NNN. Remove this section if none. -->
Refs #7744
## Changes
<!-- List concrete changes. If this adds an abstraction, configuration,
fallback, migration, or compatibility path, name its current requirement
and consumer, explain why a direct change is insufficient, and identify
the test that protects it. -->
- Extends the runtime-provider `stateMutation` contract to plan schema
v2 and registers a real Docker implementation. Hermes Shields requires a
canonical AgentDefinition protection projection, exact serialized plan
digest, phase-bearing fence, rollback posture, activation proof, and
explicit release. A direct filesystem mutation cannot prove that it
applies to the current sandbox lifecycle, runtime, mount namespace,
state root, or configuration projection. This behavior is covered by
`state-mutation.test.ts`, `runtime-provider-contract.test.ts`, and
`docker-state-mutation.test.ts`.
- Adds durable persisted-engine lifecycle claims, exact request/receipt
validation, Docker operation authority, and exclusion of ordinary
privileged container execution for the full SSH or OpenShell/fallback
operation lifetime. Process-local locking or a check immediately before
spawning is insufficient because provider publication, subprocess
execution, fallback, cleanup, and controller recovery can cross process
boundaries. This behavior is covered by
`persisted-engine-lifecycle.test.ts`,
`docker-operation-authority.test.ts`, `command-transport.test.ts`,
`privileged-exec.test.ts`, and `privileged-exec-exclusion.test.ts`.
- Packages fixed, root-owned mutation control, startup-gate, and Hermes
publisher helpers plus an exact capability descriptor in the managed
image. The image-side controller validates the durable request,
publishes or rolls back the protection posture, records activation
evidence, and gates gateway startup and recovery. Host-only mutation is
insufficient because publication and restart recovery must be enforced
inside the image that owns the state mount. This behavior is covered by
`runtime-state-mutation-control.test.ts`,
`runtime-state-mutation-startup-gate.test.ts`,
`runtime-state-mutation-hermes-publisher.test.ts`,
`hermes-final-image-layout.test.ts`, and
`hermes-gateway-supervisor-recovery.test.ts`.
- Wires Hermes Shields as the named production consumer, including
retained-fence recovery, rollback publication, activation proof
handling, timer process identity, and transition-lock coordination. Only
a current managed Hermes Docker image with the exact root-owned
capability uses the provider protocol; images without it retain the
existing compatibility path. A global switch is insufficient while older
images remain valid deployments. This behavior is covered by
`hermes-runtime-state-mutation.test.ts`, `flow.test.ts`,
`legacy-hermes-compat.test.ts`, `openclaw-transition.test.ts`,
`policy-transition.test.ts`, `timer-bound-lock.test.ts`, and
`timer-process.test.ts`.
- Hardens Docker operation authority by binding the qualified
executable, interpreter chain, credential and SSH helpers, endpoint,
PATH semantics, and privileged-execution lease through every capture and
spawn. The installer now preserves already-present user-local OpenShell
and npm PATH entries so persisted authority survives status and cleanup
without weakening real helper or endpoint drift checks.
- Restricts qualification-only Docker loopback publication to a freshly
queried live Docker Engine >= 28.3.3 authority. Registry and
model-server publications each consume a separate single-use authority;
ordinary managed llama.cpp onboarding retains the private bridge,
disables contradictory inherited image healthchecks, and performs no
Docker port publication.
- Documents provider selection, durable fencing and phases, restart hold
and recovery, rollback, ledger state, host files, troubleshooting, and
the runtime trust boundary.
## Type of Change
- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)
## Quality Gates
<!-- Check one tests line and one docs line. Check other lines when
applicable. Add every requested justification or approval reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent Codex
Desktop code and security reviews found no actionable findings in the
B4-C2 change set; final stabilization reviews accepted and fixed timer
identity caching plus guardian stop polling, dismissed two
false-positive seam/import findings with evidence, and verified
analyzer-only Python cleanups preserve fail-closed behavior. The
post-review code and security reviews found no actionable findings after
all five valid final CodeRabbit findings were fixed; focused suites plus
npm run validate:pr passed at exact head 2cd934e.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:
## Documentation Writer Review
<!-- Required for code and documentation changes after the changes and
applicable validation are complete. Keep one review checkbox and one
instance of each visible or hidden field. For Evidence, list changed
documentation paths. For documentation-only changes, also state that the
writing rules and documentation style were reviewed. For other results,
explain why no documentation change is needed or why the review is
blocked. For Agent, use a consistent product and surface name, such as
Codex Desktop, Codex CLI, Claude Code, or Cursor. After committing all
review changes, put `git rev-parse --short HEAD` and `git rev-parse
--short HEAD:AGENTS.md` in the hidden metadata below. Rerun the review
and refresh that metadata after any new commit. This receipt is advisory
during the data-collection pilot. -->
- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: docs/inference/set-up-llama-cpp.mdx;
docs/manage-sandboxes/runtime-controls.mdx; docs/reference/commands.mdx;
docs/reference/host-files-and-state.mdx;
docs/reference/troubleshooting.mdx; docs/security/tcb-boundary.mdx
- Agent: Codex Desktop
<!-- docs-review-head-sha: 2cd934e -->
<!-- docs-review-agents-blob-sha: c4923a3 -->
## DGX Station Hardware Evidence
<!-- Required only when scripts/prepare-dgx-station-host.sh changes.
Maintainers must review the linked evidence before approving or merging.
This is human-reviewed evidence, not authenticated hardware provenance.
Exceptional bypasses use existing repository governance and must be
documented on the PR. -->
- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:
## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification:
- Latest-base installer, qualification, runtime-provider, Hermes, and
lifecycle matrix: 8 files, 231 passed. Earlier full affected matrix: 20
files, 391 passed, 1 intentionally skipped.
- Docker authority, health, and publication regressions: 80 of 80 passed
in the final focused rerun; forged/reused publication tokens, npm PATH
idempotence, inherited healthcheck suppression, and create-boundary
propagation are covered.
- Independent code, security, and documentation reviews passed at exact
final head 2cd934e. Security validation covered 279 passed tests with
one expected skip; the compiler probe confirms the sync-or-Promise
recovery signature matches runtime behavior.
- Post-integration validation passed: 526/526 changed CLI tests, 247/247
integration tests, and 156/156 protected composition tests. Exact final
head 2cd934e passed 279 focused tests with one expected skip, npm run
typecheck:cli, npm run typecheck, and npm run validate:pr. The local
installer/package-contract rerun passed 96/98; its two
installer-preflight failures reproduce on untouched base under the
ambient Homebrew trust state and are outside this diff.
- `npm run validate:pr`, timer tests (22/22), mutation
controller/startup-gate/publisher tests (16/16), and protected-readiness
support tests (49/49) passed through exact head 2cd934e against
runtime base e677fcb; pre-push hooks also passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Exact-head canonical CI
run 31421039363 and managed-image all-agent activation run 31421040652
passed at prior head d4271fa. Current exact-head canonical CI run
31447285073, managed-image run 31447285090, and trusted protected E2E
run 31447308972 are running at candidate 2cd934e against runtime base
e677fcb and trusted workflow 2afba61.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] npm run docs builds without warnings (doc changes only)
- `npm run docs` completed with 0 errors. Fern reported two non-PR
warnings: the unauthenticated redirects check was skipped, and the
existing light-mode accent contrast is 2.41:1.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added managed Hermes runtime-state mutation support with protected
transitions, rollback, recovery, and startup gating.
* Added durable lifecycle tracking and fail-closed validation for
runtime changes.
* Added coordinated execution leases for privileged sandbox and
transport operations.
* Added Docker authority checks and secure loopback publishing for
supported inference workflows.
* Added safer Docker launch configuration for host-local inference.
* **Bug Fixes**
* Improved timer authorization, process identity validation, recovery
handling, and installer PATH preservation.
* **Documentation**
* Added guidance for runtime controls, recovery, lifecycle state,
troubleshooting, and inference setup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>1 parent 9257fe0 commit 96b1d67
94 files changed
Lines changed: 22319 additions & 1279 deletions
File tree
- agents/hermes
- docs
- inference
- manage-sandboxes
- reference
- security
- scripts
- checks
- src/lib
- actions/sandbox
- adapters/sandbox
- domain/uninstall
- inference/llama-cpp
- onboard
- managed-workload
- runtime-provider
- sandbox
- shields
- state/mcp-lifecycle-lock
- test
- e2e/support
- helpers
- package-contract
- cli
- support
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
123 | 126 | | |
| 127 | + | |
124 | 128 | | |
125 | 129 | | |
126 | 130 | | |
| |||
400 | 404 | | |
401 | 405 | | |
402 | 406 | | |
403 | | - | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
404 | 411 | | |
405 | 412 | | |
406 | 413 | | |
| |||
415 | 422 | | |
416 | 423 | | |
417 | 424 | | |
418 | | - | |
| 425 | + | |
419 | 426 | | |
420 | | - | |
421 | | - | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
422 | 430 | | |
423 | 431 | | |
| 432 | + | |
424 | 433 | | |
425 | 434 | | |
426 | 435 | | |
| |||
1243 | 1252 | | |
1244 | 1253 | | |
1245 | 1254 | | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
1246 | 1261 | | |
1247 | 1262 | | |
1248 | 1263 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
22 | 121 | | |
23 | 122 | | |
24 | 123 | | |
| |||
127 | 226 | | |
128 | 227 | | |
129 | 228 | | |
| 229 | + | |
130 | 230 | | |
131 | 231 | | |
132 | 232 | | |
| |||
2908 | 3008 | | |
2909 | 3009 | | |
2910 | 3010 | | |
| 3011 | + | |
| 3012 | + | |
| 3013 | + | |
| 3014 | + | |
| 3015 | + | |
2911 | 3016 | | |
2912 | 3017 | | |
2913 | 3018 | | |
| |||
3136 | 3241 | | |
3137 | 3242 | | |
3138 | 3243 | | |
| 3244 | + | |
3139 | 3245 | | |
3140 | 3246 | | |
3141 | 3247 | | |
| |||
3228 | 3334 | | |
3229 | 3335 | | |
3230 | 3336 | | |
| 3337 | + | |
3231 | 3338 | | |
3232 | 3339 | | |
3233 | 3340 | | |
| |||
3243 | 3350 | | |
3244 | 3351 | | |
3245 | 3352 | | |
| 3353 | + | |
3246 | 3354 | | |
3247 | 3355 | | |
3248 | 3356 | | |
| |||
3371 | 3479 | | |
3372 | 3480 | | |
3373 | 3481 | | |
| 3482 | + | |
3374 | 3483 | | |
3375 | 3484 | | |
3376 | 3485 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
86 | 89 | | |
87 | 90 | | |
88 | 91 | | |
| |||
135 | 138 | | |
136 | 139 | | |
137 | 140 | | |
138 | | - | |
| 141 | + | |
139 | 142 | | |
140 | 143 | | |
141 | 144 | | |
| |||
147 | 150 | | |
148 | 151 | | |
149 | 152 | | |
150 | | - | |
| 153 | + | |
151 | 154 | | |
152 | 155 | | |
153 | 156 | | |
154 | 157 | | |
| 158 | + | |
155 | 159 | | |
156 | 160 | | |
157 | 161 | | |
| 162 | + | |
| 163 | + | |
158 | 164 | | |
159 | 165 | | |
160 | 166 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
96 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
97 | 123 | | |
98 | 124 | | |
99 | 125 | | |
| |||
0 commit comments