Skip to content

fix(sandbox): give each Hermes sandbox its own OpenAI-compatible API port - #8577

Open
laitingsheng wants to merge 4 commits into
mainfrom
fix/hermes-api-port-per-sandbox
Open

fix(sandbox): give each Hermes sandbox its own OpenAI-compatible API port#8577
laitingsheng wants to merge 4 commits into
mainfrom
fix/hermes-api-port-per-sandbox

Conversation

@laitingsheng

@laitingsheng laitingsheng commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

The Hermes OpenAI-compatible API port was a fixed host-wide 8642, so only one Hermes sandbox per host could own its host forward. A snapshot clone therefore received no inference forward, and gateway restart on the clone exited 1 with a forward recovery failure it could never repair. Each Hermes sandbox now allocates its own API port from 8642 through 8652, and every consumer resolves that port instead of the manifest default.

Related Issue

Fixes #8543

Changes

  • src/lib/core/ports.ts adds HERMES_API_PORT_RANGE_START/END and isHermesApiPort. 8642 becomes the range start rather than a host-wide reservation.
  • src/lib/onboard/hermes-api-port.ts is new: allocation, resolveOnboardHermesApiPort, resolveSandboxHermesApiPort, and retargetHermesApiPortInUrl. Resolution order is explicit environment value, then the registered value, then a fresh allocation, and the result is published back to NEMOCLAW_HERMES_API_PORT. That publication is the mechanism that lets the sandbox-create environment, the registry row, and the host forward agree on one value without threading an argument through the onboarding entrypoint, matching how ensureAgentDashboardForward already propagates the dashboard port through CHAT_UI_URL. Only a sandbox with no registry row takes a fresh allocation, because a registered sandbox without a port predates this change and is already bound to the default; src/lib/onboard/hermes-api-port.test.ts protects both branches.
  • src/lib/onboard/dashboard-port.ts extracts findAvailablePortInRange so the dashboard and API allocators share one forward-list, registry, and host-bind view, and adds getRegistryOccupiedHermesApiPorts for the cross-gateway view.
  • The registry carries hermesApiPort, sanitised in src/lib/state/registry.ts and validated fail-closed in src/lib/state/gateway-registry.ts so a corrupt value cannot drop occupancy and hand out a colliding port.
  • src/lib/actions/sandbox/snapshot.ts allocates the clone's own port before any destructive step, so range exhaustion aborts before deleteSandboxForRestore removes a --force destination, and overrides the value the source-entry spread would otherwise inherit.
  • src/lib/actions/sandbox/forward-recovery.ts resolves manifest-declared forward ports against the sandbox's own ports. This fixes both halves of the reported failure: recovery no longer demands the manifest dashboard port from a sandbox that was allocated another, and it targets the sandbox's own API port.
  • agents/hermes/start.sh reads NEMOCLAW_HERMES_API_PORT and publishes it as a root-owned 0444 marker at /run/nemoclaw/hermes-api-port. A one-shot openshell sandbox exec does not inherit the supervisor environment, so agents/hermes/mcp-config-transaction.py, agents/hermes/plugin/__init__.py, and scripts/managed-gateway-control.py read that marker. It is deliberately not writable by the sandbox user, because a writable source would let the agent redirect the relay health probes. The MCP reload phase identifiers drop their port literals, since the port now varies.
  • The sandbox agent rejection message, the onboarding ready summary, and scripts/install.sh resolve the sandbox's port instead of naming 8642.
  • The reserved-port guards in src/lib/onboard/preflight-ports.ts, src/lib/onboard/hermes-dashboard.ts, and the managed-startup profile validation cover the whole range rather than the single literal.
  • ci/source-architecture-budget.json raises four limits. Introducing one module adds one edge each to snapshot.ts fan-out, src/lib/onboard root files, and core/ports.ts/state/registry.ts fan-in. Two other increases were avoidable and were removed instead: getSandboxHealthProbeUrl moved into forward-recovery.ts, which already owned that dependency, and snapshot.ts no longer imports core/ports.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: the change adds a root-owned 0444 marker at /run/nemoclaw/hermes-api-port that three in-sandbox helpers read, and it widens the reserved-port set in the Shields-sealed managed-startup profile validation from one literal to a range. A maintainer should confirm the marker's trust boundary and the reservation change before merge.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: docs/get-started/quickstart-hermes.mdx, docs/reference/commands.mdx, docs/reference/troubleshooting.mdx, docs/deployment/deploy-to-headless-server.mdx, docs/deployment/set-up-mcp-bridge.mdx, plus the changed comments, CLI strings, and test titles across the diff. The subagent reviewed the writing rules and documentation style over three passes. Pass 1 raised six blocking findings, three of which were functional: the sandbox agent rejection message and the onboarding ready summary still emitted the manifest default, and the added docs pointed operators at status, which prints no port. Pass 2 raised five newly introduced items, including a guard that applied the Hermes port to any agent and unit tests that had begun reading the real sandbox registry. Pass 3 raised one blocking finding: the remedy for a pre-change sandbox named recreate, which preserves the registry row, where only destroy removes it. All are applied. The final pass reported nothing blocking.
  • Agent: Claude Code

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: npx vitest run over the six affected suites — 193 passed. npm run checks:repository — exit 0. npm run typecheck:cli — exit 0. npm --prefix nemoclaw run typecheck — exit 0.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: not run locally; left to CI. An earlier serial npx vitest run --project cli on this branch reported 1155 files and 14423 tests passing, but that run predates the final two commits.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — npm run docs exits 0 with 0 errors and 2 warnings. Both warnings are pre-existing learn.microsoft.com language-reference links in docs/reference/troubleshooting.mdx that this change does not touch, so the box stays clear rather than claiming a warning-free build.
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Tinson Lai tinsonl@nvidia.com

Summary by CodeRabbit

  • New Features

    • Hermes now supports per-sandbox OpenAI-compatible API ports, automatically allocated within ports 8642–8652.
    • Added NEMOCLAW_HERMES_API_PORT override support with validation and collision handling.
    • Forwarding, health checks, onboarding, cloning, and recovery now use each sandbox’s assigned API port.
    • Port assignments are preserved and discovered automatically, including support for existing sandboxes.
  • Bug Fixes

    • Prevented dashboard/API port conflicts across Hermes sandboxes.
    • Improved recovery when forwards are missing or ports differ from defaults.
  • Documentation

    • Updated deployment, onboarding, command reference, quickstart, and troubleshooting guidance for dynamic ports.

…port

The API port was a fixed host-wide 8642, so a second Hermes sandbox on one
host could never own an inference forward. A snapshot clone therefore received
no inference forward and its gateway restart could not converge.

Allocate the port per sandbox from 8642-8652, persist it on the registry row,
and publish it to the sandbox through the create environment. The entrypoint
re-publishes it as a root-owned read-only marker because a one-shot sandbox
exec does not inherit the supervisor environment. Forward recovery now resolves
manifest-declared ports against the sandbox's own ports instead of demanding
the agent defaults a sibling sandbox already holds.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
The first pass left three host-side surfaces on the manifest default: the
`sandbox agent` rejection message, the onboarding ready summary, and the docs.
Each told an operator to forward a port their sandbox does not use.

Relaunch also reached the allocator through the sandbox-create environment. A
sandbox registered before the port became per-sandbox has no recorded value, so
the allocator saw its own live forward holding the default and moved it, leaving
the sandbox bound to one port while the host forwarded another. A registered
sandbox without a port now keeps the default; only an unregistered sandbox takes
a fresh allocation.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
…er-sandbox

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

# Conflicts:
#	ci/source-architecture-budget.json
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bdfc02ef-8c1a-4b72-850b-d5b5cd149e4b

📥 Commits

Reviewing files that changed from the base of the PR and between 525b49e and d1232f9.

📒 Files selected for processing (1)
  • agents/hermes/mcp-config-transaction.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • agents/hermes/mcp-config-transaction.py

📝 Walkthrough

Walkthrough

Hermes now assigns API ports per sandbox from ports 8642–8652. The selected port is stored in registry state, passed to sandbox startup, published through a runtime marker, and used by forwarding, recovery, cloning, and health checks.

Changes

Hermes API port allocation and propagation

Layer / File(s) Summary
Port contracts and allocation
src/lib/core/ports.ts, src/lib/onboard/hermes-api-port.ts, src/lib/onboard/dashboard-port.ts, src/lib/state/*
Added range-based allocation, occupancy tracking, validation, persistence, and dashboard collision checks.
Onboarding and registry propagation
src/lib/onboard/*, src/lib/agent/onboard.ts, src/lib/actions/sandbox/agent/*
Sandbox creation, registration, forwarding, passthrough diagnostics, and status rendering now use sandbox-specific API ports.
Recovery and snapshot cloning
src/lib/actions/sandbox/{forward-recovery,process-recovery,snapshot}*
Recovery retargets Hermes health probes. Snapshot clones receive independent API ports before creation.
Runtime discovery and health checks
agents/hermes/*, scripts/{install.sh,managed-gateway-control.py}
Hermes publishes its API port in a root-owned marker. Runtime health and forward restoration discover that port with fallback to 8642.
Documentation and regression updates
docs/{deployment,get-started,reference}/*, test/hermes-mcp-reload-convergence.test.ts, ci/source-architecture-budget.json
Updated port, forwarding, troubleshooting, and normalized health-phase documentation and tests.

Estimated code review effort: 4 (Complex) | ~60 minutes

Suggested labels: area: inference

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 48.84% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the primary change: assigning each Hermes sandbox its own OpenAI-compatible API port.
Linked Issues check ✅ Passed The changes allocate and persist clone-specific Hermes API ports, restore inference forwards, and support independent gateway recovery as required by issue #8543.
Out of Scope Changes check ✅ Passed The code, tests, documentation, and validation updates directly support per-sandbox Hermes API port allocation and recovery.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/hermes-api-port-per-sandbox

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit d1232f9 in the fix/hermes-api-port-... branch remains at 96%, unchanged from commit aeaaf1a in the main branch.


Updated August 08, 2026 02:42 UTC

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@laitingsheng laitingsheng added integration: hermes Hermes integration behavior area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression feature PR adds or expands user-visible functionality labels Aug 7, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lib/onboard/dashboard-port.ts (1)

318-344: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Reject explicit Hermes API/dashboard port overlaps before sandbox creation.

assertDashboardPortNotReserved rejects dashboard ports in 8642–8652, but resolveOnboardHermesApiPort accepts any valid NEMOCLAW_HERMES_API_PORT without checking the resolved dashboard port. Thus NEMOCLAW_HERMES_API_PORT=18789 can map both resources to one host port. Reject this equality and add regression tests for both cases.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/dashboard-port.ts` around lines 318 - 344, Update
resolveOnboardHermesApiPort to reject configurations where the resolved Hermes
API port equals the resolved dashboard port, including explicit
NEMOCLAW_HERMES_API_PORT values and automatically selected ports. Ensure the
rejection occurs before sandbox creation and add regression tests covering both
explicit overlap and overlap produced by resolution.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@agents/hermes/start.sh`:
- Around line 173-196: Use the shared Hermes API-port validity contract of
8642–8652 at all affected sites: update agents/hermes/start.sh lines 173-196,
agents/hermes/mcp-config-transaction.py lines 88-106,
agents/hermes/plugin/__init__.py lines 1035-1051,
scripts/managed-gateway-control.py lines 1125-1151, and scripts/install.sh lines
465-485 to reject values outside that inclusive range while preserving existing
defaults and error handling.

In `@docs/deployment/deploy-to-headless-server.mdx`:
- Line 292: Update all listed documentation examples to reflect dynamic Hermes
API-port allocation: in docs/deployment/deploy-to-headless-server.mdx lines
292-292, qualify 8642 as the first-sandbox default and revise adjacent curl
guidance; in docs/reference/commands.mdx lines 1216-1217, replace fixed-port
wording and update the Hermes agent example to use the port from openshell
forward list; in docs/reference/commands.mdx lines 1918-1919, use the selected
API port for the gateway-token example; and in
docs/reference/troubleshooting.mdx lines 3132-3133, qualify the 8642
troubleshooting path and update its health/client examples accordingly.

In `@docs/reference/troubleshooting.mdx`:
- Around line 3163-3167: Update the troubleshooting guidance around the final
destroy/re-onboard instruction to distinguish new second Hermes sandboxes from
existing pre-change sandboxes. State that new sandboxes receive the next free
API port automatically, while destroy and re-onboard is required only for an
existing sandbox with a registry row lacking hermesApiPort.

In `@src/lib/agent/onboard.ts`:
- Around line 559-577: The API dashboard branch that calls
printAdditionalForwardPorts must pass sandboxName so Hermes resolves its
registered sandbox-specific API port instead of the manifest default. Update
that kind === "api" call site, and add coverage for the Hermes dashboard.kind
=== "api" path verifying the sandbox-specific port is announced.

---

Outside diff comments:
In `@src/lib/onboard/dashboard-port.ts`:
- Around line 318-344: Update resolveOnboardHermesApiPort to reject
configurations where the resolved Hermes API port equals the resolved dashboard
port, including explicit NEMOCLAW_HERMES_API_PORT values and automatically
selected ports. Ensure the rejection occurs before sandbox creation and add
regression tests covering both explicit overlap and overlap produced by
resolution.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dc739679-b1ab-436c-a70f-c20c3887833e

📥 Commits

Reviewing files that changed from the base of the PR and between 71a7e7c and 525b49e.

📒 Files selected for processing (37)
  • agents/hermes/mcp-config-transaction.py
  • agents/hermes/plugin/__init__.py
  • agents/hermes/start.sh
  • ci/source-architecture-budget.json
  • docs/deployment/deploy-to-headless-server.mdx
  • docs/deployment/set-up-mcp-bridge.mdx
  • docs/get-started/quickstart-hermes.mdx
  • docs/reference/commands.mdx
  • docs/reference/troubleshooting.mdx
  • scripts/install.sh
  • scripts/managed-gateway-control.py
  • src/commands/sandbox/agent.ts
  • src/lib/actions/sandbox/agent/passthrough-help.ts
  • src/lib/actions/sandbox/agent/passthrough.test.ts
  • src/lib/actions/sandbox/agent/passthrough.ts
  • src/lib/actions/sandbox/forward-recovery-declared-ports.test.ts
  • src/lib/actions/sandbox/forward-recovery.ts
  • src/lib/actions/sandbox/process-recovery.ts
  • src/lib/actions/sandbox/snapshot-restore-clone-ports.test.ts
  • src/lib/actions/sandbox/snapshot.ts
  • src/lib/agent/onboard.test.ts
  • src/lib/agent/onboard.ts
  • src/lib/core/ports.ts
  • src/lib/onboard/agent-dashboard-forward.ts
  • src/lib/onboard/dashboard-port.ts
  • src/lib/onboard/hermes-api-port.test.ts
  • src/lib/onboard/hermes-api-port.ts
  • src/lib/onboard/hermes-dashboard.ts
  • src/lib/onboard/managed-startup-profile.test.ts
  • src/lib/onboard/managed-startup/profile.ts
  • src/lib/onboard/preflight-ports.ts
  • src/lib/onboard/sandbox-create-launch.ts
  • src/lib/onboard/sandbox-registration.ts
  • src/lib/state/gateway-registry.ts
  • src/lib/state/registry.ts
  • src/lib/state/registry/types.ts
  • test/hermes-mcp-reload-convergence.test.ts

Comment thread agents/hermes/start.sh
Comment on lines +173 to +196
# The API port is a per-sandbox host resource: the host forwards the same
# number it is exposed on here, so two sandboxes on one host need two values.
# NemoClaw allocates the port and passes it in; the default keeps a sandbox
# whose create environment carries no value on the original port.
_api_port_raw="${NEMOCLAW_HERMES_API_PORT:-}"
if [ -z "$_api_port_raw" ]; then
PUBLIC_PORT=8642
else
PUBLIC_PORT="$(printf '%s' "$_api_port_raw" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')"
_api_port_valid=1
case "$PUBLIC_PORT" in
*[!0-9]* | '') _api_port_valid=0 ;;
esac
if [ "$_api_port_valid" -eq 1 ] && { [ "$PUBLIC_PORT" -lt 1024 ] || [ "$PUBLIC_PORT" -gt 65535 ]; }; then
_api_port_valid=0
fi
if [ "$_api_port_valid" -ne 1 ]; then
echo "[SECURITY] Invalid NEMOCLAW_HERMES_API_PORT='${NEMOCLAW_HERMES_API_PORT}' - must be an integer between 1024 and 65535" >&2
exit 1
fi
fi

if [ "$_dashboard_port" -eq "$PUBLIC_PORT" ]; then
echo "[SECURITY] Invalid Hermes dashboard port ${PUBLIC_PORT} - reserved for the Hermes OpenAI-compatible API" >&2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Use one Hermes API-port validity contract.

The PR defines Hermes API ports as 8642–8652. These consumers instead accept all unprivileged TCP ports. This permits unsupported values to bypass allocation and occupancy protection.

  • agents/hermes/start.sh#L173-L196: reject NEMOCLAW_HERMES_API_PORT values outside 8642–8652.
  • agents/hermes/mcp-config-transaction.py#L88-L106: reject marker values outside 8642–8652.
  • agents/hermes/plugin/__init__.py#L1035-L1051: reject marker values outside 8642–8652.
  • scripts/managed-gateway-control.py#L1125-L1151: reject marker values outside 8642–8652.
  • scripts/install.sh#L465-L485: reject registry values outside 8642–8652.
📍 Affects 5 files
  • agents/hermes/start.sh#L173-L196 (this comment)
  • agents/hermes/mcp-config-transaction.py#L88-L106
  • agents/hermes/plugin/__init__.py#L1035-L1051
  • scripts/managed-gateway-control.py#L1125-L1151
  • scripts/install.sh#L465-L485
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@agents/hermes/start.sh` around lines 173 - 196, Use the shared Hermes
API-port validity contract of 8642–8652 at all affected sites: update
agents/hermes/start.sh lines 173-196, agents/hermes/mcp-config-transaction.py
lines 88-106, agents/hermes/plugin/__init__.py lines 1035-1051,
scripts/managed-gateway-control.py lines 1125-1151, and scripts/install.sh lines
465-485 to reject values outside that inclusive range while preserving existing
defaults and error handling.

```

The Hermes OpenAI-compatible API uses the loopback forward on port `8642`.
The Hermes OpenAI-compatible API uses the loopback forward on the sandbox's API port, `8642` for the first Hermes sandbox on a host.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Keep public examples aligned with dynamic Hermes API-port allocation.

These passages introduce fallback allocation, but their surrounding examples still treat 8642 as fixed. Users can query or forward the wrong port. Use the assigned port from openshell forward list, or explicitly scope each example to an available first-sandbox default.

  • docs/deployment/deploy-to-headless-server.mdx#L292-L292: qualify 8642 as the default and update the adjacent curl guidance.
  • docs/reference/commands.mdx#L1216-L1217: update the Hermes agent example and the preceding fixed-port wording.
  • docs/reference/commands.mdx#L1918-L1919: update the gateway-token example to use the selected API port.
  • docs/reference/troubleshooting.mdx#L3132-L3133: qualify the 8642 troubleshooting path and its health/client examples.
📍 Affects 3 files
  • docs/deployment/deploy-to-headless-server.mdx#L292-L292 (this comment)
  • docs/reference/commands.mdx#L1216-L1217
  • docs/reference/commands.mdx#L1918-L1919
  • docs/reference/troubleshooting.mdx#L3132-L3133
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/deployment/deploy-to-headless-server.mdx` at line 292, Update all listed
documentation examples to reflect dynamic Hermes API-port allocation: in
docs/deployment/deploy-to-headless-server.mdx lines 292-292, qualify 8642 as the
first-sandbox default and revise adjacent curl guidance; in
docs/reference/commands.mdx lines 1216-1217, replace fixed-port wording and
update the Hermes agent example to use the port from openshell forward list; in
docs/reference/commands.mdx lines 1918-1919, use the selected API port for the
gateway-token example; and in docs/reference/troubleshooting.mdx lines
3132-3133, qualify the 8642 troubleshooting path and update its health/client
examples accordingly.

Comment on lines +3163 to +3167
If `openshell forward list` does not show the sandbox's API port, run `nemohermes <name> connect --probe-only` (or `nemohermes <name> recover`) to ask the recovery path to re-establish every manifest-declared agent forward port that has gone missing.
Recovery targets each sandbox's own ports.
A second Hermes sandbox on the same host receives the next free API port, so check which sandbox owns each row before assuming a missing `8642` row belongs to the sandbox you are debugging.
A Hermes sandbox onboarded before the API port became per-sandbox carries no allocated port and keeps `8642`.
Run `nemohermes <name> destroy` and onboard the sandbox again when a second Hermes sandbox needs its own API forward, because NemoClaw allocates a port only for a sandbox that has no registry row.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Clarify when destroy and re-onboard is required.

A newly created second Hermes sandbox has no registry row and should receive the next free API port. Destroy and re-onboard is needed only to migrate an existing pre-change sandbox that already has a registry row without hermesApiPort. As written, this guidance can cause an unnecessary destroy.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/reference/troubleshooting.mdx` around lines 3163 - 3167, Update the
troubleshooting guidance around the final destroy/re-onboard instruction to
distinguish new second Hermes sandboxes from existing pre-change sandboxes.
State that new sandboxes receive the next free API port automatically, while
destroy and re-onboard is required only for an existing sandbox with a registry
row lacking hermesApiPort.

Comment thread src/lib/agent/onboard.ts
Comment on lines +559 to +577
sandboxName?: string,
): void {
const declared = Array.isArray(agent.forward_ports) ? agent.forward_ports : [];
if (declared.length === 0) return;
const apiPort = agent.healthProbe?.port;
for (const port of declared) {
if (!Number.isInteger(port) || port < 1024 || port > 65535) continue;
if (port === primaryPort || port === agent.forwardPort) continue;
const isApi = port === apiPort;
const declaredApiPort = agent.healthProbe?.port;
// The manifest names Hermes' default API port. This sandbox owns its own, so
// announce the port the operator actually has to forward. Only Hermes
// allocates a per-sandbox API port; every other agent keeps its declared one.
const sandboxApiPort =
agent.name === "hermes"
? resolveSandboxHermesApiPort(
(sandboxName ? registry.getSandbox(sandboxName) : undefined) ?? {},
)
: 0;
for (const declaredPort of declared) {
if (!Number.isInteger(declaredPort) || declaredPort < 1024 || declaredPort > 65535) continue;
if (declaredPort === primaryPort || declaredPort === agent.forwardPort) continue;
const isApi = declaredPort === declaredApiPort;
const port = isApi && agent.name === "hermes" ? sandboxApiPort : declaredPort;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Pass sandboxName from the API dashboard path.

The kind === "api" branch at Line 486 calls printAdditionalForwardPorts without sandboxName. For Hermes, this makes Line 570 resolve {} and announce manifest port 8642 instead of the sandbox-specific registered port.

Pass sandboxName in that branch. Add a Hermes dashboard.kind === "api" test.

Proposed fix
-    printAdditionalForwardPorts(agent, info.port, deps.buildControlUiUrls);
+    printAdditionalForwardPorts(agent, info.port, deps.buildControlUiUrls, sandboxName);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/agent/onboard.ts` around lines 559 - 577, The API dashboard branch
that calls printAdditionalForwardPorts must pass sandboxName so Hermes resolves
its registered sandbox-specific API port instead of the manifest default. Update
that kind === "api" call site, and add coverage for the Hermes dashboard.kind
=== "api" path verifying the sandbox-specific port is announced.

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Blocking findings reported

Advisor assessment: Blockers require maintainer review
Next action: Review the blockers below.
Findings: 1 blocker · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 1 blocker · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 0 warnings · 0 suggestions

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

3 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • justified — per-sandbox API port at src/lib/actions/sandbox/snapshot.ts:287: Keep this term when the allocated Hermes API port must be distinguished from a default or sibling port.
  • justified — root-owned read-only marker at agents/hermes/mcp-config-transaction.py:94: Keep the full term where ownership and write restriction establish the probe trust boundary.
  • established — host bind at docs/reference/commands.mdx:1919: Use host bind consistently for the host-side endpoint shown by `openshell forward list`.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite against this exact revision.

Recommended E2E: cloud-onboard, full-e2e, hermes-e2e, hermes-inference-switch, managed-image-multiarch-startup, managed-image-protected-runtime, security-posture, bedrock-runtime-compatible-anthropic, channels-stop-start, dashboard-remote-bind, device-auth-health, hermes-shields-config, issue-4462-scope-upgrade-approval, onboard-repair, onboard-resume, openclaw-inference-switch, rebuild-openclaw, state-backup-restore

Blockers

PRA-1 Blocker — Publish or fail closed on the Hermes API-port binding in managed startup

  • Location: agents/hermes/start.sh:3319
  • Category: security
  • Problem: The trusted API-port marker is written only after the root-only branch begins. The supported OpenShell-managed non-root topology does not write it, but MCP reload, plugin status, and managed gateway control use the marker and silently fall back to port 8642.
  • Impact: A managed-topology sandbox user can bind fallback port 8642, causing MCP reload or lifecycle health checks to inspect a user-controlled listener rather than the configured public relay.
  • Fix: Make the managed supervisor publish an immutable port binding before helpers can run, and have all probe consumers fail closed when the binding is absent. Keep the root-path marker behavior aligned with that binding.
  • Verification: Inspect a managed non-root Hermes startup with NEMOCLAW_HERMES_API_PORT=8643 and verify the marker exists, is supervisor-owned and not writable by the sandbox user, and that each helper probes 8643.
  • Test coverage: Add an integration-level startup test for managed non-root Hermes with API port 8643 that verifies marker ownership and mode, MCP reload and gateway control probe 8643, and a listener on 8642 cannot satisfy readiness.
  • Evidence: agents/hermes/start.sh:3238-3267 enters and exits the non-root startup branch before the marker write at lines 3319-3321. agents/hermes/mcp-config-transaction.py:99-106 falls back to 8642 when the marker is absent or invalid. scripts/managed-gateway-control.py:1100-1112 reads the same marker and falls back to 8642 for Hermes readiness checks. agents/hermes/start.sh:3314-3318 states that a sandbox-user-writable source would let the agent redirect relay probes.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@copy-pr-bot

copy-pr-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression feature PR adds or expands user-visible functionality integration: hermes Hermes integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Ubuntu 24.04][Sandbox] snapshot clone cannot restore its inference host forward

1 participant