Skip to content
12 changes: 12 additions & 0 deletions docs/inference/configure-inference-timeouts.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,18 @@ $$nemoclaw onboard
This setting is baked into the sandbox image.
Recreate an existing sandbox to apply a new value.

Each key bounds a different deadline.
`agents.defaults.timeoutSeconds` bounds one agent run, and `$$nemoclaw <name> agent --timeout <seconds>` overrides it for a single run.
`models.providers.<provider-id>.timeoutSeconds` bounds one provider request, and no flag overrides it.
Raise the provider key when a turn times out while waiting for the model server, because a longer `--timeout` does not extend the provider request.

To change a deadline on an existing sandbox instead of recreating it, lower shields first and write the key directly.

```bash
$$nemoclaw <sandbox-name> shields down
$$nemoclaw <sandbox-name> config set --key agents.defaults.timeoutSeconds --value 1800 --restart
```

</AgentOnly>

<AgentOnly variant="hermes">
Expand Down
15 changes: 14 additions & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1288,6 +1288,18 @@ Because a delivered turn always writes to one of the two streams, the wrapper re
The wrapper prints recovery guidance to `stderr` and exits with status `1`.
Pressing `Ctrl+C` interrupts the OpenShell child, and sending `SIGTERM` to the host wrapper forwards `SIGTERM` to that child.
NemoClaw waits for OpenShell to stop the in-sandbox turn, replays captured output, and returns status `130` for `SIGINT` or `143` for `SIGTERM`.
When the forwarded argv sets `openclaw agent --timeout <seconds>`, both captured paths bound the OpenShell command at that value plus 30 seconds.
The extra seconds let the in-sandbox turn report its own timeout first, so the host bound catches only a turn that stops answering.

These leave the OpenShell wait unbounded:

- `--timeout 0`.
- A value NemoClaw cannot read as a positive whole number of seconds.
- An argv without `--timeout`.
- A `--timeout` after the `--` argv terminator, which OpenClaw reads as payload rather than as its own flag.

When the captured output reports that the turn's deadline fired, the wrapper replays the partial output and writes deadline guidance to `stderr`.
It exits with status `1` instead of the upstream status `0`.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
The diagnostic shell-quotes the sandbox name and forwarded arguments, then redacts detected credential values before writing the recovery command to `stderr`.
If redaction changes the recovery command, the diagnostic tells you not to replay it; otherwise, it labels the command as runnable inside the sandbox.
For a registered sandbox, both captured paths pin the sandbox's recorded gateway with an explicit `-g`.
Expand All @@ -1297,11 +1309,12 @@ Raw `stderr`, including structured JSON diagnostics, is forwarded unchanged.
NemoClaw appends failed-tool or untrusted-child provenance only from the `stdout` JSON.
The wrapper reads completion markers only from the final matching OpenClaw response envelope: a local `{ payloads, meta }` response or a gateway `{ status, result: { payloads, meta } }` response.
It ignores earlier JSON progress or log records.
It exits with status `1` when that metadata contains `error.kind: "incomplete_turn"`, `livenessState: "abandoned"`, or `replayInvalid: true`, even when the envelope reports success.
It exits with status `1` when that metadata contains `error.kind: "incomplete_turn"`, `livenessState: "abandoned"`, `replayInvalid: true`, or a `timeoutPhase` value, even when the envelope reports success.
Marker-shaped values inside tool results, tool-call arguments, or other descendants do not change the exit status.
A turn can run every tool successfully and still become abandoned before it produces a reply.
The wrapper writes the unchanged JSON trace to `stdout` before it reports the incomplete turn, so the partial tool trace remains available.
The wrapper writes the verdict, the detected markers, and verify-before-retry guidance to `stderr`.
A `timeoutPhase` value names the phase the deadline fired in, so the wrapper writes deadline guidance in place of the generic incomplete-turn text.
Tool calls in a partial trace may have already applied side effects, so verify what the turn changed before you retry it.
The wrapper passes through an upstream non-zero exit status unchanged.
Literal `--json` values consumed by flags such as `-m` or `--reply-channel`, or arguments after `--`, stay on the normal passthrough path.
Expand Down
111 changes: 111 additions & 0 deletions src/lib/actions/sandbox/agent/passthrough-dispatch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,15 @@ import { describe, expect, it, vi } from "vitest";

import {
type AgentDispatchChild,
AGENT_DISPATCH_DEADLINE_BUFFER_SECONDS,
agentDispatchDeadlineSeconds,
agentDispatchStdio,
isSilentAgentDispatch,
isTimedOutAgentDispatch,
requestedAgentTimeoutSeconds,
runAgentDispatch,
SILENT_AGENT_DISPATCH_EXIT_CODE,
TIMED_OUT_AGENT_TURN_EXIT_CODE,
} from "./passthrough-dispatch";
import { computeExitCode, type SandboxExecSignalSource } from "../exec";

Expand Down Expand Up @@ -159,3 +164,109 @@ describe("SILENT_AGENT_DISPATCH_EXIT_CODE", () => {
expect(SILENT_AGENT_DISPATCH_EXIT_CODE).toBe(1);
});
});

describe("requestedAgentTimeoutSeconds", () => {
const agent = (...args: string[]) => ["openclaw", "agent", ...args];

it("reads a separated --timeout value (#8723)", () => {
expect(requestedAgentTimeoutSeconds(agent("--agent", "main", "--timeout", "30"))).toBe(30);
});

it("reads an equals-form --timeout value (#8723)", () => {
expect(requestedAgentTimeoutSeconds(agent("--timeout=45", "-m", "hi"))).toBe(45);
});

it("requests no deadline when the argv carries no --timeout (#8723)", () => {
expect(requestedAgentTimeoutSeconds(agent("--agent", "main", "-m", "hi"))).toBeNull();
});

it("returns null for --timeout 0 so the host stays unbounded (#8723)", () => {
expect(requestedAgentTimeoutSeconds(agent("--timeout", "0"))).toBeNull();
});

it("ignores a --timeout consumed as another option's value (#8723)", () => {
expect(requestedAgentTimeoutSeconds(agent("-m", "--timeout", "--agent", "main"))).toBeNull();
});

it("ignores anything past the -- terminator (#8723)", () => {
expect(requestedAgentTimeoutSeconds(agent("--", "--timeout", "30"))).toBeNull();
});

it("refuses a value that cannot be a deadline (#8723)", () => {
for (const raw of ["-5", "1.5", "abc", "", "1e3"]) {
expect(requestedAgentTimeoutSeconds(agent("--timeout", raw))).toBeNull();
}
expect(requestedAgentTimeoutSeconds(agent("--timeout"))).toBeNull();
});
});

describe("agentDispatchDeadlineSeconds", () => {
it("outlasts the requested deadline so the turn reports its own timeout (#8723)", () => {
expect(agentDispatchDeadlineSeconds(["openclaw", "agent", "--timeout", "30"])).toBe(
30 + AGENT_DISPATCH_DEADLINE_BUFFER_SECONDS,
);
});

it("leaves the transport unbounded when no deadline was requested (#8723)", () => {
expect(agentDispatchDeadlineSeconds(["openclaw", "agent", "-m", "hi"])).toBeUndefined();
});

it("holds the deadline buffer above the longest aborted-run finish measured (#8723)", () => {
expect(AGENT_DISPATCH_DEADLINE_BUFFER_SECONDS).toBeGreaterThan(20);
});

it("stays unbounded when the buffered deadline leaves the safe-integer range (#8723)", () => {
const ceiling = String(Number.MAX_SAFE_INTEGER);
expect(requestedAgentTimeoutSeconds(["openclaw", "agent", "--timeout", ceiling])).toBe(
Number.MAX_SAFE_INTEGER,
);
// The buffer would round past the ceiling, so the argv would carry a
// deadline that differs from the one the caller asked for.
expect(agentDispatchDeadlineSeconds(["openclaw", "agent", "--timeout", ceiling])).toBeUndefined();
});

it("still bounds the largest deadline that survives the buffer (#8723)", () => {
const largest = String(Number.MAX_SAFE_INTEGER - AGENT_DISPATCH_DEADLINE_BUFFER_SECONDS);
expect(agentDispatchDeadlineSeconds(["openclaw", "agent", "--timeout", largest])).toBe(
Number.MAX_SAFE_INTEGER,
);
});
});

describe("isTimedOutAgentDispatch", () => {
const timeoutReport =
"Request timed out before a response was generated. Please try again, or increase `agents.defaults.timeoutSeconds` in your config.";

it("classifies the timeout report OpenClaw writes to stdout (#8723)", () => {
expect(isTimedOutAgentDispatch(`${timeoutReport}\n`, "")).toBe(true);
});

it("classifies a timeout report that arrives below tool-failure lines (#8723)", () => {
const captured = `LLM request failed.\nTool Call failed\n${timeoutReport}\n`;
expect(isTimedOutAgentDispatch(captured, "")).toBe(true);
});

it("classifies a timeout report routed to stderr instead (#8723)", () => {
expect(isTimedOutAgentDispatch("", `${timeoutReport}\n`)).toBe(true);
});

it("keeps classifying when the configuration advice is reworded upstream (#8723)", () => {
const reworded = "Request timed out before a response was generated. Raise the deadline.";
expect(isTimedOutAgentDispatch(reworded, "")).toBe(true);
});

it("leaves an ordinary answer unclassified (#8723)", () => {
expect(isTimedOutAgentDispatch("PONG\n", "openclaw warning\n")).toBe(false);
});

it("leaves an unrelated timed-out message unclassified (#8723)", () => {
const mcpFailure = "McpError: MCP error -32001: Request timed out\n";
expect(isTimedOutAgentDispatch(mcpFailure, "")).toBe(false);
});
});

describe("TIMED_OUT_AGENT_TURN_EXIT_CODE", () => {
it("reports a turn failure rather than success (#8723)", () => {
expect(TIMED_OUT_AGENT_TURN_EXIT_CODE).toBe(1);
});
});
137 changes: 137 additions & 0 deletions src/lib/actions/sandbox/agent/passthrough-dispatch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,27 @@
// - Removal condition: none while NemoClaw owns the host-side OpenShell
// child lifecycle.
//
// 9. Timed-out turn guard (deadline contract).
//
// - Invalid state: the turn's deadline fires, OpenClaw reports the timeout,
// and the dispatch still exits 0 (#8723). Measured on three platforms and
// on both transports, so a run that never answered is indistinguishable
// from one that did, and a CI job or evaluation harness records the
// timed-out turn as a pass. The empty-dispatch guard above cannot catch
// it: the timeout report itself makes the streams non-empty.
// - Source boundary: OpenClaw owns the deadline and the exit code, and that
// code is the same whether this wrapper or a bare `openshell sandbox exec`
// runs the turn. NemoClaw owns what it reports to its own caller, so it
// classifies a timeout the way it already classifies an embedded-fallback
// run rather than forwarding a success.
// - Detection differs per transport because the evidence does. The JSON
// transport reads the declared `meta.timeoutPhase` field, in
// `openClawAgentIncompleteTurnSignal`. The non-JSON transport has only
// text, so it matches the sentence OpenClaw prints, exactly as the
// embedded-fallback branch matches its own banner.
// - Removal condition: drop this guard when `openclaw agent` exits non-zero
// for a turn whose deadline fired.
//
// Regression tests: `passthrough-dispatch.test.ts` owns the classifier and the
// supervised process lifecycle; `passthrough-help.test.ts` owns the diagnostic
// text.
Expand Down Expand Up @@ -230,3 +251,119 @@ export function isSilentAgentDispatch(
): boolean {
return !result.error && result.status === 0 && stdout.length === 0 && stderr.length === 0;
}

/**
* Exit code for a turn whose deadline fired without producing a result.
* Matches the wrapper's other non-recoverable dispatch failures.
*/
export const TIMED_OUT_AGENT_TURN_EXIT_CODE = 1;

/**
* The sentence OpenClaw prints when a turn's deadline fires.
*
* Read from the OpenClaw 2026.7.1 bundle, where it is a single string literal
* in one file, and observed verbatim on stdout, sometimes below tool-failure
* lines. Only the invariant clause is matched so the configuration advice that
* follows it can be reworded upstream without disabling the guard.
*/
const OPENCLAW_AGENT_TIMEOUT_PATTERN =
/(?:^|\r?\n)Request timed out before a response was generated[^\r\n]*(?:\r?\n)?$/i;

/**
* True when the captured output reports that the turn's deadline fired.
*
* Text is the only evidence the non-JSON transport has. OpenClaw writes the
* report as the final line, so matching that position avoids treating a normal
* reply that quotes or explains the sentence as a timeout. Callers gate on an
* otherwise successful exit, so an upstream non-zero code is never rewritten.
*/
export function isTimedOutAgentDispatch(stdout: string, stderr: string): boolean {
return OPENCLAW_AGENT_TIMEOUT_PATTERN.test(stdout) || OPENCLAW_AGENT_TIMEOUT_PATTERN.test(stderr);
}

/** Documented `openclaw agent` options that consume the next argv element. */
export const OPENCLAW_AGENT_VALUE_FLAGS = new Set([
"-a",
"--agent",
"-m",
"--message",
"--model",
"--provider",
"--reply-channel",
"--session-id",
"--session-key",
"--thinking",
"--timeout",
"--to",
]);

/** Documented `openclaw agent` options that consume no argv element. */
export const OPENCLAW_AGENT_BOOLEAN_FLAGS = new Set(["--deliver"]);

/**
* Extra seconds added to a requested `--timeout` before the host transport
* stops waiting.
*
* The in-sandbox turn owns the deadline and answers first while it can still
* write to stderr: it reports the timeout, names the config key, and exits.
* Only a turn that stops answering reaches the host bound, so the extra seconds
* must outlast an ordinary late finish.
*
* This value is a choice, not a derivation. #8723 timed five aborted runs
* finishing 0.1 s to 20.8 s after their deadline, and four further aborted runs
* recorded no finish at all, so no measurement establishes an upper bound.
* Below roughly five seconds the host truncates the turn's own timeout report;
* above roughly a minute the host bound no longer catches a turn that stops
* answering. Thirty is inside that range and above every post-deadline finish
* #8723 recorded. Choose another value inside that range if a slower model or a
* busier host requires it.
*/
export const AGENT_DISPATCH_DEADLINE_BUFFER_SECONDS = 30;

/**
* The `--timeout` an `openclaw agent` argv requests, or null when the argv
* requests none.
*
* Mirrors the documented flag grammar only far enough to read one value.
* Anything unrecognized, malformed, or past a `--` terminator returns null so
* the host keeps the wait unbounded rather than shortening a turn without
* evidence. `--timeout 0` disables the deadline upstream and returns null here
* for the same reason.
*/
export function requestedAgentTimeoutSeconds(argv: readonly string[]): number | null {
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index] as string;
if (arg === "--") return null;
if (arg === "--timeout") return parseDeadlineSeconds(argv[index + 1]);
if (arg.startsWith("--timeout=")) return parseDeadlineSeconds(arg.slice("--timeout=".length));
if (OPENCLAW_AGENT_VALUE_FLAGS.has(arg)) {
index += 1;
continue;
}
}
return null;
}

function parseDeadlineSeconds(raw: string | undefined): number | null {
if (raw === undefined || !/^\d+$/.test(raw)) return null;
const seconds = Number(raw);
return Number.isSafeInteger(seconds) && seconds > 0 ? seconds : null;
}

/**
* The host transport deadline for an `openclaw agent` argv, or undefined when
* the argv requested none. Undefined leaves `openshell sandbox exec` on its own
* default, which is no timeout.
*
* A requested deadline near the safe-integer ceiling stays unbounded rather
* than becoming a bound the host cannot represent. Past that ceiling the buffer
* addition rounds, so the wait would silently differ from the number written to
* the command line. That matches how this module treats every other value it
* cannot read.
*/
export function agentDispatchDeadlineSeconds(argv: readonly string[]): number | undefined {
const requested = requestedAgentTimeoutSeconds(argv);
if (requested === null) return undefined;
const deadline = requested + AGENT_DISPATCH_DEADLINE_BUFFER_SECONDS;
return Number.isSafeInteger(deadline) ? deadline : undefined;
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Loading
Loading