[Epic] Support llama.cpp as a managed local inference server

[ericksoa]

Summary

Add llama.cpp (llama-server) as a first-class NemoClaw-managed local inference server.

Today an operator can start llama-server and select Other OpenAI-compatible endpoint. Issue #2412 and PR #5505 document that path. Issue #7209 records validation through the generic path. That evidence does not establish first-class lifecycle or product support: NemoClaw does not identify, install, pin, configure, start, recover, diagnose, or qualify the server or its GGUF model.

This epic defines the product decision and delivery work required to add that ownership. The generic compatible-endpoint path remains available for self-managed or unsupported llama.cpp configurations.

Implementation status

Discussion #7636 and this epic are accepted for experimental implementation. This is not a llama.cpp support activation.

• Declarative serving catalog foundation: #8151 / PR #8152.
• Cooperative existing-server attachment: #8161 / PR #8167.
• Typed declarative llama.cpp catalog contract: #8181 / PR #8183 (merged).
• Production install-llama-cpp YAML recipe and preset: #8173 / PR #8224 (merged).
• Managed artifact, acquisition, lifecycle, qualification, activation, and complete documentation.
  • Owned image contract: #8231 / PR #8235.
  • Trusted digest-first publication and supply-chain evidence contract: #8250 / PR #8252.
  • Dormant protected DGX Spark qualification lane: #8260 / PR #8266.
  • Activation-safe qualification test fixtures for the future YAML-only transition: PR #8273 (merged).
  • YAML-only DGX Spark activation, accepted physical evidence, owned-index publication, and production recipe digest pin.
  • Declarative GGUF identity and verification-plan foundation: #8279 / PR #8294.
  • Exact standard-cache GGUF acquisition, verification, and executor-only filesystem identity: PR #8387 (merged).
  • Dormant Docker lifecycle binding and minimum provider-neutral authority seam: #8395 / PR #8418 (merged).
  • Crash-recoverable exact receipt publication: #8414 / PR #8422 (merged).
  • Experimental managed Docker onboarding, routing, diagnostics, recovery, and cleanup on DGX Spark: #8433 / PR #8448 (merged).
  • Declarative bounded llama.cpp protocol/model qualification foundation: PR #8515 (merged; agent tuples remain unqualified).

Support activation remains a separate maintainer decision after the full managed artifact and compatibility evidence gates pass.

Problem

Treating llama.cpp as an arbitrary compatible endpoint leaves these gaps:

• Onboarding cannot reliably fingerprint llama.cpp, vLLM, or another service on the same common port.
• Users must install and start the server, choose its flags, acquire a GGUF file, and keep the process running.
• NemoClaw has no immutable server or model identity to use for resume, recovery, diagnostics, or rollback.
• A responding endpoint does not establish model-template compatibility, structured tool calling, GPU execution, or agent compatibility.
• The server can fall back to CPU execution unless the launch and validation contract rejects that state.
• The server exposes UI and slot surfaces by default, and its official container binds broadly. Those defaults do not match NemoClaw's supported security boundary.
• Current status and doctor paths cannot separate server load, authentication, model identity, GPU offload, and sandbox-route failures.

Desired behavior

An operator can attach to a cooperatively fingerprinted running llama.cpp server or select a NemoClaw-managed llama.cpp recipe. NemoClaw resolves and verifies the approved image and GGUF artifacts, starts the managed server on a private authenticated route, registers llama-cpp-local, and completes the qualified agent flow through inference.local.

Status, restart, recovery, rebuild, and restore retain the exact server, model, route, and ownership identities. A server or model outside the accepted matrix remains available only through compatible-endpoint.

Product decision

Discussion #7636 and this epic are accepted as the implementation scope for an experimental managed llama.cpp provider. This authorizes the scoped implementation. It does not declare llama.cpp supported.

NemoClaw may advertise llama.cpp support only after the activation criteria pass and maintainers record the support decision. Before production artifact selection and support activation, maintainers must name:

• the product owner for the support matrix and activation decision;
• the engineering owner for provider, lifecycle, and agent compatibility;
• the OCI image and serving-catalog owner;
• the security and dependency-update owner;
• the llama.cpp update cadence and urgent-security-update process; and
• the compatibility, upgrade, rollback, and deprecation policy for pinned server/model recipes.

Minimal v1 scope

Dimension	v1 contract
Server	Upstream llama-server; no custom fork.
Deployment	One NemoClaw-owned container, one host, one allocated NVIDIA GPU, one model, one OpenShell gateway, one registered sandbox/trust principal, and one request slot.
Platforms	CUDA on linux/amd64 and linux/arm64. Each activated host profile requires its own live E2E evidence.
Container runtime	Docker first. Podman activates only after the host-local inference contracts from #7744 support the same lifecycle and evidence.
Models	Repository-owned recipes for exact GGUF file, revision, digest, quantization, context, chat template, license, and agent capabilities.
API	OpenAI Chat Completions only for agent traffic. Force preferredInferenceApi=openai-completions during selection and recovery. Use /v1/models, /health, /props, and /metrics only for bounded discovery and diagnostics.
Agents	Qualify OpenClaw, Hermes, and LangChain Deep Agents Code separately. One agent's tool-call result does not qualify another agent.
Support state	Keep the provider experimental until every claimed platform, model, and agent tuple passes the activation criteria.

Use these initial identifiers:

• Interactive and noninteractive selection: llama-cpp for a cooperatively fingerprinted running server.
• Managed install selection: install-llama-cpp.
• OpenShell provider ID and durable provider value: llama-cpp-local.
• Display label: Local llama.cpp.
• Dedicated gateway credential: NEMOCLAW_LLAMACPP_LOCAL_TOKEN.
• Reserved v1 server port: 8081.

Both first-class paths must use reserved port 8081. They must fail on a conflict instead of moving silently. Other ports remain available through the generic compatible-endpoint path. The implementation must add 8081 to port validation, the local-inference policy, bundled-loopback handling, and route compatibility. It must treat llama-cpp-local as an endpoint-bearing provider.

Architecture and ownership

Serving catalog and readiness

The implementation must extend the managed serving design in Discussion #7636. It must not add another model registry, host-probe contract, or lifecycle format.

• Consume the versioned system-readiness report from #7407.
• Use readiness facts for operating system, architecture, container runtime, NVIDIA GPU, available memory, driver compatibility, and storage admission.
• Put product selection in a llama.cpp serving preset and complete server/model behavior in a serving recipe.
• Compile only allowlisted launch settings. YAML must not contain shell fragments or arbitrary command templates.
• Record the resolved plan and immutable identities before the first managed side effect.
• Fail before acquisition or launch when required readiness is absent, unknown, stale, or incompatible.

If the catalog foundation from Discussion #7636 has not landed, the existing-server attachment slice may use the current provider modules. It must not create a temporary catalog or durable lifecycle contract that later needs a second migration.

Prerequisite status:

• Versioned serving catalog/compiler: #8151 / PR #8152.
• Versioned system-readiness schema: #7407 / PR #7544.
• Provider-neutral host-local serving receipt: PR #8060.
• Minimum provider-neutral exact-engine, receipt, and journal seam folded into clean PR #8418; broader PRs feat(runtime): persist engine lifecycle and state mutation contract #8058 and feat(runtime): add durable host-local inference lifecycle #8069 are not dependencies of llama.cpp slice 6.

The typed schema/compiler work in #8181 completed through PR #8183. Production YAML completed through #8173 / PR #8224. The owned artifact is tracked by #8231. Clean lifecycle delivery completed through PR #8418 and PR #8422; those PRs were not blocked on #8058 or #8069. Ownership, compatibility-policy, and update-cadence decisions remain activation gates. The existing-server attachment completed through #8161 / PR #8167 without claiming ownership of the server or model.

Server and model artifacts

Each supported recipe must pin:

• llama.cpp source commit and immutable OCI image digest;
• CUDA base and supported driver boundary;
• GGUF repository, exact revision, exact filename, SHA-256 digest, expected size, quantization, and model license;
• logical model ID, chat template, context size, slot count, GPU/offload policy, and bounded request limits;
• supported agent and protocol capabilities;
• health, readiness, disabled idle sleep, graceful-stop, and recovery behavior.

The shipping image must be NemoClaw-owned. It must run as a non-root user with a read-only root filesystem and read-only model mount. It must provide bounded writable temporary storage for CUDA JIT cache and required temporary files. Publication must include multiarchitecture runtime tests, an SBOM, provenance, signing, and vulnerability scanning. A digest-pinned upstream image may be used only for an implementation spike.

Reuse the Hugging Face acquisition mechanism and standard cache already used by the DGX Spark vLLM install path. It must request the exact YAML-declared revision and file, then verify the canonical local file size and digest before use. Server startup must use that verified local file in offline mode. A gated-model credential may exist only for the shared acquisition operation and must not enter the llama.cpp image, server environment, logs, registry, snapshot, or sandbox.

Routing and security

The managed server must be reachable only through the private host/gateway topology required by https://inference.local/v1.

• Require a dedicated provider credential and keep it separate from OPENAI_API_KEY.
• Do not expose an unauthenticated listener to the LAN or public network.
• Disable or gateway-block the Web UI, public slot inspection, router mode, Model Context Protocol (MCP) proxy, server-side --tools, server-side agent mode, multimodal projection, and model-initiated downloads.
• Pass a verified local --model, --no-mmproj, and offline mode. Reject non-text content before forwarding and deny server egress during runtime.
• Keep prompt and response bodies out of production logs.
• Apply request-body, token, time, concurrency, process, memory, and storage bounds outside or around llama-server as required.
• Pin --sleep-idle-seconds -1 so /health cannot report success for an unloaded model. Treat health as load/readiness evidence only.
• Define a cooperative fingerprint from bounded server metadata and local runtime evidence. Treat it as selection evidence, not an authentication or security identity.
• Verify the non-path model alias, template capability, and GPU execution separately.

The two first-class paths have separate credential ownership:

• The managed path generates a key, persists it with mode 0600 in the gateway-scoped NemoClaw host state, injects it through a secret file, and records only a hash in the receipt. Rotation must update the server and OpenShell provider atomically.
• The existing-server path requires an operator-supplied native llama.cpp key through NEMOCLAW_LLAMACPP_LOCAL_TOKEN. A server without native key enforcement remains on the generic compatible-endpoint path.

State and lifecycle

Persist one ownership receipt for the managed server. The receipt must include the serving plan digest, image digest, model digest, runtime identity, reserved port, provider identity, and lifecycle generation.

• start and restart must reuse the verified local artifact from the standard Hugging Face cache.
• recover must replace only a NemoClaw-owned missing or failed server.
• rebuild, snapshot/restore, and clone must preserve the provider/model recipe and rebind the OpenShell route without copying provider credentials into sandbox state.
• Snapshots must record model identity, not duplicate GGUF bytes.
• Restore must verify the exact file in the standard Hugging Face cache or rerun the shared acquisition effect.
• Uninstall must not remove files from the shared Hugging Face cache. It must not remove a foreign process, container, image, or model file.
• Deep Agents Code provider changes must retain its current re-onboard boundary. OpenClaw and Hermes may use nemoclaw inference set after their agent-specific validation passes.
• A second sandbox or a conflicting llama.cpp route on the same gateway must fail before effects. Multi-sandbox sharing requires a later accepted issue.

Required delivery slices

• 1. Accept the product contract and support matrix. Discussion RFC: Managed inference serving specifications #7636 and the maintainer decisions recorded in this epic confirm the identifiers, experimental state, initial DGX Spark/GB10 host profile, NVIDIA Nemotron GGUF recipe, Chat Completions and agent scope, ownership boundary, and activation exit criteria. This remains an experimental implementation decision, not a support claim.
• 2. Add cooperative server fingerprinting and existing-server attachment. Distinguish llama.cpp, vLLM, and arbitrary OpenAI-compatible servers in supported deployments. Require port 8081, a non-path alias, and an operator-supplied native key. Missing, conflicting, or ambiguous signals route to manual compatible-endpoint selection instead of a guessed local provider. Completed by #8161 / PR #8167.
• 3. Add the serving catalog entries and compiler validation. Define llama.cpp recipes and presets, exact artifact fields, agent capabilities, allowed launch settings, and diagnostics. Reuse Discussion RFC: Managed inference serving specifications #7636 contracts.
  • Declarative catalog/compiler foundation: #8151 / PR #8152.
  • Typed llama.cpp schema/compiler extension: #8181 / PR #8183 (merged).
  • Production repository-owned YAML recipe and preset: #8173 / PR #8224 (merged).
• 4. Publish the managed OCI artifact. Tracked by #8231. Select the owned registry and workflow. Build pinned linux/amd64 and linux/arm64 CUDA images, compile out optional UI assets where supported, runtime-disable and gateway-block unused surfaces, run as non-root, publish immutable evidence, and prove real GPU execution.
  • Owned, pinned, non-root multiarchitecture image contract and secretless native PR builds: PR #8235.
  • Trusted digest-first publication, SBOM, provenance, signing, scan, and receipt contract: #8250 / PR #8252.
  • Dormant trusted-main DGX Spark qualification and evidence lane: #8260 / PR #8266.
  • Provision and activate the protected Spark tuple through YAML, accept physical evidence, publish the validated owned index, and pin its digest into the repository-owned recipe.
• 5. Reuse GGUF acquisition and verify local input. Tracked by #8279. Reuse the existing Hugging Face acquisition mechanism and standard cache. Select the exact YAML-declared file and revision, run storage preflight, isolate the credential, verify the canonical local file, and pass a typed verified artifact to llama.cpp.
  • YAML/schema/type model identity, deterministic verification-plan compiler, and local-entry verification foundation: PR #8294.
  • Production host-local llama.cpp materializer consumed by protected DGX Spark qualification: PR #8338 (merged).
  • Provider-neutral reuse of the existing Hugging Face acquisition effect: PR #8371 (merged).
  • Exact standard-cache GGUF resolution, digest verification, and pre-materialization filesystem-identity revalidation: PR #8387 (merged).
• 6. Implement launch and lifecycle. Clean dormant lifecycle binding completed through #8395 / PR #8418; crash-recoverable receipt publication completed through #8414 / PR #8422. The dormant controller now materializes the bounded command, allocates one GPU, rejects CPU fallback, persists exact ownership, and provides crash-safe lifecycle and receipt boundaries for activation work.
• 7. Integrate onboarding, routing, and runtime mutation. Add provider aliases, menus, interactive and noninteractive flows, model discovery, dedicated credentials, inference.local routing, endpoint-aware compatibility, network policy, status, doctor, recovery, and agent-specific inference set behavior. Force Chat Completions and skip the Responses probe. Existing-server onboarding landed in PR #8167; experimental managed Docker onboarding, routing, status, doctor, recovery, cleanup, and provider-scoped lifecycle integration landed in #8433 / PR #8448.
• 8. Qualify protocol and model behavior. Test synchronous and streaming Chat Completions, cancellation, usage, structured output, function-call emission, tool-result continuation, context limits, authentication, malformed requests, and timeouts for each supported model/agent tuple. Maintainer-accepted for delivery sequencing through PRs test(inference): qualify llama.cpp protocol behavior #8515 and test(inference): activate OpenClaw llama.cpp qualification #8579; physical DGX Spark validation is deferred until hardware is available.
  • Declarative, bounded protocol/model qualification foundation: PR #8515. YAML selects the exact probes and bounds; protected CI validates the runner and receipt contract.
  • Accept the OpenClaw tuple for delivery sequencing through inference.local; physical DGX Spark validation remains deferred and the current recipe intentionally remains agents: [] until the separate support-activation decision.
• 9. Add security, observability, and protected E2E. Verify disabled surfaces, request limits, log redaction, readiness, metrics, GPU telemetry, failure diagnostics, lifecycle recovery, and exact-head live E2E for every activated platform and container runtime. PR feat(inference): attach existing llama.cpp servers #8167 supplies attachment-path credential, probe, port, route-containment, and protected exact-head E2E evidence; managed image, lifecycle, telemetry, and activated-platform evidence remain.
• 10. Publish user and operator documentation. Compare llama.cpp with Ollama, vLLM, and NVIDIA NIM. Document existing-server and managed paths, shared Hugging Face cache use, model lifecycle, security boundary, diagnostics, upgrade, rollback, and the generic compatible-endpoint fallback. Existing-server command/reference documentation landed in PR feat(inference): attach existing llama.cpp servers #8167; complete managed-path and operator documentation landed in PR #8610.

Acceptance criteria

Selection and routing

• Interactive onboarding offers a cooperatively fingerprinted running llama.cpp server and the managed install path when the readiness report permits them.
• NEMOCLAW_PROVIDER=llama-cpp NEMOCLAW_MODEL=<served-id> NEMOCLAW_LLAMACPP_LOCAL_TOKEN=<key> nemoclaw onboard --non-interactive --yes-i-accept-third-party-software attaches to a server that meets the fingerprint, alias, port, and authentication contract.
• NEMOCLAW_PROVIDER=install-llama-cpp NEMOCLAW_LLAMACPP_RECIPE=<recipe-id> nemoclaw onboard --non-interactive --yes-i-accept-third-party-software materializes the exact managed recipe and persists its logical model alias as the runtime model ID.
• Both paths register llama-cpp-local with a dedicated credential and route agent traffic through https://inference.local/v1.
• Cooperative fingerprint tests do not misclassify llama.cpp as vLLM or vLLM as llama.cpp. An arbitrary, spoofed, or ambiguous server stays on the generic path.
• The managed server sets a stable, non-path model alias. An attached server must report such an alias in its inventory; otherwise it stays on the generic path.
• Provider compatibility includes the endpoint. A route that has the same model but a different port or server identity is incompatible.
• A second sandbox request on the same gateway fails before server, provider, route, or registry mutation.

Artifact and execution evidence

• Status reports the immutable server and model identities, selected recipe, host profile, lifecycle state, load/readiness state, and route health without exposing a credential.
• A managed server starts without network access after its image and GGUF file are present.
• Runtime evidence proves that the selected process uses the allocated NVIDIA GPU. CPU-only fallback fails with an actionable diagnostic.
• Insufficient GPU memory, disk space, driver compatibility, checksum mismatch, model-template mismatch, and port conflict fail before registration or leave a complete rollback.
• Restart and recovery use the recorded image and model digests. They do not resolve a mutable upstream tag again.
• Recovery reads the gateway-scoped secret state. It does not depend on the original shell environment.

Agent compatibility

• Every supported tuple records the agent, exact model, GGUF quantization, chat template, context, llama.cpp image digest, platform, and container runtime.
• Each tuple completes a normal response, streaming response, structured function call, tool-result continuation, and multi-turn agent flow through inference.local.
• The validation checks the emitted tool-call structure and the agent's execution of the intended tool. Server startup alone is not sufficient.
• Selection, recovery, and sandbox configuration set openai-completions and replace any retained Responses preference.
• The v1 path does not advertise OpenAI Responses API parity, embeddings, reranking, multimodal input, or parallel tool calls.
• Deep Agents Code re-onboards when its provider changes. Its llama.cpp qualification remains separate from OpenClaw and Hermes.

Security and lifecycle

• No supported configuration exposes the server, health endpoints, model inventory, UI, or slot state to an untrusted network.
• The runtime is non-root, the model mount is read-only, and server-side filesystem or shell tools are disabled.
• Credentials and gated-model tokens do not appear in process arguments, logs, snapshots, registry output, or sandbox configuration.
• Ownership checks prevent cleanup from deleting foreign or shared resources.
• Rebuild, snapshot/restore, clone, destroy, and uninstall have behavior tests and live evidence for every supported runtime.
• The existing generic compatible-endpoint flow and current Ollama, vLLM, and NVIDIA NIM paths do not regress.

Constraints and non-goals

The first supported release does not include:

• arbitrary model URLs, unverified GGUF files, or runtime -hf model resolution;
• a custom llama.cpp fork;
• CPU-only, Metal, Vulkan, ROCm, Windows-native, or macOS-native execution;
• multi-GPU, distributed, multi-model, router, hot-swap, or multi-tenant serving;
• server-side agents, MCP, filesystem tools, shell tools, Web UI, or public ingress;
• embeddings, reranking, multimodal input, or general llama.cpp flag passthrough;
• OpenAI Responses API compatibility claims;
• automatic conversion from another model format to GGUF;
• a product-support claim based only on generic compatible-endpoint reachability.

These items may use the self-managed compatible-endpoint path or a later accepted issue.

Decisions required before activation

• Select the first model, exact GGUF file, quantization, chat template, context, and license.
• Select the first linux/amd64 and linux/arm64 NVIDIA host profiles.
• Set measurable startup, response, memory, and recovery thresholds for each tuple.
• Confirm the [Epic] Support native Podman with buildless managed onboarding #7744 dependency and protected E2E evidence required for later Podman activation.
• Select the owned image registry path and publication workflow.
• Confirm the exact cooperative fingerprint signals.

Related work

• Discussion #7636: managed inference serving specifications.

• #7407: versioned system-readiness report and qualifications.

• #7744: buildless onboarding and runtime-neutral Docker/Podman lifecycle.

• #6401: onboarding architecture.

• #2412 and PR #5505: current self-managed llama.cpp/GGUF path.

• #7209: generic compatible-endpoint validation record.

• PR #498: closed incomplete custom-fork provider attempt. This epic uses upstream llama.cpp and is not a continuation of that fork.

• #8151 and PR #8152: merged declarative serving catalog/compiler foundation.

• #8161 and PR #8167: merged cooperative existing-server attachment.

• #8181 and PR #8183: merged typed declarative llama.cpp catalog contract.

• #8173 and PR #8224: merged production repository-owned llama.cpp YAML recipe and preset.

• #8231: owned multiarchitecture llama.cpp server image publication and GPU evidence.

• PR #8060: merged provider-neutral host-local inference receipts.

• PR #8058: separate broader provider lifecycle work; not a dependency of llama.cpp slice 6.

Definition of done

Close this epic only when all accepted child issues are complete and the exact shipped artifacts pass the full acceptance criteria above. Generic compatible-endpoint connectivity, unit tests, or one successful prompt do not complete the epic.

[ericksoa]

Implementation direction for the weekly rollout:

• Target the complete accepted stack within one week. Keep each change independently reviewable and dependency ordered.
• Treat the serving catalog/compiler as the first prerequisite slice. Do not add llama.cpp-specific lifecycle behavior before that contract lands.
• Author every llama.cpp serving setting in repository-owned declarative YAML. This includes artifact identities, model behavior, protocol capabilities, ports, resources, request bounds, launch settings, health checks, security controls, and materializer/lifecycle adapter bindings.
• TypeScript may validate trusted repository YAML, compile deterministic canonical JSON, resolve a preset against the versioned readiness report, and execute an allowlisted adapter. It must not duplicate serving defaults in code or environment-variable tables.
• YAML must not contain shell fragments, executable steps, arbitrary command templates, or unbounded flag passthrough. Runtime paths must consume compiled trusted artifacts, not operator-supplied YAML.
• The catalog/compiler slice adds no llama.cpp recipe, onboarding option, default, lifecycle ownership, or support claim. It only establishes the contract required by the managed slices.

After the prerequisite merges, continue one slice at a time through server attachment, llama.cpp recipes, owned artifacts, GGUF acquisition, lifecycle, routing, qualification, protected E2E, and documentation. Support activation remains a separate maintainer decision after the evidence gates pass.