FEAT: support oci:// model URIs via llmman serve

[ericcurtin]

Motivation

model_uri accepted only file://. This adds oci://, so a model published as a CNCF ModelPack artifact in a container registry can be registered as a custom model:

{"model_name": "my-model", "model_specs": [{"model_uri": "oci://ghcr.io/org/model:tag", "...": "..."}]}

The same registries, credentials, mirroring and air-gap tooling a deployment already uses for container images then apply to models.

Implementation

• xinference/model/llmman.py (new): stdlib-only client for a running llmman serve daemon, so no new dependency. llmman already implements the ModelPack media types, registry auth, resumable blob download and a content-addressed store, so none of that is reimplemented. GET /api/version probes reachability and identity; POST /api/pull streams NDJSON so a multi-gigabyte fetch is not silent (an error can arrive in-band at HTTP 200, and a stream ending without success is a failure too); llmman resolve --no-pull reports where the bytes landed, keeping the daemon the only thing that touches the network.
• xinference/model/oci_utils.py (new): the oci:// scheme handling.
• xinference/model/utils.py: scheme dispatch is centralized in resolve_model_uri(), used by CacheManager._cache_from_uri(), LLMCacheManager.cache_uri() and cache_from_uri() — so LLM, embedding and rerank all accept oci://. The cache entry is a symlink into llmman's store, exactly the shape the file:// branch produces, so everything downstream is unchanged.

Only oci:// is claimed: file://, bare paths and every other scheme reach exactly the branch they did before. A pull needs both the daemon reachable and the binary on PATH (or XINFERENCE_LLMMAN_BIN); each missing piece has its own actionable error, and neither is required unless an oci:// URI is used.

Review feedback addressed

• LLMCacheManager.cache_uri() — the production path, also reused by the embedding and rerank cache managers — is wired up, not just the unused utils.cache_from_uri(). Covered by test_llm_cache_manager_caches_an_oci_uri, which fails without that change.
• codespell: unparseable -> unparsable.
• endpoint() now keeps the LLMMAN_HOST scheme long enough to apply llmman's own default-port rules (http:// -> 80, https:// -> 443, anything else -> 17434) and falls back to the default port for an out-of-range or non-numeric one, as llmman's u16 parse does. The origin stays HTTP either way, matching llmman's daemon client. Regression cases added for http://host, https://host and rejected ports.

Testing

pytest xinference/model/tests/test_llmman.py xinference/model/tests/test_oci_utils.py   # 43 passed

test_llmman.py runs against a real HTTP server on a loopback port, not mocks, so the NDJSON streaming contract is genuinely exercised: /api/version accepted and a non-llmman server rejected, nothing listening reported actionably; pull success with forwarded byte progress and the exact request body asserted; an in-band error at HTTP 200; a stream ending without success; a non-JSON diagnostic tolerated.

test_oci_utils.py covers parse_uri, the resolve output contract plus eight malformed-output cases, every LLMMAN_HOST form (scheme default ports, bad ports, wildcard-to-loopback, IPv6), binary default/override and the missing-binary error, an empty reference rejected without touching the daemon, and LLMCacheManager.cache() end to end on an oci:// URI.

xinference/model/tests also run with no new failures; the remaining failures there are pre-existing in this environment (missing modelscope) and reproduce on main. pre-commit is clean on the changed files.

Not covered: an end-to-end launch against a live llmman serve backed by a real registry.

Docs

A model_uri note in doc/source/models/custom.rst and XINFERENCE_LLMMAN_BIN in doc/source/getting_started/environments.rst, with catalog entries for all nine locales, compiled with python doc/build_i18n.py --all and built in each language.

Disclosure: written with AI assistance.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[qinxuye]

The actual LLM custom-model path does not use the xinference.model.utils.cache_from_uri function changed here. create_llm_model_instance constructs LLMCacheManager, and LLMCacheManager.cache_uri() still accepts only file://; an oci://ghcr.io/org/model:tag spec therefore fails immediately with ValueError: Unsupported URL scheme: oci before llmman is called. The llama.cpp embedding and rerank cache managers reuse this same method.

Please wire OCI resolution into LLMCacheManager.cache_uri() (or centralize the URI handling so both paths share it), and add a test that calls LLMCacheManager.cache() with an OCI URI to cover the production path.

Also, the changed-file pre-commit run currently fails codespell at xinference/model/llmman.py:115: unparseable should be unparsable.

[ericcurtin]

Thanks @qinxuye, both points were addressed.

[qinxuye]

The pull and resolve steps can target different stores. pull() sends /api/pull to the daemon selected by LLMMAN_HOST, but resolve() then runs a local llmman resolve --no-pull; upstream resolve reads the local process's LLMMAN_MODELS/default store and never contacts that daemon. Therefore a remote daemon, or a local daemon started with a different LLMMAN_MODELS, can complete the pull successfully and then fail here with "not found in local store." Please make resolution use the same daemon/store and return a path accessible to Xinference, or explicitly constrain and validate the co-located shared-store invariant. Please also add a regression test with distinct daemon and client stores; mocking pull_and_resolve currently hides this mismatch.