Xinference makes it easy to self-host open-source models, but some workloads still need models that are impractical to host locally: closed frontier models (GPT-5, Gemini), models requiring serious GPU resources (video generation, large multimodal), or teams that simply don't want to manage hardware for every model.
I built AnyLLM, an OpenAI-compatible API gateway with 24+ Chinese and international models behind a single key: DeepSeek V4, Qwen3.7-Plus, GLM-5.2, Kimi K2.5, MiniMax M3 — alongside GPT-5 and Gemini. International cards and crypto accepted, pass-through pricing with zero markup, built-in routing and failover.
Why this complements Xinference
Same API surface. Both expose OpenAI-compatible endpoints with Function Calling, so apps built on Xinference can call AnyLLM for frontier models with just a base URL change — no SDK changes, no new client code.
Hybrid deployment. Keep latency-sensitive or privacy-critical models self-hosted in Xinference; route frontier or burst-capacity calls to AnyLLM. One consistent API across both.
2M context without 2M GPUs. GLM-5.2's 2M-token context window is out of reach for most self-hosters — it's one API call away via AnyLLM.
Proposal
1.A short docs section: "Hybrid deployment: combining Xinference with an API gateway" using AnyLLM as the example.
2.Or a mention on the ecosystem/integrations page alongside Dify, RAGFlow, and LangChain.
Happy to open a PR for the docs section.
(Context: I'm the developer of AnyLLM — happy to provide a sandbox key for testing.)
Xinference makes it easy to self-host open-source models, but some workloads still need models that are impractical to host locally: closed frontier models (GPT-5, Gemini), models requiring serious GPU resources (video generation, large multimodal), or teams that simply don't want to manage hardware for every model.
I built AnyLLM, an OpenAI-compatible API gateway with 24+ Chinese and international models behind a single key: DeepSeek V4, Qwen3.7-Plus, GLM-5.2, Kimi K2.5, MiniMax M3 — alongside GPT-5 and Gemini. International cards and crypto accepted, pass-through pricing with zero markup, built-in routing and failover.
Why this complements Xinference
Same API surface. Both expose OpenAI-compatible endpoints with Function Calling, so apps built on Xinference can call AnyLLM for frontier models with just a base URL change — no SDK changes, no new client code.
Hybrid deployment. Keep latency-sensitive or privacy-critical models self-hosted in Xinference; route frontier or burst-capacity calls to AnyLLM. One consistent API across both.
2M context without 2M GPUs. GLM-5.2's 2M-token context window is out of reach for most self-hosters — it's one API call away via AnyLLM.
Proposal
1.A short docs section: "Hybrid deployment: combining Xinference with an API gateway" using AnyLLM as the example.
2.Or a mention on the ecosystem/integrations page alongside Dify, RAGFlow, and LangChain.
Happy to open a PR for the docs section.
(Context: I'm the developer of AnyLLM — happy to provide a sandbox key for testing.)