Skip to content

docs: deploy LitGPT on Nebius Serverless Endpoints - #2342

Open
SalikovAlex wants to merge 2 commits into
Lightning-AI:mainfrom
SalikovAlex:codex/nebius-serverless-endpoints
Open

SalikovAlex wants to merge 2 commits into
Lightning-AI:mainfrom
SalikovAlex:codex/nebius-serverless-endpoints

Conversation

@SalikovAlex

@SalikovAlex SalikovAlex commented Sep 28, 2026 •

Copy link
Copy Markdown

LitGPT's serving guide does not show how to deploy its OpenAI-compatible API to an authenticated Nebius Serverless Endpoint. This adds a linked tutorial and a self-contained container recipe using SmolLM2-135M-Instruct, pinned runtime dependencies and model revision, managed HTTPS with bearer-token authentication, readiness and streaming examples, and cleanup instructions.

Related scope proposal: #2341. This remains a draft for feedback on provider-specific placement.

Live validation exposed a deployment constraint: the endpoint service rejects image references longer than 64 characters with a Compute label error, including normal digest references. The tutorial therefore uses a short unique tag, verifies its remote digest against the build output, and explains that the tag must not be overwritten. It also covers Nebius Container Registry within the same project.

Validation:

  • Built the Linux/AMD64 Docker image, including dependency checks and conversion of the pinned Hugging Face model revision.
  • Passed local CPU readiness, JSON generation and SSE streaming, including inside the built image with networking disabled.
  • Ran one H100 endpoint in Nebius eu-north1 with CLI 0.12.279, LitGPT 0.5.13, LitServe 0.2.19 and PyTorch 2.7.1 on September 28, 2026. Logs confirmed CUDA model initialization.
  • Authenticated /health: HTTP 200 ok. Missing and incorrect tokens: HTTP 401 on both health and chat-completions routes.
  • Authorized chat completion: nonempty JSON response. Streaming: SSE content and terminal [DONE].
  • Deleted the endpoint after the smoke test and confirmed its absence. Removed both temporary registry image manifests. No test endpoint remains.
  • Passed repository pre-commit hooks, shell syntax checks for all 12 tutorial Bash blocks, local link checks and git diff --check.

This is a single-GPU deployment smoke test, not a throughput or scaling benchmark.

@SalikovAlex
SalikovAlex marked this pull request as ready for review September 28, 2026 14:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant