Skip to content

ci: wire GPU checks to patched vLLM nightly image - #183

Merged
yubofredwang merged 1 commit into
mainfrom
codex/fix-nightly-vllm-pp-ci-image
Aug 27, 2026
Merged

ci: wire GPU checks to patched vLLM nightly image#183
yubofredwang merged 1 commit into
mainfrom
codex/fix-nightly-vllm-pp-ci-image

Conversation

@yubofredwang

Copy link
Copy Markdown
Collaborator

Summary

The GPU CI dispatcher currently launches with VLLM_USE_V2_MODEL_RUNNER=1, but the nightly Docker publisher still builds the old v0.22.1 image. That image does not support the extract_hidden_states V2 path, causing the GPU smoke to fail during vLLM configuration before training starts.

This PR:

  • switches the nightly publisher to the validated e9d1398d9edfd90fcc1cf783805240e3effec013 patched vLLM Dockerfile;
  • publishes the immutable nightly-<short-sha>-vllm-pp and moving nightly-vllm-pp tags;
  • records the TorchSpec source SHA in the image label;
  • adds a trusted Slurm preflight that checks the vLLM build commit, base image tag, V2 runner setting, and extract-hidden-states patch markers;
  • documents that TORCHSPEC_CI_IMAGE must point to an ARM64 .sqsh made from nightly-vllm-pp.

Validation

  • git diff --check
  • bash -n tools/ci/dispatch_slurm.sh tools/ci/gpu_2gpu.sbatch
  • embedded runtime Python syntax compilation
  • YAML parsing for all three affected workflows

Required post-merge infrastructure update

The repository secret TORCHSPEC_CI_IMAGE must be updated to the Inkwell ARM64 .sqsh imported from lightseekorg/torchspec:nightly-vllm-pp. The new preflight will report the expected and actual image metadata if the old .sqsh remains configured.

Signed-off-by: Yubo Wang <yubowang2019@gmail.com>
@yubofredwang
yubofredwang merged commit 43dec00 into main Aug 27, 2026
1 check passed
@yubofredwang
yubofredwang deleted the codex/fix-nightly-vllm-pp-ci-image branch August 27, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant