Repository navigation
Add support for Reflection Beam - #286
Conversation
…y & High Throughput)
The High Interactivity recipe runs 8 aggregated workers (prefillWorker.replicas: 0), so it belongs under inference/a4xmax/aggregated-serving/dynamo/ with its own README. - Ship one worker serving config (laser_beam_high_int_agg.yaml); drop the unused prefill copy. - Replace the hardcoded Beam HF cache snapshot path in the shared chart with an optional workload.modelPath value. - Install aiperf from a pinned upstream commit: the 0.13.0 PyPI release lacks --warmup-requests-per-lane. Drop AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES, which no aiperf version reads. - Remove the LASER_BEAM High Interactivity section from the disaggregated README and link to the aggregated recipe.
Introduce the generic llm-d + vLLM 0.19.1rc1 aggregated serving recipe on GKE G4 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB) for reflection/Beam-1.0-501B-A23B-NVFP4 (LASER_BEAM) at 1,048,576 (1M) token context length: - src/helm-charts/g4/llm-d-inference: Generic, cluster-agnostic Helm chart packaging the llm-d Router (Envoy + llm-d-epp v0.5.1 with PrefixCachePlugin, UtilizationDetector, TokenLoadScorer, MaxScorePicker, and self-contained ServiceAccount/RBAC templates) alongside N x TP=4/TP=8 vLLM model servers with host-DRAM KV cache offloading (OffloadingConnector / CPUOffloadingSpec) and NUMA pinning. - src/frameworks/g4/llmd-configs: Reference EndpointPickerConfig and vLLM 1M-context configuration profiles. - inference/g4/aggregated-serving/llm-d: Recipe values.yaml, namespace-agnostic standalone Kubernetes manifests (router.yaml + modelserver-2x-tp4-1m.yaml), AgentX 60-minute benchmark client manifest (bench_client.yaml), and complete documentation + 1M-context benchmark results in README.md.
Add A4X-MAX Dynamo recipes for Beam-1.0-501B NVFP4 (High Interactivity & High Throughput)
Add Helm-based G4 llm-d + vLLM aggregated serving recipe for Beam
…ng) and G4 (llm-d + vLLM) - Rename configuration and Helm values files to `beam_*` and use public model name Reflection Beam (`reflection/Beam-1.0-501B-A23B-NVFP4` and `reflection/Beam-1.0-501B-A23B-DFlash2`) - Remove benchmark performance tables from public recipe READMEs - Update container image references to `lmsysorg/sglang:beam` and `vllm/vllm-openai:reflection-beam`
Add Reflection Beam 501B inference recipes for A4X-MAX (Dynamo + SGLa…
Reflection beam recipes
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces new inference recipes and Helm charts for serving Large Language Models on GKE, specifically adding an aggregated multi-node recipe with NVIDIA Dynamo on A4X-MAX and a 1M-context multi-turn recipe with llm-d and vLLM on G4. Key feedback highlights several critical issues: a template rendering panic risk due to unsafe navigation of nested values, an invalid vLLM subcommand and missing CPU device flag for the background token-producer, and an incorrect volume name that prevents GKE GCS FUSE from utilizing local SSD caching. Additionally, it is recommended to avoid running background processes inside the main container and to pre-install packages like hf_transfer in the base image rather than installing them at runtime.
| {{- if (.Values.dynamo.rdma).claimTemplateName }} | ||
| - name: rdma | ||
| resourceClaimTemplateName: all-mrdma | ||
| resourceClaimTemplateName: {{ .Values.dynamo.rdma.claimTemplateName }} | ||
| {{- end }} |
There was a problem hiding this comment.
Safe navigation is required when accessing (.Values.dynamo.rdma).claimTemplateName to prevent nil pointer evaluation errors if dynamo.rdma is not defined.
{{- if and .Values.dynamo (and .Values.dynamo.rdma .Values.dynamo.rdma.claimTemplateName) }}
- name: rdma
resourceClaimTemplateName: {{ .Values.dynamo.rdma.claimTemplateName }}
{{- end }}| {{- if (.Values.dynamo.rdma).claimTemplateName }} | ||
| - name: rdma | ||
| resourceClaimTemplateName: all-mrdma | ||
| resourceClaimTemplateName: {{ .Values.dynamo.rdma.claimTemplateName }} | ||
| {{- end }} |
There was a problem hiding this comment.
Safe navigation is required when accessing (.Values.dynamo.rdma).claimTemplateName to prevent nil pointer evaluation errors if dynamo.rdma is not defined.
{{- if and .Values.dynamo (and .Values.dynamo.rdma .Values.dynamo.rdma.claimTemplateName) }}
- name: rdma
resourceClaimTemplateName: {{ .Values.dynamo.rdma.claimTemplateName }}
{{- end }}| - name: local-ssd | ||
| emptyDir: {} | ||
| hostPath: | ||
| path: /mnt/disks/gke-local-ssd/gcsfuse-cache | ||
| type: DirectoryOrCreate |
There was a problem hiding this comment.
The GKE GCS FUSE CSI driver specifically looks for a volume named gke-gcsfuse-cache to mount as the local SSD cache directory in the sidecar container. Using the name local-ssd will prevent the CSI driver from utilizing the local SSD for caching, causing it to fallback to slow ephemeral storage or fail due to disk space limitations.
- name: gke-gcsfuse-cache
hostPath:
path: /mnt/disks/gke-local-ssd/gcsfuse-cache
type: DirectoryOrCreate| echo "--- VERIFYING NCCL ENV VARS IN SHELL ---" | ||
| env | grep NCCL_ | ||
| echo "--- END VERIFICATION ---" | ||
| pip install hf_transfer --break-system-packages || true |
There was a problem hiding this comment.
Installing hf_transfer at runtime via pip install inside the container startup script is a reliability and security risk. It requires internet access during startup, slows down container initialization, and can cause non-deterministic startup failures if PyPI or the network is unavailable. It is highly recommended to pre-install hf_transfer in the base container image instead.
| echo "--- VERIFYING NCCL ENV VARS IN SHELL ---" | ||
| env | grep NCCL_ | ||
| echo "--- END VERIFICATION ---" | ||
| pip install hf_transfer --break-system-packages || true |
There was a problem hiding this comment.
Installing hf_transfer at runtime via pip install inside the container startup script is a reliability and security risk. It requires internet access during startup, slows down container initialization, and can cause non-deterministic startup failures if PyPI or the network is unavailable. It is highly recommended to pre-install hf_transfer in the base container image instead.
Adds recipes supporting Reflection Beam across the following configurations:
A4X-Max + Dynamo + sgLang
G4 + llm-d + vLLM