From 0a88f56480a69d1a8607919c040e36693db814b4 Mon Sep 17 00:00:00 2001 From: Christopher Pirillo Date: Thu, 15 Oct 2026 23:29:56 +0000 Subject: [PATCH 1/8] Staging muh lazers --- .../disaggregated-serving/dynamo/README.md | 293 ++++++++++++++---- .../dynamo/bench_client.yaml | 25 +- 2 files changed, 256 insertions(+), 62 deletions(-) diff --git a/inference/a4xmax/disaggregated-serving/dynamo/README.md b/inference/a4xmax/disaggregated-serving/dynamo/README.md index e21bf32d..2a26ea81 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/README.md +++ b/inference/a4xmax/disaggregated-serving/dynamo/README.md @@ -1,8 +1,8 @@ # Disaggregated Multi-Node Inference with NVIDIA Dynamo on A4X-MAX GKE -This document outlines the steps to deploy and serve Large Language Models (LLMs) using [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) disaggregated inference platform on [A4X-MAX (GB300) GKE Node pools](https://cloud.google.com/kubernetes-engine). +This document outlines the steps to deploy and benchmark Large Language Models (LLMs) using the [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) disaggregated inference platform on [A4X-MAX (GB300) GKE Node pools](https://cloud.google.com/kubernetes-engine). -Dynamo provides a disaggregated architecture that separates prefill and decode operations for optimized inference performance, supporting both single-node (4 GPUs) and multi-node NVL72 (72 GPUs) configurations. Dynamo also supports various inference framework backends like [vLLM](https://docs.nvidia.com/dynamo/latest/components/backends/vllm/README.html) and [SGLang](https://docs.nvidia.com/dynamo/latest/components/backends/sglang/README.html). In this recipe, we will focus on serving using the SGLang backend. +Dynamo provides a disaggregated architecture that separates prefill and decode operations for optimized inference performance, supporting multi-node configurations from 2 nodes (8 GPUs) up to full NVL72 domains (18 nodes / 72 GPUs). Dynamo supports multiple inference framework backends including [vLLM](https://docs.nvidia.com/dynamo/latest/components/backends/vllm/README.html) and [SGLang](https://docs.nvidia.com/dynamo/latest/components/backends/sglang/README.html). In this recipe, we focus on serving with the SGLang backend and profiling across both agentic multi-turn (`aiperf` / `agentx`) and synthetic workloads. ## Table of Contents @@ -15,11 +15,16 @@ Dynamo provides a disaggregated architecture that separates prefill and decode o * [2.4. Create Secrets](#create-secrets) * [2.5. Install Dynamo Platform](#install-platform) * [2.6. Setup GCS Bucket for GKE](#setup-gcsfuse) - * [2.7. Configure Dynamo Image ](#configure-dynamo-image) + * [2.7. Configure Dynamo Image](#configure-dynamo-image) * [3. Deploy with SGLang Backend](#deploy-sglang) - * [3.1. SGLang Deployment without DeepEP(8 GPUs)](#sglang-wo-deepep) - * [3.2. SGLang Deployment with DeepEP(72 GPUs)](#sglang-deepep) -* [4. Inference Request](#inference-request) + * [3.1. Supported Models and Deployment Categories](#supported-models) + * [3.2. Category 1: High Interactivity / Low Latency Deployment](#sglang-high-interactivity) + * [3.3. Category 2: High Throughput Multi-Node Deployment](#sglang-high-throughput) +* [4. Inference Request and Benchmarking](#inference-request) + * [4.1. Verify Deployment Readiness](#verify-readiness) + * [4.2. Deploy the Benchmark Client](#deploy-bench-client) + * [4.3. Category 1: Agentic Multi-Turn Profiling with AIPerf (AgentX Dataset)](#benchmark-aiperf-agentx) + * [4.4. Category 2: Synthetic Random-Dataset Benchmarking](#benchmark-synthetic) * [5. Monitoring and Troubleshooting](#monitoring) * [6. Cleanup](#cleanup) @@ -32,8 +37,8 @@ This recipe has been tested with the following configuration: * **GKE Cluster**: * GPU node pools with [a4x-maxgpu-4g-metal](https://docs.cloud.google.com/compute/docs/gpus#gb300-gpus) machines: - * For 1p1d deployment: 2 machines with 4 GPUs each (8 GPUs total) - * For 10p8d deployment: 18 machines with 4 GPUs each (72 GPUs total) + * For 1P1D low-latency deployment: 2 machines with 4 GPUs each (8 GPUs total) + * For multi-node high-throughput deployment (e.g., 4P4D / 3P5D on 32 GPUs, or 10P8D on 72 GPUs): 8 to 18 machines with 4 GPUs each (32 to 72 GPUs total) * [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) enabled * [Cloud Storage FUSE CSI driver for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cloud-storage-fuse-csi-driver) enabled @@ -49,7 +54,7 @@ This recipe has been tested with the following configuration: git clone https://github.com/ai-hypercomputer/gpu-recipes.git cd gpu-recipes export REPO_ROOT=$(pwd) -export RECIPE_ROOT=$REPO_ROOT/inference/a4x/disaggregated-serving/dynamo +export RECIPE_ROOT=$REPO_ROOT/inference/a4xmax/disaggregated-serving/dynamo ``` @@ -63,7 +68,7 @@ export NAMESPACE=dynamo-cloud export NGC_API_KEY= export HF_TOKEN= export RELEASE_VERSION=1.0.1 -export GCS_BUCKET= +export GCS_BUCKET= # Set the project for gcloud commands gcloud config set project $PROJECT_ID @@ -75,10 +80,10 @@ Replace the following values: | -------- | ----------- | ------- | | `PROJECT_ID` | Your Google Cloud Project ID | `gcp-project-12345` | | `CLUSTER_REGION` | The GCP region where your GKE cluster is located | `us-central1` | -| `CLUSTER_NAME` | The name of your GKE cluster | `a4x-cluster` | +| `CLUSTER_NAME` | The name of your GKE cluster | `a4xmax-cluster` | | `NGC_API_KEY` | Your NVIDIA NGC API key (get from [NGC](https://ngc.nvidia.com)) | `nvapi-xxx...` | | `HF_TOKEN` | Your Hugging Face access token | `hf_xxx...` | -| `GCS_BUCKET` | Your GCS bucket name | `gs://xxx` | +| `GCS_BUCKET` | Your GCS bucket name (without `gs://` prefix for Helm `bucketName`) | `my-model-bucket` | ### 2.3. Connect to your GKE Cluster @@ -133,7 +138,7 @@ Verify the installation: kubectl get pods -n ${NAMESPACE} ``` -Before moving forward, verify that all platform Pods have successfully transitioned to the `Running` state. The expected output should look similar to this, with every pod fully `Ready` and `Running`: +Before moving forward, verify that all platform Pods have transitioned to the `Running` state. The expected output should look similar to this, with every pod `Ready` and `Running`: ```bash $ kubectl get pods NAME READY STATUS RESTARTS AGE @@ -159,7 +164,7 @@ Find the service account (usually annotated to default): kubectl get serviceaccounts ${NAMESPACE} -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.iam\.gke\.io/gcp-service-account}{"\n"}{end}' ``` -Config the service account email: +Configure the service account email: ```bash export SERVICE_ACCOUNT_EMAIL=$(kubectl get serviceaccount/default -n ${NAMESPACE} -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}') ``` @@ -168,25 +173,25 @@ Authorize the service account: ```bash gcloud iam service-accounts add-iam-policy-binding ${SERVICE_ACCOUNT_EMAIL} \ --role roles/iam.workloadIdentityUser \ - --member "serviceAccount:project_id.svc.id.goog[${NAMESPACE}/default]" + --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${NAMESPACE}/default]" ``` Grant read access to the bucket: ```bash -gcloud storage buckets add-iam-policy-binding ${GCS_BUCKET} \ +gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \ --member "serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \ --role "roles/storage.objectViewer" ``` -Downloading model files into the gcs bucket and set your gcs bucket name in values.yaml file. +Stage your model checkpoint files into the GCS bucket so they mount under `/data/model/` via GCS FUSE. ### 2.7. Configure Dynamo Image -Instead of building the Dynamo container image manually, we will use the official pre-built SGLang runtime release artifact. +Instead of building the Dynamo container image manually, use the pre-built SGLang runtime release artifact (or a custom model-specific runtime image if required by your model). For more details, refer to the [NVIDIA Release Artifacts documentation](https://docs.nvidia.com/dynamo/dev/resources/release-artifacts#sglang-runtime). -Set your environment to use the pre-built image: +Set your environment to use the runtime image: ```bash export DYNAMO_IMAGE="nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.0.1-cuda13" ``` @@ -196,61 +201,150 @@ export DYNAMO_IMAGE="nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.0.1-cuda13" [Back to Top](#table-of-contents) -Deploy Dynamo with SGLang backend for high-performance inference. + +### 3.1. Supported Models and Deployment Categories - -### 3.1. SGLang Deployment without DeepEP (8 GPUs) +This recipe is organized into deployment categories that pair a Helm topology (`values_*.yaml`) with model-specific prefill and decode serving configurations under `$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/`. You can deploy any supported model by setting its `HF_MODEL_ID`, `VALUES_FILE`, `PREFILL_CONFIG`, and `DECODE_CONFIG`. -Two nodes deployment uses 8 GPUs across 2 A4X-MAX machines, targeting low latency. +| Model Name | Hugging Face ID / Path (`HF_MODEL_ID`) | Deployment Category | Topology & GPUs | Helm Values File (`VALUES_FILE`) | Prefill Config (`PREFILL_CONFIG`) | Decode Config (`DECODE_CONFIG`) | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| **DeepSeek-R1 671B (FP8)** | `deepseek-ai/DeepSeek-R1` | Low-Latency / without DeepEP | 1P1D (2 nodes / 8 GPUs) | `values_wo_deepep.yaml` | `deepseek-r1-fp8-1p1d-prefill.yaml` | `deepseekr1-fp8-1p1d-decode.yaml` | +| **DeepSeek-R1 671B (FP8)** | `deepseek-ai/DeepSeek-R1` | High-Throughput / with DeepEP | 10P8D (18 nodes / 72 GPUs) | `values_deepep.yaml` | `deepseekr1-fp8-10p8d-prefill.yaml` | `deepseekr1-fp8-10p8d-decode.yaml` | +| **LASER_BEAM** | `` | High Interactivity | 1P1D (2 nodes / 8 GPUs) | `laser_beam_high_int.yaml` | `laser_beam_high_int_prefill.yaml` | `laser_beam_high_int_decode.yaml` | +| **LASER_BEAM** | `` | High Throughput | Multi-Node (32–72 GPUs) | `laser_beam_high_throughput.yaml` | `laser_beam_high_throughput_prefill.yaml` | `laser_beam_high_throughput_decode.yaml` | -#### DeepSeekR1 671B Model +> [!TIP] +> To add or deploy another model, stage its weights in `gs://${GCS_BUCKET}`, add its prefill and decode YAML files under `$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/`, and pass `--set workload.model=${HF_MODEL_ID}` along with `--set-file prefill_serving_config=...` and `--set-file decode_serving_config=...` to `helm install`. -Deploy DeepSeekR1-671B across 2 nodes for testing and validation. Note the use of `--set-file prefill_serving_config` and `--set-file decode_serving_config` pointing to the correct model config file. + +### 3.2. Category 1: High Interactivity / Low Latency Deployment + +This category deploys A4X-MAX nodes targeting low Time-to-First-Token (TTFT) and low Time-Per-Output-Token (TPOT) for interactive workloads. + +#### DeepSeek-R1 671B (FP8) + +1. Set the model and configuration variables: + +```bash +export HF_MODEL_ID="deepseek-ai/DeepSeek-R1" +export VALUES_FILE="values_wo_deepep.yaml" +export PREFILL_CONFIG="deepseek-r1-fp8-1p1d-prefill.yaml" +export DECODE_CONFIG="deepseekr1-fp8-1p1d-decode.yaml" +export RELEASE_NAME="$USER-dynamo-a4xmax-1p1d" +``` + +2. Install the Helm release: + +```bash +cd $RECIPE_ROOT +helm install -f ${VALUES_FILE} \ + --set workload.model=${HF_MODEL_ID} \ + --set workload.image=${DYNAMO_IMAGE} \ + --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ + --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${PREFILL_CONFIG} \ + --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${DECODE_CONFIG} \ + ${RELEASE_NAME} \ + $REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment +``` + +#### LASER_BEAM (High Interactivity) + +1. Set the model and configuration variables: + +```bash +export HF_MODEL_ID="" +export VALUES_FILE="laser_beam_high_int.yaml" +export PREFILL_CONFIG="laser_beam_high_int_prefill.yaml" +export DECODE_CONFIG="laser_beam_high_int_decode.yaml" +export RELEASE_NAME="$USER-dynamo-a4xmax-laser-beam-high-int" +``` + +2. Install the Helm release: + +```bash +cd $RECIPE_ROOT +helm install -f ${VALUES_FILE} \ + --set workload.model=${HF_MODEL_ID} \ + --set workload.image=${DYNAMO_IMAGE} \ + --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ + --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${PREFILL_CONFIG} \ + --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${DECODE_CONFIG} \ + ${RELEASE_NAME} \ + $REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment +``` + + +### 3.3. Category 2: High Throughput Multi-Node Deployment + +This category deploys multi-node A4X-MAX topologies across an NVL72 ComputeDomain to maximize total and output token throughput at high concurrency. + +#### DeepSeek-R1 671B (FP8) + +1. Set the model and configuration variables: + +```bash +export HF_MODEL_ID="deepseek-ai/DeepSeek-R1" +export VALUES_FILE="values_deepep.yaml" +export PREFILL_CONFIG="deepseekr1-fp8-10p8d-prefill.yaml" +export DECODE_CONFIG="deepseekr1-fp8-10p8d-decode.yaml" +export RELEASE_NAME="$USER-dynamo-a4xmax-multi-node" +``` + +2. Install the Helm release: ```bash cd $RECIPE_ROOT -helm install -f values_wo_deepep.yaml \ ---set workload.image=${DYNAMO_IMAGE} \ ---set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ ---set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml \ ---set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml \ -$USER-dynamo-a4xmax-1p1d \ -$REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment +helm install -f ${VALUES_FILE} \ + --set workload.model=${HF_MODEL_ID} \ + --set workload.image=${DYNAMO_IMAGE} \ + --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ + --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${PREFILL_CONFIG} \ + --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${DECODE_CONFIG} \ + ${RELEASE_NAME} \ + $REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment ``` - -### 3.2. SGLang Deployment with DeepEP (72 GPUs) +#### LASER_BEAM (High Throughput) -Multi-node deployment uses 72 GPUs across 18 A4X-MAX machines, providing increased capacity for larger models or higher throughput. +1. Set the model and configuration variables: -#### DeepSeekR1 671B Model +```bash +export HF_MODEL_ID="" +export VALUES_FILE="laser_beam_high_throughput.yaml" +export PREFILL_CONFIG="laser_beam_high_throughput_prefill.yaml" +export DECODE_CONFIG="laser_beam_high_throughput_decode.yaml" +export RELEASE_NAME="$USER-dynamo-a4xmax-laser-beam-high-throughput" +``` -Deploy DeepSeekR1-671B across 18 nodes for production workloads. Note the use of `--set-file prefill_serving_config` and `--set-file decode_serving_config` pointing to the correct model config file for a multi node deployment scenario: +2. Install the Helm release: ```bash cd $RECIPE_ROOT -helm install -f values_deepep.yaml \ ---set workload.image=${DYNAMO_IMAGE} \ ---set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ ---set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml \ ---set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml \ -$USER-dynamo-a4xmax-multi-node \ -$REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment +helm install -f ${VALUES_FILE} \ + --set workload.model=${HF_MODEL_ID} \ + --set workload.image=${DYNAMO_IMAGE} \ + --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ + --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${PREFILL_CONFIG} \ + --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${DECODE_CONFIG} \ + ${RELEASE_NAME} \ + $REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment ``` -## 4. Inference Request +## 4. Inference Request and Benchmarking + [Back to Top](#table-of-contents) -Before sending inference requests, verify that all pods have fully initialized and are in the `Running` state. + +### 4.1. Verify Deployment Readiness -Run the following command to check the status of the pods in your namespace: +Before sending inference or benchmark requests, verify that all platform and worker pods have initialized and are in the `Running` state: ```bash kubectl get pods -n ${NAMESPACE} ``` -The expected output should look similar to the example below. Ensure that all platform components and `dynamo-disagg` worker pods (decode, prefill, and frontend) show a `Running` status and that all containers within them are `Ready` (e.g., `1/1` or `2/2`). +Ensure that all platform components and `dynamo-disagg` worker pods (decode, prefill, and frontend) show a `Running` status and that all containers within them are `Ready` (e.g., `1/1` or `2/2`): ```text NAME READY STATUS RESTARTS AGE @@ -275,22 +369,108 @@ podgroup-controller-f57775757-tjjk9 1/1 Runnin queue-controller-5c8756b857-dqvh2 1/1 Running 0 117m ``` -We can then deploy the benchmark client and send benchark request. -Deploy the benchmark client like this: + +### 4.2. Deploy the Benchmark Client + +Deploy the standalone `bench-client` pod, which installs `aiperf` (for AgentX multi-turn profiling) alongside the Dynamo synthetic benchmarking tools: + ```bash +cd $RECIPE_ROOT kubectl apply -f bench_client.yaml -n ${NAMESPACE} ``` -And send the request like this: +Wait for `bench-client` to finish setup: +```bash +kubectl logs -f pod/bench-client -n ${NAMESPACE} +``` + +Proceed once you see `--- SETUP COMPLETE. POD IS READY. ---`. + + +### 4.3. Category 1: Agentic Multi-Turn Profiling with AIPerf (AgentX Dataset) + +Use [`aiperf`](https://github.com/ai-dynamo/aiperf) with the public SemiAnalysis AgentX coding trajectory dataset (`semianalysis_cc_traces_weka_062126`) to evaluate long-context, multi-turn prefix caching and decode interactivity. + +1. Configure target service, model, and concurrency variables for the benchmark client (defaults to `CONCURRENCY=32`): + +```bash +# Replace FRONTEND_HOST with your Dynamo frontend service or pod IP +export FRONTEND_HOST="${RELEASE_NAME}-frontend" +export MODEL_NAME="${HF_MODEL_ID}" +export TOKENIZER_PATH="${HF_MODEL_ID}" +export CONCURRENCY=32 +``` + +2. Run `aiperf` AgentX profiling from the `bench-client` pod: ```bash -kubectl exec -it bench-client -- bash -c "cd /workspace/dynamo/examples/backends/sglang/slurm_jobs/scripts/vllm && python3 -u benchmark_serving.py --host $USER-dynamo-a4xmax-1p1d-frontend --port 8000 --model deepseek-ai/DeepSeek-R1 --tokenizer deepseek-ai/DeepSeek-R1 --backend 'dynamo' --endpoint /v1/completions --disable-tqdm --dataset-name random --num-prompts 2560 --random-input-len 8192 --random-output-len 1024 --random-range-ratio 0.8 --ignore-eos --request-rate inf --percentile-metrics ttft,tpot,itl,e2el --max-concurrency 512" +kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " + aiperf profile \ + --scenario agentx \ + --url http://${FRONTEND_HOST}:8000 \ + --endpoint /v1/chat/completions \ + --model ${MODEL_NAME} \ + --tokenizer ${TOKENIZER_PATH} \ + --concurrency ${CONCURRENCY} \ + --benchmark-duration 3600 \ + --benchmark-grace-period inf \ + --failed-request-threshold 0.10 \ + --warmup-requests-per-lane 10 \ + --trace-idle-gap-cap-seconds 300 \ + --warmup-grace-period 1800 \ + --agentic-warmup-grace-period 1800 \ + --tokenizer-trust-remote-code \ + --server-metrics http://${FRONTEND_HOST}:6100/metrics \ + --output-artifact-dir /logs/agentic/aiperf_artifacts/stage_${CONCURRENCY} \ + --public-dataset semianalysis_cc_traces_weka_062126 +" ``` -Or we can send a benchmark request to a frontend pod like this: + +### 4.4. Category 2: Synthetic Random-Dataset Benchmarking + +For fixed-shape synthetic throughput and latency testing (e.g., `ISL=8192`, `OSL=1024`), use either the Dynamo `benchmark_serving.py` script in `bench-client` or `sglang.bench_serving`. + +#### 4.4.1. Low-Latency Deployment (1P1D, 8 GPUs) via `bench-client` + +```bash +kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " + cd /workspace/dynamo/examples/backends/sglang/slurm_jobs/scripts/vllm && \ + python3 -u benchmark_serving.py \ + --host $USER-dynamo-a4xmax-1p1d-frontend \ + --port 8000 \ + --model ${HF_MODEL_ID} \ + --tokenizer ${HF_MODEL_ID} \ + --backend 'dynamo' \ + --endpoint /v1/completions \ + --disable-tqdm \ + --dataset-name random \ + --num-prompts 2560 \ + --random-input-len 8192 \ + --random-output-len 1024 \ + --random-range-ratio 0.8 \ + --ignore-eos \ + --request-rate inf \ + --percentile-metrics ttft,tpot,itl,e2el \ + --max-concurrency 512 +" +``` + +#### 4.4.2. High-Throughput Multi-Node Deployment (10P8D, 72 GPUs) via `sglang.bench_serving` ```bash -kubectl exec -n ${NAMESPACE} $USER-dynamo-multi-node-serving-frontend -- python3 -u -m sglang.bench_serving --backend sglang-oai-chat --base-url http://localhost:8000 --model "deepseek-ai/DeepSeek-R1" --tokenizer /data/model/deepseek-ai/DeepSeek-R1 --dataset-name random --num-prompts 10240 --random-input-len 8192 --random-range-ratio 0.8 --random-output-len 1024 --max-concurrency 2048 +kubectl exec -n ${NAMESPACE} $USER-dynamo-multi-node-serving-frontend -- \ + python3 -u -m sglang.bench_serving \ + --backend sglang-oai-chat \ + --base-url http://localhost:8000 \ + --model "${HF_MODEL_ID}" \ + --tokenizer "/data/model/${HF_MODEL_ID}" \ + --dataset-name random \ + --num-prompts 10240 \ + --random-input-len 8192 \ + --random-range-ratio 0.8 \ + --random-output-len 1024 \ + --max-concurrency 2048 ``` @@ -339,7 +519,8 @@ helm list -n ${NAMESPACE} --filter $USER-dynamo- Uninstall specific deployments: ```bash -helm uninstall $USER-dynamo-multi-node-serving -n ${NAMESPACE} +helm uninstall ${RELEASE_NAME} -n ${NAMESPACE} +kubectl delete -f $RECIPE_ROOT/bench_client.yaml -n ${NAMESPACE} --ignore-not-found=true ``` Uninstall Dynamo platform (if no longer needed): @@ -355,4 +536,4 @@ kubectl delete namespace ${NAMESPACE} Clean up downloaded charts: ```bash rm -f dynamo-platform-${RELEASE_VERSION}.tgz -``` \ No newline at end of file +``` diff --git a/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml b/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml index 04064202..3e14046a 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml +++ b/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml @@ -31,17 +31,30 @@ spec: - | echo "--- STARTING SETUP ---" - # 1. Install Git - apt-get update && apt-get install -y git + # 1. Install System Dependencies + apt-get update && apt-get install -y git curl jq - # 2. Install Python Dependencies - pip install -q transformers aiohttp numpy requests tqdm pandas datasets Pillow + # 2. Install Python Dependencies (including AIPerf for AgentX profiling and synthetic benchmark deps) + pip install -q --upgrade pip + pip install -q aiperf transformers aiohttp numpy requests tqdm pandas datasets Pillow huggingface_hub hf_transfer - # 3. Clone the Repo (Specific Branch) + # 3. Create Artifact Directory for AIPerf Profiling Outputs + mkdir -p /logs/agentic/aiperf_artifacts + + # 4. Clone the Dynamo Repo for Synthetic Benchmarking Scripts echo "Cloning repo..." git clone --single-branch --branch ishan/sa-1.1-sgl-dsr1-fp8 https://github.com/ai-dynamo/dynamo.git /workspace/dynamo echo "--- SETUP COMPLETE. POD IS READY. ---" - # 4. Keep the pod alive indefinitely + # 5. Keep the pod alive indefinitely sleep infinity + env: + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + optional: true + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" From c9bd8bc2e6d791f11adcf12bcf487b9b45008978 Mon Sep 17 00:00:00 2001 From: joeywan-google Date: Fri, 16 Oct 2026 05:57:57 +0000 Subject: [PATCH 2/8] Add A4X-MAX Dynamo recipes for Beam-1.0-501B NVFP4 (High Interactivity & High Throughput) --- .../disaggregated-serving/dynamo/README.md | 24 +- .../dynamo/bench_client.yaml | 26 +- .../dynamo/laser_beam_high_int.yaml | 239 ++++++++++++ .../dynamo/laser_beam_high_throughput.yaml | 342 ++++++++++++++++++ .../laser_beam_high_int_decode.yaml | 52 +++ .../laser_beam_high_int_prefill.yaml | 52 +++ .../laser_beam_high_throughput_decode.yaml | 50 +++ .../laser_beam_high_throughput_prefill.yaml | 50 +++ .../templates/dynamo-all-mrdma.yaml | 4 +- .../templates/dynamo-graph-deployment.yaml | 212 +++++++++-- 10 files changed, 1017 insertions(+), 34 deletions(-) create mode 100644 inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_int.yaml create mode 100644 inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_throughput.yaml create mode 100644 src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_decode.yaml create mode 100644 src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_prefill.yaml create mode 100644 src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_decode.yaml create mode 100644 src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_prefill.yaml diff --git a/inference/a4xmax/disaggregated-serving/dynamo/README.md b/inference/a4xmax/disaggregated-serving/dynamo/README.md index 2a26ea81..d39fd709 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/README.md +++ b/inference/a4xmax/disaggregated-serving/dynamo/README.md @@ -210,8 +210,8 @@ This recipe is organized into deployment categories that pair a Helm topology (` | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | **DeepSeek-R1 671B (FP8)** | `deepseek-ai/DeepSeek-R1` | Low-Latency / without DeepEP | 1P1D (2 nodes / 8 GPUs) | `values_wo_deepep.yaml` | `deepseek-r1-fp8-1p1d-prefill.yaml` | `deepseekr1-fp8-1p1d-decode.yaml` | | **DeepSeek-R1 671B (FP8)** | `deepseek-ai/DeepSeek-R1` | High-Throughput / with DeepEP | 10P8D (18 nodes / 72 GPUs) | `values_deepep.yaml` | `deepseekr1-fp8-10p8d-prefill.yaml` | `deepseekr1-fp8-10p8d-decode.yaml` | -| **LASER_BEAM** | `` | High Interactivity | 1P1D (2 nodes / 8 GPUs) | `laser_beam_high_int.yaml` | `laser_beam_high_int_prefill.yaml` | `laser_beam_high_int_decode.yaml` | -| **LASER_BEAM** | `` | High Throughput | Multi-Node (32–72 GPUs) | `laser_beam_high_throughput.yaml` | `laser_beam_high_throughput_prefill.yaml` | `laser_beam_high_throughput_decode.yaml` | +| **LASER_BEAM** | `reflection/Beam-1.0-501B-A23B-NVFP4` | High Interactivity (DFlash2) | 8x Aggregated (8 nodes / 32 GPUs) | `laser_beam_high_int.yaml` | `laser_beam_high_int_prefill.yaml` | `laser_beam_high_int_decode.yaml` | +| **LASER_BEAM** | `reflection/Beam-1.0-501B-A23B-NVFP4` | High Throughput (Disaggregated) | 4P4D (8 nodes / 32 GPUs) | `laser_beam_high_throughput.yaml` | `laser_beam_high_throughput_prefill.yaml` | `laser_beam_high_throughput_decode.yaml` | > [!TIP] > To add or deploy another model, stage its weights in `gs://${GCS_BUCKET}`, add its prefill and decode YAML files under `$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/`, and pass `--set workload.model=${HF_MODEL_ID}` along with `--set-file prefill_serving_config=...` and `--set-file decode_serving_config=...` to `helm install`. @@ -252,7 +252,8 @@ helm install -f ${VALUES_FILE} \ 1. Set the model and configuration variables: ```bash -export HF_MODEL_ID="" +export HF_MODEL_ID="reflection/Beam-1.0-501B-A23B-NVFP4" +export DYNAMO_IMAGE="us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64" export VALUES_FILE="laser_beam_high_int.yaml" export PREFILL_CONFIG="laser_beam_high_int_prefill.yaml" export DECODE_CONFIG="laser_beam_high_int_decode.yaml" @@ -309,7 +310,8 @@ helm install -f ${VALUES_FILE} \ 1. Set the model and configuration variables: ```bash -export HF_MODEL_ID="" +export HF_MODEL_ID="reflection/Beam-1.0-501B-A23B-NVFP4" +export DYNAMO_IMAGE="us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64" export VALUES_FILE="laser_beam_high_throughput.yaml" export PREFILL_CONFIG="laser_beam_high_throughput_prefill.yaml" export DECODE_CONFIG="laser_beam_high_throughput_decode.yaml" @@ -391,11 +393,12 @@ Proceed once you see `--- SETUP COMPLETE. POD IS READY. ---`. Use [`aiperf`](https://github.com/ai-dynamo/aiperf) with the public SemiAnalysis AgentX coding trajectory dataset (`semianalysis_cc_traces_weka_062126`) to evaluate long-context, multi-turn prefix caching and decode interactivity. -1. Configure target service, model, and concurrency variables for the benchmark client (defaults to `CONCURRENCY=32`): +1. Configure target service, worker metrics endpoints, model, and concurrency variables for the benchmark client (for example `CONCURRENCY=32`, `96`, or `192`): ```bash # Replace FRONTEND_HOST with your Dynamo frontend service or pod IP -export FRONTEND_HOST="${RELEASE_NAME}-frontend" +export FRONTEND_HOST="$(kubectl get svc -n ${NAMESPACE} -o name | grep frontend | head -n1 | cut -d/ -f2)" +export WORKER_METRICS="$(kubectl get pods -n ${NAMESPACE} -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.podIP}{"\n"}{end}' | awk '/-decode-|-prefill-/ && $2 != "" {printf "%s:9090/metrics ", $2}')" export MODEL_NAME="${HF_MODEL_ID}" export TOKENIZER_PATH="${HF_MODEL_ID}" export CONCURRENCY=32 @@ -405,22 +408,23 @@ export CONCURRENCY=32 ```bash kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " + export AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES=0 + export AIPERF_DATASET_CONFIGURATION_TIMEOUT=1800 + export AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=1800 aiperf profile \ - --scenario agentx \ + --scenario inferencex-agentx-mvp \ --url http://${FRONTEND_HOST}:8000 \ --endpoint /v1/chat/completions \ --model ${MODEL_NAME} \ --tokenizer ${TOKENIZER_PATH} \ --concurrency ${CONCURRENCY} \ --benchmark-duration 3600 \ - --benchmark-grace-period inf \ --failed-request-threshold 0.10 \ --warmup-requests-per-lane 10 \ --trace-idle-gap-cap-seconds 300 \ --warmup-grace-period 1800 \ - --agentic-warmup-grace-period 1800 \ --tokenizer-trust-remote-code \ - --server-metrics http://${FRONTEND_HOST}:6100/metrics \ + --server-metrics ${WORKER_METRICS} \ --output-artifact-dir /logs/agentic/aiperf_artifacts/stage_${CONCURRENCY} \ --public-dataset semianalysis_cc_traces_weka_062126 " diff --git a/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml b/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml index 3e14046a..25d3dd4f 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml +++ b/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml @@ -20,9 +20,19 @@ metadata: app: bench-client spec: restartPolicy: Never + nodeSelector: + kubernetes.io/arch: arm64 + tolerations: + - key: "kubernetes.io/arch" + operator: "Equal" + value: "arm64" + effect: "NoSchedule" + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" containers: - name: benchmark - image: python:3.10 + image: python:3.12 workingDir: /workspace command: ["/bin/bash", "-c"] # This script runs ONCE when the pod starts to set everything up. @@ -34,7 +44,7 @@ spec: # 1. Install System Dependencies apt-get update && apt-get install -y git curl jq - # 2. Install Python Dependencies (including AIPerf for AgentX profiling and synthetic benchmark deps) + # 2. Install Python Dependencies (including AIPerf >=0.12.0 for AgentX profiling and synthetic benchmark deps) pip install -q --upgrade pip pip install -q aiperf transformers aiohttp numpy requests tqdm pandas datasets Pillow huggingface_hub hf_transfer @@ -58,3 +68,15 @@ spec: optional: true - name: HF_HUB_ENABLE_HF_TRANSFER value: "1" + - name: AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES + value: "0" + - name: AIPERF_DATASET_CONFIGURATION_TIMEOUT + value: "1800" + - name: AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT + value: "1800" + resources: + requests: + cpu: "4" + memory: "20Gi" + limits: + memory: "300Gi" diff --git a/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_int.yaml b/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_int.yaml new file mode 100644 index 00000000..6225809e --- /dev/null +++ b/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_int.yaml @@ -0,0 +1,239 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +dynamo: + namespace: dynamo-cloud + releaseVersion: "1.0.1" + deploymentName: dynamo-agg8w-dflash + computeDomain: + name: a4x-max-domain + numNodes: 8 + resourceClaimTemplateName: a4x-max-compute-domain-channel + rdma: + claimTemplateName: + serviceAccountName: + frontend: + image: us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64 + replicas: 1 + envs: + - name: DYN_DISCOVERY_BACKEND + value: kubernetes + - name: DYN_EVENT_PLANE + value: zmq + - name: DYN_REQUEST_PLANE + value: tcp + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + - name: DYN_ROUTER_MODE + value: kv + - name: DYN_CHAT_PROCESSOR + value: sglang + - name: DYN_KV_CACHE_BLOCK_SIZE + value: "64" + livenessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 150 + failureThreshold: 100 + readinessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 300 + failureThreshold: 100 + decodeWorker: + nodeCount: 1 + replicas: 8 + envs: + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: LD_LIBRARY_PATH + value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" + - name: UCX_TLS + value: "cuda_copy,cuda_ipc,tcp" + - name: UCX_CUDA_IPC_ENABLE_MNNVL + value: "true" + - name: UCX_CUDA_COPY_DMABUF + value: "true" + - name: UCX_MEMTYPE_CACHE + value: "false" + - name: UCX_IB_PCI_RELAXED_ORDERING + value: "true" + - name: UCX_NET_DEVICES + value: "eth0" + - name: UCX_IB_GID_INDEX + value: "3" + - name: UCX_LOG_LEVEL + value: "info" + - name: NIXL_NUM_TRANSPORT_THREADS + value: "4" + - name: GLOO_SOCKET_IFNAME + value: eth0 + - name: TP_SOCKET_IFNAME + value: eth0 + - name: PYTHONUNBUFFERED + value: "1" + - name: DYN_SKIP_SGLANG_LOG_FORMATTING + value: "1" + - name: DYN_DISCOVERY_BACKEND + value: kubernetes + - name: DYN_EVENT_PLANE + value: zmq + - name: DYN_REQUEST_PLANE + value: tcp + - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT + value: "1800" + - name: TORCH_CUDA_ARCH_LIST + value: "10.3a" + - name: PYTORCH_ALLOC_CONF + value: "expandable_segments:True" + - name: PYTORCH_CUDA_ALLOC_CONF + value: "expandable_segments:True" + - name: SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN + value: "1" + - name: SGLANG_MOE_NVFP4_DISPATCH + value: "1" + - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN + value: "1" + - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2 + value: "1" + - name: SGLANG_ENABLE_JIT_DEEPGEMM + value: "false" + - name: SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK + value: "1024" + - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE + value: "100000" + - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT + value: "100000" + - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT + value: "100000" + - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT + value: "100000" + - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM + value: "1" + - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL + value: "NVLINK" + - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER + value: "0" + - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK + value: "1" + - name: SGLANG_DISABLE_CUSTOM_ALL_REDUCE + value: "0" + - name: MOONCAKE_PROTOCOL + value: "tcp" + - name: MC_TE_METRIC + value: "true" + - name: MC_FORCE_MNNVL + value: "1" + - name: NCCL_DEBUG + value: "INFO" + - name: NCCL_IB_DISABLE + value: "1" + - name: NCCL_NET + value: "Socket" + - name: NCCL_SOCKET_IFNAME + value: "eth0" + - name: NCCL_CROSS_NIC + value: "1" + - name: NCCL_MNNVL_ENABLE + value: "1" + - name: NCCL_CUMEM_ENABLE + value: "1" + - name: NCCL_VMM_ENABLE + value: "1" + - name: NCCL_NVLS_ENABLE + value: "1" + - name: NCCL_NET_GDR_C2C + value: "1" + - name: NCCL_NET_GDR_LEVEL + value: "PIX" + livenessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 60 + readinessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 60 + startupProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 1800 + prefillWorker: + nodeCount: 1 + replicas: 0 + envs: [] + livenessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 60 + readinessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 60 + startupProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 1800 + +secrets: + ngc: + secretName: nvcr-secret + huggingface: + secretName: hf-token-secret + secretData: + token: "hf_api_token" + +volumes: + useGcs: true + gcsfuse: + bucketName: + ssdMountPath: "/ssd" + gcsMounts: + mountPath: "/data/model" + +service: + type: ClusterIP + ports: + frontend: 8000 + worker: 9090 + +workload: + model: reflection/Beam-1.0-501B-A23B-NVFP4 + image: us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64 + framework: sglang + configFile: serving-args.yaml + configPath: /workload/configs + +network: + subnetworks: [] + gibVersion: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.0.7 + ncclSettings: + - name: NCCL_DEBUG + value: "VERSION" + +quantizations: + - "nvfp4" diff --git a/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_throughput.yaml b/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_throughput.yaml new file mode 100644 index 00000000..75f0e128 --- /dev/null +++ b/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_throughput.yaml @@ -0,0 +1,342 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +dynamo: + namespace: dynamo-cloud + releaseVersion: "1.0.1" + deploymentName: dynamo-disagg4p4d + computeDomain: + name: a4x-max-domain + numNodes: 8 + resourceClaimTemplateName: a4x-max-compute-domain-channel + rdma: + claimTemplateName: + serviceAccountName: + frontend: + image: us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64 + replicas: 1 + envs: + - name: DYN_DISCOVERY_BACKEND + value: kubernetes + - name: DYN_EVENT_PLANE + value: zmq + - name: DYN_REQUEST_PLANE + value: tcp + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + - name: DYN_ROUTER_MODE + value: kv + - name: DYN_CHAT_PROCESSOR + value: sglang + - name: DYN_KV_CACHE_BLOCK_SIZE + value: "64" + livenessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 150 + failureThreshold: 100 + readinessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 300 + failureThreshold: 100 + decodeWorker: + nodeCount: 1 + replicas: 4 + envs: + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: LD_LIBRARY_PATH + value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" + - name: UCX_TLS + value: "cuda_copy,cuda_ipc,tcp" + - name: UCX_CUDA_IPC_ENABLE_MNNVL + value: "true" + - name: UCX_CUDA_COPY_DMABUF + value: "true" + - name: UCX_MEMTYPE_CACHE + value: "false" + - name: UCX_IB_PCI_RELAXED_ORDERING + value: "true" + - name: UCX_NET_DEVICES + value: "eth0" + - name: UCX_IB_GID_INDEX + value: "3" + - name: UCX_LOG_LEVEL + value: "info" + - name: NIXL_NUM_TRANSPORT_THREADS + value: "4" + - name: GLOO_SOCKET_IFNAME + value: eth0 + - name: TP_SOCKET_IFNAME + value: eth0 + - name: PYTHONUNBUFFERED + value: "1" + - name: DYN_SKIP_SGLANG_LOG_FORMATTING + value: "1" + - name: DYN_DISCOVERY_BACKEND + value: kubernetes + - name: DYN_EVENT_PLANE + value: zmq + - name: DYN_REQUEST_PLANE + value: tcp + - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT + value: "1800" + - name: TORCH_CUDA_ARCH_LIST + value: "10.3a" + - name: PYTORCH_ALLOC_CONF + value: "expandable_segments:True" + - name: PYTORCH_CUDA_ALLOC_CONF + value: "expandable_segments:True" + - name: SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN + value: "1" + - name: SGLANG_MOE_NVFP4_DISPATCH + value: "1" + - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN + value: "1" + - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2 + value: "1" + - name: SGLANG_ENABLE_JIT_DEEPGEMM + value: "false" + - name: SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK + value: "1024" + - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE + value: "100000" + - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT + value: "100000" + - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT + value: "100000" + - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT + value: "100000" + - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM + value: "1" + - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL + value: "NVLINK" + - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER + value: "0" + - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK + value: "1" + - name: SGLANG_DISABLE_CUSTOM_ALL_REDUCE + value: "0" + - name: MOONCAKE_PROTOCOL + value: "tcp" + - name: MC_TE_METRIC + value: "true" + - name: MC_FORCE_MNNVL + value: "1" + - name: NCCL_DEBUG + value: "INFO" + - name: NCCL_IB_DISABLE + value: "1" + - name: NCCL_NET + value: "Socket" + - name: NCCL_SOCKET_IFNAME + value: "eth0" + - name: NCCL_CROSS_NIC + value: "1" + - name: NCCL_MNNVL_ENABLE + value: "1" + - name: NCCL_CUMEM_ENABLE + value: "1" + - name: NCCL_VMM_ENABLE + value: "1" + - name: NCCL_NVLS_ENABLE + value: "1" + - name: NCCL_NET_GDR_C2C + value: "1" + - name: NCCL_NET_GDR_LEVEL + value: "PIX" + livenessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 60 + readinessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 60 + startupProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 1800 + prefillWorker: + nodeCount: 1 + replicas: 4 + envs: + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: LD_LIBRARY_PATH + value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" + - name: UCX_TLS + value: "cuda_copy,cuda_ipc,tcp" + - name: UCX_CUDA_IPC_ENABLE_MNNVL + value: "true" + - name: UCX_CUDA_COPY_DMABUF + value: "true" + - name: UCX_MEMTYPE_CACHE + value: "false" + - name: UCX_IB_PCI_RELAXED_ORDERING + value: "true" + - name: UCX_NET_DEVICES + value: "eth0" + - name: UCX_IB_GID_INDEX + value: "3" + - name: UCX_LOG_LEVEL + value: "info" + - name: NIXL_LOG_LEVEL + value: "info" + - name: NIXL_NUM_TRANSPORT_THREADS + value: "4" + - name: GLOO_SOCKET_IFNAME + value: eth0 + - name: TP_SOCKET_IFNAME + value: eth0 + - name: PYTHONUNBUFFERED + value: "1" + - name: DYN_SKIP_SGLANG_LOG_FORMATTING + value: "1" + - name: DYN_DISCOVERY_BACKEND + value: kubernetes + - name: DYN_EVENT_PLANE + value: zmq + - name: DYN_REQUEST_PLANE + value: tcp + - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT + value: "1800" + - name: TORCH_CUDA_ARCH_LIST + value: "10.3a" + - name: PYTORCH_ALLOC_CONF + value: "expandable_segments:True" + - name: PYTORCH_CUDA_ALLOC_CONF + value: "expandable_segments:True" + - name: SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN + value: "1" + - name: SGLANG_MOE_NVFP4_DISPATCH + value: "1" + - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN + value: "1" + - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2 + value: "1" + - name: SGLANG_ENABLE_JIT_DEEPGEMM + value: "false" + - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE + value: "100000" + - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT + value: "100000" + - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT + value: "100000" + - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL + value: "NVLINK" + - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER + value: "0" + - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK + value: "1" + - name: SGLANG_DISABLE_CUSTOM_ALL_REDUCE + value: "0" + - name: MOONCAKE_PROTOCOL + value: "tcp" + - name: MC_TE_METRIC + value: "true" + - name: MC_FORCE_MNNVL + value: "1" + - name: NCCL_DEBUG + value: "INFO" + - name: NCCL_IB_DISABLE + value: "1" + - name: NCCL_NET + value: "Socket" + - name: NCCL_SOCKET_IFNAME + value: "eth0" + - name: NCCL_CROSS_NIC + value: "1" + - name: NCCL_MNNVL_ENABLE + value: "1" + - name: NCCL_CUMEM_ENABLE + value: "1" + - name: NCCL_VMM_ENABLE + value: "1" + - name: NCCL_NVLS_ENABLE + value: "1" + - name: NCCL_NET_GDR_C2C + value: "1" + - name: NCCL_NET_GDR_LEVEL + value: "PIX" + livenessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 60 + readinessProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 60 + startupProbe: + initialDelaySeconds: 10 + periodSeconds: 60 + timeoutSeconds: 30 + failureThreshold: 1800 + +secrets: + ngc: + secretName: nvcr-secret + huggingface: + secretName: hf-token-secret + secretData: + token: "hf_api_token" + +volumes: + useGcs: true + gcsfuse: + bucketName: + ssdMountPath: "/ssd" + gcsMounts: + mountPath: "/data/model" + +service: + type: ClusterIP + ports: + frontend: 8000 + worker: 9090 + +workload: + model: reflection/Beam-1.0-501B-A23B-NVFP4 + image: us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64 + framework: sglang + configFile: serving-args.yaml + configPath: /workload/configs + +network: + subnetworks: [] + gibVersion: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.0.7 + ncclSettings: + - name: NCCL_DEBUG + value: "VERSION" + +quantizations: + - "nvfp4" diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_decode.yaml b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_decode.yaml new file mode 100644 index 00000000..2df08e63 --- /dev/null +++ b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_decode.yaml @@ -0,0 +1,52 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +served-model-name: reflection/Beam-1.0-501B-A23B-NVFP4 +host: "0.0.0.0" +port: "9090" +trust-remote-code: true +tp-size: "4" +dp: "1" +ep-size: "4" +attention-backend: "trtllm_mha" +moe-runner-backend: "flashinfer_trtllm" +bf16-gemm-backend: "cutedsl" +kv-cache-dtype: "fp8_e4m3" +page-size: "64" +swa-full-tokens-ratio: "0.03" +mem-fraction-static: "0.9" +context-length: "1048576" +chunked-prefill-size: "16384" +max-prefill-tokens: "16384" +max-running-requests: "256" +watchdog-timeout: "1000000" +dist-timeout: "3600" +enable-metrics: true +enable-flashinfer-allreduce-fusion: true +flashinfer-allreduce-fusion-backend: "mnnvl" +cuda-graph-max-bs-decode: "256" +cuda-graph-bs-decode: "[1, 2, 4, 8, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256]" +cuda-graph-bs-prefill: "[256, 512, 1024, 2048, 4096, 8192]" +speculative-algorithm: "DFLASH" +speculative-draft-model-path: "/data/model/huggingface_model_cache/models--reflection--dflash2-drafter-preview-final/snapshots/8d26c952ec6a82a9a18825d6126c9f6948149ad9" +speculative-eagle-topk: "1" +speculative-num-draft-tokens: "8" +speculative-num-steps: "1" +reasoning-parser: "beam" +tool-call-parser: "beam" +default-chat-template-kwargs: '{"allow_missing_thinking":true,"reasoning_effort":"max"}' +preferred-sampling-params: '{"temperature":1.0,"top_k":40,"top_p":0.95}' +event-plane: "zmq" +request-plane: "tcp" +kv-events-config: '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_prefill.yaml b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_prefill.yaml new file mode 100644 index 00000000..2df08e63 --- /dev/null +++ b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_prefill.yaml @@ -0,0 +1,52 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +served-model-name: reflection/Beam-1.0-501B-A23B-NVFP4 +host: "0.0.0.0" +port: "9090" +trust-remote-code: true +tp-size: "4" +dp: "1" +ep-size: "4" +attention-backend: "trtllm_mha" +moe-runner-backend: "flashinfer_trtllm" +bf16-gemm-backend: "cutedsl" +kv-cache-dtype: "fp8_e4m3" +page-size: "64" +swa-full-tokens-ratio: "0.03" +mem-fraction-static: "0.9" +context-length: "1048576" +chunked-prefill-size: "16384" +max-prefill-tokens: "16384" +max-running-requests: "256" +watchdog-timeout: "1000000" +dist-timeout: "3600" +enable-metrics: true +enable-flashinfer-allreduce-fusion: true +flashinfer-allreduce-fusion-backend: "mnnvl" +cuda-graph-max-bs-decode: "256" +cuda-graph-bs-decode: "[1, 2, 4, 8, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256]" +cuda-graph-bs-prefill: "[256, 512, 1024, 2048, 4096, 8192]" +speculative-algorithm: "DFLASH" +speculative-draft-model-path: "/data/model/huggingface_model_cache/models--reflection--dflash2-drafter-preview-final/snapshots/8d26c952ec6a82a9a18825d6126c9f6948149ad9" +speculative-eagle-topk: "1" +speculative-num-draft-tokens: "8" +speculative-num-steps: "1" +reasoning-parser: "beam" +tool-call-parser: "beam" +default-chat-template-kwargs: '{"allow_missing_thinking":true,"reasoning_effort":"max"}' +preferred-sampling-params: '{"temperature":1.0,"top_k":40,"top_p":0.95}' +event-plane: "zmq" +request-plane: "tcp" +kv-events-config: '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_decode.yaml b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_decode.yaml new file mode 100644 index 00000000..28804c82 --- /dev/null +++ b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_decode.yaml @@ -0,0 +1,50 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +served-model-name: reflection/Beam-1.0-501B-A23B-NVFP4 +disaggregation-mode: decode +disaggregation-transfer-backend: "mooncake" +host: "0.0.0.0" +port: "9090" +trust-remote-code: true +tp-size: "4" +dp: "1" +ep-size: "4" +attention-backend: "trtllm_mha" +moe-runner-backend: "flashinfer_trtllm" +bf16-gemm-backend: "cutedsl" +kv-cache-dtype: "fp8_e4m3" +page-size: "64" +swa-full-tokens-ratio: "0.03" +mem-fraction-static: "0.9" +context-length: "1048576" +chunked-prefill-size: "32768" +max-prefill-tokens: "32768" +max-running-requests: "256" +stream-interval: "64" +watchdog-timeout: "1000000" +dist-timeout: "3600" +enable-metrics: true +enable-flashinfer-allreduce-fusion: true +flashinfer-allreduce-fusion-backend: "mnnvl" +cuda-graph-max-bs-decode: "256" +cuda-graph-bs-decode: "[1, 2, 4, 8, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256]" +cuda-graph-bs-prefill: "[256, 512, 1024, 2048, 4096, 8192]" +reasoning-parser: "beam" +tool-call-parser: "beam" +default-chat-template-kwargs: '{"allow_missing_thinking":true,"reasoning_effort":"max"}' +preferred-sampling-params: '{"temperature":1.0,"top_k":40,"top_p":0.95}' +event-plane: "zmq" +request-plane: "tcp" +kv-events-config: '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_prefill.yaml b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_prefill.yaml new file mode 100644 index 00000000..ea63a11e --- /dev/null +++ b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_prefill.yaml @@ -0,0 +1,50 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +served-model-name: reflection/Beam-1.0-501B-A23B-NVFP4 +disaggregation-mode: prefill +disaggregation-transfer-backend: "mooncake" +host: "0.0.0.0" +port: "9090" +trust-remote-code: true +tp-size: "4" +dp: "1" +ep-size: "4" +attention-backend: "trtllm_mha" +moe-runner-backend: "flashinfer_trtllm" +bf16-gemm-backend: "cutedsl" +kv-cache-dtype: "fp8_e4m3" +page-size: "64" +swa-full-tokens-ratio: "0.03" +mem-fraction-static: "0.9" +context-length: "1048576" +chunked-prefill-size: "32768" +max-prefill-tokens: "32768" +max-running-requests: "256" +stream-interval: "64" +watchdog-timeout: "1000000" +dist-timeout: "3600" +enable-metrics: true +enable-flashinfer-allreduce-fusion: true +flashinfer-allreduce-fusion-backend: "mnnvl" +cuda-graph-max-bs-decode: "256" +cuda-graph-bs-decode: "[1, 2, 4, 8, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256]" +cuda-graph-bs-prefill: "[256, 512, 1024, 2048, 4096, 8192]" +reasoning-parser: "beam" +tool-call-parser: "beam" +default-chat-template-kwargs: '{"allow_missing_thinking":true,"reasoning_effort":"max"}' +preferred-sampling-params: '{"temperature":1.0,"top_k":40,"top_p":0.95}' +event-plane: "zmq" +request-plane: "tcp" +kv-events-config: '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' diff --git a/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-all-mrdma.yaml b/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-all-mrdma.yaml index d9b200c6..90f71794 100644 --- a/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-all-mrdma.yaml +++ b/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-all-mrdma.yaml @@ -12,10 +12,11 @@ # See the License for the specific language governing permissions and # limitations under the License. +{{- if or (.Values.rdma).claimTemplateName (.Values.dynamo.rdma).claimTemplateName }} apiVersion: resource.k8s.io/v1 kind: ResourceClaimTemplate metadata: - name: {{ .Values.rdma.claimTemplateName | default "all-mrdma" }} + name: {{ (.Values.rdma).claimTemplateName | default (.Values.dynamo.rdma).claimTemplateName | default "all-mrdma" }} namespace: {{ .Values.dynamo.namespace }} spec: spec: @@ -26,3 +27,4 @@ spec: deviceClassName: mrdma.google.com allocationMode: ExactCount count: 8 +{{- end }} diff --git a/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml b/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml index 6efa6b33..07aa3176 100644 --- a/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml +++ b/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml @@ -24,14 +24,39 @@ spec: services: Frontend: dynamoNamespace: {{ .Values.dynamo.namespace }} + runtimeVersionOverride: {{ .Values.dynamo.releaseVersion | default "1.0.1" | quote }} componentType: frontend replicas: {{ .Values.dynamo.frontend.replicas }} + {{- if .Values.dynamo.frontend.envs }} + envs: + {{- toYaml .Values.dynamo.frontend.envs | nindent 8 }} + {{- end }} + {{- if .Values.dynamo.frontend.livenessProbe }} + livenessProbe: + httpGet: + path: /live + port: 8000 + initialDelaySeconds: {{ .Values.dynamo.frontend.livenessProbe.initialDelaySeconds }} + periodSeconds: {{ .Values.dynamo.frontend.livenessProbe.periodSeconds }} + timeoutSeconds: {{ .Values.dynamo.frontend.livenessProbe.timeoutSeconds }} + failureThreshold: {{ .Values.dynamo.frontend.livenessProbe.failureThreshold }} + {{- end }} + {{- if .Values.dynamo.frontend.readinessProbe }} + readinessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: {{ .Values.dynamo.frontend.readinessProbe.initialDelaySeconds }} + periodSeconds: {{ .Values.dynamo.frontend.readinessProbe.periodSeconds }} + timeoutSeconds: {{ .Values.dynamo.frontend.readinessProbe.timeoutSeconds }} + failureThreshold: {{ .Values.dynamo.frontend.readinessProbe.failureThreshold }} + {{- end }} resources: requests: - cpu: "5" + cpu: "16" memory: "50Gi" limits: - cpu: "5" + cpu: "16" memory: "50Gi" extraPodMetadata: annotations: @@ -44,6 +69,12 @@ spec: gke-gcsfuse/cache-path: "/gcs-cache" {{- end }} extraPodSpec: + automountServiceAccountToken: true + {{- if .Values.dynamo.serviceAccountName }} + serviceAccountName: {{ .Values.dynamo.serviceAccountName }} + {{- end }} + nodeSelector: + kubernetes.io/arch: arm64 tolerations: - key: "kubernetes.io/arch" operator: "Equal" @@ -54,7 +85,9 @@ spec: effect: "NoSchedule" volumes: - name: local-ssd - emptyDir: {} + hostPath: + path: /mnt/disks/gke-local-ssd/gcsfuse-cache + type: DirectoryOrCreate {{- if eq .Values.volumes.useGcs true }} - name: gcs-model-volume csi: @@ -64,6 +97,9 @@ spec: mountOptions: "implicit-dirs,file-cache:enable-parallel-downloads:true,file-cache:parallel-downloads-per-file:100,file-cache:max-parallel-downloads:-1,file-cache:download-chunk-size-mb:50,file-cache:max-size-mb:-1" {{- end }} mainContainer: + securityContext: + privileged: true + runAsUser: 0 image: {{ .Values.dynamo.frontend.image }} {{- if eq .Values.volumes.useGcs true }} volumeMounts: @@ -85,9 +121,12 @@ spec: nodeCount: {{ .Values.dynamo.decodeWorker.nodeCount }} {{- end }} dynamoNamespace: {{ .Values.dynamo.namespace }} + runtimeVersionOverride: {{ .Values.dynamo.releaseVersion | default "1.0.1" | quote }} envFromSecret: {{ .Values.secrets.huggingface.secretName }} componentType: worker + {{- if gt (int .Values.dynamo.prefillWorker.replicas) 0 }} subComponentType: decode + {{- end }} replicas: {{ .Values.dynamo.decodeWorker.replicas }} livenessProbe: httpGet: @@ -112,7 +151,9 @@ spec: gpu: "4" claims: - name: compute-domain-channel + {{- if (.Values.dynamo.rdma).claimTemplateName }} - name: rdma + {{- end }} envs: - name: SERVER_ARGS_FILE value: {{ .Values.workload.configPath }}/{{ .Values.workload.configFile }} @@ -120,6 +161,10 @@ spec: - name: MODEL_PATH value: {{ .Values.volumes.gcsMounts.mountPath }}/{{ .Values.workload.model }} {{- end }} + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid {{- if .Values.dynamo.decodeWorker.envs }} {{- toYaml .Values.dynamo.decodeWorker.envs | nindent 8 }} {{- end }} @@ -132,14 +177,21 @@ spec: gke-gcsfuse/volumes: "true" {{- end }} extraPodSpec: + {{- if not (.Values.dynamo.rdma).claimTemplateName }} + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + {{- end }} + automountServiceAccountToken: true {{- if .Values.dynamo.serviceAccountName }} serviceAccountName: {{ .Values.dynamo.serviceAccountName }} {{- end }} resourceClaims: - name: compute-domain-channel resourceClaimTemplateName: {{ .Values.dynamo.computeDomain.resourceClaimTemplateName }} + {{- if (.Values.dynamo.rdma).claimTemplateName }} - name: rdma - resourceClaimTemplateName: all-mrdma + resourceClaimTemplateName: {{ .Values.dynamo.rdma.claimTemplateName }} + {{- end }} tolerations: - key: "kubernetes.io/arch" operator: "Equal" @@ -157,14 +209,28 @@ spec: operator: In values: - arm64 + podAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: nvidia.com/dynamo-component-type + operator: In + values: ["worker"] + topologyKey: cloud.google.com/gce-topology-subblock + nodeSelector: + cloud.google.com/gke-accelerator: nvidia-gb300 + cloud.google.com/gke-gpu: "true" + kubernetes.io/arch: arm64 mainContainer: securityContext: - privileged: true - runAsUser: 0 - capabilities: - add: ["IPC_LOCK"] + privileged: true + runAsUser: 0 + capabilities: + add: ["IPC_LOCK"] image: {{ .Values.workload.image }} - workingDir: /sgl-workspace/dynamo/components/backends/sglang + workingDir: /tmp startupProbe: failureThreshold: {{ .Values.dynamo.decodeWorker.startupProbe.failureThreshold }} httpGet: @@ -179,16 +245,38 @@ spec: args: - | set -e + echo "precedence ::ffff:0:0/96 100" >> /etc/gai.conf 2>/dev/null || true nvidia-smi export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas echo "Using ptxas: $TRITON_PTXAS_PATH" + [ -f /usr/local/gib/scripts/set_nccl_env.sh ] && . /usr/local/gib/scripts/set_nccl_env.sh || true + export NCCL_SOCKET_IFNAME=${NCCL_SOCKET_IFNAME:-eth0} + export NCCL_IB_DISABLE=${NCCL_IB_DISABLE:-1} + export NCCL_CROSS_NIC=${NCCL_CROSS_NIC:-1} + export NCCL_NET_GDR_LEVEL=${NCCL_NET_GDR_LEVEL:-PIX} + export GLOO_SOCKET_IFNAME=${GLOO_SOCKET_IFNAME:-eth0} + export TP_SOCKET_IFNAME=${TP_SOCKET_IFNAME:-eth0} + export NCCL_DEBUG=${NCCL_DEBUG:-INFO} + export SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE=${SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE:-100000} + export SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL=${SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL:-10.0} + export SGLANG_DISAGGREGATION_QUEUE_SIZE=${SGLANG_DISAGGREGATION_QUEUE_SIZE:-1024} + export SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=${SGLANG_DISAGGREGATION_THREAD_POOL_SIZE:-1024} + export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=${SGLANG_MOONCAKE_CUSTOM_MEM_POOL:-NVLINK} + export MOONCAKE_PROTOCOL=${MOONCAKE_PROTOCOL:-tcp} + if [ "${MC_FORCE_MNNVL:-}" = "0" ] || [ "${MC_FORCE_MNNVL:-}" = "false" ]; then + unset MC_FORCE_MNNVL + fi echo "--- VERIFYING NCCL ENV VARS IN SHELL ---" env | grep NCCL_ echo "--- END VERIFICATION ---" + pip install hf_transfer --break-system-packages || true ARGS=() if [ -n "$MODEL_PATH" ]; then + if [ ! -d "$MODEL_PATH" ] && [ -d "/data/model/huggingface_model_cache/models--reflection--Beam-1.0-501B-A23B-NVFP4-final/snapshots/81a5ab6b525d43f0a11ddd176412ee0f320ba9e0" ]; then + MODEL_PATH="/data/model/huggingface_model_cache/models--reflection--Beam-1.0-501B-A23B-NVFP4-final/snapshots/81a5ab6b525d43f0a11ddd176412ee0f320ba9e0" + fi echo "Adding model path from env var: $MODEL_PATH" ARGS+=("--model-path" "$MODEL_PATH") else @@ -200,15 +288,23 @@ spec: while IFS=': ' read -r key value || [ -n "$key" ]; do [[ -z "$key" || "$key" == \#* ]] && continue key=$(echo "$key" | xargs) - value=$(echo "$value" | xargs) + value=$(echo "$value" | sed -e 's/^[[:space:]'\''"]*//' -e 's/[[:space:]'\''"]*$//') if [ -n "$key" ]; then if [[ "$value" == "true" ]]; then ARGS+=("--$key") elif [[ "$value" == "false" ]]; then - ARGS+=("--$key" "false") + : elif [ -n "$value" ]; then - ARGS+=("--$key" "$value") + if [[ "$value" =~ ^\[.*\]$ ]]; then + clean_value=$(echo "$value" | tr -d '[],') + ARGS+=("--$key") + for val in $clean_value; do + ARGS+=("$val") + done + else + ARGS+=("--$key" "$value") + fi else ARGS+=("--$key") fi @@ -226,6 +322,8 @@ spec: {{- end }} - name: library-dir-host mountPath: /usr/local/nvidia + - name: gib + mountPath: /usr/local/gib - name: serving-configuration mountPath: {{ .Values.workload.configPath | default "/workload/configs" }} volumes: @@ -236,10 +334,15 @@ spec: volumeAttributes: bucketName: {{ .Values.volumes.gcsfuse.bucketName }} mountOptions: implicit-dirs,file-cache:enable-parallel-downloads:true,file-cache:parallel-downloads-per-file:100,file-cache:max-parallel-downloads:-1,file-cache:download-chunk-size-mb:10,file-cache:max-size-mb:-1 + fileCachePath: "/ssd/gcsfuse-cache" + fileCacheCapacityInBytes: "150000000000" {{- end }} - name: library-dir-host hostPath: path: /home/kubernetes/bin/nvidia + - name: gib + hostPath: + path: /home/kubernetes/bin/gib - name: serving-configuration configMap: name: "{{ .Release.Name }}-decode-config" @@ -247,12 +350,14 @@ spec: - key: serving-configuration path: {{ .Values.workload.configFile | default "serving-args.yaml" }} + {{- if gt (int .Values.dynamo.prefillWorker.replicas) 0 }} Prefill: {{- if gt (int .Values.dynamo.prefillWorker.nodeCount) 1 }} multinode: nodeCount: {{ .Values.dynamo.prefillWorker.nodeCount }} {{- end }} dynamoNamespace: {{ .Values.dynamo.namespace }} + runtimeVersionOverride: {{ .Values.dynamo.releaseVersion | default "1.0.1" | quote }} envFromSecret: {{ .Values.secrets.huggingface.secretName }} componentType: worker subComponentType: prefill @@ -282,7 +387,9 @@ spec: gpu: "4" claims: - name: compute-domain-channel + {{- if (.Values.dynamo.rdma).claimTemplateName }} - name: rdma + {{- end }} envs: - name: SERVER_ARGS_FILE value: {{ .Values.workload.configPath }}/{{ .Values.workload.configFile }} @@ -290,6 +397,10 @@ spec: - name: MODEL_PATH value: {{ .Values.volumes.gcsMounts.mountPath }}/{{ .Values.workload.model }} {{- end }} + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid {{- if .Values.dynamo.prefillWorker.envs }} {{- toYaml .Values.dynamo.prefillWorker.envs | nindent 8 }} {{- end }} @@ -302,14 +413,21 @@ spec: gke-gcsfuse/volumes: "true" {{- end }} extraPodSpec: + {{- if not (.Values.dynamo.rdma).claimTemplateName }} + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + {{- end }} + automountServiceAccountToken: true {{- if .Values.dynamo.serviceAccountName }} serviceAccountName: {{ .Values.dynamo.serviceAccountName }} {{- end }} resourceClaims: - name: compute-domain-channel resourceClaimTemplateName: {{ .Values.dynamo.computeDomain.resourceClaimTemplateName }} + {{- if (.Values.dynamo.rdma).claimTemplateName }} - name: rdma - resourceClaimTemplateName: all-mrdma + resourceClaimTemplateName: {{ .Values.dynamo.rdma.claimTemplateName }} + {{- end }} tolerations: - key: "kubernetes.io/arch" operator: "Equal" @@ -327,16 +445,30 @@ spec: operator: In values: - arm64 + podAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: nvidia.com/dynamo-component-type + operator: In + values: ["worker"] + topologyKey: cloud.google.com/gce-topology-subblock + nodeSelector: + cloud.google.com/gke-accelerator: nvidia-gb300 + cloud.google.com/gke-gpu: "true" + kubernetes.io/arch: arm64 mainContainer: securityContext: - privileged: true - runAsUser: 0 - capabilities: - add: ["IPC_LOCK"] + privileged: true + runAsUser: 0 + capabilities: + add: ["IPC_LOCK"] stdin: true tty: true image: {{ .Values.workload.image }} - workingDir: /sgl-workspace/dynamo/components/backends/sglang + workingDir: /tmp startupProbe: failureThreshold: {{ .Values.dynamo.prefillWorker.startupProbe.failureThreshold }} httpGet: @@ -349,16 +481,38 @@ spec: args: - | set -e + echo "precedence ::ffff:0:0/96 100" >> /etc/gai.conf 2>/dev/null || true nvidia-smi export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas echo "Using ptxas: $TRITON_PTXAS_PATH" + [ -f /usr/local/gib/scripts/set_nccl_env.sh ] && . /usr/local/gib/scripts/set_nccl_env.sh || true + export NCCL_SOCKET_IFNAME=${NCCL_SOCKET_IFNAME:-eth0} + export NCCL_IB_DISABLE=${NCCL_IB_DISABLE:-1} + export NCCL_CROSS_NIC=${NCCL_CROSS_NIC:-1} + export NCCL_NET_GDR_LEVEL=${NCCL_NET_GDR_LEVEL:-PIX} + export GLOO_SOCKET_IFNAME=${GLOO_SOCKET_IFNAME:-eth0} + export TP_SOCKET_IFNAME=${TP_SOCKET_IFNAME:-eth0} + export NCCL_DEBUG=${NCCL_DEBUG:-INFO} + export SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE=${SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE:-100000} + export SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL=${SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL:-10.0} + export SGLANG_DISAGGREGATION_QUEUE_SIZE=${SGLANG_DISAGGREGATION_QUEUE_SIZE:-1024} + export SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=${SGLANG_DISAGGREGATION_THREAD_POOL_SIZE:-1024} + export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=${SGLANG_MOONCAKE_CUSTOM_MEM_POOL:-NVLINK} + export MOONCAKE_PROTOCOL=${MOONCAKE_PROTOCOL:-tcp} + if [ "${MC_FORCE_MNNVL:-}" = "0" ] || [ "${MC_FORCE_MNNVL:-}" = "false" ]; then + unset MC_FORCE_MNNVL + fi echo "--- VERIFYING NCCL ENV VARS IN SHELL ---" env | grep NCCL_ echo "--- END VERIFICATION ---" + pip install hf_transfer --break-system-packages || true ARGS=() if [ -n "$MODEL_PATH" ]; then + if [ ! -d "$MODEL_PATH" ] && [ -d "/data/model/huggingface_model_cache/models--reflection--Beam-1.0-501B-A23B-NVFP4-final/snapshots/81a5ab6b525d43f0a11ddd176412ee0f320ba9e0" ]; then + MODEL_PATH="/data/model/huggingface_model_cache/models--reflection--Beam-1.0-501B-A23B-NVFP4-final/snapshots/81a5ab6b525d43f0a11ddd176412ee0f320ba9e0" + fi echo "Adding model path from env var: $MODEL_PATH" ARGS+=("--model-path" "$MODEL_PATH") else @@ -370,15 +524,23 @@ spec: while IFS=': ' read -r key value || [ -n "$key" ]; do [[ -z "$key" || "$key" == \#* ]] && continue key=$(echo "$key" | xargs) - value=$(echo "$value" | xargs) + value=$(echo "$value" | sed -e 's/^[[:space:]'\''"]*//' -e 's/[[:space:]'\''"]*$//') if [ -n "$key" ]; then if [[ "$value" == "true" ]]; then ARGS+=("--$key") elif [[ "$value" == "false" ]]; then - ARGS+=("--$key" "false") + : elif [ -n "$value" ]; then - ARGS+=("--$key" "$value") + if [[ "$value" =~ ^\[.*\]$ ]]; then + clean_value=$(echo "$value" | tr -d '[],') + ARGS+=("--$key") + for val in $clean_value; do + ARGS+=("$val") + done + else + ARGS+=("--$key" "$value") + fi else ARGS+=("--$key") fi @@ -396,6 +558,8 @@ spec: {{- end }} - name: library-dir-host mountPath: /usr/local/nvidia + - name: gib + mountPath: /usr/local/gib - name: serving-configuration mountPath: {{ .Values.workload.configPath | default "/workload/configs" }} volumes: @@ -406,13 +570,19 @@ spec: volumeAttributes: bucketName: {{ .Values.volumes.gcsfuse.bucketName }} mountOptions: implicit-dirs,file-cache:enable-parallel-downloads:true,file-cache:parallel-downloads-per-file:100,file-cache:max-parallel-downloads:-1,file-cache:download-chunk-size-mb:10,file-cache:max-size-mb:-1 + fileCachePath: "/ssd/gcsfuse-cache" + fileCacheCapacityInBytes: "150000000000" {{- end }} - name: library-dir-host hostPath: path: /home/kubernetes/bin/nvidia + - name: gib + hostPath: + path: /home/kubernetes/bin/gib - name: serving-configuration configMap: name: "{{ .Release.Name }}-prefill-config" items: - key: serving-configuration path: {{ .Values.workload.configFile | default "serving-args.yaml" }} + {{- end }} From 4c67d569652b63c015a1570ebeb8ccc0c443003e Mon Sep 17 00:00:00 2001 From: joeywan-google Date: Fri, 16 Oct 2026 06:37:15 +0000 Subject: [PATCH 3/8] Move LASER_BEAM High Interactivity recipe to aggregated-serving/dynamo The High Interactivity recipe runs 8 aggregated workers (prefillWorker.replicas: 0), so it belongs under inference/a4xmax/aggregated-serving/dynamo/ with its own README. - Ship one worker serving config (laser_beam_high_int_agg.yaml); drop the unused prefill copy. - Replace the hardcoded Beam HF cache snapshot path in the shared chart with an optional workload.modelPath value. - Install aiperf from a pinned upstream commit: the 0.13.0 PyPI release lacks --warmup-requests-per-lane. Drop AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES, which no aiperf version reads. - Remove the LASER_BEAM High Interactivity section from the disaggregated README and link to the aggregated recipe. --- .../aggregated-serving/dynamo/README.md | 348 ++++++++++++++++++ .../dynamo/bench_client.yaml | 78 ++++ .../dynamo/laser_beam_high_int.yaml | 0 .../dynamo/orchestrator-values.yaml | 55 +++ .../disaggregated-serving/dynamo/README.md | 36 +- .../dynamo/bench_client.yaml | 8 +- ...code.yaml => laser_beam_high_int_agg.yaml} | 0 .../laser_beam_high_int_prefill.yaml | 52 --- .../templates/dynamo-graph-deployment.yaml | 10 +- 9 files changed, 491 insertions(+), 96 deletions(-) create mode 100644 inference/a4xmax/aggregated-serving/dynamo/README.md create mode 100644 inference/a4xmax/aggregated-serving/dynamo/bench_client.yaml rename inference/a4xmax/{disaggregated-serving => aggregated-serving}/dynamo/laser_beam_high_int.yaml (100%) create mode 100644 inference/a4xmax/aggregated-serving/dynamo/orchestrator-values.yaml rename src/frameworks/a4xmax/dynamo-configs/{laser_beam_high_int_decode.yaml => laser_beam_high_int_agg.yaml} (100%) delete mode 100644 src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_prefill.yaml diff --git a/inference/a4xmax/aggregated-serving/dynamo/README.md b/inference/a4xmax/aggregated-serving/dynamo/README.md new file mode 100644 index 00000000..36664af4 --- /dev/null +++ b/inference/a4xmax/aggregated-serving/dynamo/README.md @@ -0,0 +1,348 @@ +# Aggregated Multi-Node Inference with NVIDIA Dynamo on A4X-MAX GKE + +This document outlines the steps to deploy and benchmark Large Language Models (LLMs) using [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) in aggregated mode on [A4X-MAX (GB300) GKE Node pools](https://cloud.google.com/kubernetes-engine). + +In aggregated mode, every Dynamo worker runs both prefill and decode for its requests. The Dynamo frontend uses a KV-aware router to send each request to the worker that already holds the longest matching prefix in its KV cache. This recipe uses the [SGLang](https://docs.nvidia.com/dynamo/latest/components/backends/sglang/README.html) backend and profiles an agentic multi-turn workload with [`aiperf`](https://github.com/ai-dynamo/aiperf). For prefill/decode disaggregated serving, see the [disaggregated Dynamo recipe](../../disaggregated-serving/dynamo/README.md). + + +## Table of Contents + +* [1. Test Environment](#test-environment) +* [2. Environment Setup (One-Time)](#environment-setup) + * [2.1. Clone the Repository](#clone-repo) + * [2.2. Configure Environment Variables](#configure-vars) + * [2.3. Connect to your GKE Cluster](#connect-cluster) + * [2.4. Create Secrets](#create-secrets) + * [2.5. Install Dynamo Platform](#install-platform) + * [2.6. Setup GCS Bucket for GKE](#setup-gcsfuse) + * [2.7. Configure Dynamo Image](#configure-dynamo-image) +* [3. Deploy with SGLang Backend](#deploy-sglang) + * [3.1. Supported Models and Deployment Categories](#supported-models) + * [3.2. High Interactivity Deployment](#sglang-high-interactivity) +* [4. Inference Request and Benchmarking](#inference-request) + * [4.1. Verify Deployment Readiness](#verify-readiness) + * [4.2. Deploy the Benchmark Client](#deploy-bench-client) + * [4.3. Agentic Multi-Turn Profiling with AIPerf (AgentX Dataset)](#benchmark-aiperf-agentx) +* [5. Monitoring and Troubleshooting](#monitoring) +* [6. Cleanup](#cleanup) + + +## 1. Test Environment + +[Back to Top](#table-of-contents) + +This recipe has been tested with the following configuration: + +* **GKE Cluster**: + * GPU node pools with [a4x-maxgpu-4g-metal](https://docs.cloud.google.com/compute/docs/gpus#gb300-gpus) machines: + * 8 machines with 4 GPUs each (32 GPUs total), one Dynamo worker per machine + * [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) enabled + * [Cloud Storage FUSE CSI driver for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cloud-storage-fuse-csi-driver) enabled + + +## 2. Environment Setup (One-Time) + +[Back to Top](#table-of-contents) + + +### 2.1. Clone the Repository + +```bash +git clone https://github.com/ai-hypercomputer/gpu-recipes.git +cd gpu-recipes +export REPO_ROOT=$(pwd) +export RECIPE_ROOT=$REPO_ROOT/inference/a4xmax/aggregated-serving/dynamo +``` + + +### 2.2. Configure Environment Variables + +```bash +export PROJECT_ID= +export CLUSTER_REGION= +export CLUSTER_NAME= +export NAMESPACE=dynamo-cloud +export NGC_API_KEY= +export HF_TOKEN= +export RELEASE_VERSION=1.0.1 +export GCS_BUCKET= + +# Set the project for gcloud commands +gcloud config set project $PROJECT_ID +``` + +Replace the following values: + +| Variable | Description | Example | +| -------- | ----------- | ------- | +| `PROJECT_ID` | Your Google Cloud Project ID | `gcp-project-12345` | +| `CLUSTER_REGION` | The GCP region where your GKE cluster is located | `us-central1` | +| `CLUSTER_NAME` | The name of your GKE cluster | `a4xmax-cluster` | +| `NGC_API_KEY` | Your NVIDIA NGC API key (get from [NGC](https://ngc.nvidia.com)) | `nvapi-xxx...` | +| `HF_TOKEN` | Your Hugging Face access token | `hf_xxx...` | +| `GCS_BUCKET` | Your GCS bucket name (without `gs://` prefix for Helm `bucketName`) | `my-model-bucket` | + + +### 2.3. Connect to your GKE Cluster + +```bash +gcloud container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_REGION +``` + + +### 2.4. Create Secrets + +Create the namespace: +```bash +kubectl create namespace ${NAMESPACE} +kubectl config set-context --current --namespace=$NAMESPACE +``` + +Create the Docker registry secret for NVIDIA Container Registry: +```bash +kubectl create secret docker-registry nvcr-secret \ + --namespace=${NAMESPACE} \ + --docker-server=nvcr.io \ + --docker-username='$oauthtoken' \ + --docker-password=${NGC_API_KEY} +``` + +Create the secret for the Hugging Face token: +```bash +kubectl create secret generic hf-token-secret \ + --from-literal=HF_TOKEN=${HF_TOKEN} \ + -n ${NAMESPACE} +``` + + +### 2.5. Install Dynamo Platform (One-Time Setup) + +Skip this step if the Dynamo Platform is already installed in `${NAMESPACE}`. + +Add the NVIDIA Helm repository: +```bash +helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ + --username='$oauthtoken' --password=${NGC_API_KEY} +helm repo update +``` + +Install the Dynamo Platform with Grove & Kai scheduler enabled: +```bash +cd $RECIPE_ROOT +helm install https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz \ + --namespace ${NAMESPACE} -f orchestrator-values.yaml --no-hooks --timeout 10m +``` + +Verify that all platform pods are `Running` and `Ready`: +```bash +kubectl get pods -n ${NAMESPACE} +``` + + +### 2.6. Setup GCS Bucket for GKE (One-Time Setup) + +It is recommended to utilize [gcsfuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-setup) to facilitate model access and mitigate [Hugging Face rate limiting](https://huggingface.co/docs/hub/en/rate-limits#hub-rate-limits) issues. + +Configure the service account email: +```bash +export SERVICE_ACCOUNT_EMAIL=$(kubectl get serviceaccount/default -n ${NAMESPACE} -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}') +``` + +Authorize the service account: +```bash +gcloud iam service-accounts add-iam-policy-binding ${SERVICE_ACCOUNT_EMAIL} \ + --role roles/iam.workloadIdentityUser \ + --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${NAMESPACE}/default]" +``` + +Grant read access to the bucket: +```bash +gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \ + --member "serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \ + --role "roles/storage.objectViewer" +``` + +The bucket mounts at `/data/model` in every worker. Stage the following checkpoints: + +| Checkpoint | Bucket path | Path in the worker | +| :--- | :--- | :--- | +| Target model | `gs://${GCS_BUCKET}//` | `/data/model/` (or set `--set workload.modelPath=`) | +| DFlash2 drafter | `gs://${GCS_BUCKET}/huggingface_model_cache/models--reflection--dflash2-drafter-preview-final/snapshots/8d26c952ec6a82a9a18825d6126c9f6948149ad9/` | Value of `speculative-draft-model-path` in the serving config | + + +### 2.7. Configure Dynamo Image + +This recipe uses a model-specific SGLang runtime image that includes Dynamo `1.0.1` and SGLang support for the model: + +```bash +export DYNAMO_IMAGE="us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64" +``` + +The chart sets `runtimeVersionOverride` to `dynamo.releaseVersion` (`1.0.1`) because the Dynamo operator can't infer the runtime version from a custom image tag. + + +## 3. Deploy with SGLang Backend + +[Back to Top](#table-of-contents) + + +### 3.1. Supported Models and Deployment Categories + +This recipe reuses the Dynamo Helm chart at `$REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment`. Setting `dynamo.prefillWorker.replicas: 0` in the values file renders only unified workers (no `Prefill` service and no `subComponentType`). The worker serving configuration is passed through `decode_serving_config`. + +| Model Name | Hugging Face ID / Path (`HF_MODEL_ID`) | Deployment Category | Topology & GPUs | Helm Values File (`VALUES_FILE`) | Serving Config (`SERVING_CONFIG`) | +| :--- | :--- | :--- | :--- | :--- | :--- | +| **LASER_BEAM** | `reflection/Beam-1.0-501B-A23B-NVFP4` | High Interactivity (DFlash2) | 8 aggregated workers (8 nodes / 32 GPUs) | `laser_beam_high_int.yaml` | `laser_beam_high_int_agg.yaml` | + + +### 3.2. High Interactivity Deployment + +This category deploys 8 aggregated SGLang workers (`TP=4`, `EP=4`, one per node) behind the Dynamo KV-aware router (`DYN_ROUTER_MODE=kv`, `DYN_KV_CACHE_BLOCK_SIZE=64`). Each worker uses DFlash2 speculative decoding with 8 draft tokens to lower Time-Per-Output-Token (TPOT). + +#### LASER_BEAM (High Interactivity) + +1. Set the model and configuration variables: + +```bash +export HF_MODEL_ID="reflection/Beam-1.0-501B-A23B-NVFP4" +export VALUES_FILE="laser_beam_high_int.yaml" +export SERVING_CONFIG="laser_beam_high_int_agg.yaml" +export RELEASE_NAME="$USER-dynamo-a4xmax-laser-beam-high-int" +``` + +2. Install the Helm release: + +```bash +cd $RECIPE_ROOT +helm install -f ${VALUES_FILE} \ + --set workload.model=${HF_MODEL_ID} \ + --set workload.image=${DYNAMO_IMAGE} \ + --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ + --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${SERVING_CONFIG} \ + ${RELEASE_NAME} \ + $REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment +``` + + +## 4. Inference Request and Benchmarking + +[Back to Top](#table-of-contents) + + +### 4.1. Verify Deployment Readiness + +Before sending inference or benchmark requests, verify that the frontend pod and all 8 worker pods are `Running` and that all containers are `Ready`: + +```bash +kubectl get pods -n ${NAMESPACE} +``` + + +### 4.2. Deploy the Benchmark Client + +Deploy the standalone `bench-client` pod, which installs `aiperf` for AgentX multi-turn profiling: + +```bash +cd $RECIPE_ROOT +kubectl apply -f bench_client.yaml -n ${NAMESPACE} +``` + +Wait for `bench-client` to finish setup: +```bash +kubectl logs -f pod/bench-client -n ${NAMESPACE} +``` + +Proceed once you see `--- SETUP COMPLETE. POD IS READY. ---`. + +> [!NOTE] +> The client pod requests a `300Gi` memory limit and schedules on an `arm64` GPU node. A `48Gi` limit was OOM-killed at `CONCURRENCY=192` while loading the AgentX trace dataset. + + +### 4.3. Agentic Multi-Turn Profiling with AIPerf (AgentX Dataset) + +Use [`aiperf`](https://github.com/ai-dynamo/aiperf) with the public SemiAnalysis AgentX coding trajectory dataset (`semianalysis_cc_traces_weka_062126`) to evaluate long-context, multi-turn prefix caching and decode interactivity. + +1. Configure the target service, worker metrics endpoints, model, and concurrency (for example `CONCURRENCY=32`, `96`, or `192`): + +```bash +export FRONTEND_HOST="$(kubectl get svc -n ${NAMESPACE} -o name | grep frontend | head -n1 | cut -d/ -f2)" +export WORKER_METRICS="$(kubectl get pods -n ${NAMESPACE} -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.podIP}{"\n"}{end}' | awk '/-decode-/ && $2 != "" {printf "%s:9090/metrics ", $2}')" +export MODEL_NAME="${HF_MODEL_ID}" +export TOKENIZER_PATH="${HF_MODEL_ID}" +export CONCURRENCY=32 +``` + +2. Run `aiperf` AgentX profiling from the `bench-client` pod: + +```bash +kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " + aiperf profile \ + --scenario inferencex-agentx-mvp \ + --url http://${FRONTEND_HOST}:8000 \ + --endpoint /v1/chat/completions \ + --model ${MODEL_NAME} \ + --tokenizer ${TOKENIZER_PATH} \ + --concurrency ${CONCURRENCY} \ + --benchmark-duration 3600 \ + --failed-request-threshold 0.10 \ + --warmup-requests-per-lane 10 \ + --trace-idle-gap-cap-seconds 300 \ + --warmup-grace-period 1800 \ + --tokenizer-trust-remote-code \ + --server-metrics ${WORKER_METRICS} \ + --output-artifact-dir /logs/agentic/aiperf_artifacts/stage_${CONCURRENCY} \ + --public-dataset semianalysis_cc_traces_weka_062126 +" +``` + + +## 5. Monitoring and Troubleshooting + +[Back to Top](#table-of-contents) + +Find the exact pod names: +```bash +kubectl get pods -n ${NAMESPACE} +``` + +Frontend logs: +```bash +kubectl logs -f -n ${NAMESPACE} +``` + +Worker logs: +```bash +kubectl logs -f -n ${NAMESPACE} +``` + +Common issues: + +* **Pods stuck in Pending**: Check that 8 `a4x-maxgpu-4g-metal` nodes are free in the same NVL72 domain. +* **Worker fails to load the drafter**: Check that the drafter checkpoint is staged at the `speculative-draft-model-path` in `laser_beam_high_int_agg.yaml`. +* **Slow first startup**: The first load reads the checkpoint through GCS FUSE into the local SSD file cache. Later restarts on the same node read from the cache. + + +## 6. Cleanup + +[Back to Top](#table-of-contents) + +List deployed releases: +```bash +helm list -n ${NAMESPACE} --filter $USER-dynamo- +``` + +Uninstall the deployment and the benchmark client: +```bash +helm uninstall ${RELEASE_NAME} -n ${NAMESPACE} +kubectl delete -f $RECIPE_ROOT/bench_client.yaml -n ${NAMESPACE} --ignore-not-found=true +``` + +Uninstall Dynamo platform (if no longer needed): +```bash +helm uninstall dynamo-platform -n ${NAMESPACE} +``` + +Delete namespace and secrets: +```bash +kubectl delete namespace ${NAMESPACE} +``` diff --git a/inference/a4xmax/aggregated-serving/dynamo/bench_client.yaml b/inference/a4xmax/aggregated-serving/dynamo/bench_client.yaml new file mode 100644 index 00000000..1c21764c --- /dev/null +++ b/inference/a4xmax/aggregated-serving/dynamo/bench_client.yaml @@ -0,0 +1,78 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: v1 +kind: Pod +metadata: + name: bench-client + labels: + app: bench-client +spec: + restartPolicy: Never + nodeSelector: + kubernetes.io/arch: arm64 + tolerations: + - key: "kubernetes.io/arch" + operator: "Equal" + value: "arm64" + effect: "NoSchedule" + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: benchmark + image: python:3.12 + workingDir: /workspace + command: ["/bin/bash", "-c"] + # This script runs ONCE when the pod starts to set everything up. + # Then it sleeps forever so the pod stays open for you. + args: + - | + echo "--- STARTING SETUP ---" + + # 1. Install System Dependencies + apt-get update && apt-get install -y git curl jq + + # 2. Install Python Dependencies. AgentX profiling needs `--warmup-requests-per-lane`, + # which is not in the aiperf 0.13.0 PyPI release, so install aiperf from a pinned commit. + pip install -q --upgrade pip + pip install -q "aiperf @ git+https://github.com/ai-dynamo/aiperf.git@1dfaea5fef059d32b7198b3da5325ae415be80c9" + pip install -q transformers aiohttp numpy requests tqdm pandas datasets Pillow huggingface_hub hf_transfer + + # 3. Create Artifact Directory for AIPerf Profiling Outputs + mkdir -p /logs/agentic/aiperf_artifacts + + echo "--- SETUP COMPLETE. POD IS READY. ---" + + # 4. Keep the pod alive indefinitely + sleep infinity + env: + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + optional: true + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: AIPERF_DATASET_CONFIGURATION_TIMEOUT + value: "1800" + - name: AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT + value: "1800" + resources: + requests: + cpu: "4" + memory: "20Gi" + limits: + memory: "300Gi" diff --git a/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_int.yaml b/inference/a4xmax/aggregated-serving/dynamo/laser_beam_high_int.yaml similarity index 100% rename from inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_int.yaml rename to inference/a4xmax/aggregated-serving/dynamo/laser_beam_high_int.yaml diff --git a/inference/a4xmax/aggregated-serving/dynamo/orchestrator-values.yaml b/inference/a4xmax/aggregated-serving/dynamo/orchestrator-values.yaml new file mode 100644 index 00000000..0785014e --- /dev/null +++ b/inference/a4xmax/aggregated-serving/dynamo/orchestrator-values.yaml @@ -0,0 +1,55 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +global: + grove: + install: true + enabled: true + kai-scheduler: + install: true + enabled: true + +grove: + tolerations: + - key: "kubernetes.io/arch" + operator: "Equal" + value: "arm64" + effect: "NoSchedule" + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" + +kai-scheduler: + global: + tolerations: + - key: "kubernetes.io/arch" + operator: "Equal" + value: "arm64" + effect: "NoSchedule" + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" + binder: + resources: + limits: + memory: "1024Mi" + requests: + memory: "1024Mi" + +nats: + config: + jetstream: + fileStore: + pvc: + enabled: false diff --git a/inference/a4xmax/disaggregated-serving/dynamo/README.md b/inference/a4xmax/disaggregated-serving/dynamo/README.md index d39fd709..c06e1d27 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/README.md +++ b/inference/a4xmax/disaggregated-serving/dynamo/README.md @@ -210,8 +210,10 @@ This recipe is organized into deployment categories that pair a Helm topology (` | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | **DeepSeek-R1 671B (FP8)** | `deepseek-ai/DeepSeek-R1` | Low-Latency / without DeepEP | 1P1D (2 nodes / 8 GPUs) | `values_wo_deepep.yaml` | `deepseek-r1-fp8-1p1d-prefill.yaml` | `deepseekr1-fp8-1p1d-decode.yaml` | | **DeepSeek-R1 671B (FP8)** | `deepseek-ai/DeepSeek-R1` | High-Throughput / with DeepEP | 10P8D (18 nodes / 72 GPUs) | `values_deepep.yaml` | `deepseekr1-fp8-10p8d-prefill.yaml` | `deepseekr1-fp8-10p8d-decode.yaml` | -| **LASER_BEAM** | `reflection/Beam-1.0-501B-A23B-NVFP4` | High Interactivity (DFlash2) | 8x Aggregated (8 nodes / 32 GPUs) | `laser_beam_high_int.yaml` | `laser_beam_high_int_prefill.yaml` | `laser_beam_high_int_decode.yaml` | -| **LASER_BEAM** | `reflection/Beam-1.0-501B-A23B-NVFP4` | High Throughput (Disaggregated) | 4P4D (8 nodes / 32 GPUs) | `laser_beam_high_throughput.yaml` | `laser_beam_high_throughput_prefill.yaml` | `laser_beam_high_throughput_decode.yaml` | +| **LASER_BEAM** | `reflection/Beam-1.0-501B-A23B-NVFP4` | High Throughput | 4P4D (8 nodes / 32 GPUs) | `laser_beam_high_throughput.yaml` | `laser_beam_high_throughput_prefill.yaml` | `laser_beam_high_throughput_decode.yaml` | + +> [!NOTE] +> The LASER_BEAM High Interactivity recipe runs 8 aggregated workers, so it lives in the [aggregated Dynamo recipe](../../aggregated-serving/dynamo/README.md). > [!TIP] > To add or deploy another model, stage its weights in `gs://${GCS_BUCKET}`, add its prefill and decode YAML files under `$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/`, and pass `--set workload.model=${HF_MODEL_ID}` along with `--set-file prefill_serving_config=...` and `--set-file decode_serving_config=...` to `helm install`. @@ -247,33 +249,6 @@ helm install -f ${VALUES_FILE} \ $REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment ``` -#### LASER_BEAM (High Interactivity) - -1. Set the model and configuration variables: - -```bash -export HF_MODEL_ID="reflection/Beam-1.0-501B-A23B-NVFP4" -export DYNAMO_IMAGE="us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64" -export VALUES_FILE="laser_beam_high_int.yaml" -export PREFILL_CONFIG="laser_beam_high_int_prefill.yaml" -export DECODE_CONFIG="laser_beam_high_int_decode.yaml" -export RELEASE_NAME="$USER-dynamo-a4xmax-laser-beam-high-int" -``` - -2. Install the Helm release: - -```bash -cd $RECIPE_ROOT -helm install -f ${VALUES_FILE} \ - --set workload.model=${HF_MODEL_ID} \ - --set workload.image=${DYNAMO_IMAGE} \ - --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ - --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${PREFILL_CONFIG} \ - --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/${DECODE_CONFIG} \ - ${RELEASE_NAME} \ - $REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment -``` - ### 3.3. Category 2: High Throughput Multi-Node Deployment @@ -408,9 +383,6 @@ export CONCURRENCY=32 ```bash kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " - export AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES=0 - export AIPERF_DATASET_CONFIGURATION_TIMEOUT=1800 - export AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=1800 aiperf profile \ --scenario inferencex-agentx-mvp \ --url http://${FRONTEND_HOST}:8000 \ diff --git a/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml b/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml index 25d3dd4f..a12e4ea9 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml +++ b/inference/a4xmax/disaggregated-serving/dynamo/bench_client.yaml @@ -44,9 +44,11 @@ spec: # 1. Install System Dependencies apt-get update && apt-get install -y git curl jq - # 2. Install Python Dependencies (including AIPerf >=0.12.0 for AgentX profiling and synthetic benchmark deps) + # 2. Install Python Dependencies. AgentX profiling needs `--warmup-requests-per-lane`, + # which is not in the aiperf 0.13.0 PyPI release, so install aiperf from a pinned commit. pip install -q --upgrade pip - pip install -q aiperf transformers aiohttp numpy requests tqdm pandas datasets Pillow huggingface_hub hf_transfer + pip install -q "aiperf @ git+https://github.com/ai-dynamo/aiperf.git@1dfaea5fef059d32b7198b3da5325ae415be80c9" + pip install -q transformers aiohttp numpy requests tqdm pandas datasets Pillow huggingface_hub hf_transfer # 3. Create Artifact Directory for AIPerf Profiling Outputs mkdir -p /logs/agentic/aiperf_artifacts @@ -68,8 +70,6 @@ spec: optional: true - name: HF_HUB_ENABLE_HF_TRANSFER value: "1" - - name: AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES - value: "0" - name: AIPERF_DATASET_CONFIGURATION_TIMEOUT value: "1800" - name: AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_decode.yaml b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_agg.yaml similarity index 100% rename from src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_decode.yaml rename to src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_agg.yaml diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_prefill.yaml b/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_prefill.yaml deleted file mode 100644 index 2df08e63..00000000 --- a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_prefill.yaml +++ /dev/null @@ -1,52 +0,0 @@ -# Copyright 2026 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -served-model-name: reflection/Beam-1.0-501B-A23B-NVFP4 -host: "0.0.0.0" -port: "9090" -trust-remote-code: true -tp-size: "4" -dp: "1" -ep-size: "4" -attention-backend: "trtllm_mha" -moe-runner-backend: "flashinfer_trtllm" -bf16-gemm-backend: "cutedsl" -kv-cache-dtype: "fp8_e4m3" -page-size: "64" -swa-full-tokens-ratio: "0.03" -mem-fraction-static: "0.9" -context-length: "1048576" -chunked-prefill-size: "16384" -max-prefill-tokens: "16384" -max-running-requests: "256" -watchdog-timeout: "1000000" -dist-timeout: "3600" -enable-metrics: true -enable-flashinfer-allreduce-fusion: true -flashinfer-allreduce-fusion-backend: "mnnvl" -cuda-graph-max-bs-decode: "256" -cuda-graph-bs-decode: "[1, 2, 4, 8, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256]" -cuda-graph-bs-prefill: "[256, 512, 1024, 2048, 4096, 8192]" -speculative-algorithm: "DFLASH" -speculative-draft-model-path: "/data/model/huggingface_model_cache/models--reflection--dflash2-drafter-preview-final/snapshots/8d26c952ec6a82a9a18825d6126c9f6948149ad9" -speculative-eagle-topk: "1" -speculative-num-draft-tokens: "8" -speculative-num-steps: "1" -reasoning-parser: "beam" -tool-call-parser: "beam" -default-chat-template-kwargs: '{"allow_missing_thinking":true,"reasoning_effort":"max"}' -preferred-sampling-params: '{"temperature":1.0,"top_k":40,"top_p":0.95}' -event-plane: "zmq" -request-plane: "tcp" -kv-events-config: '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' diff --git a/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml b/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml index 07aa3176..a9c65038 100644 --- a/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml +++ b/src/helm-charts/a4xmax/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml @@ -159,7 +159,7 @@ spec: value: {{ .Values.workload.configPath }}/{{ .Values.workload.configFile }} {{- if eq .Values.volumes.useGcs true }} - name: MODEL_PATH - value: {{ .Values.volumes.gcsMounts.mountPath }}/{{ .Values.workload.model }} + value: {{ .Values.workload.modelPath | default (printf "%s/%s" .Values.volumes.gcsMounts.mountPath .Values.workload.model) }} {{- end }} - name: POD_UID valueFrom: @@ -274,9 +274,6 @@ spec: ARGS=() if [ -n "$MODEL_PATH" ]; then - if [ ! -d "$MODEL_PATH" ] && [ -d "/data/model/huggingface_model_cache/models--reflection--Beam-1.0-501B-A23B-NVFP4-final/snapshots/81a5ab6b525d43f0a11ddd176412ee0f320ba9e0" ]; then - MODEL_PATH="/data/model/huggingface_model_cache/models--reflection--Beam-1.0-501B-A23B-NVFP4-final/snapshots/81a5ab6b525d43f0a11ddd176412ee0f320ba9e0" - fi echo "Adding model path from env var: $MODEL_PATH" ARGS+=("--model-path" "$MODEL_PATH") else @@ -395,7 +392,7 @@ spec: value: {{ .Values.workload.configPath }}/{{ .Values.workload.configFile }} {{- if eq .Values.volumes.useGcs true }} - name: MODEL_PATH - value: {{ .Values.volumes.gcsMounts.mountPath }}/{{ .Values.workload.model }} + value: {{ .Values.workload.modelPath | default (printf "%s/%s" .Values.volumes.gcsMounts.mountPath .Values.workload.model) }} {{- end }} - name: POD_UID valueFrom: @@ -510,9 +507,6 @@ spec: ARGS=() if [ -n "$MODEL_PATH" ]; then - if [ ! -d "$MODEL_PATH" ] && [ -d "/data/model/huggingface_model_cache/models--reflection--Beam-1.0-501B-A23B-NVFP4-final/snapshots/81a5ab6b525d43f0a11ddd176412ee0f320ba9e0" ]; then - MODEL_PATH="/data/model/huggingface_model_cache/models--reflection--Beam-1.0-501B-A23B-NVFP4-final/snapshots/81a5ab6b525d43f0a11ddd176412ee0f320ba9e0" - fi echo "Adding model path from env var: $MODEL_PATH" ARGS+=("--model-path" "$MODEL_PATH") else From 1394897c7eb4ed709ca458bb853ccc38ed681070 Mon Sep 17 00:00:00 2001 From: joeywan-google Date: Fri, 16 Oct 2026 06:45:35 +0000 Subject: [PATCH 4/8] Add model staging commands and reference results to LASER_BEAM READMEs --- .../aggregated-serving/dynamo/README.md | 31 ++++++++++++++++--- .../disaggregated-serving/dynamo/README.md | 22 +++++++++++++ 2 files changed, 48 insertions(+), 5 deletions(-) diff --git a/inference/a4xmax/aggregated-serving/dynamo/README.md b/inference/a4xmax/aggregated-serving/dynamo/README.md index 36664af4..0dcf6eca 100644 --- a/inference/a4xmax/aggregated-serving/dynamo/README.md +++ b/inference/a4xmax/aggregated-serving/dynamo/README.md @@ -162,12 +162,23 @@ gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \ --role "roles/storage.objectViewer" ``` -The bucket mounts at `/data/model` in every worker. Stage the following checkpoints: +The bucket mounts at `/data/model` in every worker. Stage the target model and the DFlash2 drafter into the bucket: -| Checkpoint | Bucket path | Path in the worker | -| :--- | :--- | :--- | -| Target model | `gs://${GCS_BUCKET}//` | `/data/model/` (or set `--set workload.modelPath=`) | -| DFlash2 drafter | `gs://${GCS_BUCKET}/huggingface_model_cache/models--reflection--dflash2-drafter-preview-final/snapshots/8d26c952ec6a82a9a18825d6126c9f6948149ad9/` | Value of `speculative-draft-model-path` in the serving config | +```bash +pip install -U "huggingface_hub[cli]" + +# Target model: mounts at /data/model/ +hf download reflection/Beam-1.0-501B-A23B-NVFP4 --local-dir ./reflection/Beam-1.0-501B-A23B-NVFP4 +gcloud storage cp -r ./reflection gs://${GCS_BUCKET}/ + +# DFlash2 drafter: mounts at the speculative-draft-model-path in laser_beam_high_int_agg.yaml +hf download reflection/dflash2-drafter-preview-final \ + --revision 8d26c952ec6a82a9a18825d6126c9f6948149ad9 \ + --cache-dir ./huggingface_model_cache +gcloud storage cp -r ./huggingface_model_cache gs://${GCS_BUCKET}/ +``` + +The workers load the target model from `/data/model/`. If you staged it at a different path in the bucket, pass `--set workload.modelPath=/data/model/` to `helm install`. ### 2.7. Configure Dynamo Image @@ -295,6 +306,16 @@ kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " " ``` +3. Compare with the reference results below. Each concurrency is a separate 3,600 s `aiperf` run. We ran `CONCURRENCY=192` first and then `CONCURRENCY=96` on the same deployment, and ran `CONCURRENCY=32` on a fresh deployment. All runs completed with 0 failed requests. + +| Concurrency | Total tok/s (32 GPUs) | Output tok/s | TPOT p50 | TTFT p50 / p99 | Prefix-cache hit rate | Completed requests | +| :---: | :---: | :---: | :---: | :---: | :---: | :---: | +| 32 | 231,382 | 1,459 | 2.87 ms | 0.58 s / 4.37 s | 92.4% | 4,711 | +| 96 | 643,409 | 4,737 | 3.84 ms | 0.84 s / 7.46 s | 82.4% | 16,285 | +| 192 | 805,244 | 6,360 | 6.20 ms | 12.45 s / 22.45 s | 76.0% | 21,933 | + +The prefix-cache hit rate is `overall_usage_prompt_cache_read_pct` in `profile_export_aiperf.json`: cached prompt tokens divided by total prompt tokens, as reported in the response `usage` field. + ## 5. Monitoring and Troubleshooting diff --git a/inference/a4xmax/disaggregated-serving/dynamo/README.md b/inference/a4xmax/disaggregated-serving/dynamo/README.md index c06e1d27..82f3357e 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/README.md +++ b/inference/a4xmax/disaggregated-serving/dynamo/README.md @@ -185,6 +185,16 @@ gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \ Stage your model checkpoint files into the GCS bucket so they mount under `/data/model/` via GCS FUSE. +For example, to stage LASER_BEAM: + +```bash +pip install -U "huggingface_hub[cli]" +hf download reflection/Beam-1.0-501B-A23B-NVFP4 --local-dir ./reflection/Beam-1.0-501B-A23B-NVFP4 +gcloud storage cp -r ./reflection gs://${GCS_BUCKET}/ +``` + +If you stage a model at a different path in the bucket, pass `--set workload.modelPath=/data/model/` to `helm install`. + ### 2.7. Configure Dynamo Image @@ -402,6 +412,18 @@ kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " " ``` +3. Compare with the reference results below. + +**LASER_BEAM High Throughput (4P4D, 32 GPUs).** Each concurrency is a separate 3,600 s `aiperf` run on a deployment that had just completed a `CONCURRENCY=192` run. All runs completed with 0 failed requests. + +| Concurrency | Total tok/s (32 GPUs) | Output tok/s | TPOT p50 | TTFT p50 / p99 | Prefix-cache hit rate | Completed requests | +| :---: | :---: | :---: | :---: | :---: | :---: | :---: | +| 32 | 192,676 | 1,219 | 5.82 ms | 0.64 s / 2.63 s | 96.7% | 4,104 | +| 96 | 476,473 | 3,822 | 11.13 ms | 0.76 s / 6.79 s | 95.0% | 13,369 | +| 192 | 657,756 | 5,201 | 16.18 ms | 10.79 s / 34.97 s | 93.2% | 18,672 | + +The prefix-cache hit rate is `overall_usage_prompt_cache_read_pct` in `profile_export_aiperf.json`: cached prompt tokens divided by total prompt tokens, as reported in the response `usage` field. + ### 4.4. Category 2: Synthetic Random-Dataset Benchmarking From 309ccaba3471d8743a07500b4f16a18acfbdba28 Mon Sep 17 00:00:00 2001 From: Anish Shah Date: Fri, 16 Oct 2026 02:38:31 +0000 Subject: [PATCH 5/8] Add Helm-based G4 llm-d + vLLM 1M-context aggregated serving recipe Introduce the generic llm-d + vLLM 0.19.1rc1 aggregated serving recipe on GKE G4 (8x NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB) for reflection/Beam-1.0-501B-A23B-NVFP4 (LASER_BEAM) at 1,048,576 (1M) token context length: - src/helm-charts/g4/llm-d-inference: Generic, cluster-agnostic Helm chart packaging the llm-d Router (Envoy + llm-d-epp v0.5.1 with PrefixCachePlugin, UtilizationDetector, TokenLoadScorer, MaxScorePicker, and self-contained ServiceAccount/RBAC templates) alongside N x TP=4/TP=8 vLLM model servers with host-DRAM KV cache offloading (OffloadingConnector / CPUOffloadingSpec) and NUMA pinning. - src/frameworks/g4/llmd-configs: Reference EndpointPickerConfig and vLLM 1M-context configuration profiles. - inference/g4/aggregated-serving/llm-d: Recipe values.yaml, namespace-agnostic standalone Kubernetes manifests (router.yaml + modelserver-2x-tp4-1m.yaml), AgentX 60-minute benchmark client manifest (bench_client.yaml), and complete documentation + 1M-context benchmark results in README.md. --- .../g4/aggregated-serving/llm-d/README.md | 369 +++++++++++++ .../llm-d/bench_client.yaml | 62 +++ .../manifests/modelserver-2x-tp4-1m.yaml | 502 ++++++++++++++++++ .../llm-d/manifests/router.yaml | 491 +++++++++++++++++ .../g4/aggregated-serving/llm-d/values.yaml | 151 ++++++ .../laser_beam_1m_offload_epp.yaml | 91 ++++ .../laser_beam_1m_offload_vllm.yaml | 42 ++ src/helm-charts/g4/llm-d-inference/Chart.yaml | 20 + .../llm-d-inference/templates/configmaps.yaml | 347 ++++++++++++ .../g4/llm-d-inference/templates/router.yaml | 203 +++++++ .../llm-d-inference/templates/services.yaml | 100 ++++ .../templates/vllm-deployments.yaml | 319 +++++++++++ .../g4/llm-d-inference/values.yaml | 151 ++++++ 13 files changed, 2848 insertions(+) create mode 100644 inference/g4/aggregated-serving/llm-d/README.md create mode 100644 inference/g4/aggregated-serving/llm-d/bench_client.yaml create mode 100644 inference/g4/aggregated-serving/llm-d/manifests/modelserver-2x-tp4-1m.yaml create mode 100644 inference/g4/aggregated-serving/llm-d/manifests/router.yaml create mode 100644 inference/g4/aggregated-serving/llm-d/values.yaml create mode 100644 src/frameworks/g4/llmd-configs/laser_beam_1m_offload_epp.yaml create mode 100644 src/frameworks/g4/llmd-configs/laser_beam_1m_offload_vllm.yaml create mode 100644 src/helm-charts/g4/llm-d-inference/Chart.yaml create mode 100644 src/helm-charts/g4/llm-d-inference/templates/configmaps.yaml create mode 100644 src/helm-charts/g4/llm-d-inference/templates/router.yaml create mode 100644 src/helm-charts/g4/llm-d-inference/templates/services.yaml create mode 100644 src/helm-charts/g4/llm-d-inference/templates/vllm-deployments.yaml create mode 100644 src/helm-charts/g4/llm-d-inference/values.yaml diff --git a/inference/g4/aggregated-serving/llm-d/README.md b/inference/g4/aggregated-serving/llm-d/README.md new file mode 100644 index 00000000..d5785faa --- /dev/null +++ b/inference/g4/aggregated-serving/llm-d/README.md @@ -0,0 +1,369 @@ +# Aggregated 1M-Context Multi-Turn Inference with llm-d and vLLM on G4 GKE + +This document outlines the steps to deploy and benchmark `reflection/Beam-1.0-501B-A23B-NVFP4` (`LASER_BEAM`) at **1,048,576 (`1M`) context length** using the [llm-d](https://github.com/llm-d/llm-d) intelligent inference scheduling stack with the [vLLM](https://docs.vllm.ai/en/latest/) backend on [G4 (`g4-standard-384`, NVIDIA RTX PRO 6000 Blackwell Server Edition) GKE Node pools](https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus). + +`llm-d` pairs an Envoy proxy and an `ext_proc` Endpoint Picker (`EPP`) router with vLLM workers to provide real-time ZMQ KV-cache event tracking (`precise-prefix-cache-producer` on `:5556` / `:5559`), multi-tier prefix-affinity routing (`prefix-cache-affinity-filter` across GPU HBM and host DRAM), `program-aware-fairness` multi-turn scheduling (`strategy: turn-priority`), and queue/KV saturation flow control (`utilization-detector`). On `g4-standard-384` nodes (8x RTX PRO 6000 Blackwell Server Edition GPUs with 96 GB GDDR7 each + 1.44 TB host DRAM), this recipe deploys **`2x TP4 / EP4` (`8` GPUs total)** with **host DRAM KV-cache offloading (`OffloadingConnector` / `CPUOffloadingSpec`, `64 GiB` per TP4 replica / `128 GiB` per node)**, socket-local PCIe Gen5 **NUMA pinning (`numactl`)**, and **DFlash2 speculative decoding**. + + +## Table of Contents + +* [1. Test Environment & Benchmark Results](#test-environment) + * [1.1. Hardware & Software Configuration](#hardware-config) + * [1.2. Verified 60-Minute 1M-Context AgentX Benchmark Results](#benchmark-results) +* [2. Environment Setup (One-Time)](#environment-setup) + * [2.1. Clone the Repository](#clone-repo) + * [2.2. Configure Environment Variables](#configure-vars) + * [2.3. Connect to your GKE Cluster](#connect-cluster) + * [2.4. Create Secrets](#create-secrets) + * [2.5. Setup GCS Bucket for GKE](#setup-gcsfuse) + * [2.6. Configure Container Images](#configure-images) +* [3. Deploy the 1M-Context llm-d + vLLM Stack (2x TP4)](#deploy-llmd-vllm) + * [3.1. System Architecture & Key Optimizations](#system-architecture) + * [3.2. Post-Optimization 1M-Context Tuning Summary](#1m-context-optimizations) + * [3.3. Option A: Deploy with Helm](#deploy-helm) + * [3.4. Option B: Deploy with Standalone Kubernetes Manifests](#deploy-manifests) +* [4. Inference Request and Benchmarking](#inference-request) + * [4.1. Verify Deployment Readiness](#verify-readiness) + * [4.2. Deploy the Benchmark Client](#deploy-bench-client) + * [4.3. Agentic Multi-Turn Profiling with AIPerf (1M-Context AgentX Dataset)](#benchmark-aiperf-agentx) +* [5. Monitoring and Troubleshooting](#monitoring) +* [6. Cleanup](#cleanup) + + +## 1. Test Environment & Benchmark Results + +[Back to Top](#table-of-contents) + + +### 1.1. Hardware & Software Configuration + +This recipe has been tested and benchmarked with the following configuration: + +* **GKE Cluster**: + * GPU node pools with [g4-standard-384](https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus) machines (`cloud.google.com/gke-accelerator: nvidia-rtx-pro-6000`, 8x NVIDIA RTX PRO 6000 Blackwell Server Edition `SM120` GPUs with 96 GB GDDR7 each = 768 GB GPU memory per node, 384 vCPUs across 2 NUMA sockets, 1,440 GB host DRAM). + * **Serving Topology**: `2x TP4 / EP4` (`8` GPUs total across 2 four-GPU instances, NUMA-pinned per 4-GPU PCIe socket via `numactl --cpunodebind={0|1} --membind={0|1}`). + * [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) enabled. + * [Cloud Storage FUSE CSI driver for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cloud-storage-fuse-csi-driver) enabled. + + +### 1.2. Verified 60-Minute 1M-Context AgentX Benchmark Results + +Evaluated using [`aiperf`](https://github.com/ai-dynamo/aiperf) (`--scenario agentx --benchmark-duration 3600`) with the SemiAnalysis AgentX multi-turn coding trajectory dataset (`semianalysis_cc_traces_weka_062126`, up to `1M` context length) across `8` RTX PRO 6000 Blackwell GPUs (`2x TP4`): + +| Concurrency | Stack Configuration | Output Throughput (`tok/s/GPU`) | Prefix KV-Cache Hit Rate (`%`) | P90 TTFT (`s`) | Speedup / Improvement (`llm-d` vs. Baseline) | +| :--- | :--- | :--- | :--- | :--- | :--- | +| **`c=8`** | **llm-d + vLLM (DRAM Offload)** | **`39.7 tok/s/GPU`** | **`95.9%`** | — | **+13.4% Throughput**, **+3.5% KV Hit Rate** | +| **`c=8`** | Standalone vLLM Baseline | `35.0 tok/s/GPU` | `92.4%` | — | *(Reference)* | +| **`c=16`** | **llm-d + vLLM (DRAM Offload)** | **`51.3 tok/s/GPU`** | **`88.5%`** | **`10.26s`** | **+95.8% (1.96x) Throughput**, **+34.4% KV Hit**, **11.3x Lower P90 TTFT** | +| **`c=16`** | Standalone vLLM Baseline | `26.2 tok/s/GPU` | `54.1%` | `115.70s` | *(Reference — severe HBM KV eviction & recomputation)* | +| **`c=32`** | **llm-d + vLLM (DRAM Offload)** | **`25.5 tok/s/GPU`** | **`59.1%`** | **`195.98s`** | Sustains `59.1%` KV hit rate under `32` concurrent 1M-context sessions | + + +## 2. Environment Setup (One-Time) + +[Back to Top](#table-of-contents) + + +### 2.1. Clone the Repository + +```bash +git clone https://github.com/ai-hypercomputer/gpu-recipes.git +cd gpu-recipes +export REPO_ROOT=$(pwd) +export RECIPE_ROOT=$REPO_ROOT/inference/g4/aggregated-serving/llm-d +``` + + +### 2.2. Configure Environment Variables + +```bash +export PROJECT_ID= +export CLUSTER_REGION= +export CLUSTER_ZONE= +export CLUSTER_NAME= +export NAMESPACE=llmd-inference +export HF_TOKEN= +export GCS_BUCKET= +export LASER_BEAM_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-NVFP4 +export LASER_BEAM_DRAFTER_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-DFlash2 + +# Set the project for gcloud commands +gcloud config set project $PROJECT_ID +``` + +Replace the following values: + +| Variable | Description | Example | +| -------- | ----------- | ------- | +| `PROJECT_ID` | Your Google Cloud Project ID | `` | +| `CLUSTER_REGION` | The GCP region where your GKE cluster is located | `` | +| `CLUSTER_ZONE` | The GCP zone where your GKE node pool is located (if zonal) | `` | +| `CLUSTER_NAME` | The name of your GKE cluster | `` | +| `NAMESPACE` | Kubernetes namespace for the deployment | `llmd-inference` | +| `HF_TOKEN` | Your Hugging Face access token | `` | +| `GCS_BUCKET` | Your GCS bucket name (without `gs://` prefix for Helm `bucketName`) | `` | +| `LASER_BEAM_HF_MODEL_ID` | Target model Hugging Face ID | `reflection/Beam-1.0-501B-A23B-NVFP4` | +| `LASER_BEAM_DRAFTER_HF_MODEL_ID` | DFlash2 speculative drafter Hugging Face ID | `reflection/Beam-1.0-501B-A23B-DFlash2` | + + +### 2.3. Connect to your GKE Cluster + +```bash +gcloud container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_REGION +``` + + +### 2.4. Create Secrets + +Create the namespace: +```bash +kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f - +kubectl config set-context --current --namespace=$NAMESPACE +``` + +Create the secret for the Hugging Face token: +```bash +kubectl create secret generic hf-token-secret \ + --from-literal=HF_TOKEN=${HF_TOKEN} \ + -n ${NAMESPACE} +``` + + +### 2.5. Setup GCS Bucket for GKE (One-Time Setup) + +If storing model weights in Google Cloud Storage, utilize [Cloud Storage FUSE (`gcsfuse`)](https://cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-setup) to facilitate fast parallel model loading: + +```bash +export SERVICE_ACCOUNT_EMAIL=$(kubectl get serviceaccount/default -n ${NAMESPACE} -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}') + +gcloud iam service-accounts add-iam-policy-binding ${SERVICE_ACCOUNT_EMAIL} \ + --role roles/iam.workloadIdentityUser \ + --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${NAMESPACE}/default]" + +gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \ + --member "serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \ + --role "roles/storage.objectViewer" +``` + + +### 2.6. Configure Container Images + +Set your environment variables for the vLLM engine image (built with `SM120` Blackwell support, `flashinfer_cutlass` MoE kernels, DFlash2 speculative decoding, and `OffloadingConnector` KV-cache CPU offloading) and the `llm-d` Endpoint Picker (`EPP`) router image: + +```bash +export VLLM_IMAGE="us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64" +export EPP_IMAGE="ghcr.io/llm-d/llm-d-router-endpoint-picker:main" +``` + + +## 3. Deploy the 1M-Context llm-d + vLLM Stack (2x TP4) + +[Back to Top](#table-of-contents) + + +### 3.1. System Architecture & Key Optimizations + +```mermaid +flowchart LR + classDef client fill:#e3f2fd,stroke:#90caf9,stroke-width:2px; + classDef router fill:#fff3e0,stroke:#ffcc80,stroke-width:2px; + classDef worker fill:#e8f5e9,stroke:#a5d6a7,stroke-width:2px; + classDef memory fill:#f3e5f5,stroke:#ce93d8,stroke-width:2px; + + Client["AIPerf / OpenAI Client
(1M-Context AgentX)"]:::client + + subgraph RouterPod ["llm-d Router Pod ({{RELEASE_NAME}}-router)"] + direction TB + Envoy["Envoy Proxy (:8081)
ORIGINAL_DST Dynamic Routing"]:::router + EPP["llm-d EPP (:9002 / :9090)
- precise-prefix-cache-producer
- prefix-cache-affinity-filter
- program-aware-fairness
- utilization-detector"]:::router + Envoy <-->|"gRPC ext_proc (:9002)"| EPP + end + + subgraph G4Node ["G4 Node (g4-standard-384: 8x RTX PRO 6000 Blackwell 96GB + 1.44TB DRAM)"] + direction TB + vLLM0["vLLM Replica 0 (TP4, GPUs 0..3, NUMA 0)
numactl --cpunodebind=0 --membind=0
--max-model-len 1048576
--max-num-batched-tokens 16384"]:::worker + vLLM1["vLLM Replica 1 (TP4, GPUs 4..7, NUMA 1)
numactl --cpunodebind=1 --membind=1
--max-model-len 1048576
--max-num-batched-tokens 16384"]:::worker + DRAM0["Socket 0 Host DRAM KV Pool
OffloadingConnector (64 GiB)"]:::memory + DRAM1["Socket 1 Host DRAM KV Pool
OffloadingConnector (64 GiB)"]:::memory + + vLLM0 <-->|"Socket-Local PCIe Gen5 DMA"| DRAM0 + vLLM1 <-->|"Socket-Local PCIe Gen5 DMA"| DRAM1 + end + + Client -->|"HTTP :8000"| Envoy + Envoy -->|"x-gateway-destination-endpoint"| vLLM0 + Envoy -->|"x-gateway-destination-endpoint"| vLLM1 + vLLM0 -.->|"ZMQ KV Events (:5556 / :5559)"| EPP + vLLM1 -.->|"ZMQ KV Events (:5556 / :5559)"| EPP +``` + + +### 3.2. Post-Optimization 1M-Context Tuning Summary + +| Component & Optimization | Configured Value | Purpose on 1M-Context Multi-Turn Workloads | +| :--- | :--- | :--- | +| **#1 NUMA Pinning (`numactl`)** | `numactl --cpunodebind=0 --membind=0` (GPUs `0..3`), `numactl --cpunodebind=1 --membind=1` (GPUs `4..7`) | Keeps PCIe Gen5 CPU<->GPU KV-cache DMA transfers and worker threads 100% socket-local without QPI/UPI cross-socket traversal. | +| **#2 Host DRAM KV-Cache Offloading (`OffloadingConnector` / `CPUOffloadingSpec`)** | `cpu_bytes_to_use: 68719476736` (`64 GiB` per TP4 replica = `128 GiB/node` for 2x TP4/node; or `137438953472` = `128 GiB/replica` for 1x TP4/node), `blocks_per_chunk: 16`, `offload_prompt_only: true` | Extends effective KV-cache capacity into `g4-standard-384` host DRAM and publishes ZMQ tier events (`:5556` / `:5559`, `gpu: 1.0`, `cpu: 0.4`) so evicted multi-turn prefixes are reloaded via PCIe DMA instead of recomputed. | +| **#3 Engine Batching & CUDA Graphs** | `--max-num-batched-tokens 16384`, `--max-num-seqs 16`, `--block-size 64`, `FULL_AND_PIECEWISE` CUDA graphs `1..1152` (`fuse_allreduce_rms: false`) | Cuts 100K–700K prefill chunks in half (updated from `8192` to `16384`), bounds concurrent active sequences (`16`), and captures CUDA graphs up to `1152` tokens for speculative decoding. | +| **#4 DFlash2 Speculative Decoding** | `num_speculative_tokens: 8`, `draft_tensor_parallel_size: 4`, `--no-disable-hybrid-kv-cache-manager`, `--swa-bounded-replay`, `--moe-backend flashinfer_cutlass` | Accelerates decode interactivity (TPOT/ITL) using block-diffusion speculative decoding with bounded sliding-window attention replay. | +| **#5 `llm-d` EPP Routing & Flow Control (`EndpointPickerConfig`)** | `prefix-cache-affinity-filter` (`affinityThreshold: 0.75`, `peakPrefillThroughput: 16444`), `utilization-detector` (`queueDepthThreshold: 6`, `kvCacheUtilThreshold: 0.80`), `program-aware-fairness` (`turn-priority`), `speculativeIndexing: true` | Directs multi-turn agentic sessions to the pod holding their GPU/DRAM prefix (`88.5%` KV hit at `c=16` vs. `54.1%` without `llm-d`, yielding **1.96x higher throughput** and **11.3x lower P90 TTFT**). | + + +### 3.3. Option A: Deploy with Helm + +1. Set the release and model variables: + +```bash +export HF_MODEL_ID="${LASER_BEAM_HF_MODEL_ID}" +export DRAFTER_MODEL_ID="${LASER_BEAM_DRAFTER_HF_MODEL_ID}" +export RELEASE_NAME="llmd-inference" +``` + +2. Install the Helm chart (`$REPO_ROOT/src/helm-charts/g4/llm-d-inference`) using `values.yaml`: + +```bash +cd $RECIPE_ROOT +helm install ${RELEASE_NAME} \ + $REPO_ROOT/src/helm-charts/g4/llm-d-inference \ + -f values.yaml \ + --namespace ${NAMESPACE} \ + --set workload.model.name=${HF_MODEL_ID} \ + --set workload.model.drafterName=${DRAFTER_MODEL_ID} \ + --set workload.image=${VLLM_IMAGE} \ + --set llmd.router.eppImage=${EPP_IMAGE} +``` + +*(Optional)* To mount model weights from Google Cloud Storage via `gcsfuse`, append `--set volumes.useGcs=true --set volumes.gcsfuse.bucketName=${GCS_BUCKET}`. If deploying into a GKE namespace with an existing pre-bound ServiceAccount instead of creating a new Role/RoleBinding, append `--set llmd.router.rbac.create=false --set llmd.router.serviceAccount.create=false --set llmd.router.serviceAccount.name=`. + + +### 3.4. Option B: Deploy with Standalone Kubernetes Manifests + +Pre-rendered Kubernetes manifests (`RELEASE_NAME=llmd-inference`) for the `llm-d` router and the `2x TP4` `1M`-context vLLM model server replicas are also available under `$RECIPE_ROOT/manifests/`: + +```bash +cd $RECIPE_ROOT +kubectl apply -f manifests/router.yaml -n ${NAMESPACE} +kubectl apply -f manifests/modelserver-2x-tp4-1m.yaml -n ${NAMESPACE} +``` + + +## 4. Inference Request and Benchmarking + +[Back to Top](#table-of-contents) + + +### 4.1. Verify Deployment Readiness + +Before sending inference or benchmark requests, verify that the `llm-d` router (`2/2` containers: `envoy-proxy` + `router`) and both `vLLM` worker pods (`1/1`) are in the `Running` and `Ready` state: + +```bash +kubectl get pods -n ${NAMESPACE} -o wide +``` + +Expected output: +```text +NAME READY STATUS RESTARTS AGE +llmd-inference-router-xxxxx-xxxxx 2/2 Running 0 15m +llmd-inference-vllm-0-xxxxx-xxxxx 1/1 Running 0 15m +llmd-inference-vllm-1-xxxxx-xxxxx 1/1 Running 0 15m +``` + +Verify that the OpenAI `/v1/models` endpoint responds through the frontend Service: +```bash +kubectl run curl-test --rm -i --tty --restart=Never --image=curlimages/curl -n ${NAMESPACE} -- \ + curl -s http://${RELEASE_NAME}-frontend:8000/v1/models +``` + + +### 4.2. Deploy the Benchmark Client + +Deploy the standalone `bench-client` pod, which installs `aiperf` for AgentX 1M-context multi-turn profiling: + +```bash +cd $RECIPE_ROOT +kubectl apply -f bench_client.yaml -n ${NAMESPACE} +``` + +Wait for `bench-client` to finish setup: +```bash +kubectl logs -f pod/bench-client -n ${NAMESPACE} +``` + +Proceed once you see `--- SETUP COMPLETE. POD IS READY. ---`. + + +### 4.3. Agentic Multi-Turn Profiling with AIPerf (1M-Context AgentX Dataset) + +Use [`aiperf`](https://github.com/ai-dynamo/aiperf) with the SemiAnalysis AgentX coding trajectory dataset (`semianalysis_cc_traces_weka_062126`) to evaluate 1M-context multi-turn prefix caching, DRAM KV offloading, and decode interactivity at concurrencies `CONCURRENCY=8`, `16`, or `32`: + +```bash +export FRONTEND_HOST="${RELEASE_NAME}-frontend" +export MODEL_NAME="${LASER_BEAM_HF_MODEL_ID}" +export TOKENIZER_PATH="${LASER_BEAM_HF_MODEL_ID}" +export CONCURRENCY=16 + +kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " + aiperf profile \ + --scenario agentx \ + --url http://${FRONTEND_HOST}:8000 \ + --endpoint /v1/chat/completions \ + --model ${MODEL_NAME} \ + --tokenizer ${TOKENIZER_PATH} \ + --concurrency ${CONCURRENCY} \ + --benchmark-duration 3600 \ + --benchmark-grace-period inf \ + --failed-request-threshold 0.50 \ + --warmup-requests-per-lane 10 \ + --trace-idle-gap-cap-seconds 300 \ + --warmup-grace-period 1800 \ + --agentic-warmup-grace-period 1800 \ + --tokenizer-trust-remote-code \ + --server-metrics http://${RELEASE_NAME}-backends:8000/metrics \ + --output-artifact-dir /logs/agentic/aiperf_artifacts/llmd_1m_c${CONCURRENCY} \ + --public-dataset semianalysis_cc_traces_weka_062126 +" +``` + + +## 5. Monitoring and Troubleshooting + +[Back to Top](#table-of-contents) + +View logs for the `llm-d` router and `vLLM` worker pods: + +```bash +# List pods +kubectl get pods -n ${NAMESPACE} + +# llm-d EPP router logs (prefix cache hits, affinity scores, flow control admissions) +kubectl logs -f deployment/${RELEASE_NAME}-router -c router -n ${NAMESPACE} + +# Envoy proxy logs +kubectl logs -f deployment/${RELEASE_NAME}-router -c envoy-proxy -n ${NAMESPACE} + +# vLLM instance 0 & 1 logs (KV cache usage, DRAM offload bandwidth, DFlash2 acceptance rate) +kubectl logs -f deployment/${RELEASE_NAME}-vllm-0 -n ${NAMESPACE} +kubectl logs -f deployment/${RELEASE_NAME}-vllm-1 -n ${NAMESPACE} +``` + +Inspect real-time Prometheus metrics from the `llm-d` EPP router (`:9090/metrics`): +```bash +kubectl exec -it deployment/${RELEASE_NAME}-router -c router -n ${NAMESPACE} -- \ + wget -qO- http://localhost:9090/metrics | grep -E "inference_pool|prefix_cache|flow_control" +``` + + +## 6. Cleanup + +[Back to Top](#table-of-contents) + +List deployed Helm releases: +```bash +helm list -n ${NAMESPACE} +``` + +Uninstall the deployment and benchmark client: +```bash +helm uninstall ${RELEASE_NAME} -n ${NAMESPACE} +kubectl delete -f $RECIPE_ROOT/bench_client.yaml -n ${NAMESPACE} --ignore-not-found=true +``` diff --git a/inference/g4/aggregated-serving/llm-d/bench_client.yaml b/inference/g4/aggregated-serving/llm-d/bench_client.yaml new file mode 100644 index 00000000..fa532542 --- /dev/null +++ b/inference/g4/aggregated-serving/llm-d/bench_client.yaml @@ -0,0 +1,62 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: v1 +kind: Pod +metadata: + name: bench-client + labels: + app: bench-client +spec: + restartPolicy: Never + containers: + - name: benchmark + image: python:3.10 + workingDir: /workspace + command: ["/bin/bash", "-c"] + # This script runs ONCE when the pod starts to set everything up. + # Then it sleeps forever so the pod stays open for you. + args: + - | + echo "--- STARTING SETUP ---" + + # 1. Install System Dependencies + apt-get update && apt-get install -y git curl jq + + # 2. Install Python Dependencies (including AIPerf for AgentX profiling and synthetic benchmark deps) + pip install -q --upgrade pip + pip install -q aiperf transformers aiohttp numpy requests tqdm pandas datasets Pillow huggingface_hub hf_transfer + + # 3. Create Artifact Directory for AIPerf Profiling Outputs + mkdir -p /logs/agentic/aiperf_artifacts + + echo "--- SETUP COMPLETE. POD IS READY. ---" + + # 4. Keep the pod alive indefinitely + sleep infinity + env: + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + optional: true + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + volumeMounts: + - mountPath: /logs + name: logs-volume + volumes: + - name: logs-volume + emptyDir: {} diff --git a/inference/g4/aggregated-serving/llm-d/manifests/modelserver-2x-tp4-1m.yaml b/inference/g4/aggregated-serving/llm-d/manifests/modelserver-2x-tp4-1m.yaml new file mode 100644 index 00000000..0b0a0ab0 --- /dev/null +++ b/inference/g4/aggregated-serving/llm-d/manifests/modelserver-2x-tp4-1m.yaml @@ -0,0 +1,502 @@ +--- +# Source: llm-d-inference/templates/configmaps.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: "llmd-inference-vllm-config" + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server +data: + serving-configuration: |- + host: "0.0.0.0" + port: "8000" + tensor-parallel-size: "4" + enable-expert-parallel: true + trust-remote-code: true + max-model-len: "1.048576e+06" + gpu-memory-utilization: "0.90" + kv-cache-dtype: "fp8" + block-size: "64" + max-num-batched-tokens: "16384" + max-num-seqs: "16" + speculative-config: '{"method":"dflash","model":"${DRAFTER_MODEL_ID}","num_speculative_tokens":8,"draft_tensor_parallel_size":4}' + compilation-config: '{"cudagraph_capture_sizes":[1,2,4,8,16,32,48,64,96,128,192,256,384,512,640,768,896,1024,1152],"cudagraph_mode":"FULL_AND_PIECEWISE","pass_config":{"fuse_allreduce_rms":false}}' + enable-prefix-caching: true + no-disable-hybrid-kv-cache-manager: true + swa-bounded-replay: true + enable-auto-tool-choice: true + tool-call-parser: "beam" + reasoning-parser: "beam" + default-chat-template-kwargs: '{"allow_missing_thinking":true,"reasoning_effort":"max"}' + override-generation-config: '{"temperature":1.0,"top_k":40,"top_p":0.95}' + moe-backend: "flashinfer_cutlass" + kv-transfer-config: '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","engine_id":"w1_0","kv_connector_extra_config":{"spec_name":"CPUOffloadingSpec","cpu_bytes_to_use":68719476736,"blocks_per_chunk":16,"offload_prompt_only":true}}' + kv-events-config: '{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:${ZMQ_PUB_PORT}","replay_endpoint":"tcp://*:${ZMQ_REPLAY_PORT}","topic":"kv@${POD_IP}:${VLLM_HTTP_PORT}@${SERVED_MODEL_NAME}"}' +--- +# Source: llm-d-inference/templates/configmaps.yaml +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--- +# Source: llm-d-inference/templates/vllm-deployments.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: "llmd-inference-vllm-0" + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-0" +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-0" + template: + metadata: + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-0" + annotations: + spec: + nodeSelector: + cloud.google.com/gke-accelerator: nvidia-rtx-pro-6000 + tolerations: + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" + - key: "cloud.google.com/impending-node-termination" + operator: "Exists" + containers: + - name: vllm + image: us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64 + imagePullPolicy: IfNotPresent + command: ["/bin/bash", "-lc"] + args: + - | + set -euo pipefail + unset PYTORCH_CUDA_ALLOC_CONF + nvidia-smi + python3 - << 'PYEOF' + import json, os, shutil, sys, yaml + + model_id = os.environ["MODEL_ID"] + served_model = os.environ.get("SERVED_MODEL_NAME") or model_id + os.environ["SERVED_MODEL_NAME"] = served_model + + mount_dir = os.environ.get("MODEL_MOUNT_DIR", "") + if mount_dir and os.path.isdir(mount_dir): + model_target = mount_dir + else: + model_target = model_id + + drafter_mount = os.environ.get("DRAFTER_MOUNT_DIR", "") + if drafter_mount and os.path.isdir(drafter_mount): + os.environ["DRAFTER_MODEL_ID"] = drafter_mount + + numa_node = os.environ.get("NUMA_NODE", "0") + if shutil.which("numactl"): + cmd = [ + "numactl", + f"--cpunodebind={numa_node}", + f"--membind={numa_node}", + "vllm", + "serve", + model_target, + "--served-model-name", + served_model, + ] + else: + cmd = ["vllm", "serve", model_target, "--served-model-name", served_model] + + cfg_path = os.environ.get("SERVER_ARGS_FILE", "") + if cfg_path and os.path.exists(cfg_path): + with open(cfg_path, "r") as f: + cfg = yaml.safe_load(f) or {} + for k, v in cfg.items(): + if k in ("model", "served-model-name") or v is None or v is False: + continue + flag = f"--{k}" + if v is True: + cmd.append(flag) + elif isinstance(v, (dict, list)): + cmd.extend([flag, os.path.expandvars(json.dumps(v))]) + else: + cmd.extend([flag, os.path.expandvars(str(v))]) + + print("Launching vLLM with NUMA pinning:", " ".join(cmd), flush=True) + os.execvp(cmd[0], cmd) + PYEOF + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_HOST_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_HTTP_PORT + value: "8000" + - name: ZMQ_PUB_PORT + value: "5556" + - name: ZMQ_REPLAY_PORT + value: "5559" + - name: CUDA_VISIBLE_DEVICES + value: "0,1,2,3" + - name: NUMA_NODE + value: "0" + - name: MODEL_ID + value: "reflection/Beam-1.0-501B-A23B-NVFP4" + - name: DRAFTER_MODEL_ID + value: "reflection/Beam-1.0-501B-A23B-DFlash2" + - name: SERVED_MODEL_NAME + value: "reflection/Beam-1.0-501B-A23B-NVFP4" + - name: HF_HOME + value: "/ssd" + - name: SERVER_ARGS_FILE + value: "/workload/configs/serving-configuration" + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + optional: true + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: VLLM_ENGINE_READY_TIMEOUT_S + value: "3600" + - name: VLLM_RPC_TIMEOUT + value: "600000" + - name: VLLM_HTTP_TIMEOUT_KEEP_ALIVE + value: "900" + - name: VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT + value: "300" + - name: VLLM_USE_V2_MODEL_RUNNER + value: "1" + - name: VLLM_FUSED_MOE_CHUNK_SIZE + value: "32768" + - name: VLLM_SWA_DECODER_REPLAY_MAX_BS + value: "4096" + - name: VLLM_ENFORCE_STRICT_TOOL_CALLING + value: "0" + - name: VLLM_BEAM_ALWAYS_CONSTRAIN_OUTPUT + value: "0" + - name: NCCL_IB_DISABLE + value: "1" + - name: NCCL_P2P_LEVEL + value: NVL + - name: NCCL_NET_GDR_LEVEL + value: "0" + - name: PYTHONUNBUFFERED + value: "1" + ports: + - name: http + containerPort: 8000 + - name: render + containerPort: 6100 + - name: kv-events + containerPort: 5556 + - name: kv-replay + containerPort: 5559 + readinessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 30 + periodSeconds: 5 + timeoutSeconds: 5 + failureThreshold: 60 + livenessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 300 + periodSeconds: 30 + timeoutSeconds: 10 + failureThreshold: 10 + resources: + limits: + cpu: "160" + memory: 1200Gi + nvidia.com/gpu: "4" + requests: + cpu: "64" + memory: 512Gi + nvidia.com/gpu: "4" + volumeMounts: + - name: dshm + mountPath: /dev/shm + - name: serving-configuration + mountPath: /workload/configs + readOnly: true + - name: local-ssd + mountPath: /ssd + volumes: + - name: dshm + emptyDir: + medium: Memory + sizeLimit: 256Gi + - name: serving-configuration + configMap: + name: "llmd-inference-vllm-config" + - name: local-ssd + emptyDir: {} +--- +# Source: llm-d-inference/templates/vllm-deployments.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: "llmd-inference-vllm-1" + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-1" +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-1" + template: + metadata: + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-1" + annotations: + spec: + nodeSelector: + cloud.google.com/gke-accelerator: nvidia-rtx-pro-6000 + affinity: + podAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-0" + topologyKey: kubernetes.io/hostname + tolerations: + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" + - key: "cloud.google.com/impending-node-termination" + operator: "Exists" + containers: + - name: vllm + image: us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64 + imagePullPolicy: IfNotPresent + command: ["/bin/bash", "-lc"] + args: + - | + set -euo pipefail + unset PYTORCH_CUDA_ALLOC_CONF + nvidia-smi + python3 - << 'PYEOF' + import json, os, shutil, sys, yaml + + model_id = os.environ["MODEL_ID"] + served_model = os.environ.get("SERVED_MODEL_NAME") or model_id + os.environ["SERVED_MODEL_NAME"] = served_model + + mount_dir = os.environ.get("MODEL_MOUNT_DIR", "") + if mount_dir and os.path.isdir(mount_dir): + model_target = mount_dir + else: + model_target = model_id + + drafter_mount = os.environ.get("DRAFTER_MOUNT_DIR", "") + if drafter_mount and os.path.isdir(drafter_mount): + os.environ["DRAFTER_MODEL_ID"] = drafter_mount + + numa_node = os.environ.get("NUMA_NODE", "0") + if shutil.which("numactl"): + cmd = [ + "numactl", + f"--cpunodebind={numa_node}", + f"--membind={numa_node}", + "vllm", + "serve", + model_target, + "--served-model-name", + served_model, + ] + else: + cmd = ["vllm", "serve", model_target, "--served-model-name", served_model] + + cfg_path = os.environ.get("SERVER_ARGS_FILE", "") + if cfg_path and os.path.exists(cfg_path): + with open(cfg_path, "r") as f: + cfg = yaml.safe_load(f) or {} + for k, v in cfg.items(): + if k in ("model", "served-model-name") or v is None or v is False: + continue + flag = f"--{k}" + if v is True: + cmd.append(flag) + elif isinstance(v, (dict, list)): + cmd.extend([flag, os.path.expandvars(json.dumps(v))]) + else: + cmd.extend([flag, os.path.expandvars(str(v))]) + + print("Launching vLLM with NUMA pinning:", " ".join(cmd), flush=True) + os.execvp(cmd[0], cmd) + PYEOF + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_HOST_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_HTTP_PORT + value: "8000" + - name: ZMQ_PUB_PORT + value: "5556" + - name: ZMQ_REPLAY_PORT + value: "5559" + - name: CUDA_VISIBLE_DEVICES + value: "4,5,6,7" + - name: NUMA_NODE + value: "1" + - name: MODEL_ID + value: "reflection/Beam-1.0-501B-A23B-NVFP4" + - name: DRAFTER_MODEL_ID + value: "reflection/Beam-1.0-501B-A23B-DFlash2" + - name: SERVED_MODEL_NAME + value: "reflection/Beam-1.0-501B-A23B-NVFP4" + - name: HF_HOME + value: "/ssd" + - name: SERVER_ARGS_FILE + value: "/workload/configs/serving-configuration" + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + optional: true + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: VLLM_ENGINE_READY_TIMEOUT_S + value: "3600" + - name: VLLM_RPC_TIMEOUT + value: "600000" + - name: VLLM_HTTP_TIMEOUT_KEEP_ALIVE + value: "900" + - name: VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT + value: "300" + - name: VLLM_USE_V2_MODEL_RUNNER + value: "1" + - name: VLLM_FUSED_MOE_CHUNK_SIZE + value: "32768" + - name: VLLM_SWA_DECODER_REPLAY_MAX_BS + value: "4096" + - name: VLLM_ENFORCE_STRICT_TOOL_CALLING + value: "0" + - name: VLLM_BEAM_ALWAYS_CONSTRAIN_OUTPUT + value: "0" + - name: NCCL_IB_DISABLE + value: "1" + - name: NCCL_P2P_LEVEL + value: NVL + - name: NCCL_NET_GDR_LEVEL + value: "0" + - name: PYTHONUNBUFFERED + value: "1" + ports: + - name: http + containerPort: 8000 + - name: render + containerPort: 6100 + - name: kv-events + containerPort: 5556 + - name: kv-replay + containerPort: 5559 + readinessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 30 + periodSeconds: 5 + timeoutSeconds: 5 + failureThreshold: 60 + livenessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 300 + periodSeconds: 30 + timeoutSeconds: 10 + failureThreshold: 10 + resources: + limits: + cpu: "160" + memory: 1200Gi + nvidia.com/gpu: "4" + requests: + cpu: "64" + memory: 512Gi + nvidia.com/gpu: "4" + volumeMounts: + - name: dshm + mountPath: /dev/shm + - name: serving-configuration + mountPath: /workload/configs + readOnly: true + - name: local-ssd + mountPath: /ssd + volumes: + - name: dshm + emptyDir: + medium: Memory + sizeLimit: 256Gi + - name: serving-configuration + configMap: + name: "llmd-inference-vllm-config" + - name: local-ssd + emptyDir: {} +--- +# Source: llm-d-inference/templates/vllm-deployments.yaml +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/inference/g4/aggregated-serving/llm-d/manifests/router.yaml b/inference/g4/aggregated-serving/llm-d/manifests/router.yaml new file mode 100644 index 00000000..95761801 --- /dev/null +++ b/inference/g4/aggregated-serving/llm-d/manifests/router.yaml @@ -0,0 +1,491 @@ +--- +# Source: llm-d-inference/templates/configmaps.yaml +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: v1 +kind: ConfigMap +metadata: + name: "llmd-inference-epp-config" + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: router +data: + plugins-config.yaml: |- + apiVersion: llm-d.ai/v1alpha1 + kind: EndpointPickerConfig + featureGates: + - flowControl + plugins: + - name: program-aware-fairness + type: program-aware-fairness + parameters: + strategy: turn-priority + - name: token-producer + type: token-producer + parameters: + modelName: "reflection/Beam-1.0-501B-A23B-NVFP4" + vllm: + url: "http://llmd-inference-backends:6100" + - name: endpoint-notification-source + type: endpoint-notification-source + - name: precise-prefix-cache-producer + type: precise-prefix-cache-producer + parameters: + tokenProcessorConfig: + blockSizeTokens: 64 + kvEventsConfig: + topicFilter: "kv@" + concurrency: 8 + discoverPods: true + podDiscoveryConfig: + socketPort: 5556 + replaySocketPort: 5559 + indexerConfig: + kvBlockIndexConfig: + enableMetrics: true + inMemoryConfig: + podCacheSize: 128 + kvCacheBackendConfigs: + - name: gpu + weight: 1.0 + - name: cpu + weight: 0.4 + speculativeIndexing: true + - name: inflight-load-producer + type: inflight-load-producer + parameters: + prefixMatchInfoProducerName: precise-prefix-cache-producer + - name: prefix-cache-affinity-filter + type: prefix-cache-affinity-filter + parameters: + peakPrefillThroughput: 16444 + inFlightLoadProducerName: inflight-load-producer + prefixMatchInfoProducerName: precise-prefix-cache-producer + affinityThreshold: 0.75 + - name: utilization-detector + type: utilization-detector + parameters: + queueDepthThreshold: 6 + kvCacheUtilThreshold: 0.8 + - name: token-load-scorer + type: token-load-scorer + schedulingProfiles: + - name: default + plugins: + - pluginRef: prefix-cache-affinity-filter + - pluginRef: utilization-detector + - pluginRef: token-load-scorer + dataLayer: + sources: + - pluginRef: endpoint-notification-source + extractors: + - pluginRef: precise-prefix-cache-producer + flowControl: + maxBytes: "10Gi" + maxRequests: "1k" + defaultRequestTTL: "1800s" + saturationDetector: + pluginRef: utilization-detector + defaultPriorityBand: + fairnessPolicyRef: program-aware-fairness +--- +# Source: llm-d-inference/templates/configmaps.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: "llmd-inference-envoy-config" + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: router +data: + envoy.yaml: |- + static_resources: + listeners: + - name: envoy-proxy-ready + address: + socket_address: + address: 0.0.0.0 + port_value: 19001 + filter_chains: + - filters: + - name: envoy.filters.network.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + stat_prefix: envoy-ready-http + route_config: + name: local_route + virtual_hosts: + - name: ready_host + domains: ["*"] + routes: + - match: + prefix: "/ready" + direct_response: + status: 200 + body: + inline_string: "OK" + http_filters: + - name: envoy.filters.http.router + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router + - name: vllm + address: + socket_address: + address: 0.0.0.0 + port_value: 8081 + filter_chains: + - filters: + - name: envoy.filters.network.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + stat_prefix: http-8081 + stream_idle_timeout: 3600s + route_config: + name: vllm + virtual_hosts: + - name: vllm-default + domains: ["*"] + routes: + - match: + prefix: "/metrics" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/ping" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/health" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/v1/models" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/version" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/" + route: + cluster: original_destination_cluster + timeout: 86400s + http_filters: + - name: envoy.filters.http.ext_proc + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor + failure_mode_allow: false + grpc_service: + envoy_grpc: + cluster_name: ext_proc + authority: localhost:9002 + timeout: 86400s + processing_mode: + request_header_mode: SEND + response_header_mode: SEND + request_body_mode: FULL_DUPLEX_STREAMED + response_body_mode: FULL_DUPLEX_STREAMED + request_trailer_mode: SEND + response_trailer_mode: SEND + message_timeout: 1000s + - name: envoy.filters.http.router + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router + clusters: + - name: model_server_service_cluster + type: STRICT_DNS + connect_timeout: 5s + lb_policy: ROUND_ROBIN + load_assignment: + cluster_name: model_server_service_cluster + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: "llmd-inference-backends" + port_value: 8000 + - name: original_destination_cluster + type: ORIGINAL_DST + connect_timeout: 1000s + lb_policy: CLUSTER_PROVIDED + original_dst_lb_config: + use_http_header: true + http_header_name: x-gateway-destination-endpoint + - name: ext_proc + type: STATIC + connect_timeout: 86400s + lb_policy: LEAST_REQUEST + transport_socket: + name: "envoy.transport_sockets.tls" + typed_config: + "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext + common_tls_context: + alpn_protocols: ["h2"] + validation_context: {} + typed_extension_protocol_options: + envoy.extensions.upstreams.http.v3.HttpProtocolOptions: + "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions + explicit_http_config: + http2_protocol_options: {} + load_assignment: + cluster_name: ext_proc + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: 127.0.0.1 + port_value: 9002 +--- +# Source: llm-d-inference/templates/services.yaml +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: v1 +kind: Service +metadata: + name: "llmd-inference-frontend" + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: router +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: router + ports: + - name: http + port: 80 + targetPort: 8081 + - name: http-openai + port: 8000 + targetPort: 8081 + - name: grpc-ext-proc + port: 9002 + targetPort: 9002 + - name: epp-metrics + port: 9090 + targetPort: 9090 +--- +# Source: llm-d-inference/templates/services.yaml +apiVersion: v1 +kind: Service +metadata: + name: "llmd-inference-backends" + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: model-server + ports: + - name: http + port: 8000 + targetPort: 8000 + - name: http-render + port: 6100 + targetPort: 6100 + - name: zmq-kv-events + port: 5556 + targetPort: 5556 + - name: zmq-kv-replay + port: 5559 + targetPort: 5559 +--- +# Source: llm-d-inference/templates/router.yaml +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: "llmd-inference-router" + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: router +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: router + template: + metadata: + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "llmd-inference" + app.kubernetes.io/component: router + spec: + automountServiceAccountToken: true + serviceAccountName: default + containers: + - name: envoy-proxy + image: docker.io/envoyproxy/envoy:distroless-v1.33.2 + imagePullPolicy: IfNotPresent + command: ["envoy"] + args: + - "--service-node" + - "envoy-proxy" + - "--log-level" + - "warn" + - "-c" + - "/etc/envoy/envoy.yaml" + ports: + - name: http-8081 + containerPort: 8081 + protocol: TCP + - name: ready-19001 + containerPort: 19001 + protocol: TCP + readinessProbe: + httpGet: + path: /ready + port: 19001 + scheme: HTTP + periodSeconds: 5 + timeoutSeconds: 1 + resources: + limits: + cpu: "8" + memory: 16Gi + requests: + cpu: "2" + memory: 4Gi + volumeMounts: + - name: envoy-config + mountPath: /etc/envoy + readOnly: true + - name: router + image: ghcr.io/llm-d/llm-d-router-endpoint-picker:main + imagePullPolicy: IfNotPresent + env: + - name: NAMESPACE + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: metadata.namespace + - name: POD_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: metadata.name + ports: + - name: grpc + containerPort: 9002 + protocol: TCP + - name: grpc-health + containerPort: 9003 + protocol: TCP + - name: metrics + containerPort: 9090 + protocol: TCP + args: + - "--endpoint-selector=app.kubernetes.io/instance=llmd-inference,app.kubernetes.io/name=llm-d-inference,app.kubernetes.io/component=model-server" + - "--endpoint-target-ports=8000" + - "--config-file=/etc/epp/plugins-config.yaml" + - "--grpc-health-port=9003" + - "--tracing=false" + - "--metrics-endpoint-auth=false" + - "--allow-experimental-plugins=true" + readinessProbe: + grpc: + port: 9003 + service: inference-extension + periodSeconds: 5 + timeoutSeconds: 1 + resources: + limits: + cpu: "8" + memory: 16Gi + requests: + cpu: "2" + memory: 4Gi + volumeMounts: + - name: epp-config + mountPath: /etc/epp + readOnly: true + volumes: + - name: envoy-config + configMap: + name: "llmd-inference-envoy-config" + - name: epp-config + configMap: + name: "llmd-inference-epp-config" +--- +# Source: llm-d-inference/templates/vllm-deployments.yaml +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/inference/g4/aggregated-serving/llm-d/values.yaml b/inference/g4/aggregated-serving/llm-d/values.yaml new file mode 100644 index 00000000..17062817 --- /dev/null +++ b/inference/g4/aggregated-serving/llm-d/values.yaml @@ -0,0 +1,151 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +llmd: + router: + enabled: true + replicas: 1 + serviceAccount: + create: true + name: "" + rbac: + create: true + eppImage: ghcr.io/llm-d/llm-d-router-endpoint-picker:main + envoyImage: docker.io/envoyproxy/envoy:distroless-v1.33.2 + verbosity: "3" + queueDepthThreshold: 6 + kvCacheUtilThreshold: 0.80 + nodeSelector: {} + tolerations: [] + resources: + epp: + requests: + cpu: "2" + memory: "4Gi" + limits: + cpu: "8" + memory: "16Gi" + envoy: + requests: + cpu: "2" + memory: "4Gi" + limits: + cpu: "8" + memory: "16Gi" + +workload: + model: + name: reflection/Beam-1.0-501B-A23B-NVFP4 + drafterName: reflection/Beam-1.0-501B-A23B-DFlash2 + servedModelName: "" + maxModelLen: 1048576 + image: us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64 + framework: vllm + replicas: 2 + gpusPerReplica: 4 + pinSingleNode: true + nodeSelector: + cloud.google.com/gke-accelerator: nvidia-rtx-pro-6000 + # Pin each 4-GPU TP4 replica to its local PCIe/CPU NUMA node: + # - Replica 0: GPUs 0,1,2,3 -> numactl --cpunodebind=0 --membind=0 + # - Replica 1: GPUs 4,5,6,7 -> numactl --cpunodebind=1 --membind=1 + cudaVisibleDevices: + - "0,1,2,3" + - "4,5,6,7" + numaNodes: + - "0" + - "1" + shmSize: "256Gi" + configPath: /workload/configs + configFile: serving-configuration + resources: + requests: + cpu: "64" + memory: "512Gi" + nvidia.com/gpu: "4" + limits: + cpu: "160" + memory: "1200Gi" + nvidia.com/gpu: "4" + envs: + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: VLLM_ENGINE_READY_TIMEOUT_S + value: "3600" + - name: VLLM_RPC_TIMEOUT + value: "600000" + - name: VLLM_HTTP_TIMEOUT_KEEP_ALIVE + value: "900" + - name: VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT + value: "300" + - name: VLLM_USE_V2_MODEL_RUNNER + value: "1" + - name: VLLM_FUSED_MOE_CHUNK_SIZE + value: "32768" + - name: VLLM_SWA_DECODER_REPLAY_MAX_BS + value: "4096" + - name: VLLM_ENFORCE_STRICT_TOOL_CALLING + value: "0" + - name: VLLM_BEAM_ALWAYS_CONSTRAIN_OUTPUT + value: "0" + - name: NCCL_IB_DISABLE + value: "1" + - name: NCCL_P2P_LEVEL + value: "NVL" + - name: NCCL_NET_GDR_LEVEL + value: "0" + - name: PYTHONUNBUFFERED + value: "1" + readinessProbe: + initialDelaySeconds: 30 + periodSeconds: 5 + timeoutSeconds: 5 + failureThreshold: 60 + livenessProbe: + initialDelaySeconds: 300 + periodSeconds: 30 + timeoutSeconds: 10 + failureThreshold: 10 + +secrets: + huggingface: + secretName: hf-token-secret + secretKey: HF_TOKEN + +volumes: + hfHome: /ssd + ssdMountPath: /ssd + useHostPathSsd: false + useGcs: false + gcsfuse: + bucketName: "" + mountPath: /data/model + pvc: + enabled: false + claimName: "" + mountPath: /models + +service: + type: ClusterIP + ports: + frontend: 8000 + envoyHttp: 8081 + eppGrpc: 9002 + eppMetrics: 9090 + tokenProducer: 6100 + zmqPub: 5556 + zmqRouter: 5559 + vllmHttp: 8000 + vllmZmq: 5556 + vllmZmqReplay: 5559 diff --git a/src/frameworks/g4/llmd-configs/laser_beam_1m_offload_epp.yaml b/src/frameworks/g4/llmd-configs/laser_beam_1m_offload_epp.yaml new file mode 100644 index 00000000..c19c761e --- /dev/null +++ b/src/frameworks/g4/llmd-configs/laser_beam_1m_offload_epp.yaml @@ -0,0 +1,91 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: llm-d.ai/v1alpha1 +kind: EndpointPickerConfig +featureGates: +- flowControl +plugins: +- name: program-aware-fairness + type: program-aware-fairness + parameters: + strategy: turn-priority +- name: token-producer + type: token-producer + parameters: + modelName: "reflection/Beam-1.0-501B-A23B-NVFP4" + vllm: + url: "http://llmd-inference-backends:6100" +- name: endpoint-notification-source + type: endpoint-notification-source +- name: precise-prefix-cache-producer + type: precise-prefix-cache-producer + parameters: + tokenProcessorConfig: + blockSizeTokens: 64 + kvEventsConfig: + topicFilter: "kv@" + concurrency: 8 + discoverPods: true + podDiscoveryConfig: + socketPort: 5556 + replaySocketPort: 5559 + indexerConfig: + kvBlockIndexConfig: + enableMetrics: true + inMemoryConfig: + podCacheSize: 128 + kvCacheBackendConfigs: + - name: gpu + weight: 1.0 + - name: cpu + weight: 0.4 + speculativeIndexing: true +- name: inflight-load-producer + type: inflight-load-producer + parameters: + prefixMatchInfoProducerName: precise-prefix-cache-producer +- name: prefix-cache-affinity-filter + type: prefix-cache-affinity-filter + parameters: + peakPrefillThroughput: 16444 + inFlightLoadProducerName: inflight-load-producer + prefixMatchInfoProducerName: precise-prefix-cache-producer + affinityThreshold: 0.75 +- name: utilization-detector + type: utilization-detector + parameters: + queueDepthThreshold: 6 + kvCacheUtilThreshold: 0.80 +- name: token-load-scorer + type: token-load-scorer +schedulingProfiles: +- name: default + plugins: + - pluginRef: prefix-cache-affinity-filter + - pluginRef: utilization-detector + - pluginRef: token-load-scorer +dataLayer: + sources: + - pluginRef: endpoint-notification-source + extractors: + - pluginRef: precise-prefix-cache-producer +flowControl: + maxBytes: "10Gi" + maxRequests: "1k" + defaultRequestTTL: "1800s" + saturationDetector: + pluginRef: utilization-detector + defaultPriorityBand: + fairnessPolicyRef: program-aware-fairness diff --git a/src/frameworks/g4/llmd-configs/laser_beam_1m_offload_vllm.yaml b/src/frameworks/g4/llmd-configs/laser_beam_1m_offload_vllm.yaml new file mode 100644 index 00000000..25bf8a2e --- /dev/null +++ b/src/frameworks/g4/llmd-configs/laser_beam_1m_offload_vllm.yaml @@ -0,0 +1,42 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Post-optimization vLLM serving config for 1M-context AgentX + Host DRAM KV Offload +# Note on CPUOffloadingSpec cpu_bytes_to_use: +# - 68719476736 (64 GiB per TP4 replica when running 2x TP4 per node = 128 GiB/node) +# - 137438953472 (128 GiB per TP4 replica when running 1x TP4 per node) +host: "0.0.0.0" +port: "8000" +tensor-parallel-size: "4" +enable-expert-parallel: true +trust-remote-code: true +max-model-len: "1048576" +gpu-memory-utilization: "0.90" +kv-cache-dtype: "fp8" +block-size: "64" +max-num-batched-tokens: "16384" +max-num-seqs: "16" +speculative-config: '{"method":"dflash","model":"${DRAFTER_MODEL_ID}","num_speculative_tokens":8,"draft_tensor_parallel_size":4}' +compilation-config: '{"cudagraph_capture_sizes":[1,2,4,8,16,32,48,64,96,128,192,256,384,512,640,768,896,1024,1152],"cudagraph_mode":"FULL_AND_PIECEWISE","pass_config":{"fuse_allreduce_rms":false}}' +enable-prefix-caching: true +no-disable-hybrid-kv-cache-manager: true +swa-bounded-replay: true +enable-auto-tool-choice: true +tool-call-parser: "beam" +reasoning-parser: "beam" +default-chat-template-kwargs: '{"allow_missing_thinking":true,"reasoning_effort":"max"}' +override-generation-config: '{"temperature":1.0,"top_k":40,"top_p":0.95}' +moe-backend: "flashinfer_cutlass" +kv-transfer-config: '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","engine_id":"w1_0","kv_connector_extra_config":{"spec_name":"CPUOffloadingSpec","cpu_bytes_to_use":68719476736,"blocks_per_chunk":16,"offload_prompt_only":true}}' +kv-events-config: '{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:${ZMQ_PUB_PORT}","replay_endpoint":"tcp://*:${ZMQ_REPLAY_PORT}","topic":"kv@${POD_IP}:${VLLM_HTTP_PORT}@${SERVED_MODEL_NAME}"}' diff --git a/src/helm-charts/g4/llm-d-inference/Chart.yaml b/src/helm-charts/g4/llm-d-inference/Chart.yaml new file mode 100644 index 00000000..1e0e408e --- /dev/null +++ b/src/helm-charts/g4/llm-d-inference/Chart.yaml @@ -0,0 +1,20 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: v2 +name: llm-d-inference +description: A Helm chart for Aggregated Inference Serving with llm-d and vLLM on G4 (RTX PRO 6000 Blackwell Server Edition) GKE Node Pools +type: application +version: 0.1.0 +appVersion: "0.5.0" diff --git a/src/helm-charts/g4/llm-d-inference/templates/configmaps.yaml b/src/helm-charts/g4/llm-d-inference/templates/configmaps.yaml new file mode 100644 index 00000000..052a26cc --- /dev/null +++ b/src/helm-charts/g4/llm-d-inference/templates/configmaps.yaml @@ -0,0 +1,347 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- $routerEnabled := .Values.llmd.router.enabled -}} +{{- if hasKey .Values "router" -}} + {{- if hasKey .Values.router "enabled" -}} + {{- $routerEnabled = .Values.router.enabled -}} + {{- end -}} +{{- end -}} +{{- $vllmEnabled := true -}} +{{- if hasKey .Values "vllm" -}} + {{- if hasKey .Values.vllm "enabled" -}} + {{- $vllmEnabled = .Values.vllm.enabled -}} + {{- end -}} +{{- end -}} +{{- $modelName := "reflection/Beam-1.0-501B-A23B-NVFP4" -}} +{{- $drafterName := "reflection/Beam-1.0-501B-A23B-DFlash2" -}} +{{- $maxModelLen := 1048576 -}} +{{- if kindIs "map" .Values.workload.model -}} + {{- $modelName = .Values.workload.model.name | default $modelName -}} + {{- $drafterName = .Values.workload.model.drafterName | default $drafterName -}} + {{- $maxModelLen = .Values.workload.model.maxModelLen | default $maxModelLen -}} +{{- else if .Values.workload.model -}} + {{- $modelName = .Values.workload.model -}} +{{- end -}} +{{- if .Values.workload.drafterModel -}} + {{- $drafterName = .Values.workload.drafterModel -}} +{{- end -}} +{{- $ns := .Release.Namespace -}} +{{- if $routerEnabled }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: "{{ .Release.Name }}-epp-config" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router +data: + plugins-config.yaml: |- +{{- if .Values.epp_config }} +{{ .Values.epp_config | nindent 4 }} +{{- else }} + apiVersion: llm-d.ai/v1alpha1 + kind: EndpointPickerConfig + featureGates: + - flowControl + plugins: + - name: program-aware-fairness + type: program-aware-fairness + parameters: + strategy: turn-priority + - name: token-producer + type: token-producer + parameters: + modelName: {{ $modelName | quote }} + vllm: + url: {{ .Values.llmd.router.tokenProducerUrl | default (printf "http://%s-backends.%s.svc.cluster.local:%v" .Release.Name $ns (.Values.service.ports.tokenProducer | default 6100)) | quote }} + - name: endpoint-notification-source + type: endpoint-notification-source + - name: precise-prefix-cache-producer + type: precise-prefix-cache-producer + parameters: + tokenProcessorConfig: + blockSizeTokens: 64 + kvEventsConfig: + topicFilter: "kv@" + concurrency: 8 + discoverPods: true + podDiscoveryConfig: + socketPort: {{ .Values.service.ports.vllmZmq | default 5556 }} + replaySocketPort: {{ .Values.service.ports.vllmZmqReplay | default 5559 }} + indexerConfig: + kvBlockIndexConfig: + enableMetrics: true + inMemoryConfig: + podCacheSize: 128 + kvCacheBackendConfigs: + - name: gpu + weight: 1.0 + - name: cpu + weight: 0.4 + speculativeIndexing: true + - name: inflight-load-producer + type: inflight-load-producer + parameters: + prefixMatchInfoProducerName: precise-prefix-cache-producer + - name: prefix-cache-affinity-filter + type: prefix-cache-affinity-filter + parameters: + peakPrefillThroughput: 16444 + inFlightLoadProducerName: inflight-load-producer + prefixMatchInfoProducerName: precise-prefix-cache-producer + affinityThreshold: 0.75 + - name: utilization-detector + type: utilization-detector + parameters: + queueDepthThreshold: {{ .Values.llmd.router.queueDepthThreshold | default 6 }} + kvCacheUtilThreshold: {{ .Values.llmd.router.kvCacheUtilThreshold | default 0.80 }} + - name: token-load-scorer + type: token-load-scorer + schedulingProfiles: + - name: default + plugins: + - pluginRef: prefix-cache-affinity-filter + - pluginRef: utilization-detector + - pluginRef: token-load-scorer + dataLayer: + sources: + - pluginRef: endpoint-notification-source + extractors: + - pluginRef: precise-prefix-cache-producer + flowControl: + maxBytes: "10Gi" + maxRequests: "1k" + defaultRequestTTL: "1800s" + saturationDetector: + pluginRef: utilization-detector + defaultPriorityBand: + fairnessPolicyRef: program-aware-fairness +{{- end }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: "{{ .Release.Name }}-envoy-config" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router +data: + envoy.yaml: |- + static_resources: + listeners: + - name: envoy-proxy-ready + address: + socket_address: + address: 0.0.0.0 + port_value: 19001 + filter_chains: + - filters: + - name: envoy.filters.network.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + stat_prefix: envoy-ready-http + route_config: + name: local_route + virtual_hosts: + - name: ready_host + domains: ["*"] + routes: + - match: + prefix: "/ready" + direct_response: + status: 200 + body: + inline_string: "OK" + http_filters: + - name: envoy.filters.http.router + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router + - name: vllm + address: + socket_address: + address: 0.0.0.0 + port_value: {{ .Values.service.ports.envoyHttp | default 8081 }} + filter_chains: + - filters: + - name: envoy.filters.network.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + stat_prefix: http-8081 + stream_idle_timeout: 3600s + route_config: + name: vllm + virtual_hosts: + - name: vllm-default + domains: ["*"] + routes: + - match: + prefix: "/metrics" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/ping" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/health" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/v1/models" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/version" + route: + cluster: model_server_service_cluster + typed_per_filter_config: + envoy.filters.http.ext_proc: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExtProcPerRoute + disabled: true + - match: + prefix: "/" + route: + cluster: original_destination_cluster + timeout: 86400s + http_filters: + - name: envoy.filters.http.ext_proc + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor + failure_mode_allow: false + grpc_service: + envoy_grpc: + cluster_name: ext_proc + authority: localhost:9002 + timeout: 86400s + processing_mode: + request_header_mode: SEND + response_header_mode: SEND + request_body_mode: FULL_DUPLEX_STREAMED + response_body_mode: FULL_DUPLEX_STREAMED + request_trailer_mode: SEND + response_trailer_mode: SEND + message_timeout: 1000s + - name: envoy.filters.http.router + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router + clusters: + - name: model_server_service_cluster + type: STRICT_DNS + connect_timeout: 5s + lb_policy: ROUND_ROBIN + load_assignment: + cluster_name: model_server_service_cluster + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: "{{ .Release.Name }}-backends.{{ $ns }}.svc.cluster.local" + port_value: {{ .Values.service.ports.vllmHttp | default 8000 }} + - name: original_destination_cluster + type: ORIGINAL_DST + connect_timeout: 1000s + lb_policy: CLUSTER_PROVIDED + original_dst_lb_config: + use_http_header: true + http_header_name: x-gateway-destination-endpoint + - name: ext_proc + type: STATIC + connect_timeout: 86400s + lb_policy: LEAST_REQUEST + transport_socket: + name: "envoy.transport_sockets.tls" + typed_config: + "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext + common_tls_context: + alpn_protocols: ["h2"] + validation_context: {} + typed_extension_protocol_options: + envoy.extensions.upstreams.http.v3.HttpProtocolOptions: + "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions + explicit_http_config: + http2_protocol_options: {} + load_assignment: + cluster_name: ext_proc + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: 127.0.0.1 + port_value: {{ .Values.service.ports.eppGrpc | default 9002 }} +{{- end }} +--- +{{- if and $vllmEnabled (gt (int .Values.workload.replicas) 0) }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: "{{ .Release.Name }}-vllm-config" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: model-server +data: + {{ .Values.workload.configFile | default "serving-configuration" }}: |- +{{- if .Values.vllm_serving_config }} +{{ .Values.vllm_serving_config | nindent 4 }} +{{- else }} + host: "0.0.0.0" + port: "{{ .Values.service.ports.vllmHttp | default 8000 }}" + tensor-parallel-size: "{{ .Values.workload.gpusPerReplica | default 4 }}" + enable-expert-parallel: true + trust-remote-code: true + max-model-len: "{{ $maxModelLen }}" + gpu-memory-utilization: "0.90" + kv-cache-dtype: "fp8" + block-size: "64" + max-num-batched-tokens: "16384" + max-num-seqs: "16" + speculative-config: '{"method":"dflash","model":"${DRAFTER_MODEL_ID}","num_speculative_tokens":8,"draft_tensor_parallel_size":{{ .Values.workload.gpusPerReplica | default 4 }}}' + compilation-config: '{"cudagraph_capture_sizes":[1,2,4,8,16,32,48,64,96,128,192,256,384,512,640,768,896,1024,1152],"cudagraph_mode":"FULL_AND_PIECEWISE","pass_config":{"fuse_allreduce_rms":false}}' + enable-prefix-caching: true + no-disable-hybrid-kv-cache-manager: true + swa-bounded-replay: true + enable-auto-tool-choice: true + tool-call-parser: "beam" + reasoning-parser: "beam" + default-chat-template-kwargs: '{"allow_missing_thinking":true,"reasoning_effort":"max"}' + override-generation-config: '{"temperature":1.0,"top_k":40,"top_p":0.95}' + moe-backend: "flashinfer_cutlass" + kv-transfer-config: '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","engine_id":"w1_0","kv_connector_extra_config":{"spec_name":"CPUOffloadingSpec","cpu_bytes_to_use":68719476736,"blocks_per_chunk":16,"offload_prompt_only":true}}' + kv-events-config: '{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:${ZMQ_PUB_PORT}","replay_endpoint":"tcp://*:${ZMQ_REPLAY_PORT}","topic":"kv@${POD_IP}:${VLLM_HTTP_PORT}@${SERVED_MODEL_NAME}"}' +{{- end }} +{{- end }} diff --git a/src/helm-charts/g4/llm-d-inference/templates/router.yaml b/src/helm-charts/g4/llm-d-inference/templates/router.yaml new file mode 100644 index 00000000..b83543dc --- /dev/null +++ b/src/helm-charts/g4/llm-d-inference/templates/router.yaml @@ -0,0 +1,203 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- $routerEnabled := .Values.llmd.router.enabled -}} +{{- if hasKey .Values "router" -}} + {{- if hasKey .Values.router "enabled" -}} + {{- $routerEnabled = .Values.router.enabled -}} + {{- end -}} +{{- end -}} +{{- $ns := .Release.Namespace -}} +{{- $saCreate := true -}} +{{- $saName := printf "%s-router-sa" .Release.Name -}} +{{- if hasKey .Values.llmd.router "serviceAccount" -}} + {{- if hasKey .Values.llmd.router.serviceAccount "create" -}} + {{- $saCreate = .Values.llmd.router.serviceAccount.create -}} + {{- end -}} + {{- if .Values.llmd.router.serviceAccount.name -}} + {{- $saName = .Values.llmd.router.serviceAccount.name -}} + {{- else if not $saCreate -}} + {{- $saName = "default" -}} + {{- end -}} +{{- end -}} +{{- if .Values.llmd.router.serviceAccountName -}} + {{- $saName = .Values.llmd.router.serviceAccountName -}} +{{- end -}} +{{- if $routerEnabled }} +{{- if $saCreate }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ $saName }} + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router +--- +{{- end }} +{{- if and .Values.llmd.router.rbac .Values.llmd.router.rbac.create }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: "{{ .Release.Name }}-k8s-discovery" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router +rules: +- apiGroups: [""] + resources: ["pods", "services", "endpoints"] + verbs: ["get", "list", "watch"] +- apiGroups: ["discovery.k8s.io"] + resources: ["endpointslices"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: "{{ .Release.Name }}-k8s-discovery-binding" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router +subjects: +- kind: ServiceAccount + name: {{ $saName }} + namespace: {{ $ns }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: "{{ .Release.Name }}-k8s-discovery" +--- +{{- end }} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: "{{ .Release.Name }}-router" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router +spec: + replicas: {{ .Values.llmd.router.replicas | default 1 }} + selector: + matchLabels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router + template: + metadata: + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router + spec: + automountServiceAccountToken: true + serviceAccountName: {{ $saName }} + {{- with .Values.llmd.router.nodeSelector }} + nodeSelector: +{{ toYaml . | indent 8 }} + {{- end }} + {{- with .Values.llmd.router.tolerations }} + tolerations: +{{ toYaml . | indent 8 }} + {{- end }} + containers: + - name: envoy-proxy + image: {{ .Values.llmd.router.envoyImage }} + imagePullPolicy: IfNotPresent + command: ["envoy"] + args: + - "--service-node" + - "envoy-proxy" + - "--log-level" + - "warn" + - "-c" + - "/etc/envoy/envoy.yaml" + ports: + - name: http-8081 + containerPort: {{ .Values.service.ports.envoyHttp | default 8081 }} + protocol: TCP + - name: ready-19001 + containerPort: 19001 + protocol: TCP + readinessProbe: + httpGet: + path: /ready + port: 19001 + scheme: HTTP + periodSeconds: 5 + timeoutSeconds: 1 + resources: +{{ toYaml .Values.llmd.router.resources.envoy | indent 10 }} + volumeMounts: + - name: envoy-config + mountPath: /etc/envoy + readOnly: true + - name: router + image: {{ .Values.llmd.router.eppImage }} + imagePullPolicy: IfNotPresent + env: + - name: NAMESPACE + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: metadata.namespace + - name: POD_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: metadata.name + ports: + - name: grpc + containerPort: {{ .Values.service.ports.eppGrpc | default 9002 }} + protocol: TCP + - name: grpc-health + containerPort: 9003 + protocol: TCP + - name: metrics + containerPort: {{ .Values.service.ports.eppMetrics | default 9090 }} + protocol: TCP + args: + - "--endpoint-selector={{ .Values.llmd.router.endpointSelector | default (printf "app.kubernetes.io/instance=%s,app.kubernetes.io/name=llm-d-inference,app.kubernetes.io/component=model-server" .Release.Name) }}" + - "--endpoint-target-ports={{ .Values.service.ports.vllmHttp | default 8000 }}" + - "--config-file=/etc/epp/plugins-config.yaml" + - "--grpc-health-port=9003" + - "--tracing=false" + - "--metrics-endpoint-auth=false" + - "--allow-experimental-plugins=true" + readinessProbe: + grpc: + port: 9003 + service: inference-extension + periodSeconds: 5 + timeoutSeconds: 1 + resources: +{{ toYaml .Values.llmd.router.resources.epp | indent 10 }} + volumeMounts: + - name: epp-config + mountPath: /etc/epp + readOnly: true + volumes: + - name: envoy-config + configMap: + name: "{{ .Release.Name }}-envoy-config" + - name: epp-config + configMap: + name: "{{ .Release.Name }}-epp-config" +{{- end }} diff --git a/src/helm-charts/g4/llm-d-inference/templates/services.yaml b/src/helm-charts/g4/llm-d-inference/templates/services.yaml new file mode 100644 index 00000000..a91e4ebb --- /dev/null +++ b/src/helm-charts/g4/llm-d-inference/templates/services.yaml @@ -0,0 +1,100 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- $routerEnabled := .Values.llmd.router.enabled -}} +{{- if hasKey .Values "router" -}} + {{- if hasKey .Values.router "enabled" -}} + {{- $routerEnabled = .Values.router.enabled -}} + {{- end -}} +{{- end -}} +{{- $ns := .Release.Namespace -}} +{{- if $routerEnabled }} +apiVersion: v1 +kind: Service +metadata: + name: "{{ .Release.Name }}-frontend" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router +spec: + type: {{ .Values.service.type | default "ClusterIP" }} + selector: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: router + ports: + - name: http + port: 80 + targetPort: {{ .Values.service.ports.envoyHttp | default 8081 }} + - name: http-openai + port: {{ .Values.service.ports.frontend | default 8000 }} + targetPort: {{ .Values.service.ports.envoyHttp | default 8081 }} + - name: grpc-ext-proc + port: {{ .Values.service.ports.eppGrpc | default 9002 }} + targetPort: {{ .Values.service.ports.eppGrpc | default 9002 }} + - name: epp-metrics + port: {{ .Values.service.ports.eppMetrics | default 9090 }} + targetPort: {{ .Values.service.ports.eppMetrics | default 9090 }} +--- +apiVersion: v1 +kind: Service +metadata: + name: "{{ .Release.Name }}-backends" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: model-server +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: model-server + ports: + - name: http + port: {{ .Values.service.ports.vllmHttp | default 8000 }} + targetPort: {{ .Values.service.ports.vllmHttp | default 8000 }} + - name: http-render + port: {{ .Values.service.ports.tokenProducer | default 6100 }} + targetPort: {{ .Values.service.ports.tokenProducer | default 6100 }} + - name: zmq-kv-events + port: {{ .Values.service.ports.vllmZmq | default 5556 }} + targetPort: {{ .Values.service.ports.vllmZmq | default 5556 }} + - name: zmq-kv-replay + port: {{ .Values.service.ports.vllmZmqReplay | default 5559 }} + targetPort: {{ .Values.service.ports.vllmZmqReplay | default 5559 }} +{{- else }} +apiVersion: v1 +kind: Service +metadata: + name: "{{ .Release.Name }}-frontend" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: model-server +spec: + type: {{ .Values.service.type | default "ClusterIP" }} + selector: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: model-server + ports: + - name: http-openai + port: {{ .Values.service.ports.frontend | default 8000 }} + targetPort: {{ .Values.service.ports.vllmHttp | default 8000 }} +{{- end }} diff --git a/src/helm-charts/g4/llm-d-inference/templates/vllm-deployments.yaml b/src/helm-charts/g4/llm-d-inference/templates/vllm-deployments.yaml new file mode 100644 index 00000000..fcff6f11 --- /dev/null +++ b/src/helm-charts/g4/llm-d-inference/templates/vllm-deployments.yaml @@ -0,0 +1,319 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- $vllmEnabled := true -}} +{{- if hasKey .Values "vllm" -}} + {{- if hasKey .Values.vllm "enabled" -}} + {{- $vllmEnabled = .Values.vllm.enabled -}} + {{- end -}} +{{- end -}} +{{- $routerEnabled := .Values.llmd.router.enabled -}} +{{- if hasKey .Values "router" -}} + {{- if hasKey .Values.router "enabled" -}} + {{- $routerEnabled = .Values.router.enabled -}} + {{- end -}} +{{- end -}} +{{- $modelName := "reflection/Beam-1.0-501B-A23B-NVFP4" -}} +{{- $drafterName := "reflection/Beam-1.0-501B-A23B-DFlash2" -}} +{{- $servedModelName := "" -}} +{{- $maxModelLen := 1048576 -}} +{{- if kindIs "map" .Values.workload.model -}} + {{- $modelName = .Values.workload.model.name | default $modelName -}} + {{- $drafterName = .Values.workload.model.drafterName | default $drafterName -}} + {{- $servedModelName = .Values.workload.model.servedModelName | default "" -}} + {{- $maxModelLen = .Values.workload.model.maxModelLen | default $maxModelLen -}} +{{- else if .Values.workload.model -}} + {{- $modelName = .Values.workload.model -}} +{{- end -}} +{{- if .Values.workload.drafterModel -}} + {{- $drafterName = .Values.workload.drafterModel -}} +{{- end -}} +{{- if .Values.workload.servedModelName -}} + {{- $servedModelName = .Values.workload.servedModelName -}} +{{- end -}} +{{- $ns := .Release.Namespace -}} +{{- $root := . -}} +{{- if and $vllmEnabled (gt (int .Values.workload.replicas) 0) }} +{{- range $i := until (int .Values.workload.replicas) }} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: "{{ $root.Release.Name }}-vllm-{{ $i }}" + namespace: {{ $ns }} + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ $root.Release.Name }}" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-{{ $i }}" +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ $root.Release.Name }}" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-{{ $i }}" + template: + metadata: + labels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ $root.Release.Name }}" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-{{ $i }}" + annotations: + {{- if and $root.Values.volumes.useGcs $root.Values.volumes.gcsfuse.bucketName }} + gke-gcsfuse/volumes: "true" + gke-gcsfuse/cpu-limit: "0" + gke-gcsfuse/memory-limit: "0" + gke-gcsfuse/ephemeral-storage-limit: "0" + {{- end }} + spec: + {{- with $root.Values.workload.nodeSelector }} + nodeSelector: +{{ toYaml . | indent 8 }} + {{- end }} + {{- if and $root.Values.workload.pinSingleNode (gt $i 0) }} + affinity: + podAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ $root.Release.Name }}" + app.kubernetes.io/component: model-server + app.kubernetes.io/part-of: "replica-0" + topologyKey: kubernetes.io/hostname + {{- else if not $root.Values.workload.pinSingleNode }} + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: llm-d-inference + app.kubernetes.io/instance: "{{ $root.Release.Name }}" + app.kubernetes.io/component: model-server + topologyKey: kubernetes.io/hostname + {{- end }} + tolerations: + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" + - key: "cloud.google.com/impending-node-termination" + operator: "Exists" + containers: + - name: vllm + image: {{ $root.Values.workload.image }} + imagePullPolicy: IfNotPresent + command: ["/bin/bash", "-lc"] + args: + - | + set -euo pipefail + unset PYTORCH_CUDA_ALLOC_CONF + nvidia-smi + {{- if $routerEnabled }} + CUDA_VISIBLE_DEVICES="" nohup vllm launch render \ + "${MODEL_MOUNT_DIR:-${MODEL_ID}}" \ + --served-model-name "${SERVED_MODEL_NAME}" \ + --host 0.0.0.0 \ + --port {{ $root.Values.service.ports.tokenProducer | default 6100 }} \ + --trust-remote-code \ + --max-model-len {{ $maxModelLen }} \ + --enable-auto-tool-choice \ + --tool-call-parser beam \ + --reasoning-parser beam \ + --default-chat-template-kwargs '{"allow_missing_thinking":true,"reasoning_effort":"max"}' > /tmp/vllm-render.log 2>&1 & + {{- end }} + python3 - << 'PYEOF' + import json, os, shutil, sys, yaml + + model_id = os.environ["MODEL_ID"] + served_model = os.environ.get("SERVED_MODEL_NAME") or model_id + os.environ["SERVED_MODEL_NAME"] = served_model + + mount_dir = os.environ.get("MODEL_MOUNT_DIR", "") + if mount_dir and os.path.isdir(mount_dir): + model_target = mount_dir + else: + model_target = model_id + + drafter_mount = os.environ.get("DRAFTER_MOUNT_DIR", "") + if drafter_mount and os.path.isdir(drafter_mount): + os.environ["DRAFTER_MODEL_ID"] = drafter_mount + + numa_node = os.environ.get("NUMA_NODE", "0") + if shutil.which("numactl"): + cmd = [ + "numactl", + f"--cpunodebind={numa_node}", + f"--membind={numa_node}", + "vllm", + "serve", + model_target, + "--served-model-name", + served_model, + ] + else: + cmd = ["vllm", "serve", model_target, "--served-model-name", served_model] + + cfg_path = os.environ.get("SERVER_ARGS_FILE", "") + if cfg_path and os.path.exists(cfg_path): + with open(cfg_path, "r") as f: + cfg = yaml.safe_load(f) or {} + for k, v in cfg.items(): + if k in ("model", "served-model-name") or v is None or v is False: + continue + flag = f"--{k}" + if v is True: + cmd.append(flag) + elif isinstance(v, (dict, list)): + cmd.extend([flag, os.path.expandvars(json.dumps(v))]) + else: + cmd.extend([flag, os.path.expandvars(str(v))]) + + print("Launching vLLM with NUMA pinning:", " ".join(cmd), flush=True) + os.execvp(cmd[0], cmd) + PYEOF + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_HOST_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: VLLM_HTTP_PORT + value: "{{ $root.Values.service.ports.vllmHttp | default 8000 }}" + - name: ZMQ_PUB_PORT + value: "{{ $root.Values.service.ports.vllmZmq | default 5556 }}" + - name: ZMQ_REPLAY_PORT + value: "{{ $root.Values.service.ports.vllmZmqReplay | default 5559 }}" + - name: CUDA_VISIBLE_DEVICES + value: {{ index $root.Values.workload.cudaVisibleDevices $i | quote }} + - name: NUMA_NODE + value: {{ index $root.Values.workload.numaNodes $i | default "0" | quote }} + - name: MODEL_ID + value: {{ $modelName | quote }} + - name: DRAFTER_MODEL_ID + value: {{ $drafterName | quote }} + - name: SERVED_MODEL_NAME + value: {{ default $modelName $servedModelName | quote }} + - name: HF_HOME + value: {{ $root.Values.volumes.hfHome | default "/ssd" | quote }} + - name: SERVER_ARGS_FILE + value: "{{ $root.Values.workload.configPath | default "/workload/configs" }}/{{ $root.Values.workload.configFile | default "serving-configuration" }}" + {{- if and $root.Values.volumes.useGcs $root.Values.volumes.gcsfuse.bucketName }} + - name: MODEL_MOUNT_DIR + value: "{{ $root.Values.volumes.gcsfuse.mountPath | default "/data/model" }}/{{ $modelName }}" + - name: DRAFTER_MOUNT_DIR + value: "{{ $root.Values.volumes.gcsfuse.mountPath | default "/data/model" }}/{{ $drafterName }}" + {{- else if and $root.Values.volumes.pvc $root.Values.volumes.pvc.enabled $root.Values.volumes.pvc.claimName }} + - name: MODEL_MOUNT_DIR + value: "{{ $root.Values.volumes.pvc.mountPath | default "/models" }}/{{ $modelName }}" + - name: DRAFTER_MOUNT_DIR + value: "{{ $root.Values.volumes.pvc.mountPath | default "/models" }}/{{ $drafterName }}" + {{- end }} + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: {{ $root.Values.secrets.huggingface.secretName }} + key: {{ $root.Values.secrets.huggingface.secretKey | default "HF_TOKEN" }} + optional: true + {{- if $root.Values.workload.envs }} +{{ toYaml $root.Values.workload.envs | indent 8 }} + {{- end }} + ports: + - name: http + containerPort: {{ $root.Values.service.ports.vllmHttp | default 8000 }} + - name: render + containerPort: {{ $root.Values.service.ports.tokenProducer | default 6100 }} + - name: kv-events + containerPort: {{ $root.Values.service.ports.vllmZmq | default 5556 }} + - name: kv-replay + containerPort: {{ $root.Values.service.ports.vllmZmqReplay | default 5559 }} + readinessProbe: + httpGet: + path: /health + port: {{ $root.Values.service.ports.vllmHttp | default 8000 }} + initialDelaySeconds: {{ $root.Values.workload.readinessProbe.initialDelaySeconds }} + periodSeconds: {{ $root.Values.workload.readinessProbe.periodSeconds }} + timeoutSeconds: {{ $root.Values.workload.readinessProbe.timeoutSeconds }} + failureThreshold: {{ $root.Values.workload.readinessProbe.failureThreshold }} + livenessProbe: + httpGet: + path: /health + port: {{ $root.Values.service.ports.vllmHttp | default 8000 }} + initialDelaySeconds: {{ $root.Values.workload.livenessProbe.initialDelaySeconds }} + periodSeconds: {{ $root.Values.workload.livenessProbe.periodSeconds }} + timeoutSeconds: {{ $root.Values.workload.livenessProbe.timeoutSeconds }} + failureThreshold: {{ $root.Values.workload.livenessProbe.failureThreshold }} + resources: +{{ toYaml $root.Values.workload.resources | indent 10 }} + volumeMounts: + - name: dshm + mountPath: /dev/shm + - name: serving-configuration + mountPath: {{ $root.Values.workload.configPath | default "/workload/configs" }} + readOnly: true + {{- if $root.Values.volumes.ssdMountPath }} + - name: local-ssd + mountPath: {{ $root.Values.volumes.ssdMountPath }} + {{- end }} + {{- if and $root.Values.volumes.useGcs $root.Values.volumes.gcsfuse.bucketName }} + - name: gcs-model-volume + mountPath: {{ $root.Values.volumes.gcsfuse.mountPath | default "/data/model" }} + readOnly: true + {{- end }} + {{- if and $root.Values.volumes.pvc $root.Values.volumes.pvc.enabled $root.Values.volumes.pvc.claimName }} + - name: pvc-model-volume + mountPath: {{ $root.Values.volumes.pvc.mountPath | default "/models" }} + readOnly: true + {{- end }} + volumes: + - name: dshm + emptyDir: + medium: Memory + sizeLimit: {{ $root.Values.workload.shmSize | default "256Gi" }} + - name: serving-configuration + configMap: + name: "{{ $root.Release.Name }}-vllm-config" + {{- if $root.Values.volumes.ssdMountPath }} + - name: local-ssd + {{- if $root.Values.volumes.useHostPathSsd }} + hostPath: + path: /mnt/stateful_partition/kube-ephemeral-ssd + type: DirectoryOrCreate + {{- else }} + emptyDir: {} + {{- end }} + {{- end }} + {{- if and $root.Values.volumes.useGcs $root.Values.volumes.gcsfuse.bucketName }} + - name: gcs-model-volume + csi: + driver: gcsfuse.csi.storage.gke.io + volumeAttributes: + bucketName: {{ $root.Values.volumes.gcsfuse.bucketName }} + mountOptions: "implicit-dirs,file-cache:enable-parallel-downloads:true,file-cache:parallel-downloads-per-file:100,file-cache:max-parallel-downloads:-1,file-cache:download-chunk-size-mb:50,file-cache:max-size-mb:-1" + {{- end }} + {{- if and $root.Values.volumes.pvc $root.Values.volumes.pvc.enabled $root.Values.volumes.pvc.claimName }} + - name: pvc-model-volume + persistentVolumeClaim: + claimName: {{ $root.Values.volumes.pvc.claimName }} + readOnly: true + {{- end }} +{{- end }} +{{- end }} diff --git a/src/helm-charts/g4/llm-d-inference/values.yaml b/src/helm-charts/g4/llm-d-inference/values.yaml new file mode 100644 index 00000000..17062817 --- /dev/null +++ b/src/helm-charts/g4/llm-d-inference/values.yaml @@ -0,0 +1,151 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +llmd: + router: + enabled: true + replicas: 1 + serviceAccount: + create: true + name: "" + rbac: + create: true + eppImage: ghcr.io/llm-d/llm-d-router-endpoint-picker:main + envoyImage: docker.io/envoyproxy/envoy:distroless-v1.33.2 + verbosity: "3" + queueDepthThreshold: 6 + kvCacheUtilThreshold: 0.80 + nodeSelector: {} + tolerations: [] + resources: + epp: + requests: + cpu: "2" + memory: "4Gi" + limits: + cpu: "8" + memory: "16Gi" + envoy: + requests: + cpu: "2" + memory: "4Gi" + limits: + cpu: "8" + memory: "16Gi" + +workload: + model: + name: reflection/Beam-1.0-501B-A23B-NVFP4 + drafterName: reflection/Beam-1.0-501B-A23B-DFlash2 + servedModelName: "" + maxModelLen: 1048576 + image: us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64 + framework: vllm + replicas: 2 + gpusPerReplica: 4 + pinSingleNode: true + nodeSelector: + cloud.google.com/gke-accelerator: nvidia-rtx-pro-6000 + # Pin each 4-GPU TP4 replica to its local PCIe/CPU NUMA node: + # - Replica 0: GPUs 0,1,2,3 -> numactl --cpunodebind=0 --membind=0 + # - Replica 1: GPUs 4,5,6,7 -> numactl --cpunodebind=1 --membind=1 + cudaVisibleDevices: + - "0,1,2,3" + - "4,5,6,7" + numaNodes: + - "0" + - "1" + shmSize: "256Gi" + configPath: /workload/configs + configFile: serving-configuration + resources: + requests: + cpu: "64" + memory: "512Gi" + nvidia.com/gpu: "4" + limits: + cpu: "160" + memory: "1200Gi" + nvidia.com/gpu: "4" + envs: + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: VLLM_ENGINE_READY_TIMEOUT_S + value: "3600" + - name: VLLM_RPC_TIMEOUT + value: "600000" + - name: VLLM_HTTP_TIMEOUT_KEEP_ALIVE + value: "900" + - name: VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT + value: "300" + - name: VLLM_USE_V2_MODEL_RUNNER + value: "1" + - name: VLLM_FUSED_MOE_CHUNK_SIZE + value: "32768" + - name: VLLM_SWA_DECODER_REPLAY_MAX_BS + value: "4096" + - name: VLLM_ENFORCE_STRICT_TOOL_CALLING + value: "0" + - name: VLLM_BEAM_ALWAYS_CONSTRAIN_OUTPUT + value: "0" + - name: NCCL_IB_DISABLE + value: "1" + - name: NCCL_P2P_LEVEL + value: "NVL" + - name: NCCL_NET_GDR_LEVEL + value: "0" + - name: PYTHONUNBUFFERED + value: "1" + readinessProbe: + initialDelaySeconds: 30 + periodSeconds: 5 + timeoutSeconds: 5 + failureThreshold: 60 + livenessProbe: + initialDelaySeconds: 300 + periodSeconds: 30 + timeoutSeconds: 10 + failureThreshold: 10 + +secrets: + huggingface: + secretName: hf-token-secret + secretKey: HF_TOKEN + +volumes: + hfHome: /ssd + ssdMountPath: /ssd + useHostPathSsd: false + useGcs: false + gcsfuse: + bucketName: "" + mountPath: /data/model + pvc: + enabled: false + claimName: "" + mountPath: /models + +service: + type: ClusterIP + ports: + frontend: 8000 + envoyHttp: 8081 + eppGrpc: 9002 + eppMetrics: 9090 + tokenProducer: 6100 + zmqPub: 5556 + zmqRouter: 5559 + vllmHttp: 8000 + vllmZmq: 5556 + vllmZmqReplay: 5559 From c5277b0fabbe5b1545b055af3ce8a4c620819af0 Mon Sep 17 00:00:00 2001 From: Christopher Pirillo Date: Fri, 16 Oct 2026 13:46:15 +0000 Subject: [PATCH 6/8] Add Reflection Beam 501B inference recipes for A4X-MAX (Dynamo + SGLang) and G4 (llm-d + vLLM) - Rename configuration and Helm values files to `beam_*` and use public model name Reflection Beam (`reflection/Beam-1.0-501B-A23B-NVFP4` and `reflection/Beam-1.0-501B-A23B-DFlash2`) - Remove benchmark performance tables from public recipe READMEs - Update container image references to `lmsysorg/sglang:beam` and `vllm/vllm-openai:reflection-beam` --- .../aggregated-serving/dynamo/README.md | 38 ++++++----------- ..._beam_high_int.yaml => beam_high_int.yaml} | 4 +- .../disaggregated-serving/dynamo/README.md | 30 ++++--------- ...oughput.yaml => beam_high_throughput.yaml} | 4 +- .../g4/aggregated-serving/llm-d/README.md | 42 +++++++------------ .../manifests/modelserver-2x-tp4-1m.yaml | 4 +- .../g4/aggregated-serving/llm-d/values.yaml | 2 +- ...gh_int_agg.yaml => beam_high_int_agg.yaml} | 2 +- ....yaml => beam_high_throughput_decode.yaml} | 0 ...yaml => beam_high_throughput_prefill.yaml} | 0 ...load_epp.yaml => beam_1m_offload_epp.yaml} | 0 ...ad_vllm.yaml => beam_1m_offload_vllm.yaml} | 0 .../g4/llm-d-inference/values.yaml | 2 +- 13 files changed, 45 insertions(+), 83 deletions(-) rename inference/a4xmax/aggregated-serving/dynamo/{laser_beam_high_int.yaml => beam_high_int.yaml} (96%) rename inference/a4xmax/disaggregated-serving/dynamo/{laser_beam_high_throughput.yaml => beam_high_throughput.yaml} (97%) rename src/frameworks/a4xmax/dynamo-configs/{laser_beam_high_int_agg.yaml => beam_high_int_agg.yaml} (91%) rename src/frameworks/a4xmax/dynamo-configs/{laser_beam_high_throughput_decode.yaml => beam_high_throughput_decode.yaml} (100%) rename src/frameworks/a4xmax/dynamo-configs/{laser_beam_high_throughput_prefill.yaml => beam_high_throughput_prefill.yaml} (100%) rename src/frameworks/g4/llmd-configs/{laser_beam_1m_offload_epp.yaml => beam_1m_offload_epp.yaml} (100%) rename src/frameworks/g4/llmd-configs/{laser_beam_1m_offload_vllm.yaml => beam_1m_offload_vllm.yaml} (100%) diff --git a/inference/a4xmax/aggregated-serving/dynamo/README.md b/inference/a4xmax/aggregated-serving/dynamo/README.md index 0dcf6eca..b344304d 100644 --- a/inference/a4xmax/aggregated-serving/dynamo/README.md +++ b/inference/a4xmax/aggregated-serving/dynamo/README.md @@ -162,20 +162,18 @@ gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \ --role "roles/storage.objectViewer" ``` -The bucket mounts at `/data/model` in every worker. Stage the target model and the DFlash2 drafter into the bucket: +The bucket mounts at `/data/model` in every worker. Stage the target model ([`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4)) and the DFlash2 drafter ([`reflection/Beam-1.0-501B-A23B-DFlash2`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2)) into the bucket: ```bash pip install -U "huggingface_hub[cli]" # Target model: mounts at /data/model/ hf download reflection/Beam-1.0-501B-A23B-NVFP4 --local-dir ./reflection/Beam-1.0-501B-A23B-NVFP4 -gcloud storage cp -r ./reflection gs://${GCS_BUCKET}/ -# DFlash2 drafter: mounts at the speculative-draft-model-path in laser_beam_high_int_agg.yaml -hf download reflection/dflash2-drafter-preview-final \ - --revision 8d26c952ec6a82a9a18825d6126c9f6948149ad9 \ - --cache-dir ./huggingface_model_cache -gcloud storage cp -r ./huggingface_model_cache gs://${GCS_BUCKET}/ +# DFlash2 drafter: mounts at the speculative-draft-model-path in beam_high_int_agg.yaml +hf download reflection/Beam-1.0-501B-A23B-DFlash2 --local-dir ./reflection/Beam-1.0-501B-A23B-DFlash2 + +gcloud storage cp -r ./reflection gs://${GCS_BUCKET}/ ``` The workers load the target model from `/data/model/`. If you staged it at a different path in the bucket, pass `--set workload.modelPath=/data/model/` to `helm install`. @@ -186,7 +184,7 @@ The workers load the target model from `/data/model/`. If you stage This recipe uses a model-specific SGLang runtime image that includes Dynamo `1.0.1` and SGLang support for the model: ```bash -export DYNAMO_IMAGE="us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64" +export DYNAMO_IMAGE="lmsysorg/sglang:beam" ``` The chart sets `runtimeVersionOverride` to `dynamo.releaseVersion` (`1.0.1`) because the Dynamo operator can't infer the runtime version from a custom image tag. @@ -203,22 +201,22 @@ This recipe reuses the Dynamo Helm chart at `$REPO_ROOT/src/helm-charts/a4xmax/i | Model Name | Hugging Face ID / Path (`HF_MODEL_ID`) | Deployment Category | Topology & GPUs | Helm Values File (`VALUES_FILE`) | Serving Config (`SERVING_CONFIG`) | | :--- | :--- | :--- | :--- | :--- | :--- | -| **LASER_BEAM** | `reflection/Beam-1.0-501B-A23B-NVFP4` | High Interactivity (DFlash2) | 8 aggregated workers (8 nodes / 32 GPUs) | `laser_beam_high_int.yaml` | `laser_beam_high_int_agg.yaml` | +| **Reflection Beam 501B (NVFP4)** | [`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4) | High Interactivity (DFlash2) | 8 aggregated workers (8 nodes / 32 GPUs) | `beam_high_int.yaml` | `beam_high_int_agg.yaml` | ### 3.2. High Interactivity Deployment -This category deploys 8 aggregated SGLang workers (`TP=4`, `EP=4`, one per node) behind the Dynamo KV-aware router (`DYN_ROUTER_MODE=kv`, `DYN_KV_CACHE_BLOCK_SIZE=64`). Each worker uses DFlash2 speculative decoding with 8 draft tokens to lower Time-Per-Output-Token (TPOT). +This category deploys 8 aggregated SGLang workers (`TP=4`, `EP=4`, one per node) behind the Dynamo KV-aware router (`DYN_ROUTER_MODE=kv`, `DYN_KV_CACHE_BLOCK_SIZE=64`). Each worker uses DFlash2 speculative decoding ([`reflection/Beam-1.0-501B-A23B-DFlash2`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2)) with 8 draft tokens to lower Time-Per-Output-Token (TPOT). -#### LASER_BEAM (High Interactivity) +#### Reflection Beam 501B (High Interactivity) 1. Set the model and configuration variables: ```bash export HF_MODEL_ID="reflection/Beam-1.0-501B-A23B-NVFP4" -export VALUES_FILE="laser_beam_high_int.yaml" -export SERVING_CONFIG="laser_beam_high_int_agg.yaml" -export RELEASE_NAME="$USER-dynamo-a4xmax-laser-beam-high-int" +export VALUES_FILE="beam_high_int.yaml" +export SERVING_CONFIG="beam_high_int_agg.yaml" +export RELEASE_NAME="$USER-dynamo-a4xmax-beam-high-int" ``` 2. Install the Helm release: @@ -306,16 +304,6 @@ kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " " ``` -3. Compare with the reference results below. Each concurrency is a separate 3,600 s `aiperf` run. We ran `CONCURRENCY=192` first and then `CONCURRENCY=96` on the same deployment, and ran `CONCURRENCY=32` on a fresh deployment. All runs completed with 0 failed requests. - -| Concurrency | Total tok/s (32 GPUs) | Output tok/s | TPOT p50 | TTFT p50 / p99 | Prefix-cache hit rate | Completed requests | -| :---: | :---: | :---: | :---: | :---: | :---: | :---: | -| 32 | 231,382 | 1,459 | 2.87 ms | 0.58 s / 4.37 s | 92.4% | 4,711 | -| 96 | 643,409 | 4,737 | 3.84 ms | 0.84 s / 7.46 s | 82.4% | 16,285 | -| 192 | 805,244 | 6,360 | 6.20 ms | 12.45 s / 22.45 s | 76.0% | 21,933 | - -The prefix-cache hit rate is `overall_usage_prompt_cache_read_pct` in `profile_export_aiperf.json`: cached prompt tokens divided by total prompt tokens, as reported in the response `usage` field. - ## 5. Monitoring and Troubleshooting @@ -339,7 +327,7 @@ kubectl logs -f -n ${NAMESPACE} Common issues: * **Pods stuck in Pending**: Check that 8 `a4x-maxgpu-4g-metal` nodes are free in the same NVL72 domain. -* **Worker fails to load the drafter**: Check that the drafter checkpoint is staged at the `speculative-draft-model-path` in `laser_beam_high_int_agg.yaml`. +* **Worker fails to load the drafter**: Check that the drafter checkpoint is staged at the `speculative-draft-model-path` in `beam_high_int_agg.yaml`. * **Slow first startup**: The first load reads the checkpoint through GCS FUSE into the local SSD file cache. Later restarts on the same node read from the cache. diff --git a/inference/a4xmax/aggregated-serving/dynamo/laser_beam_high_int.yaml b/inference/a4xmax/aggregated-serving/dynamo/beam_high_int.yaml similarity index 96% rename from inference/a4xmax/aggregated-serving/dynamo/laser_beam_high_int.yaml rename to inference/a4xmax/aggregated-serving/dynamo/beam_high_int.yaml index 6225809e..093d1fc5 100644 --- a/inference/a4xmax/aggregated-serving/dynamo/laser_beam_high_int.yaml +++ b/inference/a4xmax/aggregated-serving/dynamo/beam_high_int.yaml @@ -24,7 +24,7 @@ dynamo: claimTemplateName: serviceAccountName: frontend: - image: us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64 + image: lmsysorg/sglang:beam replicas: 1 envs: - name: DYN_DISCOVERY_BACKEND @@ -223,7 +223,7 @@ service: workload: model: reflection/Beam-1.0-501B-A23B-NVFP4 - image: us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64 + image: lmsysorg/sglang:beam framework: sglang configFile: serving-args.yaml configPath: /workload/configs diff --git a/inference/a4xmax/disaggregated-serving/dynamo/README.md b/inference/a4xmax/disaggregated-serving/dynamo/README.md index 82f3357e..2cccf80d 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/README.md +++ b/inference/a4xmax/disaggregated-serving/dynamo/README.md @@ -185,7 +185,7 @@ gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \ Stage your model checkpoint files into the GCS bucket so they mount under `/data/model/` via GCS FUSE. -For example, to stage LASER_BEAM: +For example, to stage Reflection Beam ([`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4)): ```bash pip install -U "huggingface_hub[cli]" @@ -220,10 +220,10 @@ This recipe is organized into deployment categories that pair a Helm topology (` | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | **DeepSeek-R1 671B (FP8)** | `deepseek-ai/DeepSeek-R1` | Low-Latency / without DeepEP | 1P1D (2 nodes / 8 GPUs) | `values_wo_deepep.yaml` | `deepseek-r1-fp8-1p1d-prefill.yaml` | `deepseekr1-fp8-1p1d-decode.yaml` | | **DeepSeek-R1 671B (FP8)** | `deepseek-ai/DeepSeek-R1` | High-Throughput / with DeepEP | 10P8D (18 nodes / 72 GPUs) | `values_deepep.yaml` | `deepseekr1-fp8-10p8d-prefill.yaml` | `deepseekr1-fp8-10p8d-decode.yaml` | -| **LASER_BEAM** | `reflection/Beam-1.0-501B-A23B-NVFP4` | High Throughput | 4P4D (8 nodes / 32 GPUs) | `laser_beam_high_throughput.yaml` | `laser_beam_high_throughput_prefill.yaml` | `laser_beam_high_throughput_decode.yaml` | +| **Reflection Beam 501B (NVFP4)** | [`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4) | High Throughput | 4P4D (8 nodes / 32 GPUs) | `beam_high_throughput.yaml` | `beam_high_throughput_prefill.yaml` | `beam_high_throughput_decode.yaml` | > [!NOTE] -> The LASER_BEAM High Interactivity recipe runs 8 aggregated workers, so it lives in the [aggregated Dynamo recipe](../../aggregated-serving/dynamo/README.md). +> The Reflection Beam High Interactivity recipe runs 8 aggregated workers, so it lives in the [aggregated Dynamo recipe](../../aggregated-serving/dynamo/README.md). > [!TIP] > To add or deploy another model, stage its weights in `gs://${GCS_BUCKET}`, add its prefill and decode YAML files under `$REPO_ROOT/src/frameworks/a4xmax/dynamo-configs/`, and pass `--set workload.model=${HF_MODEL_ID}` along with `--set-file prefill_serving_config=...` and `--set-file decode_serving_config=...` to `helm install`. @@ -290,17 +290,17 @@ helm install -f ${VALUES_FILE} \ $REPO_ROOT/src/helm-charts/a4xmax/inference-templates/dynamo-deployment ``` -#### LASER_BEAM (High Throughput) +#### Reflection Beam 501B (High Throughput) 1. Set the model and configuration variables: ```bash export HF_MODEL_ID="reflection/Beam-1.0-501B-A23B-NVFP4" -export DYNAMO_IMAGE="us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64" -export VALUES_FILE="laser_beam_high_throughput.yaml" -export PREFILL_CONFIG="laser_beam_high_throughput_prefill.yaml" -export DECODE_CONFIG="laser_beam_high_throughput_decode.yaml" -export RELEASE_NAME="$USER-dynamo-a4xmax-laser-beam-high-throughput" +export DYNAMO_IMAGE="lmsysorg/sglang:beam" +export VALUES_FILE="beam_high_throughput.yaml" +export PREFILL_CONFIG="beam_high_throughput_prefill.yaml" +export DECODE_CONFIG="beam_high_throughput_decode.yaml" +export RELEASE_NAME="$USER-dynamo-a4xmax-beam-high-throughput" ``` 2. Install the Helm release: @@ -412,18 +412,6 @@ kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " " ``` -3. Compare with the reference results below. - -**LASER_BEAM High Throughput (4P4D, 32 GPUs).** Each concurrency is a separate 3,600 s `aiperf` run on a deployment that had just completed a `CONCURRENCY=192` run. All runs completed with 0 failed requests. - -| Concurrency | Total tok/s (32 GPUs) | Output tok/s | TPOT p50 | TTFT p50 / p99 | Prefix-cache hit rate | Completed requests | -| :---: | :---: | :---: | :---: | :---: | :---: | :---: | -| 32 | 192,676 | 1,219 | 5.82 ms | 0.64 s / 2.63 s | 96.7% | 4,104 | -| 96 | 476,473 | 3,822 | 11.13 ms | 0.76 s / 6.79 s | 95.0% | 13,369 | -| 192 | 657,756 | 5,201 | 16.18 ms | 10.79 s / 34.97 s | 93.2% | 18,672 | - -The prefix-cache hit rate is `overall_usage_prompt_cache_read_pct` in `profile_export_aiperf.json`: cached prompt tokens divided by total prompt tokens, as reported in the response `usage` field. - ### 4.4. Category 2: Synthetic Random-Dataset Benchmarking diff --git a/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_throughput.yaml b/inference/a4xmax/disaggregated-serving/dynamo/beam_high_throughput.yaml similarity index 97% rename from inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_throughput.yaml rename to inference/a4xmax/disaggregated-serving/dynamo/beam_high_throughput.yaml index 75f0e128..e2c9ed43 100644 --- a/inference/a4xmax/disaggregated-serving/dynamo/laser_beam_high_throughput.yaml +++ b/inference/a4xmax/disaggregated-serving/dynamo/beam_high_throughput.yaml @@ -24,7 +24,7 @@ dynamo: claimTemplateName: serviceAccountName: frontend: - image: us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64 + image: lmsysorg/sglang:beam replicas: 1 envs: - name: DYN_DISCOVERY_BACKEND @@ -326,7 +326,7 @@ service: workload: model: reflection/Beam-1.0-501B-A23B-NVFP4 - image: us-central1-docker.pkg.dev/supercomputer-testing/pirillo-gcr/sglang-beam:dev-cu13-d4be2f8c-arm64 + image: lmsysorg/sglang:beam framework: sglang configFile: serving-args.yaml configPath: /workload/configs diff --git a/inference/g4/aggregated-serving/llm-d/README.md b/inference/g4/aggregated-serving/llm-d/README.md index d5785faa..e0074d59 100644 --- a/inference/g4/aggregated-serving/llm-d/README.md +++ b/inference/g4/aggregated-serving/llm-d/README.md @@ -1,15 +1,14 @@ # Aggregated 1M-Context Multi-Turn Inference with llm-d and vLLM on G4 GKE -This document outlines the steps to deploy and benchmark `reflection/Beam-1.0-501B-A23B-NVFP4` (`LASER_BEAM`) at **1,048,576 (`1M`) context length** using the [llm-d](https://github.com/llm-d/llm-d) intelligent inference scheduling stack with the [vLLM](https://docs.vllm.ai/en/latest/) backend on [G4 (`g4-standard-384`, NVIDIA RTX PRO 6000 Blackwell Server Edition) GKE Node pools](https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus). +This document outlines the steps to deploy and benchmark Reflection Beam ([`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4)) with the DFlash2 drafter ([`reflection/Beam-1.0-501B-A23B-DFlash2`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2)) at **1,048,576 (`1M`) context length** using the [llm-d](https://github.com/llm-d/llm-d) intelligent inference scheduling stack with the [vLLM](https://docs.vllm.ai/en/latest/) backend on [G4 (`g4-standard-384`, NVIDIA RTX PRO 6000 Blackwell Server Edition) GKE Node pools](https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus). `llm-d` pairs an Envoy proxy and an `ext_proc` Endpoint Picker (`EPP`) router with vLLM workers to provide real-time ZMQ KV-cache event tracking (`precise-prefix-cache-producer` on `:5556` / `:5559`), multi-tier prefix-affinity routing (`prefix-cache-affinity-filter` across GPU HBM and host DRAM), `program-aware-fairness` multi-turn scheduling (`strategy: turn-priority`), and queue/KV saturation flow control (`utilization-detector`). On `g4-standard-384` nodes (8x RTX PRO 6000 Blackwell Server Edition GPUs with 96 GB GDDR7 each + 1.44 TB host DRAM), this recipe deploys **`2x TP4 / EP4` (`8` GPUs total)** with **host DRAM KV-cache offloading (`OffloadingConnector` / `CPUOffloadingSpec`, `64 GiB` per TP4 replica / `128 GiB` per node)**, socket-local PCIe Gen5 **NUMA pinning (`numactl`)**, and **DFlash2 speculative decoding**. ## Table of Contents -* [1. Test Environment & Benchmark Results](#test-environment) +* [1. Test Environment](#test-environment) * [1.1. Hardware & Software Configuration](#hardware-config) - * [1.2. Verified 60-Minute 1M-Context AgentX Benchmark Results](#benchmark-results) * [2. Environment Setup (One-Time)](#environment-setup) * [2.1. Clone the Repository](#clone-repo) * [2.2. Configure Environment Variables](#configure-vars) @@ -30,14 +29,14 @@ This document outlines the steps to deploy and benchmark `reflection/Beam-1.0-50 * [6. Cleanup](#cleanup) -## 1. Test Environment & Benchmark Results +## 1. Test Environment [Back to Top](#table-of-contents) ### 1.1. Hardware & Software Configuration -This recipe has been tested and benchmarked with the following configuration: +This recipe has been tested with the following configuration: * **GKE Cluster**: * GPU node pools with [g4-standard-384](https://cloud.google.com/compute/docs/gpus#rtx-6000-gpus) machines (`cloud.google.com/gke-accelerator: nvidia-rtx-pro-6000`, 8x NVIDIA RTX PRO 6000 Blackwell Server Edition `SM120` GPUs with 96 GB GDDR7 each = 768 GB GPU memory per node, 384 vCPUs across 2 NUMA sockets, 1,440 GB host DRAM). @@ -45,19 +44,6 @@ This recipe has been tested and benchmarked with the following configuration: * [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) enabled. * [Cloud Storage FUSE CSI driver for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cloud-storage-fuse-csi-driver) enabled. - -### 1.2. Verified 60-Minute 1M-Context AgentX Benchmark Results - -Evaluated using [`aiperf`](https://github.com/ai-dynamo/aiperf) (`--scenario agentx --benchmark-duration 3600`) with the SemiAnalysis AgentX multi-turn coding trajectory dataset (`semianalysis_cc_traces_weka_062126`, up to `1M` context length) across `8` RTX PRO 6000 Blackwell GPUs (`2x TP4`): - -| Concurrency | Stack Configuration | Output Throughput (`tok/s/GPU`) | Prefix KV-Cache Hit Rate (`%`) | P90 TTFT (`s`) | Speedup / Improvement (`llm-d` vs. Baseline) | -| :--- | :--- | :--- | :--- | :--- | :--- | -| **`c=8`** | **llm-d + vLLM (DRAM Offload)** | **`39.7 tok/s/GPU`** | **`95.9%`** | — | **+13.4% Throughput**, **+3.5% KV Hit Rate** | -| **`c=8`** | Standalone vLLM Baseline | `35.0 tok/s/GPU` | `92.4%` | — | *(Reference)* | -| **`c=16`** | **llm-d + vLLM (DRAM Offload)** | **`51.3 tok/s/GPU`** | **`88.5%`** | **`10.26s`** | **+95.8% (1.96x) Throughput**, **+34.4% KV Hit**, **11.3x Lower P90 TTFT** | -| **`c=16`** | Standalone vLLM Baseline | `26.2 tok/s/GPU` | `54.1%` | `115.70s` | *(Reference — severe HBM KV eviction & recomputation)* | -| **`c=32`** | **llm-d + vLLM (DRAM Offload)** | **`25.5 tok/s/GPU`** | **`59.1%`** | **`195.98s`** | Sustains `59.1%` KV hit rate under `32` concurrent 1M-context sessions | - ## 2. Environment Setup (One-Time) @@ -84,8 +70,8 @@ export CLUSTER_NAME= export NAMESPACE=llmd-inference export HF_TOKEN= export GCS_BUCKET= -export LASER_BEAM_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-NVFP4 -export LASER_BEAM_DRAFTER_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-DFlash2 +export BEAM_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-NVFP4 +export BEAM_DRAFTER_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-DFlash2 # Set the project for gcloud commands gcloud config set project $PROJECT_ID @@ -102,8 +88,8 @@ Replace the following values: | `NAMESPACE` | Kubernetes namespace for the deployment | `llmd-inference` | | `HF_TOKEN` | Your Hugging Face access token | `` | | `GCS_BUCKET` | Your GCS bucket name (without `gs://` prefix for Helm `bucketName`) | `` | -| `LASER_BEAM_HF_MODEL_ID` | Target model Hugging Face ID | `reflection/Beam-1.0-501B-A23B-NVFP4` | -| `LASER_BEAM_DRAFTER_HF_MODEL_ID` | DFlash2 speculative drafter Hugging Face ID | `reflection/Beam-1.0-501B-A23B-DFlash2` | +| `BEAM_HF_MODEL_ID` | Target model Hugging Face ID ([`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4)) | `reflection/Beam-1.0-501B-A23B-NVFP4` | +| `BEAM_DRAFTER_HF_MODEL_ID` | DFlash2 speculative drafter Hugging Face ID ([`reflection/Beam-1.0-501B-A23B-DFlash2`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2)) | `reflection/Beam-1.0-501B-A23B-DFlash2` | ### 2.3. Connect to your GKE Cluster @@ -151,7 +137,7 @@ gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \ Set your environment variables for the vLLM engine image (built with `SM120` Blackwell support, `flashinfer_cutlass` MoE kernels, DFlash2 speculative decoding, and `OffloadingConnector` KV-cache CPU offloading) and the `llm-d` Endpoint Picker (`EPP`) router image: ```bash -export VLLM_IMAGE="us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64" +export VLLM_IMAGE="vllm/vllm-openai:reflection-beam" export EPP_IMAGE="ghcr.io/llm-d/llm-d-router-endpoint-picker:main" ``` @@ -206,7 +192,7 @@ flowchart LR | **#2 Host DRAM KV-Cache Offloading (`OffloadingConnector` / `CPUOffloadingSpec`)** | `cpu_bytes_to_use: 68719476736` (`64 GiB` per TP4 replica = `128 GiB/node` for 2x TP4/node; or `137438953472` = `128 GiB/replica` for 1x TP4/node), `blocks_per_chunk: 16`, `offload_prompt_only: true` | Extends effective KV-cache capacity into `g4-standard-384` host DRAM and publishes ZMQ tier events (`:5556` / `:5559`, `gpu: 1.0`, `cpu: 0.4`) so evicted multi-turn prefixes are reloaded via PCIe DMA instead of recomputed. | | **#3 Engine Batching & CUDA Graphs** | `--max-num-batched-tokens 16384`, `--max-num-seqs 16`, `--block-size 64`, `FULL_AND_PIECEWISE` CUDA graphs `1..1152` (`fuse_allreduce_rms: false`) | Cuts 100K–700K prefill chunks in half (updated from `8192` to `16384`), bounds concurrent active sequences (`16`), and captures CUDA graphs up to `1152` tokens for speculative decoding. | | **#4 DFlash2 Speculative Decoding** | `num_speculative_tokens: 8`, `draft_tensor_parallel_size: 4`, `--no-disable-hybrid-kv-cache-manager`, `--swa-bounded-replay`, `--moe-backend flashinfer_cutlass` | Accelerates decode interactivity (TPOT/ITL) using block-diffusion speculative decoding with bounded sliding-window attention replay. | -| **#5 `llm-d` EPP Routing & Flow Control (`EndpointPickerConfig`)** | `prefix-cache-affinity-filter` (`affinityThreshold: 0.75`, `peakPrefillThroughput: 16444`), `utilization-detector` (`queueDepthThreshold: 6`, `kvCacheUtilThreshold: 0.80`), `program-aware-fairness` (`turn-priority`), `speculativeIndexing: true` | Directs multi-turn agentic sessions to the pod holding their GPU/DRAM prefix (`88.5%` KV hit at `c=16` vs. `54.1%` without `llm-d`, yielding **1.96x higher throughput** and **11.3x lower P90 TTFT**). | +| **#5 `llm-d` EPP Routing & Flow Control (`EndpointPickerConfig`)** | `prefix-cache-affinity-filter` (`affinityThreshold: 0.75`, `peakPrefillThroughput: 16444`), `utilization-detector` (`queueDepthThreshold: 6`, `kvCacheUtilThreshold: 0.80`), `program-aware-fairness` (`turn-priority`), `speculativeIndexing: true` | Directs multi-turn agentic sessions to the pod holding their GPU or host DRAM prefix while balancing in-flight prefill and queue load. | ### 3.3. Option A: Deploy with Helm @@ -214,8 +200,8 @@ flowchart LR 1. Set the release and model variables: ```bash -export HF_MODEL_ID="${LASER_BEAM_HF_MODEL_ID}" -export DRAFTER_MODEL_ID="${LASER_BEAM_DRAFTER_HF_MODEL_ID}" +export HF_MODEL_ID="${BEAM_HF_MODEL_ID}" +export DRAFTER_MODEL_ID="${BEAM_DRAFTER_HF_MODEL_ID}" export RELEASE_NAME="llmd-inference" ``` @@ -298,8 +284,8 @@ Use [`aiperf`](https://github.com/ai-dynamo/aiperf) with the SemiAnalysis AgentX ```bash export FRONTEND_HOST="${RELEASE_NAME}-frontend" -export MODEL_NAME="${LASER_BEAM_HF_MODEL_ID}" -export TOKENIZER_PATH="${LASER_BEAM_HF_MODEL_ID}" +export MODEL_NAME="${BEAM_HF_MODEL_ID}" +export TOKENIZER_PATH="${BEAM_HF_MODEL_ID}" export CONCURRENCY=16 kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c " diff --git a/inference/g4/aggregated-serving/llm-d/manifests/modelserver-2x-tp4-1m.yaml b/inference/g4/aggregated-serving/llm-d/manifests/modelserver-2x-tp4-1m.yaml index 0b0a0ab0..0b4b893b 100644 --- a/inference/g4/aggregated-serving/llm-d/manifests/modelserver-2x-tp4-1m.yaml +++ b/inference/g4/aggregated-serving/llm-d/manifests/modelserver-2x-tp4-1m.yaml @@ -89,7 +89,7 @@ spec: operator: "Exists" containers: - name: vllm - image: us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64 + image: vllm/vllm-openai:reflection-beam imagePullPolicy: IfNotPresent command: ["/bin/bash", "-lc"] args: @@ -312,7 +312,7 @@ spec: operator: "Exists" containers: - name: vllm - image: us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64 + image: vllm/vllm-openai:reflection-beam imagePullPolicy: IfNotPresent command: ["/bin/bash", "-lc"] args: diff --git a/inference/g4/aggregated-serving/llm-d/values.yaml b/inference/g4/aggregated-serving/llm-d/values.yaml index 17062817..1d8b1522 100644 --- a/inference/g4/aggregated-serving/llm-d/values.yaml +++ b/inference/g4/aggregated-serving/llm-d/values.yaml @@ -50,7 +50,7 @@ workload: drafterName: reflection/Beam-1.0-501B-A23B-DFlash2 servedModelName: "" maxModelLen: 1048576 - image: us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64 + image: vllm/vllm-openai:reflection-beam framework: vllm replicas: 2 gpusPerReplica: 4 diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_agg.yaml b/src/frameworks/a4xmax/dynamo-configs/beam_high_int_agg.yaml similarity index 91% rename from src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_agg.yaml rename to src/frameworks/a4xmax/dynamo-configs/beam_high_int_agg.yaml index 2df08e63..7f47626a 100644 --- a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_int_agg.yaml +++ b/src/frameworks/a4xmax/dynamo-configs/beam_high_int_agg.yaml @@ -39,7 +39,7 @@ cuda-graph-max-bs-decode: "256" cuda-graph-bs-decode: "[1, 2, 4, 8, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256]" cuda-graph-bs-prefill: "[256, 512, 1024, 2048, 4096, 8192]" speculative-algorithm: "DFLASH" -speculative-draft-model-path: "/data/model/huggingface_model_cache/models--reflection--dflash2-drafter-preview-final/snapshots/8d26c952ec6a82a9a18825d6126c9f6948149ad9" +speculative-draft-model-path: "/data/model/reflection/Beam-1.0-501B-A23B-DFlash2" speculative-eagle-topk: "1" speculative-num-draft-tokens: "8" speculative-num-steps: "1" diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_decode.yaml b/src/frameworks/a4xmax/dynamo-configs/beam_high_throughput_decode.yaml similarity index 100% rename from src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_decode.yaml rename to src/frameworks/a4xmax/dynamo-configs/beam_high_throughput_decode.yaml diff --git a/src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_prefill.yaml b/src/frameworks/a4xmax/dynamo-configs/beam_high_throughput_prefill.yaml similarity index 100% rename from src/frameworks/a4xmax/dynamo-configs/laser_beam_high_throughput_prefill.yaml rename to src/frameworks/a4xmax/dynamo-configs/beam_high_throughput_prefill.yaml diff --git a/src/frameworks/g4/llmd-configs/laser_beam_1m_offload_epp.yaml b/src/frameworks/g4/llmd-configs/beam_1m_offload_epp.yaml similarity index 100% rename from src/frameworks/g4/llmd-configs/laser_beam_1m_offload_epp.yaml rename to src/frameworks/g4/llmd-configs/beam_1m_offload_epp.yaml diff --git a/src/frameworks/g4/llmd-configs/laser_beam_1m_offload_vllm.yaml b/src/frameworks/g4/llmd-configs/beam_1m_offload_vllm.yaml similarity index 100% rename from src/frameworks/g4/llmd-configs/laser_beam_1m_offload_vllm.yaml rename to src/frameworks/g4/llmd-configs/beam_1m_offload_vllm.yaml diff --git a/src/helm-charts/g4/llm-d-inference/values.yaml b/src/helm-charts/g4/llm-d-inference/values.yaml index 17062817..1d8b1522 100644 --- a/src/helm-charts/g4/llm-d-inference/values.yaml +++ b/src/helm-charts/g4/llm-d-inference/values.yaml @@ -50,7 +50,7 @@ workload: drafterName: reflection/Beam-1.0-501B-A23B-DFlash2 servedModelName: "" maxModelLen: 1048576 - image: us-central1-docker.pkg.dev/gpu-launchpad-playground/vllm/vllm-beam:lin-tracking-cu13-8e0922e8-sm120-x86_64 + image: vllm/vllm-openai:reflection-beam framework: vllm replicas: 2 gpusPerReplica: 4 From 057d410a3f462f5c21c5f49407282d6b13200687 Mon Sep 17 00:00:00 2001 From: Christopher Pirillo Date: Fri, 16 Oct 2026 13:52:50 +0000 Subject: [PATCH 7/8] Add Reflection Beam 501B recipes to top-level README support matrix --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index 2108de27..4f52fdb1 100644 --- a/README.md +++ b/README.md @@ -176,10 +176,20 @@ Models | GPU Machine Type | **Qwen 3 4B** | [A4X (NVIDIA GB200)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | TensorRT-LLM (1.3.0rc5) | Inference | GKE | [Link](./inference/a4x/single-host-serving/tensorrt-llm/README.md) +### Inference benchmarks A4X MAX + +| Models | GPU Machine Type | Framework | Workload Type | Orchestrator | Link to the recipe | +| ---------------- | ---------------- | --------- | ------------------- | ------------ | ------------------ | +| **Reflection Beam 501B** | [A4X MAX (NVIDIA GB300)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | Dynamo + SGLang | Aggregated Inference | GKE | [Link](./inference/a4xmax/aggregated-serving/dynamo/README.md) | +| **Reflection Beam 501B** | [A4X MAX (NVIDIA GB300)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | Dynamo + SGLang | Disaggregated Inference | GKE | [Link](./inference/a4xmax/disaggregated-serving/dynamo/README.md) | +| **DeepSeek R1 671B** | [A4X MAX (NVIDIA GB300)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#a4x-vms) | Dynamo + SGLang | Disaggregated Inference | GKE | [Link](./inference/a4xmax/disaggregated-serving/dynamo/README.md) | + + ### Inference benchmarks G4 | Models | GPU Machine Type | Framework | Workload Type | Orchestrator | Link to the recipe | | ---------------- | ---------------- | --------- | ------------------- | ------------ | ------------------ | +| **Reflection Beam 501B** | [G4 (NVIDIA RTX PRO 6000 Blackwell)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#g4-series) | llm-d + vLLM | Inference | GKE | [Link](./inference/g4/aggregated-serving/llm-d/README.md) | | **Qwen3 8B** | [G4 (NVIDIA RTX PRO 6000 Blackwell)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#g4-series) | vLLM | Inference | GCE | [Link](./inference/g4/qwen-8b/single-host-serving/vllm/README.md) | **Qwen3 30B A3B**| [G4 (NVIDIA RTX PRO 6000 Blackwell)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#g4-series) | TensorRT-LLM | Inference | GCE | [Link](./inference/g4/qwen3_30b_a3b/single-host-serving/tensorrt-llm/README.md) | **Qwen3 4B** | [G4 (NVIDIA RTX PRO 6000 Blackwell)](https://cloud.google.com/compute/docs/accelerator-optimized-machines#g4-series) | TensorRT-LLM | Inference | GCE | [Link](./inference/g4/qwen3_4b/single-host-serving/tensorrt-llm/README.md) From c0cd90c844c662f3c081fd6b3b312e19ca192bae Mon Sep 17 00:00:00 2001 From: Christopher Pirillo Date: Fri, 16 Oct 2026 13:54:29 +0000 Subject: [PATCH 8/8] Use HF_MODEL_ID and DRAFTER_HF_MODEL_ID in G4 llm-d README --- .../g4/aggregated-serving/llm-d/README.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/inference/g4/aggregated-serving/llm-d/README.md b/inference/g4/aggregated-serving/llm-d/README.md index e0074d59..55e74b27 100644 --- a/inference/g4/aggregated-serving/llm-d/README.md +++ b/inference/g4/aggregated-serving/llm-d/README.md @@ -70,8 +70,8 @@ export CLUSTER_NAME= export NAMESPACE=llmd-inference export HF_TOKEN= export GCS_BUCKET= -export BEAM_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-NVFP4 -export BEAM_DRAFTER_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-DFlash2 +export HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-NVFP4 +export DRAFTER_HF_MODEL_ID=reflection/Beam-1.0-501B-A23B-DFlash2 # Set the project for gcloud commands gcloud config set project $PROJECT_ID @@ -88,8 +88,8 @@ Replace the following values: | `NAMESPACE` | Kubernetes namespace for the deployment | `llmd-inference` | | `HF_TOKEN` | Your Hugging Face access token | `` | | `GCS_BUCKET` | Your GCS bucket name (without `gs://` prefix for Helm `bucketName`) | `` | -| `BEAM_HF_MODEL_ID` | Target model Hugging Face ID ([`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4)) | `reflection/Beam-1.0-501B-A23B-NVFP4` | -| `BEAM_DRAFTER_HF_MODEL_ID` | DFlash2 speculative drafter Hugging Face ID ([`reflection/Beam-1.0-501B-A23B-DFlash2`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2)) | `reflection/Beam-1.0-501B-A23B-DFlash2` | +| `HF_MODEL_ID` | Target model Hugging Face ID ([`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4)) | `reflection/Beam-1.0-501B-A23B-NVFP4` | +| `DRAFTER_HF_MODEL_ID` | DFlash2 speculative drafter Hugging Face ID ([`reflection/Beam-1.0-501B-A23B-DFlash2`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2)) | `reflection/Beam-1.0-501B-A23B-DFlash2` | ### 2.3. Connect to your GKE Cluster @@ -197,11 +197,9 @@ flowchart LR ### 3.3. Option A: Deploy with Helm -1. Set the release and model variables: +1. Set the release name: ```bash -export HF_MODEL_ID="${BEAM_HF_MODEL_ID}" -export DRAFTER_MODEL_ID="${BEAM_DRAFTER_HF_MODEL_ID}" export RELEASE_NAME="llmd-inference" ``` @@ -214,7 +212,7 @@ helm install ${RELEASE_NAME} \ -f values.yaml \ --namespace ${NAMESPACE} \ --set workload.model.name=${HF_MODEL_ID} \ - --set workload.model.drafterName=${DRAFTER_MODEL_ID} \ + --set workload.model.drafterName=${DRAFTER_HF_MODEL_ID} \ --set workload.image=${VLLM_IMAGE} \ --set llmd.router.eppImage=${EPP_IMAGE} ``` @@ -284,8 +282,8 @@ Use [`aiperf`](https://github.com/ai-dynamo/aiperf) with the SemiAnalysis AgentX ```bash export FRONTEND_HOST="${RELEASE_NAME}-frontend" -export MODEL_NAME="${BEAM_HF_MODEL_ID}" -export TOKENIZER_PATH="${BEAM_HF_MODEL_ID}" +export MODEL_NAME="${HF_MODEL_ID}" +export TOKENIZER_PATH="${HF_MODEL_ID}" export CONCURRENCY=16 kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c "