[syeda-anjum]

Adding a vLLM (TorchTPU backend) single-host inference recipe for Qwen/Qwen3-32B on Trillium (TPU v6e) on GKE under inference/trillium/vLLM/vLLM-TorchTPU/Qwen3-32B/single-host.

• Model: Qwen/Qwen3-32B (BF16 weights + FP8 KV Cache)
• Topology: Single ct6e-standard-8t node pool (2x4, 8 TPU v6e chips), TP=8
• Serving: vLLM with TorchTPU plugin, custom attention backend, async scheduling, and FP8 KV Cache.
• Benchmark Workloads: Validated with the SemiAnalysisAI InferenceX (benchmark_serving.py) client.
• uBench Run ID: vllm_inference-torchtpu_vllm/qwen3_32b_v6e_single_host_1k_1k-2026-10-19_035034-ca177b03-d73e-43c3-a6f6-dfda2855446b

[gemini-code-assist]

Code Review

This pull request introduces the configuration files and documentation required to serve the Qwen3-32B model using vLLM (TorchTPU) on a single-host Trillium TPU v6e node pool in GKE. It includes a step-by-step README guide, benchmark Pod definitions, and the vLLM server deployment manifest. To ensure successful deployment and accurate performance testing, the CPU and memory requests in the server deployment should be reduced slightly to account for GKE system overhead and prevent scheduling failures. Additionally, the benchmarking instructions should be updated to run the benchmark workloads sequentially rather than concurrently to avoid resource contention.

[gemini-code-assist]

Requesting 100% of the node's physical CPU (64 cores) and memory (256Gi) will cause the Pod to fail scheduling on GKE. GKE reserves a portion of the node's CPU and memory for system overhead (kubelet, kube-proxy, logging agents, etc.), meaning the allocatable resources are always less than the physical capacity. Lowering these requests/limits slightly (e.g., to 55 CPUs and 210Gi memory) ensures the Pod can be successfully scheduled on a ct6e-standard-8t node.

            limits:
              cpu: "55"
              memory: 210Gi
              google.com/tpu: "8"
            requests:
              cpu: "55"
              memory: 210Gi
              google.com/tpu: "8"

[syeda-anjum]

The premise here is incorrect. ct6e-standard-8t has 360 vCPUs and 1,440 GiB host RAM (gcloud compute machine-types describe ct6e-standard-8t), not 64 vCPU / 256 GiB — the 256 GB figure is the TPU HBM, not host memory. The cpu: 64 / memory: 256Gi request is ~18% of node allocatable and schedules fine; this exact manifest was validated end-to-end on a ct6e-standard-8t node pool. No change needed.

[gemini-code-assist]

Applying qwen3-32b-benchmarks.yaml directly as instructed will create and run all three benchmark Pods (1k/1k, 1k/8k, and 8k/1k) concurrently. This will cause severe resource contention on the single-host vLLM server, leading to high queueing latency and inaccurate benchmark results. To obtain reliable performance metrics, the benchmarks should be run sequentially. Consider updating the instructions to run them one at a time (e.g., apply one, wait for completion, delete it, and then apply the next).

[syeda-anjum]

Good catch — addressed in the latest revision. The separate qwen3-32b-benchmarks.yaml was removed; each workload is now its own inline manifest in the README with an explicit apply → logs → delete sequence (matching the Gemma4-26B recipe), so benchmarks run strictly one at a time.

[syeda-anjum]

@joeywan-google could you please review? This follows the same structure as the Trillium Gemma4-26B vLLM-TorchTPU recipe (README.md + *-server.yaml, benchmark Pods inlined in Step 6). I have replied inline to the Gemini Code Assist comments — the resource-limit and sequencing points are resolved/not applicable.