kadadsns · Aug 6, 2026 · Aug 6, 2026 · Aug 6, 2026 · Aug 6, 2026 · Aug 6, 2026
diff --git a/ASSIGNMENT_REPORT.md b/ASSIGNMENT_REPORT.md
@@ -0,0 +1,127 @@
+# LLAISYS Assignment #4 Report
+
+## Summary
+
+This pull request completes Assignment #4: CUDA integration for LLAISYS.
+
+Implemented:
+
+- CUDA Runtime APIs for device management, streams, memory allocation, and memory copies.
+- NVIDIA device resource management with a cuBLAS handle.
+- CUDA implementations for all required operators: add, argmax, embedding, linear, rearrange, rms_norm, rope, self_attention, and swiglu.
+- CUDA inference support for the Qwen2 model through the existing tensor and operator dispatch layers.
+- Xmake build configuration for NVIDIA (`--nv-gpu=y`) and MetaX MACA (`--metax-gpu=y`).
+
+## Supported Platforms and Status
+
+| Platform | Status |
+| --- | --- |
+| CPU | Passed. Assignment #0-#3 tests pass in CI. |
+| NVIDIA RTX 4090 | Passed. Runtime, all operator tests, and full model inference test pass. |
+| MetaX C500 (MACA) | Passed. Build with Makefile.metax; runtime, operator, and model inference tests pass. |
+
+## Reproduction Procedure
+
+### NVIDIA RTX 4090
+
+Test environment:
+
+- GPU: NVIDIA GeForce RTX 4090 D
+- Driver: 570.124.06
+- CUDA Toolkit: 12.8
+- Python: 3.12
+- PyTorch: NVIDIA build with CUDA 12.8 support
+- Xmake: 3.1.0
+
+Build:
+
+```bash
+xmake f --nv-gpu=y -c
+xmake -r
+xmake install
+```
+
+Run tests:
+
+```bash
+export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
+export PYTHONPATH=$PWD/python
+
+python test/test_runtime.py --device nvidia
+python test/ops/add.py --device nvidia
+python test/ops/argmax.py --device nvidia
+python test/ops/embedding.py --device nvidia
+python test/ops/linear.py --device nvidia
+python test/ops/rms_norm.py --device nvidia
+python test/ops/rope.py --device nvidia
+python test/ops/self_attention.py --device nvidia
+python test/ops/swiglu.py --device nvidia
+python test/test_infer.py --model [model_path] --test --device nvidia
+```
+
+### MetaX C500 (pending hardware validation)
+### MetaX C500 (MACA)
+
+Test environment:
+
+- MACA: 3.5.3
+- Device architecture: xcore1000
+- Python: conda 3.10 with torch 2.8.0+metax3.5.3.9
+
+Build with the standalone MetaX Makefile:
+
+```bash
+make -f Makefile.metax -j4
+make -f Makefile.metax install
+```
+
+Run tests with the same `--device nvidia` commands because MetaX reuses the `LLAISYS_DEVICE_NVIDIA` device type through the MACA CUDA-compatible runtime:
+
+```bash
+export MACA_PATH=/opt/maca-3.5.3
+export MACA_HOME=/opt/maca-3.5.3
+export LD_LIBRARY_PATH=/opt/maca-3.5.3/lib64:/opt/maca-3.5.3/mxgpu_llvm/lib64:$LD_LIBRARY_PATH
+export PYTHONPATH=$PWD/python
+
+python test/test_runtime.py --device nvidia
+python test/ops/add.py --device nvidia
+python test/ops/argmax.py --device nvidia
+python test/ops/embedding.py --device nvidia
+python test/ops/linear.py --device nvidia
+python test/ops/rms_norm.py --device nvidia
+python test/ops/rope.py --device nvidia
+python test/ops/self_attention.py --device nvidia
+python test/ops/swiglu.py --device nvidia
+python test/test_infer.py --test --device nvidia
+```
+
+## Results
+
+### CI
+
+Latest Actions run succeeded on both platforms:
+
+- Windows latest: success
+- Ubuntu latest: success
+- Assignment #0, #1, #2, and #3 all passed.
+
+### RTX 4090
+
+- Runtime test: passed.
+- Operator tests: all 8 operators passed for F32, F16, and BF16.
+- Model inference: `test/test_infer.py --test --device nvidia` passed with 128 generated tokens, matching PyTorch output exactly.
+
+### MetaX C500
+
+- Runtime test: passed.
+- Operator tests: all 8 operators passed for F32, F16, and BF16.
+- Model inference: `test/test_infer.py --test --device nvidia` passed with 128 generated tokens, matching PyTorch output exactly.
+
+## Notable Fixes
+
+- Added `src/llaisys/cuda_link.cu` so the shared library is device-linked with nvcc; without this, loading `libllaisys.so` fails with an undefined `__cudaRegisterLinkedBinary` symbol.
+- Changed F16/BF16 linear to use `cublasGemmEx` with `CUBLAS_COMPUTE_32F` and float alpha/beta for FP32 accumulation.
+- Fixed the CUDA self-attention causal mask when `kvlen > qlen` by using `key_limit = q_idx + (kvlen - qlen)`.
+- Fixed the CUDA argmax second-stage reduction to handle more than 256 block results via a grid-stride loop.
+- Fixed `test/ops/self_attention.py` so the causal mask is created on the same device as the query tensor.
+- Added `Makefile.metax`, a standalone MetaX build using mxcc with `--maca-link -fgpu-rdc` and `-lruntime_cu`.
diff --git a/ASSIGNMENT_REPORT_ZH.md b/ASSIGNMENT_REPORT_ZH.md
@@ -0,0 +1,128 @@
+# LLAISYS 作业 #4 报告
+
+## 总结
+
+本次 PR 完成作业 #4：为 LLAISYS 集成 CUDA。
+
+实现内容包括：
+
+- CUDA Runtime API：设备管理、stream、显存分配、内存拷贝
+- NVIDIA 设备资源管理，包含 cuBLAS handle
+- 全部作业算子的 CUDA 实现：`add`、`argmax`、`embedding`、`linear`、`rearrange`、`rms_norm`、`rope`、`self_attention`、`swiglu`
+- Qwen2 模型通过统一 Tensor/算子分发支持 CUDA 推理
+- Xmake 构建配置支持 NVIDIA（`--nv-gpu=y`）和 MetaX MACA（`--metax-gpu=y`）
+
+## 支持平台与状态
+
+| 平台 | 状态 |
+| --- | --- |
+| CPU | 通过。CI 中 Assignment #0-#3 全部通过。 |
+| NVIDIA RTX 4090 | 通过。Runtime、全部算子测试、完整模型推理测试均通过。 |
+| MetaX C500（MACA） | 构建配置和源码已就绪，等待真实硬件验证。 |
+| MetaX C500（MACA） | 通过。使用 Makefile.metax 构建，Runtime、全部算子测试和完整模型推理均通过。 |
+
+## 复现步骤
+
+### RTX 4090
+
+测试环境：
+
+- GPU：NVIDIA GeForce RTX 4090 D
+- 驱动：570.124.06
+- CUDA Toolkit：12.8
+- Python：3.12
+- PyTorch：NVIDIA 版，支持 CUDA 12.8
+- Xmake：3.1.0
+
+构建：
+
+```bash
+xmake f --nv-gpu=y -c
+xmake -r
+xmake install
+```
+
+测试：
+
+```bash
+export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
+export PYTHONPATH=$PWD/python
+
+python test/test_runtime.py --device nvidia
+python test/ops/add.py --device nvidia
+python test/ops/argmax.py --device nvidia
+python test/ops/embedding.py --device nvidia
+python test/ops/linear.py --device nvidia
+python test/ops/rms_norm.py --device nvidia
+python test/ops/rope.py --device nvidia
+python test/ops/self_attention.py --device nvidia
+python test/ops/swiglu.py --device nvidia
+python test/test_infer.py --model [model_path] --test --device nvidia
+```
+
+### MetaX C500（待硬件验证）
+### MetaX C500（MACA）
+
+测试环境：
+
+- MACA：3.5.3
+- 设备架构：xcore1000
+- Python：conda 3.10，torch 2.8.0+metax3.5.3.9
+
+使用独立的 MetaX Makefile 构建：
+
+```bash
+make -f Makefile.metax -j4
+make -f Makefile.metax install
+```
+
+测试命令仍使用 `--device nvidia`，因为 MetaX 通过 MACA 的 CUDA 兼容 Runtime 复用 `LLAISYS_DEVICE_NVIDIA` 设备类型：
+
+```bash
+export MACA_PATH=/opt/maca-3.5.3
+export MACA_HOME=/opt/maca-3.5.3
+export LD_LIBRARY_PATH=/opt/maca-3.5.3/lib64:/opt/maca-3.5.3/mxgpu_llvm/lib64:$LD_LIBRARY_PATH
+export PYTHONPATH=$PWD/python
+
+python test/test_runtime.py --device nvidia
+python test/ops/add.py --device nvidia
+python test/ops/argmax.py --device nvidia
+python test/ops/embedding.py --device nvidia
+python test/ops/linear.py --device nvidia
+python test/ops/rms_norm.py --device nvidia
+python test/ops/rope.py --device nvidia
+python test/ops/self_attention.py --device nvidia
+python test/ops/swiglu.py --device nvidia
+python test/test_infer.py --test --device nvidia
+```
+
+## 测试结果
+
+### CI
+
+最新 Actions 已通过：
+
+- Windows latest：成功
+- Ubuntu latest：成功
+- Assignment #0、#1、#2、#3 全部通过
+
+### RTX 4090
+
+- Runtime 测试：通过
+- 算子测试：8 个算子全部通过，覆盖 F32 / F16 / BF16
+- 模型推理：`test_infer.py --test --device nvidia` 通过，128 个 token 与 PyTorch 输出完全一致
+
+### MetaX C500
+
+- Runtime 测试：通过
+- 算子测试：8 个算子全部通过，覆盖 F32 / F16 / BF16
+- 模型推理：`test_infer.py --test --device nvidia` 通过，128 个 token 与 PyTorch 输出完全一致
+
+## 关键修复
+
+- 新增 `src/llaisys/cuda_link.cu`，让共享库使用 nvcc 完成 device link；否则加载 `libllaisys.so` 会报未定义的 `__cudaRegisterLinkedBinary` 符号。
+- Linear 的 F16/BF16 改为 `cublasGemmEx` + `CUBLAS_COMPUTE_32F`，使用 FP32 累加。
+- 修复 CUDA self-attention 在 `kvlen > qlen` 时的 causal mask 偏移：`key_limit = q_idx + (kvlen - qlen)`。
+- 修复 CUDA argmax 二级归约：当 block 数量超过 256 时，改用 grid-stride 循环处理所有 block 结果。
+- 修复 `test/ops/self_attention.py`，causal mask 改为在 query 所在设备上创建。
+- 新增 `Makefile.metax`，使用 mxcc 的 `--maca-link -fgpu-rdc` 并链接 `-lruntime_cu` 完成 MetaX 独立构建。
diff --git a/CUDA_FRAMEWORK.md b/CUDA_FRAMEWORK.md
@@ -0,0 +1,98 @@
+# CUDA 代码框架说明
+
+## 已创建的文件结构
+
+```
+src/
+├── device/
+│   └── nvidia/
+│       ├── nvidia_runtime_api.cu    # CUDA Runtime API 实现
+│       └── nvidia_resource.cu       # CUDA 资源管理
+├── ops/
+│   ├── add/nvidia/
+│   │   └── add_nvidia.cu           # 加法算子 CUDA 实现
+│   ├── argmax/nvidia/
+│   │   └── argmax_nvidia.cu        # Argmax 算子 CUDA 实现
+│   ├── embedding/nvidia/
+│   │   └── embedding_nvidia.cu     # Embedding 算子 CUDA 实现
+│   ├── linear/nvidia/
+│   │   └── linear_nvidia.cu        # 线性层算子 CUDA 实现
+│   ├── rearrange/nvidia/
+│   │   └── rearrange_nvidia.cu     # 重排算子 CUDA 实现
+│   ├── rms_norm/nvidia/
+│   │   └── rms_norm_nvidia.cu      # RMS Norm 算子 CUDA 实现
+│   ├── rope/nvidia/
+│   │   └── rope_nvidia.cu          # RoPE 算子 CUDA 实现
+│   ├── self_attention/nvidia/
+│   │   └── self_attention_nvidia.cu # 自注意力算子 CUDA 实现
+│   └── swiglu/nvidia/
+│       └── swiglu_nvidia.cu        # SwiGLU 算子 CUDA 实现
+└── xmake/
+    └── nvidia.lua                   # NVIDIA CUDA 编译配置
+```
+
+## 主要功能
+
+### 1. CUDA Runtime API (nvidia_runtime_api.cu)
+- 内存管理：malloc, free, memset
+- 内存拷贝：H2D, D2H, D2D
+- 设备管理：set_device, get_device, get_device_count
+- 同步：synchronize
+
+### 2. CUDA 资源管理 (nvidia_resource.cu)
+- cuBLAS 句柄管理
+- cuDNN 句柄管理
+- CUDA 流管理
+
+### 3. CUDA 算子实现
+- **Add**: 元素级加法
+- **Argmax**: 求最大值和索引
+- **Embedding**: 嵌入查找
+- **Linear**: 线性变换（使用 cuBLAS）
+- **Rearrange**: 张量重排
+- **RMS Norm**: RMS 归一化
+- **RoPE**: 旋转位置编码
+- **Self-Attention**: 自注意力机制
+- **SwiGLU**: SwiGLU 激活函数
+
+## 编译配置
+
+### 1. 启用 CUDA 支持
+```bash
+xmake f --nv-gpu=y -cv
+xmake
+xmake install
+```
+
+### 2. 测试 CUDA 实现
+```bash
+# 测试运行时
+python test/test_runtime.py --device nvidia
+
+# 测试算子
+python test/test_ops.py --device nvidia
+
+# 测试模型
+python test/test_infer.py --model [model_path] --device nvidia
+```
+
+## 注意事项
+
+1. **需要 CUDA Toolkit**: 确保安装了 CUDA Toolkit
+2. **GPU 驱动**: 确保安装了 NVIDIA GPU 驱动
+3. **编译器**: 需要支持 CUDA 的 C++ 编译器
+4. **内存**: 确保有足够的显存
+
+## 下一步
+
+1. 安装 CUDA Toolkit
+2. 配置编译环境
+3. 实现完整的 CUDA 算子
+4. 集成到模型
+5. 测试和优化
+
+## 参考资源
+
+- [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/)
+- [cuBLAS Library](https://docs.nvidia.com/cuda/cublas/)
+- [cuDNN Library](https://docs.nvidia.com/deeplearning/cudnn/)