diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml new file mode 100644 index 0000000..97be8d1 --- /dev/null +++ b/.github/workflows/validate.yaml @@ -0,0 +1,24 @@ +name: validate + +on: + push: + pull_request: + +permissions: + contents: read + +jobs: + go: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-go@v7 + with: + go-version-file: go.mod + cache: true + - run: go mod verify + - name: Check formatting + run: test -z "$(gofmt -l .)" + - run: go test ./... + - run: go vet ./... + - run: go build ./... diff --git a/.gitignore b/.gitignore index e7ecc12..9db1ad4 100644 --- a/.gitignore +++ b/.gitignore @@ -34,6 +34,8 @@ kubeconfig* evidence/ reports/ validation-output/ +local-run-output/ +*.kube-report.json # Secrets and environment overrides .env diff --git a/Makefile b/Makefile index 2610ce5..ea3d2d3 100644 --- a/Makefile +++ b/Makefile @@ -10,7 +10,7 @@ test: go test ./... fmt: - gofmt -w ./cmd ./internal + gofmt -w $$(find . -name '*.go' -not -path './bin/*') vet: go vet ./... diff --git a/README.md b/README.md index 152af63..785c67e 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,19 @@ `k3s-nvidia-edge` is a reusable Go package and Helm profile for installing, configuring, validating, and cleaning up a local Ubuntu 22+ k3s cluster with NVIDIA GPU support and CUDA Toolkit 12.8+. -The reusable base workflows live in `pkg/edgebase` and are imported by sibling projects such as `llm-observability-stack`. New operator workflows should use the unified organization CLI, [`edge-cli`](https://github.com/Edge-Computing-LLM/edge-cli), with commands such as `edge install infra`, `edge validate infra`, and `edge status`. The legacy `k3s-nvidia-edge` binary remains available during migration. +The reusable base workflows live in `pkg/edgebase`. New operator workflows should use the unified organization CLI, [`edge-cli`](https://github.com/Edge-Computing-LLM/edge-cli), with commands such as `edge install infra`, `edge validate infra`, and `edge install all`. The legacy `k3s-nvidia-edge` binary remains available during migration. + +This is Layer 1 of the `Edge-Computing-LLM` platform. It owns the local Linux + k3s + NVIDIA substrate. `llm-observability-stack` is Layer 2 and must not install GPU Operator, NVIDIA device plugin, or DCGM exporter in the main local NVIDIA path. + +[`qwen-gguf-observability`](https://github.com/Edge-Computing-LLM/qwen-gguf-observability) +may read the resulting node, RuntimeClass, and GPU-capacity status as a runtime +evidence companion. It does not install or manage this infrastructure layer. + +This layer is conditional. `edge install all --accelerator auto` invokes the +NVIDIA setup only when `nvidia-smi` confirms working host hardware. On CPU-only +hosts, `edge-cli` installs or validates basic k3s without deploying this chart and +then selects the CPU observability profile. Use `--accelerator nvidia` when the +GPU layer is mandatory and fallback would hide an infrastructure problem. ## Documentation @@ -10,6 +22,8 @@ The reusable base workflows live in `pkg/edgebase` and are imported by sibling p - [Installation guide](docs/installation.md) - [Commands](docs/commands.md) - [Architecture](docs/architecture.md) +- [Live validation - 2026-07-08](docs/live-validation-2026-07-08.md) +- [Live validation - 2026-07-17](docs/live-validation-2026-07-17.md) - [Reusable edgebase Go package](docs/edgebase-package.md) - [Production readiness](docs/production-readiness.md) - [Troubleshooting](docs/troubleshooting.md) @@ -18,6 +32,7 @@ The reusable base workflows live in `pkg/edgebase` and are imported by sibling p - [Bundled Helm chart](charts/k3s-nvidia-edge/README.md) - [Contributing](CONTRIBUTING.md) - [Security](SECURITY.md) +- [Qwen GGUF runtime evidence companion](https://github.com/Edge-Computing-LLM/qwen-gguf-observability) The default profile matches the working local Xubuntu 24 setup: @@ -120,6 +135,12 @@ edge install infra --yes edge validate infra ``` +This command sequence supports an otherwise empty local k3s cluster that only +has default k3s system components such as CoreDNS and local-path-provisioner. +After validation passes, install the LLM observability layer with +`edge install observability --yes` or use `edge install all --yes` for the full +ordered flow. + Legacy direct command: ```bash @@ -232,6 +253,17 @@ Main tool mapping: | NVIDIA DRA Driver | `https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu` | | cloudflared | `https://github.com/cloudflare/cloudflared` | +## Programming language boundary + +Go remains the implementation language for reusable infrastructure workflows, +typed options, dry-run enforcement, and command error handling. Helm/YAML is +the declarative boundary for the GPU Operator profile. New Bash should be +limited to short operator examples; durable infrastructure behavior belongs in +`pkg/edgebase` or `edge-cli`, not a second shell implementation. + +See the organization +[programming language and script boundaries](https://github.com/Edge-Computing-LLM/edge-cli/blob/main/docs/LANGUAGE-BOUNDARIES.md). + ## Notes This tool assumes the NVIDIA display/compute driver is already installed on the host unless `--operator-driver-enabled=true` is explicitly passed. diff --git a/docs/architecture.md b/docs/architecture.md index fa73794..7621303 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -4,6 +4,16 @@ The primary operator CLI is now [`edge-cli`](https://github.com/Edge-Computing-LLM/edge-cli). This repository keeps the base-layer implementation and assets that `edge-cli` coordinates. +This repository is Layer 1 in the organization platform. It owns the NVIDIA GPU +substrate for local k3s. Layer 2, `llm-observability-stack`, may observe DCGM +metrics and schedule Ollama with `RuntimeClass/nvidia`, but it must not install +GPU Operator, NVIDIA device plugin, Node Feature Discovery, or DCGM exporter in +the main local NVIDIA path. + +`qwen-gguf-observability` is not Layer 3 and owns no infrastructure. It is a +read-only evidence consumer that verifies selected outputs of Layer 1 together +with the Qwen runtime deployed by Layer 2. + ## Components ```text @@ -90,3 +100,11 @@ CUDA validation pod can run nvidia-smi ``` The validation pod is short-lived and requests exactly one GPU. + +When Ollama already holds the single GPU, the evidence companion reads existing +capacity, RuntimeClass, pod, Ollama, and `nvidia-smi` status. It does not launch +a competing CUDA validation pod. + +A mostly empty k3s cluster is expected before this layer is installed. CoreDNS +and local-path-provisioner remain owned by k3s; this repository adds the NVIDIA +runtime and GPU readiness layer on top. diff --git a/docs/edge-cli-migration.md b/docs/edge-cli-migration.md index 05a22ba..c5d7016 100644 --- a/docs/edge-cli-migration.md +++ b/docs/edge-cli-migration.md @@ -25,6 +25,19 @@ Those responsibilities belong to: - `edge-cli`: unified Go CLI control plane. - `llm-observability-stack`: LLMOps and observability Helm layer. +The required deployment order is infra first, observability second: + +```bash +edge install infra --yes +edge validate infra +edge install observability --yes +edge validate observability +``` + +`edge install all --yes` runs the same sequence. Future repositories should be +added as later layers behind `edge-cli`, with explicit dependencies on the +validated previous layer. + ## Preferred Commands Use `edge-cli` for new operator workflows: diff --git a/docs/live-validation-2026-07-08.md b/docs/live-validation-2026-07-08.md new file mode 100644 index 0000000..847ac49 --- /dev/null +++ b/docs/live-validation-2026-07-08.md @@ -0,0 +1,22 @@ +# Live Validation - 2026-07-08 + +Validated as Layer 1 on local Xubuntu 24, single-node k3s, NVIDIA GeForce 940M. + +Tested commands: + +```bash +edge install infra --skip-base-package-install --skip-toolkit-install --skip-k3s-install --yes +edge validate infra +bin/k3s-nvidia-edge status +``` + +Results: + +- Installed/upgraded Helm release `k3s-nvidia-edge` in namespace `gpu-operator`. +- GPU Operator, NVIDIA device plugin, DCGM exporter, Node Feature Discovery, and validator pods became healthy. +- `RuntimeClass/nvidia` existed. +- Node advertised `nvidia.com/gpu: 1` capacity and allocatable. +- CUDA validation pod ran `nvidia-smi` successfully. +- Layer 2 was installed and uninstalled without removing this base layer. + +This confirms the repo remains independently useful as the NVIDIA/k3s substrate while also working through `edge-cli`. diff --git a/docs/live-validation-2026-07-17.md b/docs/live-validation-2026-07-17.md new file mode 100644 index 0000000..c4e6e6e --- /dev/null +++ b/docs/live-validation-2026-07-17.md @@ -0,0 +1,23 @@ +# Live NVIDIA validation — 2026-07-17 + +The Layer 1 chart was installed on Ubuntu 24.04.3, k3s v1.36.2+k3s1, and a +GeForce 940M using the host driver 580.95.05. + +Validated components: + +- NVIDIA Container Toolkit 1.19.1 +- GPU Operator v26.3.3 +- Node Feature Discovery v0.18.3 +- NVIDIA device plugin v0.19.3 +- NVIDIA DCGM Exporter +- NVIDIA operator and CUDA validators +- `RuntimeClass/nvidia` +- one allocatable `nvidia.com/gpu` + +The CUDA validation pod ran `nvidia-smi` successfully inside +`nvidia/cuda:12.8.1-base-ubuntu24.04`. This confirms the complete path from host +driver through k3s containerd, RuntimeClass, device plugin, scheduler, and container. + +The toolkit reconciliation restarted k3s once. Early device-plugin events briefly +reported that the NVIDIA runtime was not configured; they resolved after the +toolkit restart and all final daemonsets were healthy.