From 0820f5ecec8808a8de0db3884ba43fc66542472a Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 17:18:59 +1100 Subject: [PATCH 01/21] docs(design): bring your own cluster and per-cluster state Add a design for proving a cluster was installed by us before trusting it, for using an operator-supplied cluster through the Kubernetes API only, and for keeping state per cluster under state/hosts/local and state/clusters/. Covers the identity gate, the state layout and migration, API-only external mode, local development with Kind, and the effect on CI lanes. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 235 +++++++++++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 docs/dev/bring-your-own-cluster.md diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md new file mode 100644 index 00000000..152c2358 --- /dev/null +++ b/docs/dev/bring-your-own-cluster.md @@ -0,0 +1,235 @@ +# Bring Your Own Cluster and Per-Cluster State + +Status: **Draft for review.** Nothing in this document is implemented yet. + +## 1. Problem + +`solo-provisioner` assumes one machine runs exactly one cluster, and that the cluster is the one it installed. The code does not check this. + +- `kube.ProbeCluster()` (`internal/kube/client.go`) only asks if a kubeconfig is reachable. It reads `$KUBECONFIG` or `~/.kube/config`, stats the file, and calls `/version` with a 3 second timeout. +- `clusterChecker.RefreshState` then sets `ClusterState.Created = true` for any reachable cluster. +- `bll/blocknode` and `bll/consensus` install handlers branch on `Created`. A block node install can run against whatever cluster the operator's shell points at, and still run host-level steps (firewall, tc shaping, bind mounts, weaver user). +- Kubeconfig resolution is duplicated in `loadKubeConfig`, `resolveKubeconfigPath`, `step_daemon_provision.newTypedClient`, Helm's own settings, the daemon (`kubeconfigPath` in six places), and two hard-coded `/etc/kubernetes/admin.conf` constants. +- All state lives in one file, `/opt/solo/weaver/state/state.yaml`, so a second cluster would overwrite the first one's state. + +We also want to run most UAT against a cluster we did not install (for example Kind), so CI does not need KVM or privileged runners for every test. + +## 2. Goals and non-goals + +Goals: + +1. Never trust a cluster as ours unless we can prove we installed it on this machine. Check this as early as possible, and stop if the proof fails. +2. Let an operator explicitly point the CLI at any cluster (`--kubeconfig`) and use it for API-level work: install, upgrade and uninstall of block node, consensus node and the other cluster components. +3. Keep state separate per cluster, so switching between clusters, or forgetting a flag, can never mix or reset state. +4. One place decides which kubeconfig is used. +5. Leave room to manage several VMs from macOS later. + +Non-goals for now: + +- SSH delegation to remote hosts (`--host`). See section 9. +- Storing state inside the cluster. +- A full darwin build of the CLI. + +## 3. Concepts + +| Term | Meaning | +|---|---| +| Host | The machine the binary runs on. Today it is always `local`. | +| Cluster ID | The UID of the `kube-system` namespace. Stable for the life of a cluster, unique, and independent of IP or context name. | +| Origin `provisioned` | `solo-provisioner` installed this cluster (kubeadm, CRI-O, kubelet) on a host. | +| Origin `external` | Someone else's cluster. We only talk to its API. | + +The mode is **derived, not declared**. The `--kubeconfig` flag only chooses which cluster to look at. The identity check (section 5) decides whether it is `provisioned` on this host or `external`. This is what stops flag flip-flopping from corrupting state. + +## 4. Cluster resolution + +Add one resolver, `kube.ResolveCluster(opts)`. It returns the REST config, the kubeconfig path, the context, the cluster ID and the origin. Every caller listed in section 1 uses it. + +New root persistent flags: + +| Flag | Meaning | +|---|---| +| `--kubeconfig ` | Kubeconfig to use. Without it, only the cluster this host provisioned is considered. | +| `--context ` | Context inside the kubeconfig. | +| `--cluster-name ` | Optional human label, stored in `clusters/index.yaml`. Never used as a key. | + +Without `--kubeconfig`, ambient `$KUBECONFIG` and `~/.kube/config` are not trusted on their own. Weaver already copies `admin.conf` to `~/.kube/config` for root and the sudo user (`internal/kube/admin.go`), so that file can be ours, but the identity check, not the path, proves it. + +## 5. Identity gate + +A check in the root pre-run, **before** any state load, RSL or reality check. If it fails, the command stops with a reason code and a hint. This is how we know reality and state can be trusted. + +### 5.1 What a `provisioned` cluster must prove + +All of these must pass: + +| Evidence | Why | +|---|---| +| Software state records kubelet and CRI-O as installed, and the sandbox binaries on disk match the recorded hash | Binary provenance | +| kubelet and CRI-O are running (systemd unit active, CRI-O socket answers), and the running executable is the sandbox binary, not one from `PATH` | A matching file is not enough | +| `/etc/kubernetes/admin.conf` and the static pod manifests under `/etc/kubernetes/manifests` exist | kubeadm control plane evidence | +| The kubeconfig server host matches this machine's IP and the IP in weaver's config | The cluster is on this host | +| The live cluster ID matches the one recorded in state (when one is recorded) | Same cluster as last time | + +The IP check alone is not enough. A Kind or minikube cluster on the same machine points at `127.0.0.1` and would pass it. The binary and process checks rule those out. + +### 5.2 Outcomes + +| Situation | Result | +|---|---| +| All checks pass | `provisioned`, host `local`, state in `hosts/local` | +| Checks fail, `--kubeconfig` given | `external`, state in `clusters/` | +| Checks fail, no `--kubeconfig` | Stop. Hint: run `solo-provisioner kube cluster install`, or pass `--kubeconfig` for an external cluster | +| Cluster ID matches the one recorded in `hosts/local`, but a local check fails (for example kubelet is down) | Stop. Hint: fix the local cluster. **Never** fall back to `external` | +| Cluster ID already has `clusters/` state and now passes the local proof | Stop. Needs an explicit step; it is never adopted silently | + +The last two rows prevent split-brain between the two state locations. + +### 5.3 Exempt commands + +These commands expect no cluster yet, so they skip the gate: + +- `solo-provisioner install` +- `solo-provisioner uninstall` +- `solo-provisioner version` +- `solo-provisioner demo` +- `solo-provisioner --help` (and `--help` on any subcommand) +- `solo-provisioner kube cluster install` + +`solo-provisioner kube cluster install` still has to refuse to proceed when a foreign cluster is configured. + +Diagnostic commands (`doctor` and `diagnose`, epic #1110) are **not** exempt, because their job is to report on the cluster. They run the identity check in **report-only mode**: a failed check is printed as a finding instead of stopping the command. The detail of those commands belongs to #1110. + +### 5.4 Clusters installed by older versions + +Old installs may lack the hashes or metadata the gate checks, and a strict gate would lock operators out of their own clusters. A one-time backfill records the missing metadata and the cluster ID when the weaker but still conclusive checks pass: local sandbox kubelet and CRI-O running, `admin.conf` present, IP match. This must ship together with the gate. + +## 6. State layout + +State is grouped by host. A cluster with no host on this machine gets its own directory. + +``` +/opt/solo/weaver/state/ +├── hosts/ +│ └── local/state.yaml # this machine + the cluster it provisioned (content unchanged) +└── clusters/ + ├── /state.yaml # one per cluster with no host here; API-only state type + └── index.yaml # complete view of every known cluster +``` + +Rules: + +- `hosts/local/state.yaml` keeps today's content: machine, software, hardware, firewall, shaping, and the provisioned cluster's workloads. It also records the provisioned cluster's ID. +- `clusters//state.yaml` uses a **separate type with API-level fields only** (cluster info, Helm releases, block node release and plugin settings, consensus node specs, teleport cluster agent). It has no machine, firewall, shaping or hostPath fields, so external-mode code cannot touch host state by accident. +- Each file records its own `clusterId` and `origin`, and the loader checks them against the live cluster. A copied file is rejected. +- External runs never write `hosts/local`. Provisioned runs never write `clusters/`. +- The key is the cluster ID, not the IP, the context name or a user-supplied name. A rebuilt cluster gets a new ID and therefore fresh state. + +### 6.1 `clusters/index.yaml` + +The index lists **every** cluster, including the provisioned one, so it is the complete view. A cluster with `state: hosts/local` has no `clusters//` directory. + +```yaml +clusters: + - uid: 3f2a91bc-... + alias: local + origin: provisioned + host: local # state: hosts/local + server: https://10.20.0.5:6443 + lastUsed: 2026-10-15T09:12:00Z + - uid: 7c1e9a04-... + alias: perfnet-eu + origin: external # no host field + state: clusters/7c1e9a04-... + context: perfnet-eu-admin + server: https://k8s.perfnet-eu.example:6443 + lastUsed: 2026-10-15T09:40:00Z +``` + +The alias binds to a cluster ID on first use. If the alias later points at a different cluster ID, the command stops with an error. `--rebind` overrides it explicitly. An alias alone can never create state. + +## 7. External mode: API only + +> In external mode weaver only talks to the Kubernetes API. It touches nothing on the host. + +- Skipped: node setup, kubeadm setup, network setup (nft, tc), bind mounts, sysctl, swap, weaver user, daemon config. +- Storage: hostPath PVs refer to a node filesystem weaver cannot own, so they are not used. Storage comes from the cluster through a `storageClassName` per volume (flag or values file). Weaver creates PVCs only, never PVs. The hostPath templates (`storage-config.yaml`, `optional-storage.yaml`) are provisioned-only. +- Preflight checks API-level facts only: server version range, access via `SelfSubjectAccessReview`, and that the CNI and StorageClass exist. +- Provisioned-only commands fail in external mode with a reason code and hint: `solo-provisioner kube cluster install`, `solo-provisioner kube cluster uninstall`, `solo-provisioner teleport node ...`, and `solo-provisioner network firewall|shape|policy ...`. +- Uninstall and reset remove only what weaver created in that cluster, tracked by labels or state. Cluster-level components are never touched. +- This also applies on a provisioned host: pointing it at a remote cluster with `--kubeconfig` runs external mode and skips the host steps. + +### 7.1 Local development with Kind + +A developer can create a Kind cluster on their own machine (including a Mac) and use it with weaver. It does not need SSH delegation (section 9). + +- Weaver did not install it, so it is `origin: external`. It gets `clusters//state.yaml` and an index entry with the context (for example `kind-dev`) and a `127.0.0.1` server address. +- Use it with `--kubeconfig` and `--context`. Only API-level work runs. Storage comes from Kind's default StorageClass. +- Each `kind create cluster` gets a new cluster ID, so a recreated cluster starts with clean state. If the same alias is reused for the new cluster, `--rebind` is needed. +- The same flow is what the CI Kind lane uses (section 10). + +Prerequisites for running this on macOS: + +- The CLI must run on darwin. Build stubs for the Linux-only code (`mount`, `modprobe`, swap) exist, but running the CLI on macOS has not been verified. +- The state directory defaults to `~/.solo-weaver/state` on macOS, because `/opt/solo/weaver` is a Linux path. + +## 8. Migration of existing deployments + +Add a startup migration, `state-hosts-layout`, registered first in `RegisterMigrations()` (`cmd/cli/commands/root.go`). See `docs/dev/migration-framework.md`. + +- `Applies`: `state/state.yaml` exists and `state/hosts/local/state.yaml` does not. +- `Execute`: create `hosts/local/` with the same owner and mode, copy the file, verify the hash, rename atomically, and keep the original as `state.yaml.pre-hosts-layout`. +- If both files exist, prefer the new one, leave the legacy one untouched, and log a warning. Never overwrite. +- Idempotent and crash-safe. `Rollback` restores the backup on a best-effort basis. +- It runs before anything opens the state manager, and before the identity gate. Migration is pure filesystem work and needs no cluster. +- All code that builds the state path (`NewState`, `state_reader.go`, the `helm-release-schema-v2` and `mgmt-ports-v1` migrations, `testutil`) moves to one resolver. + +Risks to settle: + +1. The `StateFile` field is serialized inside `state.yaml`. After the move it must be rewritten, and the loader should force it to the path actually loaded, or a later flush could write to the old location. +2. Downgrade: an older binary looks at the old path and sees a fresh machine. Migrations are one-way today; document it. A symlink does not help, because the atomic write replaces it. +3. The daemon is a separate binary. Check whether it reads the state path. +4. Two concurrent invocations during the move need a lock, or an atomic rename plus a re-check. +5. The legacy-cluster backfill in section 5.4 ships with this change. + +## 9. Later: several VMs from macOS + +Not in scope now, but the layout leaves room. + +- A developer on macOS runs a launcher that copies the matching Linux binary to a VM, verifies its digest, and runs the command there over SSH. The VM keeps its own `hosts/local` state. +- The Mac keeps `hosts//host.yaml` as a registry only: SSH target, verified machine ID, last known binary version. It never holds a copy of a VM's state. +- The Mac can still drive that VM's cluster natively through the API under `clusters//`, and delegate host-level commands with `--host `. +- In the index, such a cluster is `origin: provisioned` with `host: `. +- Open design points: version skew, binary provenance (signatures and digests), flag-to-file path rewriting, secrets in uploaded files, PTY and non-interactive mode, SSH host-key trust, and keeping a command running after a dropped connection. + +## 10. Effect on CI + +External mode is what makes a Kind lane possible on the ARC runners (`solo-linux-medium`) without privileged pods. Tests then split into: + +| Lane | Runner | Content | +|---|---|---| +| Unprivileged | `solo-linux-medium` | Unit tests, installer download and verify tests | +| Kind (external mode) | `solo-linux-medium` with `dind` | API-level UAT: chart and plugin-preset upgrade, kube client, alloy, teleport cluster, solo-operator | +| Host-level (provisioned) | Hosted runner or privileged systemd container | Network, mount, sysctl, systemd tests and the full cluster install UAT | +| Real VM | `kvm-runner`, nightly or on path filters | Boot-order and reboot tests, one full Debian lifecycle | + +## 11. Delivery plan + +1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the legacy backfill rules). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. +2. State layout: `state-hosts-layout` migration, the single path resolver, the external state type and `clusters/index.yaml`. +3. External-mode workflows, block node first, with the StorageClass storage path. +4. CI: Kind lane using `--kubeconfig`, and the test split by required privilege. + +## 12. Decisions + +1. **Gate exemptions.** The list in section 5.3 is the rule. Diagnostic commands run the gate in report-only mode. +2. **Reinstall.** When `solo-provisioner kube cluster install` finds that `hosts/local` records a different cluster ID, the old cluster section moves to `hosts/local/archive/-.yaml`. Host-scoped data (software, firewall) stays. Nothing is deleted, and the new cluster starts with an empty cluster section. +3. **External state is a local cache.** Almost all of it can be re-derived from the cluster, and the cluster wins when they disagree. Storing state in the cluster is a follow-up with its own design (RBAC, ownership). +4. **Garbage collection.** Nothing is deleted automatically, since a cluster that is only unreachable for a moment must not lose its state. A later `solo-provisioner state prune` lists clusters that are unreachable or whose ID no longer exists, and asks before deleting. +5. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State lives in `~/.solo-weaver/state`. + +## 13. Still open + +- The detail of report-only mode for `doctor` and `diagnose` (epic #1110). +- The design of in-cluster state storage, if we want it. +- The `solo-provisioner state prune` command and its safety rules. From c719c476c7c0b2619e8f77f1302da5d4578a1f91 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 17:28:05 +1100 Subject: [PATCH 02/21] docs(design): build on per-component state files; archive state at uninstall Describe the state layout in terms of the per-component files from #1231 and #1245: hosts/local holds all component files, and clusters/ holds only the cluster-scoped ones. Rewrite the migration as a move of component files that runs after the per-component split. List the component files that mix host and cluster fields, archive cluster state at uninstall, and make a stale-state install stop and ask the operator to clean up instead of moving state automatically. Order the delivery plan after #1235. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 80 ++++++++++++++++++++++-------- 1 file changed, 60 insertions(+), 20 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 152c2358..1a847f8a 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -2,6 +2,8 @@ Status: **Draft for review.** Nothing in this document is implemented yet. +Depends on the per-component state files work: #1231 (PR #1235) and #1245. This design is written in terms of those component files and should be built after they merge. See section 6. + ## 1. Problem `solo-provisioner` assumes one machine runs exactly one cluster, and that the cluster is the one it installed. The code does not check this. @@ -10,7 +12,7 @@ Status: **Draft for review.** Nothing in this document is implemented yet. - `clusterChecker.RefreshState` then sets `ClusterState.Created = true` for any reachable cluster. - `bll/blocknode` and `bll/consensus` install handlers branch on `Created`. A block node install can run against whatever cluster the operator's shell points at, and still run host-level steps (firewall, tc shaping, bind mounts, weaver user). - Kubeconfig resolution is duplicated in `loadKubeConfig`, `resolveKubeconfigPath`, `step_daemon_provision.newTypedClient`, Helm's own settings, the daemon (`kubeconfigPath` in six places), and two hard-coded `/etc/kubernetes/admin.conf` constants. -- All state lives in one file, `/opt/solo/weaver/state/state.yaml`, so a second cluster would overwrite the first one's state. +- State has no notion of which cluster it describes. Today it is one file, `/opt/solo/weaver/state/state.yaml` (being split into per-component files by #1231), so a second cluster would overwrite the first one's state. We also want to run most UAT against a cluster we did not install (for example Kind), so CI does not need KVM or privileged runners for every test. @@ -106,26 +108,58 @@ Old installs may lack the hashes or metadata the gate checks, and a strict gate ## 6. State layout -State is grouped by host. A cluster with no host on this machine gets its own directory. +### 6.0 What it builds on + +State is moving from one `state.yaml` to per-component files, each with its own hash and lock: + +- #1231 / PR #1235: `machine.yaml`, `cluster.yaml`, `blocknode.yaml`, `consensus.yaml`, `teleport.yaml`, with a startup migration. +- #1245: `machine.yaml` split further into `profile`, `firewall` and `system`. + +We build on that and do not change it. The per-component files are what make the host and cluster split structural. + +### 6.1 Layout + +State is grouped by host. A cluster with no host on this machine gets its own directory holding only the cluster-scoped component files. ``` /opt/solo/weaver/state/ ├── hosts/ -│ └── local/state.yaml # this machine + the cluster it provisioned (content unchanged) +│ └── local/ # this machine + the cluster it provisioned +│ ├── machine.yaml # host components (later profile, firewall, system) +│ ├── cluster.yaml +│ ├── blocknode.yaml +│ ├── consensus.yaml +│ └── teleport.yaml └── clusters/ - ├── /state.yaml # one per cluster with no host here; API-only state type - └── index.yaml # complete view of every known cluster + ├── / # one per cluster with no host here + │ ├── cluster.yaml # cluster-scoped components only + │ ├── blocknode.yaml + │ ├── consensus.yaml + │ └── teleport.yaml + └── index.yaml # complete view of every known cluster ``` Rules: -- `hosts/local/state.yaml` keeps today's content: machine, software, hardware, firewall, shaping, and the provisioned cluster's workloads. It also records the provisioned cluster's ID. -- `clusters//state.yaml` uses a **separate type with API-level fields only** (cluster info, Helm releases, block node release and plugin settings, consensus node specs, teleport cluster agent). It has no machine, firewall, shaping or hostPath fields, so external-mode code cannot touch host state by accident. +- `hosts/local/` keeps today's component files and content. `cluster.yaml` also records the provisioned cluster's ID. +- `clusters//` holds **only cluster-scoped components**. Host components (`machine`, and later `profile`, `firewall`, `system`) never exist there, so external-mode code cannot reach host state. Hashes and locks work per file, as in #1235. - Each file records its own `clusterId` and `origin`, and the loader checks them against the live cluster. A copied file is rejected. - External runs never write `hosts/local`. Provisioned runs never write `clusters/`. - The key is the cluster ID, not the IP, the context name or a user-supplied name. A rebuilt cluster gets a new ID and therefore fresh state. -### 6.1 `clusters/index.yaml` +### 6.2 Components that mix host and cluster fields + +Some component files hold both kinds of fields. They have to be split, or the host fields made optional and never written in external mode, before a cluster directory can hold them: + +| File | Host-bound fields | Cluster-level fields | +|---|---|---| +| `blocknode.yaml` | `Shaping`, `TrafficShapingDisabled`, `Storage` (hostPath) | Helm release info, plugin preset and list, retention | +| `consensus.yaml` | `HostPathOwners` | Node identity, images, config hashes, secrets, managed spec | +| `teleport.yaml` | Node agent | Cluster agent | + +The daemon writes `consensus.yaml` and is released separately, so it has to follow any path or layout change. + +### 6.3 `clusters/index.yaml` The index lists **every** cluster, including the provisioned one, so it is the complete view. A cluster with `state: hosts/local` has no `clusters//` directory. @@ -163,7 +197,7 @@ The alias binds to a cluster ID on first use. If the alias later points at a dif A developer can create a Kind cluster on their own machine (including a Mac) and use it with weaver. It does not need SSH delegation (section 9). -- Weaver did not install it, so it is `origin: external`. It gets `clusters//state.yaml` and an index entry with the context (for example `kind-dev`) and a `127.0.0.1` server address. +- Weaver did not install it, so it is `origin: external`. It gets a `clusters//` directory and an index entry with the context (for example `kind-dev`) and a `127.0.0.1` server address. - Use it with `--kubeconfig` and `--context`. Only API-level work runs. Storage comes from Kind's default StorageClass. - Each `kind create cluster` gets a new cluster ID, so a recreated cluster starts with clean state. If the same alias is reused for the new cluster, `--rebind` is needed. - The same flow is what the CI Kind lane uses (section 10). @@ -175,21 +209,21 @@ Prerequisites for running this on macOS: ## 8. Migration of existing deployments -Add a startup migration, `state-hosts-layout`, registered first in `RegisterMigrations()` (`cmd/cli/commands/root.go`). See `docs/dev/migration-framework.md`. +Add a startup migration, `state-hosts-layout`, registered in `RegisterMigrations()` (`cmd/cli/commands/root.go`) **after** the migration that splits `state.yaml` into component files (#1231). See `docs/dev/migration-framework.md`. The chain is: `unified-state`, then per-component split, then `state-hosts-layout`. Each step is idempotent. -- `Applies`: `state/state.yaml` exists and `state/hosts/local/state.yaml` does not. -- `Execute`: create `hosts/local/` with the same owner and mode, copy the file, verify the hash, rename atomically, and keep the original as `state.yaml.pre-hosts-layout`. -- If both files exist, prefer the new one, leave the legacy one untouched, and log a warning. Never overwrite. +- `Applies`: component files exist directly under `state/` and `state/hosts/local/` does not. +- `Execute`: create `hosts/local/` with the same owner and mode, move the component files (and their lock and hash companions) into it, and keep a backup of the originals as `state/.pre-hosts-layout/`. It moves files and does not rewrite their content. +- If both locations have files, prefer `hosts/local/`, leave the legacy files untouched, and log a warning. Never overwrite. - Idempotent and crash-safe. `Rollback` restores the backup on a best-effort basis. - It runs before anything opens the state manager, and before the identity gate. Migration is pure filesystem work and needs no cluster. -- All code that builds the state path (`NewState`, `state_reader.go`, the `helm-release-schema-v2` and `mgmt-ports-v1` migrations, `testutil`) moves to one resolver. +- All code that builds a state path (the state manager, `state_reader.go`, the existing state migrations, `testutil`, the daemon) moves to one resolver. Risks to settle: -1. The `StateFile` field is serialized inside `state.yaml`. After the move it must be rewritten, and the loader should force it to the path actually loaded, or a later flush could write to the old location. -2. Downgrade: an older binary looks at the old path and sees a fresh machine. Migrations are one-way today; document it. A symlink does not help, because the atomic write replaces it. -3. The daemon is a separate binary. Check whether it reads the state path. -4. Two concurrent invocations during the move need a lock, or an atomic rename plus a re-check. +1. Any path stored inside a state file (the old `StateFile` field is the known case) must be rewritten, and the loader should use the path it actually loaded from, or a later flush could write to the old location. +2. Downgrade: an older binary looks at the old path and sees a fresh machine. Migrations are one-way today; document it. A symlink does not help, because atomic writes replace it. +3. The daemon is a separate binary with its own release schedule. It must read the new location, and a daemon older than the CLI must not see an empty state. +4. Two concurrent invocations during the move need a lock, or an atomic rename plus a re-check. The per-component `flock` files from #1235 move together with the state files, so the move must hold all of them. 5. The legacy-cluster backfill in section 5.4 ships with this change. ## 9. Later: several VMs from macOS @@ -216,20 +250,26 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux ## 11. Delivery plan 1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the legacy backfill rules). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. -2. State layout: `state-hosts-layout` migration, the single path resolver, the external state type and `clusters/index.yaml`. +2. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, the cluster-scoped component files for `clusters//`, the splits in section 6.2, and `clusters/index.yaml`. 3. External-mode workflows, block node first, with the StorageClass storage path. 4. CI: Kind lane using `--kubeconfig`, and the test split by required privilege. ## 12. Decisions 1. **Gate exemptions.** The list in section 5.3 is the rule. Diagnostic commands run the gate in report-only mode. -2. **Reinstall.** When `solo-provisioner kube cluster install` finds that `hosts/local` records a different cluster ID, the old cluster section moves to `hosts/local/archive/-.yaml`. Host-scoped data (software, firewall) stays. Nothing is deleted, and the new cluster starts with an empty cluster section. +2. **Uninstall and reinstall.** The normal path is at uninstall: `solo-provisioner kube cluster uninstall` moves the cluster-scoped component files (`cluster`, `blocknode`, `consensus` and the cluster half of `teleport`) to `hosts/local/archive/-/`. Host-scoped components (software, firewall, shaping) stay, because the machine is still provisioned. A later `solo-provisioner kube cluster install` then starts clean. Nothing is deleted. + + If `solo-provisioner kube cluster install` finds that `hosts/local` records a cluster ID that differs from the live one (or a cluster that no longer exists), it **stops and does not touch the state**. Weaver cannot tell why: a `kubeadm reset` by hand, a failed or older uninstall, a rebuilt VM with surviving state, or the wrong kubeconfig all look the same, so moving the state automatically would be a guess. The error names the recorded and live cluster IDs and tells the operator to review and clean up the state by hand. The hint lists the files to move or remove and the paths they live in. + 3. **External state is a local cache.** Almost all of it can be re-derived from the cluster, and the cluster wins when they disagree. Storing state in the cluster is a follow-up with its own design (RBAC, ownership). 4. **Garbage collection.** Nothing is deleted automatically, since a cluster that is only unreachable for a moment must not lose its state. A later `solo-provisioner state prune` lists clusters that are unreachable or whose ID no longer exists, and asks before deleting. 5. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State lives in `~/.solo-weaver/state`. ## 13. Still open +- What `solo-provisioner kube cluster uninstall` does with state today. This decides whether archive-at-uninstall is new behaviour or a fix. +- Whether to offer an explicit command for the manual cleanup after a stale-state stop, so operators do not edit files by hand. +- How to split or gate the mixed host and cluster fields in section 6.2, and whether that belongs in #1245's follow-ups or in this work. - The detail of report-only mode for `doctor` and `diagnose` (epic #1110). - The design of in-cluster state storage, if we want it. - The `solo-provisioner state prune` command and its safety rules. From 7d36748a1fdc287e9aed7d7075aa71ae3a8df7ee Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 21:07:10 +1100 Subject: [PATCH 03/21] docs(design): add a cluster-side lease lock for mutating commands Add a Lease per component per cluster, in the component's namespace, taken by mutating commands in both provisioned and external mode so two operators cannot change the same component at once. Build it on client-go leaderelection, wrapped in internal/kube/lease, with fail-fast holder details and context cancellation when the lease is lost. Add it to the goals, decisions, delivery plan and open items. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 35 +++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 6 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 1a847f8a..100f2f2a 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -25,6 +25,7 @@ Goals: 3. Keep state separate per cluster, so switching between clusters, or forgetting a flag, can never mix or reset state. 4. One place decides which kubeconfig is used. 5. Leave room to manage several VMs from macOS later. +6. Stop two operators (or an operator and the daemon) from changing the same component of the same cluster at the same time. Non-goals for now: @@ -106,6 +107,25 @@ Diagnostic commands (`doctor` and `diagnose`, epic #1110) are **not** exempt, be Old installs may lack the hashes or metadata the gate checks, and a strict gate would lock operators out of their own clusters. A one-time backfill records the missing metadata and the cluster ID when the weaker but still conclusive checks pass: local sandbox kubelet and CRI-O running, `admin.conf` present, IP match. This must ship together with the gate. +### 5.5 Cluster-side lock + +The per-component `flock` files (#1235) only protect one machine. Two operators on different machines (for example a laptop in external mode and a CLI on the VM) can still change the same cluster at once. A Lease object in the cluster closes that gap. The idea follows the lease in `hiero-ledger/solo` (`src/core/lock/`), with a different renewal and loss model. + +Decisions: + +- **Object.** A `coordination.k8s.io/v1` Lease, one per component per cluster, named `weaver-` (for example `weaver-blocknode`), in the **component's own namespace**. Per-component leases mirror the per-component state locks, so disjoint components (for example consensus and teleport) do not block each other. The lease disappears with the namespace on uninstall. +- **Both modes.** Every mutating command takes the lease as soon as the cluster is reachable, whether the cluster is `provisioned` or `external`. Read-only commands do not. Order: cluster lease first, then the local component `flock`. +- **Library.** `k8s.io/client-go/tools/leaderelection`, wrapped in a small internal package, `internal/kube/lease`. It handles the compare-and-swap on `resourceVersion`, renewal, and loss detection. Adding it needs `go mod vendor`, as for any dependency. +- **Wrapper API.** `TryAcquire(ctx, component)` returns a handle with a context that is cancelled when the lease is lost, and `Release()`. +- **Fail fast.** Before starting the elector, read the Lease. If it is held and not expired, stop and print the holder: user, host (machine ID), command line, start time and weaver version, taken from the holder identity (JSON), and how long until it expires. If the elector loses a race, a short acquire timeout turns that into the same error. +- **Fencing.** If renewal fails past the renew deadline, the wrapper cancels the workflow's context and the command aborts. A laptop that sleeps past expiry can therefore not resume and keep changing a cluster another operator now holds. +- **Expiry only.** The lease is taken over only after it expires, because a live holder keeps renewing. There is no other automatic takeover. An explicit override flag (name to be decided) lets an operator remove a lease they know is dead. +- **Timings.** Starting values: lease duration 30 seconds, renew deadline 20 seconds, retry period 5 seconds. To be tuned against slow API servers. +- **First install.** The cluster or the component namespace may not exist yet. Cluster install is covered by the local `flock` only. For a component namespace that does not exist, create the namespace first (an `AlreadyExists` error is fine) and then take the lease. The race window is only the namespace creation. +- **RBAC.** External operators need `get`, `create`, `update` and `delete` on `leases` in the component namespace. The preflight checks this with `SelfSubjectAccessReview` and fails with a clear hint. +- **Daemon.** The daemon takes the same lease when it writes consensus state, so it excludes CLI runs. +- **Diagnostics.** `doctor` and `diagnose` show current lease holders as a finding. + ## 6. State layout ### 6.0 What it builds on @@ -250,9 +270,10 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux ## 11. Delivery plan 1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the legacy backfill rules). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. -2. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, the cluster-scoped component files for `clusters//`, the splits in section 6.2, and `clusters/index.yaml`. -3. External-mode workflows, block node first, with the StorageClass storage path. -4. CI: Kind lane using `--kubeconfig`, and the test split by required privilege. +2. Cluster-side lock (`internal/kube/lease`) and its use in the mutating commands (section 5.5). It needs the resolved cluster from step 1 and nothing from the state work. +3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, the cluster-scoped component files for `clusters//`, the splits in section 6.2, and `clusters/index.yaml`. +4. External-mode workflows, block node first, with the StorageClass storage path. +5. CI: Kind lane using `--kubeconfig`, and the test split by required privilege. ## 12. Decisions @@ -261,15 +282,17 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux If `solo-provisioner kube cluster install` finds that `hosts/local` records a cluster ID that differs from the live one (or a cluster that no longer exists), it **stops and does not touch the state**. Weaver cannot tell why: a `kubeadm reset` by hand, a failed or older uninstall, a rebuilt VM with surviving state, or the wrong kubeconfig all look the same, so moving the state automatically would be a guess. The error names the recorded and live cluster IDs and tells the operator to review and clean up the state by hand. The hint lists the files to move or remove and the paths they live in. -3. **External state is a local cache.** Almost all of it can be re-derived from the cluster, and the cluster wins when they disagree. Storing state in the cluster is a follow-up with its own design (RBAC, ownership). -4. **Garbage collection.** Nothing is deleted automatically, since a cluster that is only unreachable for a moment must not lose its state. A later `solo-provisioner state prune` lists clusters that are unreachable or whose ID no longer exists, and asks before deleting. -5. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State lives in `~/.solo-weaver/state`. +3. **Cluster-side lock.** A Lease per component per cluster, in the component's namespace, in both modes, built on `leaderelection` (section 5.5). +4. **External state is a local cache.** Almost all of it can be re-derived from the cluster, and the cluster wins when they disagree. Storing state in the cluster is a follow-up with its own design (RBAC, ownership). +5. **Garbage collection.** Nothing is deleted automatically, since a cluster that is only unreachable for a moment must not lose its state. A later `solo-provisioner state prune` lists clusters that are unreachable or whose ID no longer exists, and asks before deleting. +6. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State lives in `~/.solo-weaver/state`. ## 13. Still open - What `solo-provisioner kube cluster uninstall` does with state today. This decides whether archive-at-uninstall is new behaviour or a fix. - Whether to offer an explicit command for the manual cleanup after a stale-state stop, so operators do not edit files by hand. - How to split or gate the mixed host and cluster fields in section 6.2, and whether that belongs in #1245's follow-ups or in this work. +- The lease timings, and the name of the explicit override flag for removing a dead lease. - The detail of report-only mode for `doctor` and `diagnose` (epic #1110). - The design of in-cluster state storage, if we want it. - The `solo-provisioner state prune` command and its safety rules. From 24932846e035a8acc2c12f97799212f0cc65817e Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 21:51:12 +1100 Subject: [PATCH 04/21] docs(design): add host.yaml and provisioner.yaml; narrow the host-bound field guard Add a small hosts/local/host.yaml identity file (name, machine ID, the cluster this host provisioned) that the identity gate reads first, and a global provisioner.yaml for the provisioner and state model versions, so a machine with only external clusters (for example a Mac) still has a home for the data startup migrations depend on. Move those fields out of the host-level system file proposed in #1245. Drop the legacy backfill in favour of a catalog checksum check, explain the external-then-local case and how a cluster is identified to a human, and show the layout with the profile, firewall and system files. Rewrite the host-bound field section: same component files in cluster directories, a writer guard for host-only fields, and the profile kept with its target. Add non-goals for the cluster-side lock, and tidy the migration risks, delivery plan, decisions and open items to match. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 92 +++++++++++++++++++++++------- 1 file changed, 70 insertions(+), 22 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 100f2f2a..49cbd430 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -32,6 +32,10 @@ Non-goals for now: - SSH delegation to remote hosts (`--host`). See section 9. - Storing state inside the cluster. - A full darwin build of the CLI. +- Using the cluster-side lock (section 5.5) as a state store. It only says who is changing a component right now. +- Locking the first `solo-provisioner kube cluster install`. No cluster exists yet, so only the local `flock` applies. +- Taking over a live lease automatically. Only expiry releases it, and an explicit operator override removes a dead one. +- Protecting against changes made outside weaver (`kubectl`, Helm, other tools). The lease only coordinates weaver processes, the CLI and the daemon. ## 3. Concepts @@ -54,7 +58,7 @@ New root persistent flags: |---|---| | `--kubeconfig ` | Kubeconfig to use. Without it, only the cluster this host provisioned is considered. | | `--context ` | Context inside the kubeconfig. | -| `--cluster-name ` | Optional human label, stored in `clusters/index.yaml`. Never used as a key. | +| `--cluster-name ` | Optional human label, stored in `clusters/index.yaml`. Defaults to the context name. Never used as a key. | Without `--kubeconfig`, ambient `$KUBECONFIG` and `~/.kube/config` are not trusted on their own. Weaver already copies `admin.conf` to `~/.kube/config` for root and the sudo user (`internal/kube/admin.go`), so that file can be ours, but the identity check, not the path, proves it. @@ -68,11 +72,11 @@ All of these must pass: | Evidence | Why | |---|---| -| Software state records kubelet and CRI-O as installed, and the sandbox binaries on disk match the recorded hash | Binary provenance | +| Software state records kubelet and CRI-O as installed, and the sandbox binaries on disk match the catalog checksum for the recorded version and platform | Binary provenance | | kubelet and CRI-O are running (systemd unit active, CRI-O socket answers), and the running executable is the sandbox binary, not one from `PATH` | A matching file is not enough | | `/etc/kubernetes/admin.conf` and the static pod manifests under `/etc/kubernetes/manifests` exist | kubeadm control plane evidence | | The kubeconfig server host matches this machine's IP and the IP in weaver's config | The cluster is on this host | -| The live cluster ID matches the one recorded in state (when one is recorded) | Same cluster as last time | +| The live cluster ID matches the one recorded in `hosts/local/host.yaml` (when one is recorded; see 5.4) | Same cluster as last time | The IP check alone is not enough. A Kind or minikube cluster on the same machine points at `127.0.0.1` and would pass it. The binary and process checks rule those out. @@ -83,11 +87,22 @@ The IP check alone is not enough. A Kind or minikube cluster on the same machine | All checks pass | `provisioned`, host `local`, state in `hosts/local` | | Checks fail, `--kubeconfig` given | `external`, state in `clusters/` | | Checks fail, no `--kubeconfig` | Stop. Hint: run `solo-provisioner kube cluster install`, or pass `--kubeconfig` for an external cluster | -| Cluster ID matches the one recorded in `hosts/local`, but a local check fails (for example kubelet is down) | Stop. Hint: fix the local cluster. **Never** fall back to `external` | +| Cluster ID matches the one recorded in `hosts/local/host.yaml`, but a local check fails (for example kubelet is down) | Stop. Hint: fix the local cluster. **Never** fall back to `external` | | Cluster ID already has `clusters/` state and now passes the local proof | Stop. Needs an explicit step; it is never adopted silently | The last two rows prevent split-brain between the two state locations. +#### The last row: a cluster that was external and now passes the local proof + +This can happen when a cluster on the machine was not recognised at first (for example kubelet was down, or it was built by hand), so a command with `--kubeconfig` treated it as `external` and created `clusters//`. Later the proof passes and the same cluster ID now qualifies as `provisioned`. Two places could then describe one cluster, and the records may disagree. Picking one silently could ignore the block node and consensus information in the other, and merging automatically could overwrite newer facts with older ones. So weaver stops and asks. + +The explicit step is one of: + +1. **Discard the external copy.** The operator archives or removes `clusters//` and reruns. External state is a cache (decision 4), so `hosts/local` is rebuilt from what the cluster reports. +2. **Adopt it.** A command copies the cluster-scoped component files (`cluster`, `blocknode`, `consensus` and the cluster half of `teleport`) from `clusters//` into `hosts/local/`, then archives the external directory. + +Until a command for this exists (section 13), the error lists the manual steps. + ### 5.3 Exempt commands These commands expect no cluster yet, so they skip the gate: @@ -105,7 +120,14 @@ Diagnostic commands (`doctor` and `diagnose`, epic #1110) are **not** exempt, be ### 5.4 Clusters installed by older versions -Old installs may lack the hashes or metadata the gate checks, and a strict gate would lock operators out of their own clusters. A one-time backfill records the missing metadata and the cluster ID when the weaker but still conclusive checks pass: local sandbox kubelet and CRI-O running, `admin.conf` present, IP match. This must ship together with the gate. +A strict gate must not lock operators out of clusters that older versions installed. Two things could be missing on such a machine, and neither needs a separate migration: + +- **Recorded hashes.** Software state records only the name, version, `Installed` and `Configured`; it has never held a binary hash. The gate does not need one. The embedded catalog has the expected checksum for each software, version and platform, which the installer already checks at download. The gate verifies that the sandbox binary on disk matches the catalog checksum for the version recorded in state, and that the running process is that binary. +- **The cluster ID.** `hosts/local/host.yaml` has none on an older install. The first time all the checks in 5.1 pass, the gate records the live cluster ID. This is trust on first use, and it is safe because the binary, process, `admin.conf` and IP checks have already passed. + +Catalog gaps: the catalog is refreshed and split over time (#1262), so an old kubelet or CRI-O version may no longer be in it. If the recorded version is not in the catalog, the checksum check is skipped with a warning, and the gate relies on the other checks: the binary is in the sandbox path and the running process is that binary. A version that is in the catalog but does not match its checksum still stops the command. + +The acceptance test is the existing `uat:compat` scenario, which installs a released version and upgrades. After the upgrade the gate must pass with no manual step. ### 5.5 Cluster-side lock @@ -135,17 +157,21 @@ State is moving from one `state.yaml` to per-component files, each with its own - #1231 / PR #1235: `machine.yaml`, `cluster.yaml`, `blocknode.yaml`, `consensus.yaml`, `teleport.yaml`, with a startup migration. - #1245: `machine.yaml` split further into `profile`, `firewall` and `system`. -We build on that and do not change it. The per-component files are what make the host and cluster split structural. +We build on that. The per-component files are what make the host and cluster split structural. One change is needed in #1245: `ProvisionerState` and `Version` (the provisioner's own version, commit and executable, and the state model version) move out of `system` into a binary-level file, `provisioner.yaml`, described below. That is cheaper to agree now, while #1245 is open. ### 6.1 Layout -State is grouped by host. A cluster with no host on this machine gets its own directory holding only the cluster-scoped component files. +State is grouped by host. A cluster with no host on this machine gets its own directory holding only the cluster-scoped component files. The tree below shows the layout once #1235 and #1245 have both merged, plus the binary-level file. Until #1245 lands, `profile`, `firewall` and `system` are still one `machine.yaml`. ``` /opt/solo/weaver/state/ +├── provisioner.yaml # binary-level: provisioner version, commit, executable, state model version ├── hosts/ │ └── local/ # this machine + the cluster it provisioned -│ ├── machine.yaml # host components (later profile, firewall, system) +│ ├── host.yaml # identity: name, machine ID, the cluster this host provisioned +│ ├── profile.yaml # host components +│ ├── firewall.yaml +│ ├── system.yaml │ ├── cluster.yaml │ ├── blocknode.yaml │ ├── consensus.yaml @@ -161,15 +187,17 @@ State is grouped by host. A cluster with no host on this machine gets its own di Rules: -- `hosts/local/` keeps today's component files and content. `cluster.yaml` also records the provisioned cluster's ID. -- `clusters//` holds **only cluster-scoped components**. Host components (`machine`, and later `profile`, `firewall`, `system`) never exist there, so external-mode code cannot reach host state. Hashes and locks work per file, as in #1235. +- `provisioner.yaml` describes the binary, not a host or a cluster. It is always loadable, the binary reads it first before choosing a host or a cluster, and it exists on every machine, including a Mac with only external clusters (`~/.solo-weaver/state/provisioner.yaml`). Startup migrations depend on the recorded provisioner version, and without this file a machine with no `hosts/local` would have nowhere to record it. Host and cluster state never write it. +- `hosts/local/host.yaml` is a small identity file that the identity gate reads first, before any component file, so it needs no component lock. It holds the host `name`, `kind` (`local`; `remote` later), the `machineId` from `/etc/machine-id`, and `managedClusterId`, the cluster this host provisioned (empty when none). A later remote host (section 9) uses the same file with extra registry fields. If the recorded `machineId` no longer matches the machine, for example because the directory was copied, the command stops. If the machine has no usable `/etc/machine-id` (some containers), the check is skipped and no ID is recorded. +- `hosts/local/` otherwise keeps today's component files and content. +- `clusters//` holds **only cluster-scoped components**. Host components (`profile`, `firewall`, `system`) never exist there, so external-mode code cannot reach host state. Hashes and locks work per file, as in #1235. - Each file records its own `clusterId` and `origin`, and the loader checks them against the live cluster. A copied file is rejected. - External runs never write `hosts/local`. Provisioned runs never write `clusters/`. - The key is the cluster ID, not the IP, the context name or a user-supplied name. A rebuilt cluster gets a new ID and therefore fresh state. -### 6.2 Components that mix host and cluster fields +### 6.2 Host-bound fields inside component files -Some component files hold both kinds of fields. They have to be split, or the host fields made optional and never written in external mode, before a cluster directory can hold them: +`clusters//` holds `blocknode.yaml`, `consensus.yaml` and `teleport.yaml` for the components managed in that cluster, with the same names and schema as in `hosts/local/`. A few fields inside those files only make sense on a provisioned host: | File | Host-bound fields | Cluster-level fields | |---|---|---| @@ -177,6 +205,13 @@ Some component files hold both kinds of fields. They have to be split, or the ho | `consensus.yaml` | `HostPathOwners` | Node identity, images, config hashes, secrets, managed spec | | `teleport.yaml` | Node agent | Cluster agent | +In external mode the steps that fill the host-bound fields (tc shaping, hostPath storage, the node agent) do not run, so those fields stay empty. The question is how to make sure they can never be filled in a cluster directory. Two ways: + +1. **Optional fields plus a writer guard (the default).** The state writer refuses to write a non-empty host-bound field to any file under `clusters//`, and a unit test covers every field in the table. It keeps the schemas as they are. +2. **Split the files by owner**, as #1245 does for `machine.yaml`, for example a host-side file for shaping and storage that exists only in `hosts/local/`. The compiler then enforces it. It is more work and touches the daemon's `consensus.yaml`, so keep it as later hardening if the guard proves leaky. + +The deployment profile (`local`, `testnet`, `mainnet`) belongs to its target, because one person can manage a `local` Kind cluster, a `perfnet` cluster and a `mainnet` VM. For a provisioned host it is `hosts/local/profile.yaml`, as in #1245. For an external cluster it is a `profile` field in `clusters//cluster.yaml`. A default for new targets still comes from the config file or `--profile`; that is configuration, not state. + The daemon writes `consensus.yaml` and is released separately, so it has to follow any path or layout change. ### 6.3 `clusters/index.yaml` @@ -200,6 +235,16 @@ clusters: lastUsed: 2026-10-15T09:40:00Z ``` +#### How a cluster is identified to a human + +The cluster ID is the UID of the `kube-system` namespace, a random UUID that says nothing about the cluster. It stays the key because it is stable and unique. Everything a person reads shows more than the bare ID: + +- **Alias.** From `--cluster-name`, or the kubeconfig context name when none is given (for example `kind-dev`). A label only. +- **Short ID.** The first 8 characters of the UID. +- **Server address.** For example `https://10.20.0.5:6443`. + +Every message, log line and diagnostic shows them together, for example `perfnet-eu (7c1e9a04) at https://k8s.perfnet-eu.example:6443`. On disk, `clusters//` stays a plain UID directory, because putting the alias in the name would mean renaming it when the alias changes. The index maps one to the other, and a read-only list command prints it (section 13). + The alias binds to a cluster ID on first use. If the alias later points at a different cluster ID, the command stops with an error. `--rebind` overrides it explicitly. An alias alone can never create state. ## 7. External mode: API only @@ -232,7 +277,7 @@ Prerequisites for running this on macOS: Add a startup migration, `state-hosts-layout`, registered in `RegisterMigrations()` (`cmd/cli/commands/root.go`) **after** the migration that splits `state.yaml` into component files (#1231). See `docs/dev/migration-framework.md`. The chain is: `unified-state`, then per-component split, then `state-hosts-layout`. Each step is idempotent. - `Applies`: component files exist directly under `state/` and `state/hosts/local/` does not. -- `Execute`: create `hosts/local/` with the same owner and mode, move the component files (and their lock and hash companions) into it, and keep a backup of the originals as `state/.pre-hosts-layout/`. It moves files and does not rewrite their content. +- `Execute`: create `hosts/local/` with the same owner and mode, move the component files (and their lock and hash companions) into it, and keep a backup of the originals as `state/.pre-hosts-layout/`. It moves files and does not rewrite their content, with one exception: it copies `ProvisionerState` and `Version` out of the machine-level file into `state/provisioner.yaml`, and writes `hosts/local/host.yaml` (name, kind, machine ID, and the cluster ID if one is already recorded). - If both locations have files, prefer `hosts/local/`, leave the legacy files untouched, and log a warning. Never overwrite. - Idempotent and crash-safe. `Rollback` restores the backup on a best-effort basis. - It runs before anything opens the state manager, and before the identity gate. Migration is pure filesystem work and needs no cluster. @@ -244,14 +289,13 @@ Risks to settle: 2. Downgrade: an older binary looks at the old path and sees a fresh machine. Migrations are one-way today; document it. A symlink does not help, because atomic writes replace it. 3. The daemon is a separate binary with its own release schedule. It must read the new location, and a daemon older than the CLI must not see an empty state. 4. Two concurrent invocations during the move need a lock, or an atomic rename plus a re-check. The per-component `flock` files from #1235 move together with the state files, so the move must hold all of them. -5. The legacy-cluster backfill in section 5.4 ships with this change. ## 9. Later: several VMs from macOS Not in scope now, but the layout leaves room. - A developer on macOS runs a launcher that copies the matching Linux binary to a VM, verifies its digest, and runs the command there over SSH. The VM keeps its own `hosts/local` state. -- The Mac keeps `hosts//host.yaml` as a registry only: SSH target, verified machine ID, last known binary version. It never holds a copy of a VM's state. +- The Mac keeps `hosts//host.yaml`, the same file `hosts/local` uses, with `kind: remote` and registry fields added: SSH target, verified machine ID, last known binary version. It never holds a copy of a VM's state. - The Mac can still drive that VM's cluster natively through the API under `clusters//`, and delegate host-level commands with `--host `. - In the index, such a cluster is `origin: provisioned` with `host: `. - Open design points: version skew, binary provenance (signatures and digests), flag-to-file path rewriting, secrets in uploaded files, PTY and non-interactive mode, SSH host-key trust, and keeping a command running after a dropped connection. @@ -269,29 +313,33 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux ## 11. Delivery plan -1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the legacy backfill rules). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. +1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the handling of older installs in section 5.4). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. 2. Cluster-side lock (`internal/kube/lease`) and its use in the mutating commands (section 5.5). It needs the resolved cluster from step 1 and nothing from the state work. -3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, the cluster-scoped component files for `clusters//`, the splits in section 6.2, and `clusters/index.yaml`. +3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, `provisioner.yaml`, `hosts/local/host.yaml`, the cluster-scoped component files for `clusters//`, the host-bound field guard in section 6.2, and `clusters/index.yaml`. 4. External-mode workflows, block node first, with the StorageClass storage path. 5. CI: Kind lane using `--kubeconfig`, and the test split by required privilege. ## 12. Decisions 1. **Gate exemptions.** The list in section 5.3 is the rule. Diagnostic commands run the gate in report-only mode. -2. **Uninstall and reinstall.** The normal path is at uninstall: `solo-provisioner kube cluster uninstall` moves the cluster-scoped component files (`cluster`, `blocknode`, `consensus` and the cluster half of `teleport`) to `hosts/local/archive/-/`. Host-scoped components (software, firewall, shaping) stay, because the machine is still provisioned. A later `solo-provisioner kube cluster install` then starts clean. Nothing is deleted. +2. **Uninstall and reinstall.** The normal path is at uninstall: `solo-provisioner kube cluster uninstall` moves the cluster-scoped component files (`cluster`, `blocknode`, `consensus` and the cluster half of `teleport`) to `hosts/local/archive/-/`. Host-scoped components (software, firewall, shaping) stay, because the machine is still provisioned. It also clears the cluster ID recorded in `host.yaml`. A later `solo-provisioner kube cluster install` then starts clean. Nothing is deleted. - If `solo-provisioner kube cluster install` finds that `hosts/local` records a cluster ID that differs from the live one (or a cluster that no longer exists), it **stops and does not touch the state**. Weaver cannot tell why: a `kubeadm reset` by hand, a failed or older uninstall, a rebuilt VM with surviving state, or the wrong kubeconfig all look the same, so moving the state automatically would be a guess. The error names the recorded and live cluster IDs and tells the operator to review and clean up the state by hand. The hint lists the files to move or remove and the paths they live in. + If `solo-provisioner kube cluster install` finds that `hosts/local/host.yaml` records a cluster ID that differs from the live one (or a cluster that no longer exists), it **stops and does not touch the state**. Weaver cannot tell why: a `kubeadm reset` by hand, a failed or older uninstall, a rebuilt VM with surviving state, or the wrong kubeconfig all look the same, so moving the state automatically would be a guess. The error names the recorded and live cluster IDs and tells the operator to review and clean up the state by hand. The hint lists the files to move or remove and the paths they live in. 3. **Cluster-side lock.** A Lease per component per cluster, in the component's namespace, in both modes, built on `leaderelection` (section 5.5). 4. **External state is a local cache.** Almost all of it can be re-derived from the cluster, and the cluster wins when they disagree. Storing state in the cluster is a follow-up with its own design (RBAC, ownership). 5. **Garbage collection.** Nothing is deleted automatically, since a cluster that is only unreachable for a moment must not lose its state. A later `solo-provisioner state prune` lists clusters that are unreachable or whose ID no longer exists, and asks before deleting. -6. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State lives in `~/.solo-weaver/state`. +6. **Older installs.** No separate backfill. The gate checks binaries against the catalog checksum and records the cluster ID on its first successful pass (section 5.4). +7. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State, including `provisioner.yaml`, lives in `~/.solo-weaver/state`. ## 13. Still open - What `solo-provisioner kube cluster uninstall` does with state today. This decides whether archive-at-uninstall is new behaviour or a fix. -- Whether to offer an explicit command for the manual cleanup after a stale-state stop, so operators do not edit files by hand. -- How to split or gate the mixed host and cluster fields in section 6.2, and whether that belongs in #1245's follow-ups or in this work. +- Whether to offer explicit commands so operators do not edit state by hand: one for cleanup after a stale-state stop, and one to discard or adopt external state for a cluster that now passes the local proof (section 5.2), for example `solo-provisioner state adopt`. +- A read-only `solo-provisioner kube cluster list` that prints `clusters/index.yaml` (section 6.3). +- Whether the writer guard in section 6.2 is enough, or the files should be split by owner, and whether that belongs in #1245's follow-ups or in this work. +- Where the action history (`LastAction` and `action_history.yaml`, #1244) lives. It is probably per target (host or cluster), but that needs a look. +- Agreeing the `ProvisionerState` and `Version` move out of `system` with #1245. - The lease timings, and the name of the explicit override flag for removing a dead lease. - The detail of report-only mode for `doctor` and `diagnose` (epic #1110). - The design of in-cluster state storage, if we want it. From 9d6542cd8a031c1b74d9650730d2fde8edc20920 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 21:53:21 +1100 Subject: [PATCH 05/21] docs(design): note how the layout migration fits the per-component state work Describe how the hosts/local migration builds on #1235's component path helpers, runs under its startup-migrations lock, and leaves the state.yaml.legacy and state.yaml.diverged-* files untouched. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 49cbd430..8b28a40d 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -281,7 +281,9 @@ Add a startup migration, `state-hosts-layout`, registered in `RegisterMigrations - If both locations have files, prefer `hosts/local/`, leave the legacy files untouched, and log a warning. Never overwrite. - Idempotent and crash-safe. `Rollback` restores the backup on a best-effort basis. - It runs before anything opens the state manager, and before the identity gate. Migration is pure filesystem work and needs no cluster. -- All code that builds a state path (the state manager, `state_reader.go`, the existing state migrations, `testutil`, the daemon) moves to one resolver. +- All code that builds a state path (the state manager, `state_reader.go`, the existing state migrations, `testutil`, the daemon) moves to one resolver. #1235 already builds component paths from a directory (`ComponentFilePath(dir, id)`, `RuntimeResolver.StateDir()`), so for most callers the change is what that directory is (`state/` becomes `state/hosts/local/`). A few callers still use `models.Paths().StateDir` directly, such as the startup-migrations lock and some tests. +- The migration runs under the same startup-migrations lock that #1235 takes on the state directory. +- The split migration from #1235 can leave `state.yaml.legacy` and `state.yaml.diverged-*` files in `state/`. They stay where they are: the layout migration moves only the component files, and does not touch or delete them. Risks to settle: From 771e768ca83fcaaca534e04fe70a59afe85e72da Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 21:58:50 +1100 Subject: [PATCH 06/21] docs(design): add explicit cluster registration and Solo prior art Add `kube cluster connect` as the one command that registers a cluster (test the connection, read the cluster ID, check permissions, bind the alias), with `list` and `info`, and implicit registration for non-interactive runs. Exempt `connect` and `list` from the identity gate. Add a prior-art section on hiero-ledger/solo (alias to context map, in-cluster state ConfigMap and validator, lease) with what we use and what we avoid, and note the in-cluster state shape and cluster grouping as follow-ups. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 60 ++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 7 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 8b28a40d..acdfbe24 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -58,10 +58,42 @@ New root persistent flags: |---|---| | `--kubeconfig ` | Kubeconfig to use. Without it, only the cluster this host provisioned is considered. | | `--context ` | Context inside the kubeconfig. | -| `--cluster-name ` | Optional human label, stored in `clusters/index.yaml`. Defaults to the context name. Never used as a key. | +| `--cluster-name ` | Optional human label, stored in `clusters/index.yaml` when the cluster is registered (section 4.1). Defaults to the context name. Never used as a key. | Without `--kubeconfig`, ambient `$KUBECONFIG` and `~/.kube/config` are not trusted on their own. Weaver already copies `admin.conf` to `~/.kube/config` for root and the sudo user (`internal/kube/admin.go`), so that file can be ours, but the identity check, not the path, proves it. +### 4.1 Registering a cluster + +A cluster is registered once, by one command, and the rest of the commands refer to it: + +```bash +solo-provisioner kube cluster connect --kubeconfig [--context ] [--cluster-name ] +``` + +`connect` does these steps, in order: + +1. Load the kubeconfig and test the connection with a short deadline. If it fails, stop with a clear error. +2. Read the cluster ID (the `kube-system` UID). +3. Classify the cluster with the identity check in section 5.1: provisioned on this host, or external. +4. Check the permissions it can check at this point, with `SelfSubjectAccessReview`: API access, and creating namespaces and leases. Checks in a component's own namespace run later, when a command needs them, because that namespace may not exist yet. +5. Bind the alias to the cluster ID. If the alias is already bound to a different cluster ID, stop unless `--rebind` is given. +6. Write the `clusters/index.yaml` entry. State files are not created here. They are created when a component is first managed in that cluster. + +Related commands, all reading or editing the index only: + +| Command | Purpose | +|---|---| +| `solo-provisioner kube cluster list` | Print every known cluster: alias, short ID, origin, server, last used. | +| `solo-provisioner kube cluster info ` | Origin, server, the components managed there (block node, consensus node and so on) and their state, and the current lease holders (section 5.5). Runs the identity check in report-only mode, so it pairs with `doctor` and `diagnose` (#1110). | +| `solo-provisioner kube cluster disconnect ` | Remove the index entry. State files are left in place until an explicit prune (section 12). Later work. | + +A command that is given an unregistered cluster: + +- Interactive: stops with a hint to run `solo-provisioner kube cluster connect`. +- Non-interactive (`--non-interactive`, CI): registers the cluster implicitly with the same steps, so a Kind lane needs no extra step. + +`connect` is idempotent. Running it again for the same cluster ID refreshes the context, server and last-used time. + ## 5. Identity gate A check in the root pre-run, **before** any state load, RSL or reality check. If it fails, the command stops with a reason code and a hint. This is how we know reality and state can be trusted. @@ -85,7 +117,7 @@ The IP check alone is not enough. A Kind or minikube cluster on the same machine | Situation | Result | |---|---| | All checks pass | `provisioned`, host `local`, state in `hosts/local` | -| Checks fail, `--kubeconfig` given | `external`, state in `clusters/` | +| Checks fail, `--kubeconfig` given | `external`, state in `clusters/`. The cluster must be registered (section 4.1): an interactive run stops with a hint to run `solo-provisioner kube cluster connect`, a non-interactive run registers it first | | Checks fail, no `--kubeconfig` | Stop. Hint: run `solo-provisioner kube cluster install`, or pass `--kubeconfig` for an external cluster | | Cluster ID matches the one recorded in `hosts/local/host.yaml`, but a local check fails (for example kubelet is down) | Stop. Hint: fix the local cluster. **Never** fall back to `external` | | Cluster ID already has `clusters/` state and now passes the local proof | Stop. Needs an explicit step; it is never adopted silently | @@ -113,6 +145,7 @@ These commands expect no cluster yet, so they skip the gate: - `solo-provisioner demo` - `solo-provisioner --help` (and `--help` on any subcommand) - `solo-provisioner kube cluster install` +- `solo-provisioner kube cluster connect` (it is the step that registers any cluster) and `solo-provisioner kube cluster list` (it reads the index only) `solo-provisioner kube cluster install` still has to refuse to proceed when a foreign cluster is configured. @@ -245,7 +278,7 @@ The cluster ID is the UID of the `kube-system` namespace, a random UUID that say Every message, log line and diagnostic shows them together, for example `perfnet-eu (7c1e9a04) at https://k8s.perfnet-eu.example:6443`. On disk, `clusters//` stays a plain UID directory, because putting the alias in the name would mean renaming it when the alias changes. The index maps one to the other, and a read-only list command prints it (section 13). -The alias binds to a cluster ID on first use. If the alias later points at a different cluster ID, the command stops with an error. `--rebind` overrides it explicitly. An alias alone can never create state. +The alias binds to a cluster ID when the cluster is registered with `solo-provisioner kube cluster connect` (section 4.1). If the alias later points at a different cluster ID, the command stops with an error. `--rebind` overrides it explicitly. An alias alone can never create state. ## 7. External mode: API only @@ -263,7 +296,7 @@ The alias binds to a cluster ID on first use. If the alias later points at a dif A developer can create a Kind cluster on their own machine (including a Mac) and use it with weaver. It does not need SSH delegation (section 9). - Weaver did not install it, so it is `origin: external`. It gets a `clusters//` directory and an index entry with the context (for example `kind-dev`) and a `127.0.0.1` server address. -- Use it with `--kubeconfig` and `--context`. Only API-level work runs. Storage comes from Kind's default StorageClass. +- Register it once with `solo-provisioner kube cluster connect`, or let a non-interactive run register it. Use it with `--kubeconfig` and `--context`. Only API-level work runs. Storage comes from Kind's default StorageClass. - Each `kind create cluster` gets a new cluster ID, so a recreated cluster starts with clean state. If the same alias is reused for the new cluster, `--rebind` is needed. - The same flow is what the CI Kind lane uses (section 10). @@ -317,7 +350,7 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux 1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the handling of older installs in section 5.4). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. 2. Cluster-side lock (`internal/kube/lease`) and its use in the mutating commands (section 5.5). It needs the resolved cluster from step 1 and nothing from the state work. -3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, `provisioner.yaml`, `hosts/local/host.yaml`, the cluster-scoped component files for `clusters//`, the host-bound field guard in section 6.2, and `clusters/index.yaml`. +3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, `provisioner.yaml`, `hosts/local/host.yaml`, the cluster-scoped component files for `clusters//`, the host-bound field guard in section 6.2, `clusters/index.yaml`, and the `kube cluster connect`, `list` and `info` commands. 4. External-mode workflows, block node first, with the StorageClass storage path. 5. CI: Kind lane using `--kubeconfig`, and the test split by required privilege. @@ -334,15 +367,28 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux 6. **Older installs.** No separate backfill. The gate checks binaries against the catalog checksum and records the cluster ID on its first successful pass (section 5.4). 7. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State, including `provisioner.yaml`, lives in `~/.solo-weaver/state`. +8. **Cluster registration.** One explicit command, `solo-provisioner kube cluster connect`, registers a cluster. Non-interactive runs register implicitly. `list` and `info` read the index and the cluster. `disconnect` is later work (section 4.1). + ## 13. Still open - What `solo-provisioner kube cluster uninstall` does with state today. This decides whether archive-at-uninstall is new behaviour or a fix. - Whether to offer explicit commands so operators do not edit state by hand: one for cleanup after a stale-state stop, and one to discard or adopt external state for a cluster that now passes the local proof (section 5.2), for example `solo-provisioner state adopt`. -- A read-only `solo-provisioner kube cluster list` that prints `clusters/index.yaml` (section 6.3). +- Details of `solo-provisioner kube cluster info` (which component states it shows) and of `disconnect` (what it does with state files). Both belong with the diagnostics work (#1110) and the prune command. - Whether the writer guard in section 6.2 is enough, or the files should be split by owner, and whether that belongs in #1245's follow-ups or in this work. - Where the action history (`LastAction` and `action_history.yaml`, #1244) lives. It is probably per target (host or cluster), but that needs a look. - Agreeing the `ProvisionerState` and `Version` move out of `system` with #1245. - The lease timings, and the name of the explicit override flag for removing a dead lease. - The detail of report-only mode for `doctor` and `diagnose` (epic #1110). -- The design of in-cluster state storage, if we want it. +- The design of in-cluster state storage, if we want it. Prior art in section 14: a labelled ConfigMap with `lastUpdatedBy` and `lastUpdatedAt` (the lease holder identity already carries user, host and command), and a validator that checks the recorded components against the live cluster. +- Whether a grouping of clusters is ever needed (Solo's deployments span several clusters). It may matter for consensus orbits that span clusters. Out of scope here. - The `solo-provisioner state prune` command and its safety rules. + +## 14. Prior art: hiero-ledger/solo + +Solo (`hiero-ledger/solo`) solves part of this problem. We do not adopt its code, but we use some of its ideas. + +- **Alias to context map.** A Solo "cluster reference" is a user-chosen alias mapped to a kubeconfig context name, kept in a local config file. Commands: `cluster-ref config connect | disconnect | list | info`. `connect` first tests the connection. This is the model behind `kube cluster connect`, `list` and `info` in section 4.1. +- **State in the cluster.** Solo keeps component state in a labelled ConfigMap (`solo-remote-config`) with history and `lastUpdatedAt` and `lastUpdatedBy`, and a validator checks the recorded components against the live cluster (Helm releases, pods). That is the shape of the later in-cluster state work in section 13. +- **Deployments.** A Solo deployment groups a namespace and one or more cluster aliases. We leave that out for now. +- **What we avoid.** The alias is Solo's only identity. Nothing checks which cluster a context really points at, `connect` can silently remap an alias, and kubeconfig context names are mutable. Here the key is the cluster ID, an alias binds to one ID, and rebinding needs `--rebind`. Solo also has no notion of whether a cluster was installed by Solo. +- **Lease.** Solo's lock is a Lease with a holder identity, renewed in the background. Section 5.5 adapts it. From 70af89132a6dd77ccb141a2a0af50b6bb387f6f9 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 22:10:56 +1100 Subject: [PATCH 07/21] docs(design): clarify that provisioned clusters register automatically State that a cluster passing the provisioned proof registers without `kube cluster connect` (at install for new clusters, on the gate's first successful pass for older installs), so the unregistered-cluster rule applies only to external clusters. Describe `connect` as an optional eager registration for older installs. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index acdfbe24..0f13cad3 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -87,7 +87,9 @@ Related commands, all reading or editing the index only: | `solo-provisioner kube cluster info ` | Origin, server, the components managed there (block node, consensus node and so on) and their state, and the current lease holders (section 5.5). Runs the identity check in report-only mode, so it pairs with `doctor` and `diagnose` (#1110). | | `solo-provisioner kube cluster disconnect ` | Remove the index entry. State files are left in place until an explicit prune (section 12). Later work. | -A command that is given an unregistered cluster: +A cluster that passes the provisioned proof (section 5.1) registers automatically: `solo-provisioner kube cluster install` does it for new installs, and the gate does it on its first successful pass for older installs (section 5.4). No `connect` is needed. The rule below applies to every other cluster, which is external. + +A command that is given an unregistered external cluster: - Interactive: stops with a hint to run `solo-provisioner kube cluster connect`. - Non-interactive (`--non-interactive`, CI): registers the cluster implicitly with the same steps, so a Kind lane needs no extra step. @@ -156,7 +158,7 @@ Diagnostic commands (`doctor` and `diagnose`, epic #1110) are **not** exempt, be A strict gate must not lock operators out of clusters that older versions installed. Two things could be missing on such a machine, and neither needs a separate migration: - **Recorded hashes.** Software state records only the name, version, `Installed` and `Configured`; it has never held a binary hash. The gate does not need one. The embedded catalog has the expected checksum for each software, version and platform, which the installer already checks at download. The gate verifies that the sandbox binary on disk matches the catalog checksum for the version recorded in state, and that the running process is that binary. -- **The cluster ID.** `hosts/local/host.yaml` has none on an older install. The first time all the checks in 5.1 pass, the gate records the live cluster ID. This is trust on first use, and it is safe because the binary, process, `admin.conf` and IP checks have already passed. +- **The cluster ID.** `hosts/local/host.yaml` has none on an older install. The first time all the checks in 5.1 pass, the gate records the live cluster ID. This is trust on first use, and it is safe because the binary, process, `admin.conf` and IP checks have already passed. Running `solo-provisioner kube cluster connect` is an optional way to register eagerly and see each check reported (section 4.1); nothing stops working if it is skipped. Catalog gaps: the catalog is refreshed and split over time (#1262), so an old kubelet or CRI-O version may no longer be in it. If the recorded version is not in the catalog, the checksum check is skipped with a warning, and the gate relies on the other checks: the binary is in the sandbox path and the running process is that binary. A version that is in the catalog but does not match its checksum still stops the command. From f38a479107b2e22751536d9436b2ee5491229e97 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 22:26:09 +1100 Subject: [PATCH 08/21] docs(design): add a CI rollout order and baseline to the design State what the design gives CI (fewer KVM VMs, isolated parallel jobs, no privileged runners for most tests, external mode tested on every PR) and what it does not cover (host-level tests, the tag split, block node storage). Give the order: record a baseline, start the Kind lane with API-only tests, add block node, and build the host-level lane in parallel. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 0f13cad3..038a1fe6 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -348,6 +348,28 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux | Host-level (provisioned) | Hosted runner or privileged systemd container | Network, mount, sysctl, systemd tests and the full cluster install UAT | | Real VM | `kvm-runner`, nightly or on path filters | Boot-order and reboot tests, one full Debian lifecycle | +### 10.1 Rollout order and baseline + +What this design gives CI: + +- Most PR jobs stop booting a KVM VM. Today the PR gate boots five (four UAT jobs and the integration job), and three of the UAT jobs repeat the full cluster install. +- Jobs become isolated and parallel. A Kind cluster per job replaces the shared `/opt/solo` and one serial integration run. Where several jobs do share a cluster, the lease (section 5.5) keeps them from interfering. +- Most tests no longer need privileged runners. +- External mode is tested on every PR. One useful check comes with it: a Kind cluster on the same machine must be classified as `external` and refused as `provisioned`, which covers the IP-only trap in section 5.1. + +What it does not do: + +- Host-level tests (nft, tc, sysctl, swap, mounts, systemd, the full install and uninstall, `uat:compat`, and the block node scenarios that migrate hostPath storage) cannot run on Kind. They need the separate privileged container or hosted runner lane in the table above. +- Splitting runs by lane needs finer test tags or capability gates than `integration`. That is test infrastructure work and does not depend on this design. +- Block node on Kind depends on the StorageClass storage path (delivery step 4). + +Order: + +1. **Baseline first.** Record the current duration of each PR job (`integration`, `compat`, `lifecycle`, `purge-storage`, `plugin-preset-upgrade`) and the VM minutes used, so the gain can be measured. No timings have been taken yet. +2. **Kind lane, first candidates.** These need only the API and do not wait for block node storage: the kube client integration tests (`TestWithCluster_*`), the Helm manager and cluster checker tests, and the alloy, teleport cluster, solo-operator and ESO scenarios. Whether a scenario belongs here is confirmed one by one. +3. **Block node on Kind.** The plugin-preset upgrade scenario first, once external mode and StorageClass storage work for block node. Scenarios that touch hostPath stay host-level. +4. **Host-level lane and tag split.** This can run in parallel with steps 2 and 3. Move the host-level tests to a privileged container or hosted runner, and keep KVM for the nightly run and path-filtered jobs. + ## 11. Delivery plan 1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the handling of older installs in section 5.4). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. From 029f22854a0cee57b3b6c95978d4c3a8d77f3ccf Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 22:32:07 +1100 Subject: [PATCH 09/21] docs(design): run the KVM lane nightly and on demand, not on path filters Make KVM a nightly job that a maintainer enables by hand for a PR that needs a real VM, instead of starting it from path filters. Path filters may later suggest it but do not start it. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 038a1fe6..a49f9091 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -346,7 +346,7 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux | Unprivileged | `solo-linux-medium` | Unit tests, installer download and verify tests | | Kind (external mode) | `solo-linux-medium` with `dind` | API-level UAT: chart and plugin-preset upgrade, kube client, alloy, teleport cluster, solo-operator | | Host-level (provisioned) | Hosted runner or privileged systemd container | Network, mount, sysctl, systemd tests and the full cluster install UAT | -| Real VM | `kvm-runner`, nightly or on path filters | Boot-order and reboot tests, one full Debian lifecycle | +| Real VM | `kvm-runner`, nightly, and enabled by hand for a PR when needed | Boot-order and reboot tests, one full Debian lifecycle | ### 10.1 Rollout order and baseline @@ -368,7 +368,7 @@ Order: 1. **Baseline first.** Record the current duration of each PR job (`integration`, `compat`, `lifecycle`, `purge-storage`, `plugin-preset-upgrade`) and the VM minutes used, so the gain can be measured. No timings have been taken yet. 2. **Kind lane, first candidates.** These need only the API and do not wait for block node storage: the kube client integration tests (`TestWithCluster_*`), the Helm manager and cluster checker tests, and the alloy, teleport cluster, solo-operator and ESO scenarios. Whether a scenario belongs here is confirmed one by one. 3. **Block node on Kind.** The plugin-preset upgrade scenario first, once external mode and StorageClass storage work for block node. Scenarios that touch hostPath stay host-level. -4. **Host-level lane and tag split.** This can run in parallel with steps 2 and 3. Move the host-level tests to a privileged container or hosted runner, and keep KVM for the nightly run and path-filtered jobs. +4. **Host-level lane and tag split.** This can run in parallel with steps 2 and 3. Move the host-level tests to a privileged container or hosted runner. KVM is no longer part of the default PR gate: it runs nightly, and a maintainer enables it by hand for a PR that touches something only a real VM can check (for example a manual workflow dispatch or a PR label). Path filters can later suggest when to enable it, but do not start it automatically. ## 11. Delivery plan From d2360dfa1860758e85f91b61389d7185f7d8f4aa Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 22:34:21 +1100 Subject: [PATCH 10/21] docs(design): start the KVM lane from path filters, by hand, or nightly Replace the manual-only wording: KVM starts automatically when a PR touches files only a real VM can check (using the existing paths-filter job), by hand through a dispatch or label for anything else, and nightly as a safety net. Add the starting candidate paths and the cost trade-off of a broad filter. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index a49f9091..65467784 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -346,7 +346,7 @@ External mode is what makes a Kind lane possible on the ARC runners (`solo-linux | Unprivileged | `solo-linux-medium` | Unit tests, installer download and verify tests | | Kind (external mode) | `solo-linux-medium` with `dind` | API-level UAT: chart and plugin-preset upgrade, kube client, alloy, teleport cluster, solo-operator | | Host-level (provisioned) | Hosted runner or privileged systemd container | Network, mount, sysctl, systemd tests and the full cluster install UAT | -| Real VM | `kvm-runner`, nightly, and enabled by hand for a PR when needed | Boot-order and reboot tests, one full Debian lifecycle | +| Real VM | `kvm-runner`, nightly, started by a path filter, or enabled by hand | Boot-order and reboot tests, one full Debian lifecycle | ### 10.1 Rollout order and baseline @@ -368,7 +368,12 @@ Order: 1. **Baseline first.** Record the current duration of each PR job (`integration`, `compat`, `lifecycle`, `purge-storage`, `plugin-preset-upgrade`) and the VM minutes used, so the gain can be measured. No timings have been taken yet. 2. **Kind lane, first candidates.** These need only the API and do not wait for block node storage: the kube client integration tests (`TestWithCluster_*`), the Helm manager and cluster checker tests, and the alloy, teleport cluster, solo-operator and ESO scenarios. Whether a scenario belongs here is confirmed one by one. 3. **Block node on Kind.** The plugin-preset upgrade scenario first, once external mode and StorageClass storage work for block node. Scenarios that touch hostPath stay host-level. -4. **Host-level lane and tag split.** This can run in parallel with steps 2 and 3. Move the host-level tests to a privileged container or hosted runner. KVM is no longer part of the default PR gate: it runs nightly, and a maintainer enables it by hand for a PR that touches something only a real VM can check (for example a manual workflow dispatch or a PR label). Path filters can later suggest when to enable it, but do not start it automatically. +4. **Host-level lane and tag split.** This can run in parallel with steps 2 and 3. Move the host-level tests to a privileged container or hosted runner. KVM is no longer part of the default PR gate. It starts in three ways: + - **Path filter (automatic).** The PR touches files whose behaviour can only be verified on a real VM. The repo already has this mechanism: the `changes` job in `flow-pull-request-checks.yaml` uses `dorny/paths-filter`. Starting candidates: `internal/network/**` and the network unit workflows, `internal/templates/files/**` for sysctl, kubeadm, CRI-O and Cilium, the startup migrations (`internal/workflows/migration_*.go`, `internal/state/migration_*.go`), `internal/mount/**`, `pkg/kernel/**`, `pkg/os/**`, the daemon and its systemd unit, the release pipeline, and `taskfiles/uat.yaml`. The exact list is chosen when the lane is built. + - **By hand.** A maintainer starts it for any other PR, through a manual workflow dispatch or a PR label. + - **Nightly, always.** It is the safety net: a change that the path filter missed is caught within a day. + + A filter that is too broad costs KVM minutes, so keep the list tight and review it from time to time. The tag split above tells the filter which tests to run on KVM, so it does not rerun everything. ## 11. Delivery plan From 0cbed322c22987af42e951d23bcc664b81375443 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 22:43:09 +1100 Subject: [PATCH 11/21] docs(design): rewrite the identity gate around offline identity and command needs Separate local evidence, recorded identity (cluster ID, server address and CA fingerprint) and live identity, so a broken local cluster is still recognised as ours and never treated as external. Replace the overlapping outcome table with ordered rules that cover a wrong machine ID, a rebuilt cluster, an unreachable API and unreadable evidence. Replace the exempt command list with a per-command need (none, connect, install, host, api, host+api), so uninstall and the host-level commands work on a broken cluster. Record the identity on the first successful pass for older installs, and add a threat-model note. Refs #1287 Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 85 +++++++++++++++++++----------- 1 file changed, 53 insertions(+), 32 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 65467784..f97d62d7 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -74,7 +74,7 @@ solo-provisioner kube cluster connect --kubeconfig [--context ] [-- 1. Load the kubeconfig and test the connection with a short deadline. If it fails, stop with a clear error. 2. Read the cluster ID (the `kube-system` UID). -3. Classify the cluster with the identity check in section 5.1: provisioned on this host, or external. +3. Classify the cluster with the evidence and outcomes in sections 5.1 and 5.2: provisioned on this host, or external. 4. Check the permissions it can check at this point, with `SelfSubjectAccessReview`: API access, and creating namespaces and leases. Checks in a component's own namespace run later, when a command needs them, because that namespace may not exist yet. 5. Bind the alias to the cluster ID. If the alias is already bound to a different cluster ID, stop unless `--rebind` is given. 6. Write the `clusters/index.yaml` entry. State files are not created here. They are created when a component is first managed in that cluster. @@ -98,35 +98,55 @@ A command that is given an unregistered external cluster: ## 5. Identity gate -A check in the root pre-run, **before** any state load, RSL or reality check. If it fails, the command stops with a reason code and a hint. This is how we know reality and state can be trusted. +A check in the root pre-run, **before** any RSL or reality check. It decides which cluster a command is about and whether that cluster is this host's own. If the answer is unsafe, the command stops with a reason code and a hint. This is how we know reality and state can be trusted. -### 5.1 What a `provisioned` cluster must prove +The gate prevents accidents: running against the wrong cluster, or against state left behind by another one. It is not a security boundary. A user with root can fake any of its evidence. -All of these must pass: +The gate reads only `hosts/local/host.yaml` and the software records, read-only and without locks. It does not go through RSL, reality or the state manager. + +### 5.1 Evidence + +The gate separates evidence that needs no API from evidence that needs it, so a broken local cluster can still be recognised as ours. + +**Local evidence** (no API needed). All of these must pass for the host to count as running a provisioned cluster: | Evidence | Why | |---|---| | Software state records kubelet and CRI-O as installed, and the sandbox binaries on disk match the catalog checksum for the recorded version and platform | Binary provenance | | kubelet and CRI-O are running (systemd unit active, CRI-O socket answers), and the running executable is the sandbox binary, not one from `PATH` | A matching file is not enough | -| `/etc/kubernetes/admin.conf` and the static pod manifests under `/etc/kubernetes/manifests` exist | kubeadm control plane evidence | -| The kubeconfig server host matches this machine's IP and the IP in weaver's config | The cluster is on this host | -| The live cluster ID matches the one recorded in `hosts/local/host.yaml` (when one is recorded; see 5.4) | Same cluster as last time | +| `/etc/kubernetes/admin.conf`, the static pod manifests under `/etc/kubernetes/manifests` and `/etc/kubernetes/pki/ca.crt` exist | kubeadm control plane evidence | +| The server host in `admin.conf` resolves to an address of this machine | The cluster is on this host | + +The address check alone is not enough. A Kind or minikube cluster on the same machine points at `127.0.0.1` and would pass it. The binary and process checks rule those out. + +**Recorded identity** (also no API needed). `hosts/local/host.yaml` records, after the first successful pass, the cluster ID, the server address and the CA certificate fingerprint of the cluster this host provisioned. A kubeconfig **matches the recorded identity** when its server address or its CA fingerprint equals the recorded one. This works while the cluster is down, which the cluster ID alone cannot. -The IP check alone is not enough. A Kind or minikube cluster on the same machine points at `127.0.0.1` and would pass it. The binary and process checks rule those out. +**Live identity** (needs the API). The cluster ID, the UID of the `kube-system` namespace, read from the live cluster. + +Reading evidence can fail for a reason other than a mismatch. For example, checking the running executable (`/proc//exe`) generally needs root. The gate keeps "cannot read" apart from "does not match": an unreadable check is never taken as proof that a cluster is not ours. + +Assumption, to be confirmed against the kubeadm workflow: a host runs one single-node cluster, so the control plane is local. ### 5.2 Outcomes -| Situation | Result | -|---|---| -| All checks pass | `provisioned`, host `local`, state in `hosts/local` | -| Checks fail, `--kubeconfig` given | `external`, state in `clusters/`. The cluster must be registered (section 4.1): an interactive run stops with a hint to run `solo-provisioner kube cluster connect`, a non-interactive run registers it first | -| Checks fail, no `--kubeconfig` | Stop. Hint: run `solo-provisioner kube cluster install`, or pass `--kubeconfig` for an external cluster | -| Cluster ID matches the one recorded in `hosts/local/host.yaml`, but a local check fails (for example kubelet is down) | Stop. Hint: fix the local cluster. **Never** fall back to `external` | -| Cluster ID already has `clusters/` state and now passes the local proof | Stop. Needs an explicit step; it is never adopted silently | +The rules are checked in order. The first one that applies wins. + +| # | Situation | Result | +|---|---|---| +| 1 | The recorded `machineId` in `host.yaml` does not match this machine (for example the directory was copied) | Stop. Hint: the state belongs to another machine | +| 2 | The target matches the recorded identity, local evidence passes, the API is reachable, and the live cluster ID matches the recorded one | `provisioned`, host `local`, state in `hosts/local` | +| 3 | The target matches the recorded identity, local evidence passes, the API is reachable, but the live cluster ID differs | Stop. The cluster was rebuilt and the state is stale (decision 2 in section 12) | +| 4 | The target matches the recorded identity, but a local check fails or the API is unreachable | Stop. The hint names the failing piece (for example kubelet is installed but not running). **Never** fall back to `external` | +| 5 | The target matches the recorded identity, but a check could not be read (for example no root) | Stop. Hint: run with `sudo` | +| 6 | Nothing is recorded, local evidence passes (an older install, section 5.4), and the live cluster ID has no `clusters/` state | `provisioned`. Record the identity | +| 7 | Nothing is recorded, local evidence passes, and the live cluster ID already has `clusters/` state | Stop. Needs an explicit step; it is never adopted silently (see below) | +| 8 | Nothing is recorded, local evidence passes, but the API is unreachable | Stop. The cluster ID cannot be read. Hint: start the cluster, or run `solo-provisioner kube cluster connect` once it is up | +| 9 | The target is not this host's cluster, and `--kubeconfig` was given | `external`, state in `clusters/`. The cluster must be registered (section 4.1): an interactive run stops with a hint to run `solo-provisioner kube cluster connect`, a non-interactive run registers it first | +| 10 | The target is not this host's cluster, and no `--kubeconfig` was given | Stop. If nothing is installed here, the hint is to run `solo-provisioner kube cluster install` or pass `--kubeconfig` for an external cluster. If a provisioned install is only partly present, the hint names what is missing | -The last two rows prevent split-brain between the two state locations. +Rules 3 and 7 prevent split-brain between the state locations. Rule 4 is the one that keeps a broken local cluster from being treated as someone else's. -#### The last row: a cluster that was external and now passes the local proof +#### Rule 7: a cluster that was external and now passes the local proof This can happen when a cluster on the machine was not recognised at first (for example kubelet was down, or it was built by hand), so a command with `--kubeconfig` treated it as `external` and created `clusters//`. Later the proof passes and the same cluster ID now qualifies as `provisioned`. Two places could then describe one cluster, and the records may disagree. Picking one silently could ignore the block node and consensus information in the other, and merging automatically could overwrite newer facts with older ones. So weaver stops and asks. @@ -137,28 +157,29 @@ The explicit step is one of: Until a command for this exists (section 13), the error lists the manual steps. -### 5.3 Exempt commands +### 5.3 What each command needs -These commands expect no cluster yet, so they skip the gate: +A fixed exempt list does not fit, so each command declares what it needs, and the gate checks only that: -- `solo-provisioner install` -- `solo-provisioner uninstall` -- `solo-provisioner version` -- `solo-provisioner demo` -- `solo-provisioner --help` (and `--help` on any subcommand) -- `solo-provisioner kube cluster install` -- `solo-provisioner kube cluster connect` (it is the step that registers any cluster) and `solo-provisioner kube cluster list` (it reads the index only) +| Need | Meaning | Commands | What the gate does | +|---|---|---|---| +| `none` | No cluster and no host evidence | `solo-provisioner install`, `solo-provisioner uninstall`, `solo-provisioner version`, `solo-provisioner demo`, `--help`, `solo-provisioner kube cluster list` | Nothing | +| `connect` | Classifies the target itself | `solo-provisioner kube cluster connect` | Runs the evidence and outcomes in 5.1 and 5.2 to classify the target, and registers it instead of stopping | +| `install` | Creates the cluster | `solo-provisioner kube cluster install` | Allows no cluster, or stale state (rule 3, which this command resolves). Refuses when a cluster that is not ours is already on this host, which means `/etc/kubernetes/admin.conf` exists but the evidence does not show it is ours | +| `host` | Host-level work that does not need the API, and must also work on a broken cluster | `solo-provisioner kube cluster uninstall`, `solo-provisioner network firewall\|shape\|policy ...`, `solo-provisioner teleport node ...` | Checks the machine ID (rule 1), and that the run is not in external mode. Continues when the local cluster is unhealthy (rule 4) | +| `api` | Work through the Kubernetes API in either mode | Block node, consensus node, alloy, ESO and teleport cluster commands that do not touch the host | The full outcomes in 5.2. The result may be `provisioned` or `external` | +| `host+api` | Provisioned-only work that also needs the cluster | Commands that run host steps and API steps, such as block node install on a provisioned host | The full outcomes in 5.2, and the result must be `provisioned` | -`solo-provisioner kube cluster install` still has to refuse to proceed when a foreign cluster is configured. +Diagnostic commands (`doctor`, `diagnose` and `solo-provisioner kube cluster info`, epic #1110) need `api`, but run the gate in **report-only mode**. A failed outcome is printed as a finding and does not stop the command. The detail of those commands belongs to #1110. -Diagnostic commands (`doctor` and `diagnose`, epic #1110) are **not** exempt, because their job is to report on the cluster. They run the identity check in **report-only mode**: a failed check is printed as a finding instead of stopping the command. The detail of those commands belongs to #1110. +The exact assignment of each command is made when the gate is built, from the command tree under `cmd/cli/commands`. ### 5.4 Clusters installed by older versions A strict gate must not lock operators out of clusters that older versions installed. Two things could be missing on such a machine, and neither needs a separate migration: - **Recorded hashes.** Software state records only the name, version, `Installed` and `Configured`; it has never held a binary hash. The gate does not need one. The embedded catalog has the expected checksum for each software, version and platform, which the installer already checks at download. The gate verifies that the sandbox binary on disk matches the catalog checksum for the version recorded in state, and that the running process is that binary. -- **The cluster ID.** `hosts/local/host.yaml` has none on an older install. The first time all the checks in 5.1 pass, the gate records the live cluster ID. This is trust on first use, and it is safe because the binary, process, `admin.conf` and IP checks have already passed. Running `solo-provisioner kube cluster connect` is an optional way to register eagerly and see each check reported (section 4.1); nothing stops working if it is skipped. +- **The recorded identity.** `hosts/local/host.yaml` has no cluster ID, server address or CA fingerprint on an older install. The first time all the local evidence passes and the live cluster ID can be read (rule 6), the gate records all three. This is trust on first use, and it is safe because the binary, process, `admin.conf` and address checks have already passed. Running `solo-provisioner kube cluster connect` is an optional way to register eagerly and see each check reported (section 4.1); nothing stops working if it is skipped. Catalog gaps: the catalog is refreshed and split over time (#1262), so an old kubelet or CRI-O version may no longer be in it. If the recorded version is not in the catalog, the checksum check is skipped with a warning, and the gate relies on the other checks: the binary is in the sandbox path and the running process is that binary. A version that is in the catalog but does not match its checksum still stops the command. @@ -223,7 +244,7 @@ State is grouped by host. A cluster with no host on this machine gets its own di Rules: - `provisioner.yaml` describes the binary, not a host or a cluster. It is always loadable, the binary reads it first before choosing a host or a cluster, and it exists on every machine, including a Mac with only external clusters (`~/.solo-weaver/state/provisioner.yaml`). Startup migrations depend on the recorded provisioner version, and without this file a machine with no `hosts/local` would have nowhere to record it. Host and cluster state never write it. -- `hosts/local/host.yaml` is a small identity file that the identity gate reads first, before any component file, so it needs no component lock. It holds the host `name`, `kind` (`local`; `remote` later), the `machineId` from `/etc/machine-id`, and `managedClusterId`, the cluster this host provisioned (empty when none). A later remote host (section 9) uses the same file with extra registry fields. If the recorded `machineId` no longer matches the machine, for example because the directory was copied, the command stops. If the machine has no usable `/etc/machine-id` (some containers), the check is skipped and no ID is recorded. +- `hosts/local/host.yaml` is a small identity file that the identity gate reads first, before any component file, so it needs no component lock. It holds the host `name`, `kind` (`local`; `remote` later), the `machineId` from `/etc/machine-id`, and the identity of the cluster this host provisioned (empty when none): `managedClusterId`, `serverAddress` and `caFingerprint`, which the gate uses to recognise it even when its API is down (section 5.1). A later remote host (section 9) uses the same file with extra registry fields. If the recorded `machineId` no longer matches the machine, for example because the directory was copied, the command stops. If the machine has no usable `/etc/machine-id` (some containers), the check is skipped and no ID is recorded. - `hosts/local/` otherwise keeps today's component files and content. - `clusters//` holds **only cluster-scoped components**. Host components (`profile`, `firewall`, `system`) never exist there, so external-mode code cannot reach host state. Hashes and locks work per file, as in #1235. - Each file records its own `clusterId` and `origin`, and the loader checks them against the live cluster. A copied file is rejected. @@ -385,7 +406,7 @@ Order: ## 12. Decisions -1. **Gate exemptions.** The list in section 5.3 is the rule. Diagnostic commands run the gate in report-only mode. +1. **What each command needs.** Each command declares a need (`none`, `connect`, `install`, `host`, `api` or `host+api`), and the gate checks only that (section 5.3). Diagnostic commands run the gate in report-only mode. 2. **Uninstall and reinstall.** The normal path is at uninstall: `solo-provisioner kube cluster uninstall` moves the cluster-scoped component files (`cluster`, `blocknode`, `consensus` and the cluster half of `teleport`) to `hosts/local/archive/-/`. Host-scoped components (software, firewall, shaping) stay, because the machine is still provisioned. It also clears the cluster ID recorded in `host.yaml`. A later `solo-provisioner kube cluster install` then starts clean. Nothing is deleted. If `solo-provisioner kube cluster install` finds that `hosts/local/host.yaml` records a cluster ID that differs from the live one (or a cluster that no longer exists), it **stops and does not touch the state**. Weaver cannot tell why: a `kubeadm reset` by hand, a failed or older uninstall, a rebuilt VM with surviving state, or the wrong kubeconfig all look the same, so moving the state automatically would be a guess. The error names the recorded and live cluster IDs and tells the operator to review and clean up the state by hand. The hint lists the files to move or remove and the paths they live in. @@ -393,7 +414,7 @@ Order: 3. **Cluster-side lock.** A Lease per component per cluster, in the component's namespace, in both modes, built on `leaderelection` (section 5.5). 4. **External state is a local cache.** Almost all of it can be re-derived from the cluster, and the cluster wins when they disagree. Storing state in the cluster is a follow-up with its own design (RBAC, ownership). 5. **Garbage collection.** Nothing is deleted automatically, since a cluster that is only unreachable for a moment must not lose its state. A later `solo-provisioner state prune` lists clusters that are unreachable or whose ID no longer exists, and asks before deleting. -6. **Older installs.** No separate backfill. The gate checks binaries against the catalog checksum and records the cluster ID on its first successful pass (section 5.4). +6. **Older installs.** No separate backfill. The gate checks binaries against the catalog checksum and records the cluster identity (ID, server address and CA fingerprint) on its first successful pass (section 5.4). 7. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State, including `provisioner.yaml`, lives in `~/.solo-weaver/state`. 8. **Cluster registration.** One explicit command, `solo-provisioner kube cluster connect`, registers a cluster. Non-interactive runs register implicitly. `list` and `info` read the index and the cluster. `disconnect` is later work (section 4.1). From 4f7033a699a622263086de410757af0c397b8d16 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 23:00:17 +1100 Subject: [PATCH 12/21] docs(design): harden the identity gate, lease and provisioner.yaml Clarify that only api/host+api needs run the full outcome table and the host need is an offline-only check. Match recorded identity on the CA fingerprint, with the server address as fallback only. Give provisioner.yaml a cli and daemon section so each binary records its own version and both detect skew. Note that fencing needs ctx-cancellable steps and that the gate reads state through the #1235 loader. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index f97d62d7..ea59705e 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -102,7 +102,7 @@ A check in the root pre-run, **before** any RSL or reality check. It decides whi The gate prevents accidents: running against the wrong cluster, or against state left behind by another one. It is not a security boundary. A user with root can fake any of its evidence. -The gate reads only `hosts/local/host.yaml` and the software records, read-only and without locks. It does not go through RSL, reality or the state manager. +The gate reads only `hosts/local/host.yaml` and the software records, read-only and without locks. It does not go through RSL, reality or the state manager, but it reads these files through #1235's component loader (`ComponentFilePath`), not a second parser, so the on-disk format has one reader. Atomic writes (#1235 replaces each file by rename) make a lock-free read safe. ### 5.1 Evidence @@ -119,7 +119,7 @@ The gate separates evidence that needs no API from evidence that needs it, so a The address check alone is not enough. A Kind or minikube cluster on the same machine points at `127.0.0.1` and would pass it. The binary and process checks rule those out. -**Recorded identity** (also no API needed). `hosts/local/host.yaml` records, after the first successful pass, the cluster ID, the server address and the CA certificate fingerprint of the cluster this host provisioned. A kubeconfig **matches the recorded identity** when its server address or its CA fingerprint equals the recorded one. This works while the cluster is down, which the cluster ID alone cannot. +**Recorded identity** (also no API needed). `hosts/local/host.yaml` records, after the first successful pass, the cluster ID, the server address and the CA certificate fingerprint of the cluster this host provisioned. A kubeconfig **matches the recorded identity** when its CA fingerprint equals the recorded one. The server address is a fallback, used only when no fingerprint is recorded, which can happen on an older install. The fingerprint is the real identity. A server address can be reused: a different cluster rebuilt on the same `IP:6443` would match on the address alone, so weaver would route it through the "this is our cluster" branch (rules 2 to 5) and report it as "rebuilt, state stale" (rule 3) instead of as external (rule 9). The live cluster-ID check still stops any state corruption in that case. Preferring the fingerprint only makes the message the operator sees correct. This works while the cluster is down, which the cluster ID alone cannot. **Live identity** (needs the API). The cluster ID, the UID of the `kube-system` namespace, read from the live cluster. @@ -131,6 +131,8 @@ Assumption, to be confirmed against the kubeadm workflow: a host runs one single The rules are checked in order. The first one that applies wins. +Only the needs that reach the live cluster, `api` and `host+api` (section 5.3), evaluate this full table. The `host`, `install` and `connect` needs run a reduced check that never reads the API, so the "API unreachable" half of rules 3, 4 and 8 does not apply to them. Section 5.3 says exactly what each need checks. + | # | Situation | Result | |---|---|---| | 1 | The recorded `machineId` in `host.yaml` does not match this machine (for example the directory was copied) | Stop. Hint: the state belongs to another machine | @@ -166,7 +168,7 @@ A fixed exempt list does not fit, so each command declares what it needs, and th | `none` | No cluster and no host evidence | `solo-provisioner install`, `solo-provisioner uninstall`, `solo-provisioner version`, `solo-provisioner demo`, `--help`, `solo-provisioner kube cluster list` | Nothing | | `connect` | Classifies the target itself | `solo-provisioner kube cluster connect` | Runs the evidence and outcomes in 5.1 and 5.2 to classify the target, and registers it instead of stopping | | `install` | Creates the cluster | `solo-provisioner kube cluster install` | Allows no cluster, or stale state (rule 3, which this command resolves). Refuses when a cluster that is not ours is already on this host, which means `/etc/kubernetes/admin.conf` exists but the evidence does not show it is ours | -| `host` | Host-level work that does not need the API, and must also work on a broken cluster | `solo-provisioner kube cluster uninstall`, `solo-provisioner network firewall\|shape\|policy ...`, `solo-provisioner teleport node ...` | Checks the machine ID (rule 1), and that the run is not in external mode. Continues when the local cluster is unhealthy (rule 4) | +| `host` | Host-level work that does not need the API, and must also work on a broken cluster | `solo-provisioner kube cluster uninstall`, `solo-provisioner network firewall\|shape\|policy ...`, `solo-provisioner teleport node ...` | Offline check only: the machine ID matches (rule 1), and the run is not in external mode (no `--kubeconfig` points elsewhere). It never reads the API, so a down or unhealthy local cluster does not stop it. Rules 2 to 10 are not evaluated | | `api` | Work through the Kubernetes API in either mode | Block node, consensus node, alloy, ESO and teleport cluster commands that do not touch the host | The full outcomes in 5.2. The result may be `provisioned` or `external` | | `host+api` | Provisioned-only work that also needs the cluster | Commands that run host steps and API steps, such as block node install on a provisioned host | The full outcomes in 5.2, and the result must be `provisioned` | @@ -196,9 +198,9 @@ Decisions: - **Library.** `k8s.io/client-go/tools/leaderelection`, wrapped in a small internal package, `internal/kube/lease`. It handles the compare-and-swap on `resourceVersion`, renewal, and loss detection. Adding it needs `go mod vendor`, as for any dependency. - **Wrapper API.** `TryAcquire(ctx, component)` returns a handle with a context that is cancelled when the lease is lost, and `Release()`. - **Fail fast.** Before starting the elector, read the Lease. If it is held and not expired, stop and print the holder: user, host (machine ID), command line, start time and weaver version, taken from the holder identity (JSON), and how long until it expires. If the elector loses a race, a short acquire timeout turns that into the same error. -- **Fencing.** If renewal fails past the renew deadline, the wrapper cancels the workflow's context and the command aborts. A laptop that sleeps past expiry can therefore not resume and keep changing a cluster another operator now holds. +- **Fencing.** If renewal fails past the renew deadline, the wrapper cancels the workflow's context and the command aborts. A laptop that sleeps past expiry can therefore not resume and keep changing a cluster another operator now holds. This only stops a mutation if the step in flight honours context cancellation, so the mutating steps must return promptly on `ctx.Done()` rather than run a long host or API call to completion. Auditing the mutating steps for this is part of delivery step 2. - **Expiry only.** The lease is taken over only after it expires, because a live holder keeps renewing. There is no other automatic takeover. An explicit override flag (name to be decided) lets an operator remove a lease they know is dead. -- **Timings.** Starting values: lease duration 30 seconds, renew deadline 20 seconds, retry period 5 seconds. To be tuned against slow API servers. +- **Timings.** Starting values: lease duration 30 seconds, renew deadline 20 seconds, retry period 5 seconds. To be tuned against slow API servers, where a renewal that is merely slow, not lost, could fence a legitimate long operation; the renew deadline must stay comfortably above the worst-case round trip to the API. - **First install.** The cluster or the component namespace may not exist yet. Cluster install is covered by the local `flock` only. For a component namespace that does not exist, create the namespace first (an `AlreadyExists` error is fine) and then take the lease. The race window is only the namespace creation. - **RBAC.** External operators need `get`, `create`, `update` and `delete` on `leases` in the component namespace. The preflight checks this with `SelfSubjectAccessReview` and fails with a clear hint. - **Daemon.** The daemon takes the same lease when it writes consensus state, so it excludes CLI runs. @@ -221,7 +223,7 @@ State is grouped by host. A cluster with no host on this machine gets its own di ``` /opt/solo/weaver/state/ -├── provisioner.yaml # binary-level: provisioner version, commit, executable, state model version +├── provisioner.yaml # binary-level: cli and daemon sections, each with version, commit, executable, state model version ├── hosts/ │ └── local/ # this machine + the cluster it provisioned │ ├── host.yaml # identity: name, machine ID, the cluster this host provisioned @@ -243,7 +245,7 @@ State is grouped by host. A cluster with no host on this machine gets its own di Rules: -- `provisioner.yaml` describes the binary, not a host or a cluster. It is always loadable, the binary reads it first before choosing a host or a cluster, and it exists on every machine, including a Mac with only external clusters (`~/.solo-weaver/state/provisioner.yaml`). Startup migrations depend on the recorded provisioner version, and without this file a machine with no `hosts/local` would have nowhere to record it. Host and cluster state never write it. +- `provisioner.yaml` describes the binaries, not a host or a cluster. It is always loadable, the binary reads it first before choosing a host or a cluster, and it exists on every machine, including a Mac with only external clusters (`~/.solo-weaver/state/provisioner.yaml`). Startup migrations depend on the recorded provisioner version, and without this file a machine with no `hosts/local` would have nowhere to record it. Host and cluster state never write it. The file holds one section per binary, `cli` and `daemon`, each with that binary's version, commit, executable path and state-model version. The CLI and the daemon ship and upgrade on their own schedules, and the daemon can self-upgrade (epic #500) to a version the CLI never installed, so each binary is the authority on its own version: each writes only its own section and reads the other's. This gives both binaries a reliable view of each other's version and existence, so either can warn on a skew. Because two binaries write one file, a write is a read-modify-write under the file's lock (#1235), not a blind atomic replace, so neither section clobbers the other; reads stay lock-free. The daemon section is empty until a daemon is installed. Startup migrations are a CLI concern and target the version in the `cli` section. - `hosts/local/host.yaml` is a small identity file that the identity gate reads first, before any component file, so it needs no component lock. It holds the host `name`, `kind` (`local`; `remote` later), the `machineId` from `/etc/machine-id`, and the identity of the cluster this host provisioned (empty when none): `managedClusterId`, `serverAddress` and `caFingerprint`, which the gate uses to recognise it even when its API is down (section 5.1). A later remote host (section 9) uses the same file with extra registry fields. If the recorded `machineId` no longer matches the machine, for example because the directory was copied, the command stops. If the machine has no usable `/etc/machine-id` (some containers), the check is skipped and no ID is recorded. - `hosts/local/` otherwise keeps today's component files and content. - `clusters//` holds **only cluster-scoped components**. Host components (`profile`, `firewall`, `system`) never exist there, so external-mode code cannot reach host state. Hashes and locks work per file, as in #1235. @@ -333,7 +335,7 @@ Prerequisites for running this on macOS: Add a startup migration, `state-hosts-layout`, registered in `RegisterMigrations()` (`cmd/cli/commands/root.go`) **after** the migration that splits `state.yaml` into component files (#1231). See `docs/dev/migration-framework.md`. The chain is: `unified-state`, then per-component split, then `state-hosts-layout`. Each step is idempotent. - `Applies`: component files exist directly under `state/` and `state/hosts/local/` does not. -- `Execute`: create `hosts/local/` with the same owner and mode, move the component files (and their lock and hash companions) into it, and keep a backup of the originals as `state/.pre-hosts-layout/`. It moves files and does not rewrite their content, with one exception: it copies `ProvisionerState` and `Version` out of the machine-level file into `state/provisioner.yaml`, and writes `hosts/local/host.yaml` (name, kind, machine ID, and the cluster ID if one is already recorded). +- `Execute`: create `hosts/local/` with the same owner and mode, move the component files (and their lock and hash companions) into it, and keep a backup of the originals as `state/.pre-hosts-layout/`. It moves files and does not rewrite their content, with one exception: it copies `ProvisionerState` and `Version` out of the machine-level file into the `cli` section of `state/provisioner.yaml` (the `daemon` section stays empty until a daemon writes it, section 6.1), and writes `hosts/local/host.yaml` (name, kind, machine ID, and the cluster ID if one is already recorded). - If both locations have files, prefer `hosts/local/`, leave the legacy files untouched, and log a warning. Never overwrite. - Idempotent and crash-safe. `Rollback` restores the backup on a best-effort basis. - It runs before anything opens the state manager, and before the identity gate. Migration is pure filesystem work and needs no cluster. @@ -400,7 +402,7 @@ Order: 1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the handling of older installs in section 5.4). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. 2. Cluster-side lock (`internal/kube/lease`) and its use in the mutating commands (section 5.5). It needs the resolved cluster from step 1 and nothing from the state work. -3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, `provisioner.yaml`, `hosts/local/host.yaml`, the cluster-scoped component files for `clusters//`, the host-bound field guard in section 6.2, `clusters/index.yaml`, and the `kube cluster connect`, `list` and `info` commands. +3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, `provisioner.yaml`, `hosts/local/host.yaml`, the cluster-scoped component files for `clusters//`, the host-bound field guard in section 6.2, `clusters/index.yaml`, and the `kube cluster connect`, `list` and `info` commands. This step also gives the daemon write access to its own section of `provisioner.yaml` under the #1235 lock (section 6.1); the daemon is a separate binary, so that change ships on its release schedule, not the CLI's. 4. External-mode workflows, block node first, with the StorageClass storage path. 5. CI: Kind lane using `--kubeconfig`, and the test split by required privilege. From 1bf979c293e485fbc41317a5cd51a99b8b559d28 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Thu, 15 Oct 2026 23:13:00 +1100 Subject: [PATCH 13/21] docs(design): group the identity-gate outcomes into three branches Reshape the section 5.2 table into a precondition plus three branches (target matches recorded identity, nothing recorded, not our cluster), so the "first match wins" precedence and the split-brain reasoning read off the structure. Stable rule numbers are kept for cross-references. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 55 +++++++++++++++++++----------- 1 file changed, 36 insertions(+), 19 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index ea59705e..bcd4d733 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -119,7 +119,7 @@ The gate separates evidence that needs no API from evidence that needs it, so a The address check alone is not enough. A Kind or minikube cluster on the same machine points at `127.0.0.1` and would pass it. The binary and process checks rule those out. -**Recorded identity** (also no API needed). `hosts/local/host.yaml` records, after the first successful pass, the cluster ID, the server address and the CA certificate fingerprint of the cluster this host provisioned. A kubeconfig **matches the recorded identity** when its CA fingerprint equals the recorded one. The server address is a fallback, used only when no fingerprint is recorded, which can happen on an older install. The fingerprint is the real identity. A server address can be reused: a different cluster rebuilt on the same `IP:6443` would match on the address alone, so weaver would route it through the "this is our cluster" branch (rules 2 to 5) and report it as "rebuilt, state stale" (rule 3) instead of as external (rule 9). The live cluster-ID check still stops any state corruption in that case. Preferring the fingerprint only makes the message the operator sees correct. This works while the cluster is down, which the cluster ID alone cannot. +**Recorded identity** (also no API needed). `hosts/local/host.yaml` records, after the first successful pass, the cluster ID, the server address and the CA certificate fingerprint of the cluster this host provisioned. A kubeconfig **matches the recorded identity** when its CA fingerprint equals the recorded one. The server address is a fallback, used only when no fingerprint is recorded, which can happen on an older install. The fingerprint is the real identity. A server address can be reused: a different cluster rebuilt on the same `IP:6443` would match on the address alone, so weaver would route it through the "this is our cluster" branch (branch A) and report it as "rebuilt, state stale" (rule A.4) instead of as external (rule C.1). The live cluster-ID check still stops any state corruption in that case. Preferring the fingerprint only makes the message the operator sees correct. This works while the cluster is down, which the cluster ID alone cannot. **Live identity** (needs the API). The cluster ID, the UID of the `kube-system` namespace, read from the live cluster. @@ -129,26 +129,43 @@ Assumption, to be confirmed against the kubeadm workflow: a host runs one single ### 5.2 Outcomes -The rules are checked in order. The first one that applies wins. +Only the needs that reach the live cluster, `api` and `host+api` (section 5.3), evaluate these rules. The `host`, `install` and `connect` needs run a reduced check that never reads the API, so the "API unreachable" leaves do not apply to them. Section 5.3 says exactly what each need checks. -Only the needs that reach the live cluster, `api` and `host+api` (section 5.3), evaluate this full table. The `host`, `install` and `connect` needs run a reduced check that never reads the API, so the "API unreachable" half of rules 3, 4 and 8 does not apply to them. Section 5.3 says exactly what each need checks. +The outcomes fall into three branches: the target matches the recorded identity, nothing is recorded yet, or the target is not this host's cluster. One precondition is checked before all of them. Within a branch the rows are checked in order and the first match wins. The `#` column (for example A.1) is the stable rule id used elsewhere in this document. -| # | Situation | Result | +**Precondition (all branches):** + +| # | Condition | Result | +|---|---|---| +| P.1 | The recorded `machineId` in `host.yaml` does not match this machine (for example the directory was copied) | Stop. Hint: the state belongs to another machine | + +**Branch A — the target matches the recorded identity** (CA fingerprint; server address only as a fallback). Reads the API. + +| # | Condition | Result | +|---|---|---| +| A.1 | A local check fails, or the API is unreachable | Stop. The hint names the failing piece (for example kubelet is installed but not running). **Never** fall back to `external` | +| A.2 | A local check could not be read (for example no root) | Stop. Hint: run with `sudo` | +| A.3 | Local evidence passes, the API is reachable, and the live cluster ID matches the recorded one | `provisioned`, host `local`, state in `hosts/local` | +| A.4 | Local evidence passes, the API is reachable, but the live cluster ID differs | Stop. The cluster was rebuilt and the state is stale (decision 2 in section 12) | + +**Branch B — nothing is recorded, but local evidence passes** (an older install, section 5.4; trust on first use). Reads the API. + +| # | Condition | Result | +|---|---|---| +| B.1 | The API is unreachable | Stop. The cluster ID cannot be read. Hint: start the cluster, or run `solo-provisioner kube cluster connect` once it is up | +| B.2 | The live cluster ID already has `clusters/` state | Stop. Needs an explicit step; it is never adopted silently (see below) | +| B.3 | The live cluster ID has no `clusters/` state | `provisioned`. Record the identity | + +**Branch C — the target is not this host's cluster** (an identity is recorded but the target does not match it, or nothing is recorded and local evidence does not pass). Decided from `--kubeconfig`; no API needed. + +| # | Condition | Result | |---|---|---| -| 1 | The recorded `machineId` in `host.yaml` does not match this machine (for example the directory was copied) | Stop. Hint: the state belongs to another machine | -| 2 | The target matches the recorded identity, local evidence passes, the API is reachable, and the live cluster ID matches the recorded one | `provisioned`, host `local`, state in `hosts/local` | -| 3 | The target matches the recorded identity, local evidence passes, the API is reachable, but the live cluster ID differs | Stop. The cluster was rebuilt and the state is stale (decision 2 in section 12) | -| 4 | The target matches the recorded identity, but a local check fails or the API is unreachable | Stop. The hint names the failing piece (for example kubelet is installed but not running). **Never** fall back to `external` | -| 5 | The target matches the recorded identity, but a check could not be read (for example no root) | Stop. Hint: run with `sudo` | -| 6 | Nothing is recorded, local evidence passes (an older install, section 5.4), and the live cluster ID has no `clusters/` state | `provisioned`. Record the identity | -| 7 | Nothing is recorded, local evidence passes, and the live cluster ID already has `clusters/` state | Stop. Needs an explicit step; it is never adopted silently (see below) | -| 8 | Nothing is recorded, local evidence passes, but the API is unreachable | Stop. The cluster ID cannot be read. Hint: start the cluster, or run `solo-provisioner kube cluster connect` once it is up | -| 9 | The target is not this host's cluster, and `--kubeconfig` was given | `external`, state in `clusters/`. The cluster must be registered (section 4.1): an interactive run stops with a hint to run `solo-provisioner kube cluster connect`, a non-interactive run registers it first | -| 10 | The target is not this host's cluster, and no `--kubeconfig` was given | Stop. If nothing is installed here, the hint is to run `solo-provisioner kube cluster install` or pass `--kubeconfig` for an external cluster. If a provisioned install is only partly present, the hint names what is missing | +| C.1 | `--kubeconfig` was given | `external`, state in `clusters/`. The cluster must be registered (section 4.1): an interactive run stops with a hint to run `solo-provisioner kube cluster connect`, a non-interactive run registers it first | +| C.2 | No `--kubeconfig` was given | Stop. If nothing is installed here, the hint is to run `solo-provisioner kube cluster install` or pass `--kubeconfig` for an external cluster. If a provisioned install is only partly present, the hint names what is missing | -Rules 3 and 7 prevent split-brain between the state locations. Rule 4 is the one that keeps a broken local cluster from being treated as someone else's. +Branches A and C together prevent split-brain between the state locations: a cluster that does not match the recorded identity can only become `external`, never overwrite `hosts/local`. Rules A.4 and B.2 cover the two ways one cluster ID could end up described twice. Rule A.1 is the one that keeps a broken local cluster from being treated as someone else's. -#### Rule 7: a cluster that was external and now passes the local proof +#### Rule B.2: a cluster that was external and now passes the local proof This can happen when a cluster on the machine was not recognised at first (for example kubelet was down, or it was built by hand), so a command with `--kubeconfig` treated it as `external` and created `clusters//`. Later the proof passes and the same cluster ID now qualifies as `provisioned`. Two places could then describe one cluster, and the records may disagree. Picking one silently could ignore the block node and consensus information in the other, and merging automatically could overwrite newer facts with older ones. So weaver stops and asks. @@ -167,8 +184,8 @@ A fixed exempt list does not fit, so each command declares what it needs, and th |---|---|---|---| | `none` | No cluster and no host evidence | `solo-provisioner install`, `solo-provisioner uninstall`, `solo-provisioner version`, `solo-provisioner demo`, `--help`, `solo-provisioner kube cluster list` | Nothing | | `connect` | Classifies the target itself | `solo-provisioner kube cluster connect` | Runs the evidence and outcomes in 5.1 and 5.2 to classify the target, and registers it instead of stopping | -| `install` | Creates the cluster | `solo-provisioner kube cluster install` | Allows no cluster, or stale state (rule 3, which this command resolves). Refuses when a cluster that is not ours is already on this host, which means `/etc/kubernetes/admin.conf` exists but the evidence does not show it is ours | -| `host` | Host-level work that does not need the API, and must also work on a broken cluster | `solo-provisioner kube cluster uninstall`, `solo-provisioner network firewall\|shape\|policy ...`, `solo-provisioner teleport node ...` | Offline check only: the machine ID matches (rule 1), and the run is not in external mode (no `--kubeconfig` points elsewhere). It never reads the API, so a down or unhealthy local cluster does not stop it. Rules 2 to 10 are not evaluated | +| `install` | Creates the cluster | `solo-provisioner kube cluster install` | Allows no cluster, or stale state (rule A.4, which this command resolves). Refuses when a cluster that is not ours is already on this host, which means `/etc/kubernetes/admin.conf` exists but the evidence does not show it is ours | +| `host` | Host-level work that does not need the API, and must also work on a broken cluster | `solo-provisioner kube cluster uninstall`, `solo-provisioner network firewall\|shape\|policy ...`, `solo-provisioner teleport node ...` | Offline check only: the machine ID matches (rule P.1), and the run is not in external mode (no `--kubeconfig` points elsewhere). It never reads the API, so a down or unhealthy local cluster does not stop it. Branches A, B and C are not evaluated | | `api` | Work through the Kubernetes API in either mode | Block node, consensus node, alloy, ESO and teleport cluster commands that do not touch the host | The full outcomes in 5.2. The result may be `provisioned` or `external` | | `host+api` | Provisioned-only work that also needs the cluster | Commands that run host steps and API steps, such as block node install on a provisioned host | The full outcomes in 5.2, and the result must be `provisioned` | @@ -181,7 +198,7 @@ The exact assignment of each command is made when the gate is built, from the co A strict gate must not lock operators out of clusters that older versions installed. Two things could be missing on such a machine, and neither needs a separate migration: - **Recorded hashes.** Software state records only the name, version, `Installed` and `Configured`; it has never held a binary hash. The gate does not need one. The embedded catalog has the expected checksum for each software, version and platform, which the installer already checks at download. The gate verifies that the sandbox binary on disk matches the catalog checksum for the version recorded in state, and that the running process is that binary. -- **The recorded identity.** `hosts/local/host.yaml` has no cluster ID, server address or CA fingerprint on an older install. The first time all the local evidence passes and the live cluster ID can be read (rule 6), the gate records all three. This is trust on first use, and it is safe because the binary, process, `admin.conf` and address checks have already passed. Running `solo-provisioner kube cluster connect` is an optional way to register eagerly and see each check reported (section 4.1); nothing stops working if it is skipped. +- **The recorded identity.** `hosts/local/host.yaml` has no cluster ID, server address or CA fingerprint on an older install. The first time all the local evidence passes and the live cluster ID can be read (rule B.3), the gate records all three. This is trust on first use, and it is safe because the binary, process, `admin.conf` and address checks have already passed. Running `solo-provisioner kube cluster connect` is an optional way to register eagerly and see each check reported (section 4.1); nothing stops working if it is skipped. Catalog gaps: the catalog is refreshed and split over time (#1262), so an old kubelet or CRI-O version may no longer be in it. If the recorded version is not in the catalog, the checksum check is skipped with a warning, and the gate relies on the other checks: the binary is in the sandbox path and the running process is that binary. A version that is in the catalog but does not match its checksum still stops the command. From 62f1b1aff977777758518f6d3fa167937c6ed3a3 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Fri, 16 Oct 2026 12:16:55 +1100 Subject: [PATCH 14/21] docs(design): gate external-cluster support as provisioner-test-only experimental Add a top banner and a decision making external mode (--kubeconfig against a cluster weaver did not provision) a provisioner-test-only feature, not for production and not encouraged in operators' CI or clusters. Reuse the existing common.GateExperimental gate (feature external-cluster): external mode and kube cluster connect require --experimental or SOLO_PROVISIONER_ENABLE_EXTERNAL_CLUSTER and are refused otherwise. Reframe goal 2, section 7, the Kind section and the CI section to match. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index bcd4d733..4c74085c 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -2,6 +2,17 @@ Status: **Draft for review.** Nothing in this document is implemented yet. +> **IMPORTANT: external-cluster support is a provisioner-test-only feature.** +> Pointing weaver at a cluster it did not provision (`--kubeconfig`, "external +> mode") exists so we can test `solo-provisioner` itself against a throwaway +> cluster such as Kind, without a KVM or a privileged runner. It is **not** a +> supported way to run block node, consensus node or any other component in +> production, and we do not encourage operators to use it in their own CI or +> against their own clusters. It is gated behind the experimental feature flag +> (`--experimental` / `SOLO_PROVISIONER_ENABLE_EXTERNAL_CLUSTER`) and refuses to +> run without it. The provisioned, single-node path on a Linux host stays the +> only supported way to operate a real deployment. + Depends on the per-component state files work: #1231 (PR #1235) and #1245. This design is written in terms of those component files and should be built after they merge. See section 6. ## 1. Problem @@ -21,7 +32,7 @@ We also want to run most UAT against a cluster we did not install (for example K Goals: 1. Never trust a cluster as ours unless we can prove we installed it on this machine. Check this as early as possible, and stop if the proof fails. -2. Let an operator explicitly point the CLI at any cluster (`--kubeconfig`) and use it for API-level work: install, upgrade and uninstall of block node, consensus node and the other cluster components. +2. For testing `solo-provisioner` itself only, and behind the experimental gate, let the CLI point at a cluster it did not provision (`--kubeconfig`) and do API-level work there: install, upgrade and uninstall of block node, consensus node and the other cluster components. This is not for production use (see the banner above). 3. Keep state separate per cluster, so switching between clusters, or forgetting a flag, can never mix or reset state. 4. One place decides which kubeconfig is used. 5. Leave room to manage several VMs from macOS later. @@ -64,10 +75,10 @@ Without `--kubeconfig`, ambient `$KUBECONFIG` and `~/.kube/config` are not trust ### 4.1 Registering a cluster -A cluster is registered once, by one command, and the rest of the commands refer to it: +A cluster is registered once, by one command, and the rest of the commands refer to it. `connect` is part of the experimental, provisioner-test-only external-cluster feature (see the banner at the top), so it is hidden from `--help` and refuses to run without `--experimental` or `SOLO_PROVISIONER_ENABLE_EXTERNAL_CLUSTER`: ```bash -solo-provisioner kube cluster connect --kubeconfig [--context ] [--cluster-name ] +solo-provisioner kube cluster connect --experimental --kubeconfig [--context ] [--cluster-name ] ``` `connect` does these steps, in order: @@ -326,6 +337,8 @@ The alias binds to a cluster ID when the cluster is registered with `solo-provis > In external mode weaver only talks to the Kubernetes API. It touches nothing on the host. +External mode is experimental and provisioner-test-only (see the banner at the top). It is reached only when the experimental gate is on, through the hidden `--experimental` flag or `SOLO_PROVISIONER_ENABLE_EXTERNAL_CLUSTER=1`, reusing `common.GateExperimental` (`cmd/cli/commands/common/experimental_gate.go`). Without the gate, `--kubeconfig` pointing at a cluster that is not this host's own is refused with a reason code and a hint, and `solo-provisioner kube cluster connect` is hidden from `--help`. + - Skipped: node setup, kubeadm setup, network setup (nft, tc), bind mounts, sysctl, swap, weaver user, daemon config. - Storage: hostPath PVs refer to a node filesystem weaver cannot own, so they are not used. Storage comes from the cluster through a `storageClassName` per volume (flag or values file). Weaver creates PVCs only, never PVs. The hostPath templates (`storage-config.yaml`, `optional-storage.yaml`) are provisioned-only. - Preflight checks API-level facts only: server version range, access via `SelfSubjectAccessReview`, and that the CNI and StorageClass exist. @@ -335,7 +348,7 @@ The alias binds to a cluster ID when the cluster is registered with `solo-provis ### 7.1 Local development with Kind -A developer can create a Kind cluster on their own machine (including a Mac) and use it with weaver. It does not need SSH delegation (section 9). +To test `solo-provisioner` itself, a developer can create a Kind cluster on their own machine (including a Mac) and use it with weaver, behind the experimental gate. It does not need SSH delegation (section 9). This is for exercising weaver against a throwaway cluster, not for running a real deployment on Kind. - Weaver did not install it, so it is `origin: external`. It gets a `clusters//` directory and an index entry with the context (for example `kind-dev`) and a `127.0.0.1` server address. - Register it once with `solo-provisioner kube cluster connect`, or let a non-interactive run register it. Use it with `--kubeconfig` and `--context`. Only API-level work runs. Storage comes from Kind's default StorageClass. @@ -379,7 +392,7 @@ Not in scope now, but the layout leaves room. ## 10. Effect on CI -External mode is what makes a Kind lane possible on the ARC runners (`solo-linux-medium`) without privileged pods. Tests then split into: +This section is about `solo-provisioner`'s own CI, which is the one place external mode is meant to run. It is not a recommendation for operators' pipelines. External mode is what makes a Kind lane possible on the ARC runners (`solo-linux-medium`) without privileged pods. Tests then split into: | Lane | Runner | Content | |---|---|---| @@ -438,6 +451,8 @@ Order: 8. **Cluster registration.** One explicit command, `solo-provisioner kube cluster connect`, registers a cluster. Non-interactive runs register implicitly. `list` and `info` read the index and the cluster. `disconnect` is later work (section 4.1). +9. **External mode is experimental and provisioner-test-only.** External mode exists to test `solo-provisioner` against a throwaway cluster (such as Kind) without a KVM or privileged runner. It is not supported for running real deployments, and we do not encourage operators to use it in their own CI or clusters. It is gated with `common.GateExperimental` (feature `external-cluster`): `--kubeconfig` against a non-provisioned cluster, and `solo-provisioner kube cluster connect`, both require `--experimental` or `SOLO_PROVISIONER_ENABLE_EXTERNAL_CLUSTER`, and are refused otherwise. The provisioned single-node Linux path is unaffected and needs no flag. + ## 13. Still open - What `solo-provisioner kube cluster uninstall` does with state today. This decides whether archive-at-uninstall is new behaviour or a fix. From bda685e0c8e47b52ceff12eba1312246596724b2 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Fri, 16 Oct 2026 22:04:26 +1100 Subject: [PATCH 15/21] docs(design): rename the identity gate to the provenance gate "Identity" was overloaded across the host, the cluster, the recorded host-to-cluster binding and the target. Rename the gate to the provenance gate and adopt one noun per thing: HostIdentity (machine-id), ClusterIdentity (kube-system UID), provision record (what this host recorded about the cluster it provisioned), Target, and Origin (the provisioned/external verdict; a refusal is a stop, not an origin). Add a Terms table to section 5. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 48 +++++++++++++++++++----------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 4c74085c..7abc956a 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -57,7 +57,7 @@ Non-goals for now: | Origin `provisioned` | `solo-provisioner` installed this cluster (kubeadm, CRI-O, kubelet) on a host. | | Origin `external` | Someone else's cluster. We only talk to its API. | -The mode is **derived, not declared**. The `--kubeconfig` flag only chooses which cluster to look at. The identity check (section 5) decides whether it is `provisioned` on this host or `external`. This is what stops flag flip-flopping from corrupting state. +The mode is **derived, not declared**. The `--kubeconfig` flag only chooses which cluster to look at. The provenance check (section 5) decides whether it is `provisioned` on this host or `external`. This is what stops flag flip-flopping from corrupting state. ## 4. Cluster resolution @@ -71,7 +71,7 @@ New root persistent flags: | `--context ` | Context inside the kubeconfig. | | `--cluster-name ` | Optional human label, stored in `clusters/index.yaml` when the cluster is registered (section 4.1). Defaults to the context name. Never used as a key. | -Without `--kubeconfig`, ambient `$KUBECONFIG` and `~/.kube/config` are not trusted on their own. Weaver already copies `admin.conf` to `~/.kube/config` for root and the sudo user (`internal/kube/admin.go`), so that file can be ours, but the identity check, not the path, proves it. +Without `--kubeconfig`, ambient `$KUBECONFIG` and `~/.kube/config` are not trusted on their own. Weaver already copies `admin.conf` to `~/.kube/config` for root and the sudo user (`internal/kube/admin.go`), so that file can be ours, but the provenance check, not the path, proves it. ### 4.1 Registering a cluster @@ -95,7 +95,7 @@ Related commands, all reading or editing the index only: | Command | Purpose | |---|---| | `solo-provisioner kube cluster list` | Print every known cluster: alias, short ID, origin, server, last used. | -| `solo-provisioner kube cluster info ` | Origin, server, the components managed there (block node, consensus node and so on) and their state, and the current lease holders (section 5.5). Runs the identity check in report-only mode, so it pairs with `doctor` and `diagnose` (#1110). | +| `solo-provisioner kube cluster info ` | Origin, server, the components managed there (block node, consensus node and so on) and their state, and the current lease holders (section 5.5). Runs the provenance check in report-only mode, so it pairs with `doctor` and `diagnose` (#1110). | | `solo-provisioner kube cluster disconnect ` | Remove the index entry. State files are left in place until an explicit prune (section 12). Later work. | A cluster that passes the provisioned proof (section 5.1) registers automatically: `solo-provisioner kube cluster install` does it for new installs, and the gate does it on its first successful pass for older installs (section 5.4). No `connect` is needed. The rule below applies to every other cluster, which is external. @@ -107,14 +107,26 @@ A command that is given an unregistered external cluster: `connect` is idempotent. Running it again for the same cluster ID refreshes the context, server and last-used time. -## 5. Identity gate +## 5. Provenance gate -A check in the root pre-run, **before** any RSL or reality check. It decides which cluster a command is about and whether that cluster is this host's own. If the answer is unsafe, the command stops with a reason code and a hint. This is how we know reality and state can be trusted. +A check in the root pre-run, **before** any RSL or reality check. It decides which cluster a command is about and whether this host provisioned it. If the answer is unsafe, the command stops with a reason code and a hint. This is how we know reality and state can be trusted. The gate prevents accidents: running against the wrong cluster, or against state left behind by another one. It is not a security boundary. A user with root can fake any of its evidence. The gate reads only `hosts/local/host.yaml` and the software records, read-only and without locks. It does not go through RSL, reality or the state manager, but it reads these files through #1235's component loader (`ComponentFilePath`), not a second parser, so the on-disk format has one reader. Atomic writes (#1235 replaces each file by rename) make a lock-free read safe. +**Terms.** "Identity" is overloaded, so this design uses one noun per thing and never the bare word: + +| Term | Meaning | +|---|---| +| **HostIdentity** | the machine — `/etc/machine-id` | +| **ClusterIdentity** | the cluster — the `kube-system` UID (plus server address and CA fingerprint) | +| **Provision record** | what this host recorded about the cluster it provisioned — that cluster's `ClusterIdentity`, bound to this `HostIdentity`, in `hosts/local/host.yaml` | +| **Target** | the cluster a command is pointed at (via `--kubeconfig`, or the host's own cluster by default) | +| **Origin** | the verdict: `provisioned` or `external`. A refusal is not an origin — it is a stop | + +The gate resolves the **Target**, compares its **ClusterIdentity** against the **provision record** for this **HostIdentity**, and returns an **Origin** (or stops). + ### 5.1 Evidence The gate separates evidence that needs no API from evidence that needs it, so a broken local cluster can still be recognised as ours. @@ -130,9 +142,9 @@ The gate separates evidence that needs no API from evidence that needs it, so a The address check alone is not enough. A Kind or minikube cluster on the same machine points at `127.0.0.1` and would pass it. The binary and process checks rule those out. -**Recorded identity** (also no API needed). `hosts/local/host.yaml` records, after the first successful pass, the cluster ID, the server address and the CA certificate fingerprint of the cluster this host provisioned. A kubeconfig **matches the recorded identity** when its CA fingerprint equals the recorded one. The server address is a fallback, used only when no fingerprint is recorded, which can happen on an older install. The fingerprint is the real identity. A server address can be reused: a different cluster rebuilt on the same `IP:6443` would match on the address alone, so weaver would route it through the "this is our cluster" branch (branch A) and report it as "rebuilt, state stale" (rule A.4) instead of as external (rule C.1). The live cluster-ID check still stops any state corruption in that case. Preferring the fingerprint only makes the message the operator sees correct. This works while the cluster is down, which the cluster ID alone cannot. +**Provision record** (also no API needed). `hosts/local/host.yaml` records, after the first successful pass, the cluster ID, the server address and the CA certificate fingerprint of the cluster this host provisioned. A kubeconfig **matches the provision record** when its CA fingerprint equals the recorded one. The server address is a fallback, used only when no fingerprint is recorded, which can happen on an older install. The fingerprint is the real cluster identity. A server address can be reused: a different cluster rebuilt on the same `IP:6443` would match on the address alone, so weaver would route it through the "this is our cluster" branch (branch A) and report it as "rebuilt, state stale" (rule A.4) instead of as external (rule C.1). The live cluster-ID check still stops any state corruption in that case. Preferring the fingerprint only makes the message the operator sees correct. This works while the cluster is down, which the cluster ID alone cannot. -**Live identity** (needs the API). The cluster ID, the UID of the `kube-system` namespace, read from the live cluster. +**Live cluster identity** (needs the API). The `ClusterIdentity`, the UID of the `kube-system` namespace, read from the live cluster. Reading evidence can fail for a reason other than a mismatch. For example, checking the running executable (`/proc//exe`) generally needs root. The gate keeps "cannot read" apart from "does not match": an unreadable check is never taken as proof that a cluster is not ours. @@ -142,7 +154,7 @@ Assumption, to be confirmed against the kubeadm workflow: a host runs one single Only the needs that reach the live cluster, `api` and `host+api` (section 5.3), evaluate these rules. The `host`, `install` and `connect` needs run a reduced check that never reads the API, so the "API unreachable" leaves do not apply to them. Section 5.3 says exactly what each need checks. -The outcomes fall into three branches: the target matches the recorded identity, nothing is recorded yet, or the target is not this host's cluster. One precondition is checked before all of them. Within a branch the rows are checked in order and the first match wins. The `#` column (for example A.1) is the stable rule id used elsewhere in this document. +The outcomes fall into three branches: the target matches the provision record, nothing is recorded yet, or the target is not this host's cluster. One precondition is checked before all of them. Within a branch the rows are checked in order and the first match wins. The `#` column (for example A.1) is the stable rule id used elsewhere in this document. **Precondition (all branches):** @@ -150,7 +162,7 @@ The outcomes fall into three branches: the target matches the recorded identity, |---|---|---| | P.1 | The recorded `machineId` in `host.yaml` does not match this machine (for example the directory was copied) | Stop. Hint: the state belongs to another machine | -**Branch A — the target matches the recorded identity** (CA fingerprint; server address only as a fallback). Reads the API. +**Branch A — the target matches the provision record** (CA fingerprint; server address only as a fallback). Reads the API. | # | Condition | Result | |---|---|---| @@ -165,16 +177,16 @@ The outcomes fall into three branches: the target matches the recorded identity, |---|---|---| | B.1 | The API is unreachable | Stop. The cluster ID cannot be read. Hint: start the cluster, or run `solo-provisioner kube cluster connect` once it is up | | B.2 | The live cluster ID already has `clusters/` state | Stop. Needs an explicit step; it is never adopted silently (see below) | -| B.3 | The live cluster ID has no `clusters/` state | `provisioned`. Record the identity | +| B.3 | The live cluster ID has no `clusters/` state | `provisioned`. Write the provision record | -**Branch C — the target is not this host's cluster** (an identity is recorded but the target does not match it, or nothing is recorded and local evidence does not pass). Decided from `--kubeconfig`; no API needed. +**Branch C — the target is not this host's cluster** (a provision record exists but the target does not match it, or nothing is recorded and local evidence does not pass). Decided from `--kubeconfig`; no API needed. | # | Condition | Result | |---|---|---| | C.1 | `--kubeconfig` was given | `external`, state in `clusters/`. The cluster must be registered (section 4.1): an interactive run stops with a hint to run `solo-provisioner kube cluster connect`, a non-interactive run registers it first | | C.2 | No `--kubeconfig` was given | Stop. If nothing is installed here, the hint is to run `solo-provisioner kube cluster install` or pass `--kubeconfig` for an external cluster. If a provisioned install is only partly present, the hint names what is missing | -Branches A and C together prevent split-brain between the state locations: a cluster that does not match the recorded identity can only become `external`, never overwrite `hosts/local`. Rules A.4 and B.2 cover the two ways one cluster ID could end up described twice. Rule A.1 is the one that keeps a broken local cluster from being treated as someone else's. +Branches A and C together prevent split-brain between the state locations: a cluster that does not match the provision record can only become `external`, never overwrite `hosts/local`. Rules A.4 and B.2 cover the two ways one cluster ID could end up described twice. Rule A.1 is the one that keeps a broken local cluster from being treated as someone else's. #### Rule B.2: a cluster that was external and now passes the local proof @@ -209,7 +221,7 @@ The exact assignment of each command is made when the gate is built, from the co A strict gate must not lock operators out of clusters that older versions installed. Two things could be missing on such a machine, and neither needs a separate migration: - **Recorded hashes.** Software state records only the name, version, `Installed` and `Configured`; it has never held a binary hash. The gate does not need one. The embedded catalog has the expected checksum for each software, version and platform, which the installer already checks at download. The gate verifies that the sandbox binary on disk matches the catalog checksum for the version recorded in state, and that the running process is that binary. -- **The recorded identity.** `hosts/local/host.yaml` has no cluster ID, server address or CA fingerprint on an older install. The first time all the local evidence passes and the live cluster ID can be read (rule B.3), the gate records all three. This is trust on first use, and it is safe because the binary, process, `admin.conf` and address checks have already passed. Running `solo-provisioner kube cluster connect` is an optional way to register eagerly and see each check reported (section 4.1); nothing stops working if it is skipped. +- **The provision record.** `hosts/local/host.yaml` has no cluster ID, server address or CA fingerprint on an older install. The first time all the local evidence passes and the live cluster ID can be read (rule B.3), the gate records all three. This is trust on first use, and it is safe because the binary, process, `admin.conf` and address checks have already passed. Running `solo-provisioner kube cluster connect` is an optional way to register eagerly and see each check reported (section 4.1); nothing stops working if it is skipped. Catalog gaps: the catalog is refreshed and split over time (#1262), so an old kubelet or CRI-O version may no longer be in it. If the recorded version is not in the catalog, the checksum check is skipped with a warning, and the gate relies on the other checks: the binary is in the sandbox path and the running process is that binary. A version that is in the catalog but does not match its checksum still stops the command. @@ -254,7 +266,7 @@ State is grouped by host. A cluster with no host on this machine gets its own di ├── provisioner.yaml # binary-level: cli and daemon sections, each with version, commit, executable, state model version ├── hosts/ │ └── local/ # this machine + the cluster it provisioned -│ ├── host.yaml # identity: name, machine ID, the cluster this host provisioned +│ ├── host.yaml # host identity + provision record: name, machine ID, the cluster this host provisioned │ ├── profile.yaml # host components │ ├── firewall.yaml │ ├── system.yaml @@ -274,7 +286,7 @@ State is grouped by host. A cluster with no host on this machine gets its own di Rules: - `provisioner.yaml` describes the binaries, not a host or a cluster. It is always loadable, the binary reads it first before choosing a host or a cluster, and it exists on every machine, including a Mac with only external clusters (`~/.solo-weaver/state/provisioner.yaml`). Startup migrations depend on the recorded provisioner version, and without this file a machine with no `hosts/local` would have nowhere to record it. Host and cluster state never write it. The file holds one section per binary, `cli` and `daemon`, each with that binary's version, commit, executable path and state-model version. The CLI and the daemon ship and upgrade on their own schedules, and the daemon can self-upgrade (epic #500) to a version the CLI never installed, so each binary is the authority on its own version: each writes only its own section and reads the other's. This gives both binaries a reliable view of each other's version and existence, so either can warn on a skew. Because two binaries write one file, a write is a read-modify-write under the file's lock (#1235), not a blind atomic replace, so neither section clobbers the other; reads stay lock-free. The daemon section is empty until a daemon is installed. Startup migrations are a CLI concern and target the version in the `cli` section. -- `hosts/local/host.yaml` is a small identity file that the identity gate reads first, before any component file, so it needs no component lock. It holds the host `name`, `kind` (`local`; `remote` later), the `machineId` from `/etc/machine-id`, and the identity of the cluster this host provisioned (empty when none): `managedClusterId`, `serverAddress` and `caFingerprint`, which the gate uses to recognise it even when its API is down (section 5.1). A later remote host (section 9) uses the same file with extra registry fields. If the recorded `machineId` no longer matches the machine, for example because the directory was copied, the command stops. If the machine has no usable `/etc/machine-id` (some containers), the check is skipped and no ID is recorded. +- `hosts/local/host.yaml` is a small host-identity file that the provenance gate reads first, before any component file, so it needs no component lock. It holds the host `name`, `kind` (`local`; `remote` later), the `machineId` from `/etc/machine-id`, and the provision record of the cluster this host provisioned (empty when none): `managedClusterId`, `serverAddress` and `caFingerprint`, which the gate uses to recognise it even when its API is down (section 5.1). A later remote host (section 9) uses the same file with extra registry fields. If the recorded `machineId` no longer matches the machine, for example because the directory was copied, the command stops. If the machine has no usable `/etc/machine-id` (some containers), the check is skipped and no ID is recorded. - `hosts/local/` otherwise keeps today's component files and content. - `clusters//` holds **only cluster-scoped components**. Host components (`profile`, `firewall`, `system`) never exist there, so external-mode code cannot reach host state. Hashes and locks work per file, as in #1235. - Each file records its own `clusterId` and `origin`, and the loader checks them against the live cluster. A copied file is rejected. @@ -368,7 +380,7 @@ Add a startup migration, `state-hosts-layout`, registered in `RegisterMigrations - `Execute`: create `hosts/local/` with the same owner and mode, move the component files (and their lock and hash companions) into it, and keep a backup of the originals as `state/.pre-hosts-layout/`. It moves files and does not rewrite their content, with one exception: it copies `ProvisionerState` and `Version` out of the machine-level file into the `cli` section of `state/provisioner.yaml` (the `daemon` section stays empty until a daemon writes it, section 6.1), and writes `hosts/local/host.yaml` (name, kind, machine ID, and the cluster ID if one is already recorded). - If both locations have files, prefer `hosts/local/`, leave the legacy files untouched, and log a warning. Never overwrite. - Idempotent and crash-safe. `Rollback` restores the backup on a best-effort basis. -- It runs before anything opens the state manager, and before the identity gate. Migration is pure filesystem work and needs no cluster. +- It runs before anything opens the state manager, and before the provenance gate. Migration is pure filesystem work and needs no cluster. - All code that builds a state path (the state manager, `state_reader.go`, the existing state migrations, `testutil`, the daemon) moves to one resolver. #1235 already builds component paths from a directory (`ComponentFilePath(dir, id)`, `RuntimeResolver.StateDir()`), so for most callers the change is what that directory is (`state/` becomes `state/hosts/local/`). A few callers still use `models.Paths().StateDir` directly, such as the startup-migrations lock and some tests. - The migration runs under the same startup-migrations lock that #1235 takes on the state directory. - The split migration from #1235 can leave `state.yaml.legacy` and `state.yaml.diverged-*` files in `state/`. They stay where they are: the layout migration moves only the component files, and does not touch or delete them. @@ -430,7 +442,7 @@ Order: ## 11. Delivery plan -1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the identity gate (including the handling of older installs in section 5.4). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. +1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the provenance gate (including the handling of older installs in section 5.4). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. 2. Cluster-side lock (`internal/kube/lease`) and its use in the mutating commands (section 5.5). It needs the resolved cluster from step 1 and nothing from the state work. 3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, `provisioner.yaml`, `hosts/local/host.yaml`, the cluster-scoped component files for `clusters//`, the host-bound field guard in section 6.2, `clusters/index.yaml`, and the `kube cluster connect`, `list` and `info` commands. This step also gives the daemon write access to its own section of `provisioner.yaml` under the #1235 lock (section 6.1); the daemon is a separate binary, so that change ships on its release schedule, not the CLI's. 4. External-mode workflows, block node first, with the StorageClass storage path. @@ -446,7 +458,7 @@ Order: 3. **Cluster-side lock.** A Lease per component per cluster, in the component's namespace, in both modes, built on `leaderelection` (section 5.5). 4. **External state is a local cache.** Almost all of it can be re-derived from the cluster, and the cluster wins when they disagree. Storing state in the cluster is a follow-up with its own design (RBAC, ownership). 5. **Garbage collection.** Nothing is deleted automatically, since a cluster that is only unreachable for a moment must not lose its state. A later `solo-provisioner state prune` lists clusters that are unreachable or whose ID no longer exists, and asks before deleting. -6. **Older installs.** No separate backfill. The gate checks binaries against the catalog checksum and records the cluster identity (ID, server address and CA fingerprint) on its first successful pass (section 5.4). +6. **Older installs.** No separate backfill. The gate checks binaries against the catalog checksum and writes the provision record (cluster ID, server address and CA fingerprint) on its first successful pass (section 5.4). 7. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State, including `provisioner.yaml`, lives in `~/.solo-weaver/state`. 8. **Cluster registration.** One explicit command, `solo-provisioner kube cluster connect`, registers a cluster. Non-interactive runs register implicitly. `list` and `info` read the index and the cluster. `disconnect` is later work (section 4.1). From 59c4b35ba88b52158af1f7429ec694b63070f9f7 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Fri, 16 Oct 2026 22:08:40 +1100 Subject: [PATCH 16/21] docs(design): add startup order, migration split, and FAQ Add section 5.0 (startup order): the three state tiers, the full pre-run sequence, and why the provenance gate is pinned between the filesystem migrations it depends on and the cluster migrations that depend on it. Add section 8.1 (startup migrations split around the gate): ScopeStartup stays filesystem-only, a new ScopeStartupCluster runs cluster-touching migrations after the gate (provisioned-only, skipped in external mode), and the three Cilium migrations move there. Add a FAQ (section 15) covering the recurring questions: gate-after-migrations, no cluster work in ScopeStartup, not a security boundary, provenance vs identity, the narrow-only allowed-origins allowlist, external mode being test-only, behaviour with no cluster, and how commands opt in. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 62 +++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 7abc956a..0653a02b 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -127,6 +127,38 @@ The gate reads only `hosts/local/host.yaml` and the software records, read-only The gate resolves the **Target**, compares its **ClusterIdentity** against the **provision record** for this **HostIdentity**, and returns an **Origin** (or stops). +### 5.0 Startup order and where the gate sits + +"Before any state loading" means before the state manager, RSL and reality layer open — not before any file is read. Weaver's state access comes in three tiers, and the gate is confined to the lowest one: + +| Tier | What it is | Side effects | Gate uses it? | +|---|---|---|---| +| File readers | field- and file-level disk reads (`state_reader.go`, #1235's `ComponentFilePath`) | none; pure, lock-free, no cluster | **Yes** — the only tier the gate touches | +| State manager | `NewStateManager` + `Refresh` + `Flush*` | writes (stamps the running version), takes `flock`s, loads one resolved target | No | +| RSL + reality | runtime spec and hardware/cluster detection | probes the machine and cluster | No | + +The full sequence on every invocation: + +1. **Load `provisioner.yaml`** (file-reader tier; no host or cluster chosen yet). Gives the recorded state-model version the migration runner gates on. The read tolerates both the pre-migration location and `provisioner.yaml`, because step 3 creates it. +2. **Take the startup-migrations lock** on the state directory (#1235). +3. **Run the filesystem startup migrations** (`ScopeStartup`): `unified-state`, the per-component split (#1231), then `state-hosts-layout` (section 8). Pure filesystem, idempotent, no cluster. This creates `hosts/local/` and `provisioner.yaml`, so it must finish before the gate can read them. +4. **Release the migration lock.** +5. **Run the provenance gate** (file-reader tier only). Reads `provisioner.yaml`, `hosts/local/host.yaml` and the software records, recomputes binary checksums against the embedded catalog (no hash is persisted, section 5.4), and reads the live `ClusterIdentity` when the need requires the API. On its first successful pass the trust-on-first-use provision record is written through the normal state path (not inside the gate, which stays read-only). +6. **Run the post-gate cluster migrations** (`ScopeStartupCluster`, section 8) — only when the gate resolved `provisioned`. These may touch the cluster, because the target is now proven ours. +7. **Run the command body.** Only now do the state manager, RSL and reality open, the per-component `flock`s get taken, and the cluster-side lease (section 5.5) is acquired. + +**Why the gate is pinned in the middle.** Two dependencies point in opposite directions, so there is no single point where all migrations can run on one side of the gate: + +``` +filesystem migrations ──(gate reads their output)──▶ gate ──(cluster work needs a verified target)──▶ cluster migrations + (ScopeStartup) (step 5) (ScopeStartupCluster) +``` + +- The gate **depends on** the filesystem migrations: it reads `hosts/local/host.yaml`, which `state-hosts-layout` creates. A gate running before that migration would look at the old `state/` root and misclassify a provisioned host as "nothing recorded". +- The cluster migrations **depend on** the gate: they must not touch a cluster until the gate has proved it is ours and which one. + +So the gate cannot move earlier (its state dependency) and cluster-touching work cannot stay before it (the safety it exists to provide). Splitting the startup migrations around the gate is the only order that satisfies both. This is why `ScopeStartup` **must stay filesystem-only** — see section 8. + ### 5.1 Evidence The gate separates evidence that needs no API from evidence that needs it, so a broken local cluster can still be recognised as ours. @@ -385,6 +417,16 @@ Add a startup migration, `state-hosts-layout`, registered in `RegisterMigrations - The migration runs under the same startup-migrations lock that #1235 takes on the state directory. - The split migration from #1235 can leave `state.yaml.legacy` and `state.yaml.diverged-*` files in `state/`. They stay where they are: the layout migration moves only the component files, and does not touch or delete them. +### 8.1 Startup migrations split around the gate + +The provenance gate (section 5.0) runs between two groups of startup migrations, so the startup scope is split by what a migration touches. + +- **`ScopeStartup` — filesystem only, before the gate.** These reshape on-disk state and never read or write the cluster: `unified-state`, the per-component split (#1231), `state-hosts-layout`, and the existing host-level migrations (crio socket drop-in, network nft/shaper units, sysctl tuning, shell completion, legacy binary). The gate reads their output, so they must run first. This is an invariant: **a `ScopeStartup` migration must not take a kube client.** That is what lets the whole pre-gate phase have no dependency on the gate's verdict. + +- **`ScopeStartupCluster` — cluster-touching, after the gate.** A new scope for startup migrations that read or write the cluster. They run only when the gate resolved `provisioned`, against the resolved client, and are skipped in external mode (weaver touches nothing it did not create). The three Cilium migrations (`cilium-acceleration`, `cilium-agent-restart`, `cilium-host-legacy-routing`), today registered as `ScopeStartup`, move here: they reconfigure `kube-system/cilium-config` and the agent, which is cluster work on the cluster this host provisioned. + +This split is not optional. A cluster-touching migration left in `ScopeStartup` would run before the gate has proved the target is ours — exactly the wrong-cluster accident the gate prevents — and once external mode exists it could reconfigure Cilium on a cluster weaver does not own. The rule for any future migration is the same: if it needs the cluster, it is `ScopeStartupCluster`, not `ScopeStartup`. + Risks to settle: 1. Any path stored inside a state file (the old `StateFile` field is the known case) must be rewritten, and the loader should use the path it actually loaded from, or a later flush could write to the old location. @@ -442,7 +484,7 @@ Order: ## 11. Delivery plan -1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, and the provenance gate (including the handling of older installs in section 5.4). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. +1. Resolver, `--kubeconfig`, `--context`, `--cluster-name`, the provenance gate (including the handling of older installs in section 5.4), and its placement in the startup order (section 5.0), including moving the cluster-touching Cilium startup migrations to `ScopeStartupCluster` (section 8.1). Unit-testable on macOS with fake kubeconfigs. Update `docs/reference/global-flags.md`. 2. Cluster-side lock (`internal/kube/lease`) and its use in the mutating commands (section 5.5). It needs the resolved cluster from step 1 and nothing from the state work. 3. State layout, **after** #1231 / #1235 and #1245 merge: `state-hosts-layout` migration, the single path resolver, `provisioner.yaml`, `hosts/local/host.yaml`, the cluster-scoped component files for `clusters//`, the host-bound field guard in section 6.2, `clusters/index.yaml`, and the `kube cluster connect`, `list` and `info` commands. This step also gives the daemon write access to its own section of `provisioner.yaml` under the #1235 lock (section 6.1); the daemon is a separate binary, so that change ships on its release schedule, not the CLI's. 4. External-mode workflows, block node first, with the StorageClass storage path. @@ -488,3 +530,21 @@ Solo (`hiero-ledger/solo`) solves part of this problem. We do not adopt its code - **Deployments.** A Solo deployment groups a namespace and one or more cluster aliases. We leave that out for now. - **What we avoid.** The alias is Solo's only identity. Nothing checks which cluster a context really points at, `connect` can silently remap an alias, and kubeconfig context names are mutable. Here the key is the cluster ID, an alias binds to one ID, and rebinding needs `--rebind`. Solo also has no notion of whether a cluster was installed by Solo. - **Lease.** Solo's lock is a Lease with a holder identity, renewed in the background. Section 5.5 adapts it. + +## 15. FAQ + +**Why does the provenance gate run *after* migrations, not before?** Because the gate reads `hosts/local/host.yaml`, and the `state-hosts-layout` migration is what creates that file and moves state into it. A gate running first would read the old `state/` root and misclassify a provisioned host as "nothing recorded". So the filesystem migrations run first, then the gate (sections 5.0 and 8.1). + +**Then can a startup migration talk to the cluster?** No — not in `ScopeStartup`. That scope is filesystem-only, precisely so the whole pre-gate phase has no dependency on the gate's verdict. A migration that reads or writes the cluster runs in `ScopeStartupCluster`, after the gate, and only when the target resolved `provisioned`. The three Cilium migrations move there (section 8.1). The rule for any new migration: if it needs the cluster, it is `ScopeStartupCluster`. + +**Is the gate a security boundary?** No. It prevents *accidents* — running against the wrong cluster, or against state left by another one. A user with root can fake any of its evidence. It is about provenance and state-safety, not authorization. + +**Why "provenance" and not "identity"?** "Identity" was overloaded across the host, the cluster, the recorded host-to-cluster binding, and the target. The gate's job is to establish whether this host *provisioned* the target — its provenance — and return an `Origin`. "Identity" named only the comparison, not the decision. The vocabulary is fixed in section 5: `HostIdentity`, `ClusterIdentity`, provision record, Target, Origin. + +**What is `SOLO_PROVISIONER_ALLOWED_ORIGINS`, and can it bypass the gate?** It is a host-owned allowlist of the origins weaver may operate in (`provisioned`, `external`). It can only **narrow**: it is applied after the gate has classified the target, so it can turn an allowed origin into a refusal, but it can never grant an origin the gate did not verify, and never turns a stop into a pass. An unknown value fails closed. There is no runtime override that forces a verdict or skips the checks (section 5.5 of the implementation; the env var is the safe inverse of a bypass). + +**Is external mode a supported way to run in production?** No. It is experimental and provisioner-test-only: it exists to test `solo-provisioner` against a throwaway cluster (such as Kind) without a KVM or privileged runner. It is gated behind `--experimental` / `SOLO_PROVISIONER_ENABLE_EXTERNAL_CLUSTER` and we do not encourage operators to use it in their own CI or clusters (see the banner at the top and section 7). + +**What happens on a machine with no cluster, or on macOS?** With no provisioned install and no `--kubeconfig`, a gated command stops at rule C.2 with a hint to run `kube cluster install` or pass `--kubeconfig`. On macOS there is no provisioned host, so `hosts/local` is never created and every cluster is `external` (decision 7 in section 12). + +**How does a command get gated?** It declares a need with `SetProvenanceNeed(cmd, need)`; the shared pre-run reads it (inherited from a parent command if set there) and runs the gate after migrations. Commands that declare nothing default to `NeedNone` and are not gated. A gated command must not also skip the global pre-run checks. The need vocabulary and what each checks is in section 5.3. From ea464a918494b4a79d69742b025789018c0c54f4 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Fri, 16 Oct 2026 22:19:31 +1100 Subject: [PATCH 17/21] docs(design): note that rule ids are frozen and allocated, not positional Rule ids (P.1, A.4, ...) appear in logs, hints and the doc, so a new rule takes the next free number in its branch and keeps its evaluation position in the table rather than renumbering existing ids. The numeric suffix is allocation order, not evaluation order. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 0653a02b..90af9422 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -188,6 +188,8 @@ Only the needs that reach the live cluster, `api` and `host+api` (section 5.3), The outcomes fall into three branches: the target matches the provision record, nothing is recorded yet, or the target is not this host's cluster. One precondition is checked before all of them. Within a branch the rows are checked in order and the first match wins. The `#` column (for example A.1) is the stable rule id used elsewhere in this document. +**Rule ids are frozen.** An id appears in logs, hints and this document, so once shipped it is never renumbered or reused. A new rule takes the next free number in its branch (`A.5`, `B.4`, …) and is placed in its evaluation position in the table, even when that number is out of sequence — the numeric suffix is allocation order, not evaluation order. The branch letter (A/B/C) stays meaningful; the number is just a label. In code these ids are typed constants (`internal/provenance/rules.go`), compared on by name, with a guard test that keeps them unique. + **Precondition (all branches):** | # | Condition | Result | From bcd7e7adc4507f2e967d3983f2340b96c1262540 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Fri, 16 Oct 2026 23:19:22 +1100 Subject: [PATCH 18/21] docs(design): state is last-known cache; resource ownership via labels Rewrite decision 4: all state is a last-known cache, there is no desired state. Reality (cluster + host) is the source of truth and every field is re-derivable; reconciliation has three outcomes (match, confirmed-drift, unobservable-error), where unobservable is never treated as missing (the #1193 distinction). Add two FAQ entries covering it. Fold in resource ownership (section 7): ownership is authoritative on the resources themselves via app.kubernetes.io/managed-by: weaver plus ownerReferences; uninstall deletes only labeled resources and never touches unlabeled ones (so a foreign Helm release is left alone). weaver-remote-config is the audit log, not the ownership authority, and is written under the per-component lease. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 90af9422..9ce4ac62 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -389,7 +389,8 @@ External mode is experimental and provisioner-test-only (see the banner at the t - Storage: hostPath PVs refer to a node filesystem weaver cannot own, so they are not used. Storage comes from the cluster through a `storageClassName` per volume (flag or values file). Weaver creates PVCs only, never PVs. The hostPath templates (`storage-config.yaml`, `optional-storage.yaml`) are provisioned-only. - Preflight checks API-level facts only: server version range, access via `SelfSubjectAccessReview`, and that the CNI and StorageClass exist. - Provisioned-only commands fail in external mode with a reason code and hint: `solo-provisioner kube cluster install`, `solo-provisioner kube cluster uninstall`, `solo-provisioner teleport node ...`, and `solo-provisioner network firewall|shape|policy ...`. -- Uninstall and reset remove only what weaver created in that cluster, tracked by labels or state. Cluster-level components are never touched. +- Uninstall and reset remove only what weaver created in that cluster. **Ownership is authoritative on the resources themselves**: every weaver-created resource carries `app.kubernetes.io/managed-by: weaver` (plus `ownerReferences` where applicable), written with the resource. Uninstall deletes only labeled resources and **never touches anything unlabeled** — which is how an existing Helm release weaver did not create (it carries Helm's own `managed-by: Helm`) or any other foreign resource is left alone. +- A per-cluster `weaver-remote-config` ConfigMap is the **audit log** (who did what, when), not the ownership authority — a central ledger can drift (a failed write orphans a resource; an out-of-band delete leaves a phantom entry), so the labels on the resources are the source of truth and the log sits on top. Writes to it are made under the per-component lease (section 5.5). - This also applies on a provisioned host: pointing it at a remote cluster with `--kubeconfig` runs external mode and skips the host steps. ### 7.1 Local development with Kind @@ -500,7 +501,17 @@ Order: If `solo-provisioner kube cluster install` finds that `hosts/local/host.yaml` records a cluster ID that differs from the live one (or a cluster that no longer exists), it **stops and does not touch the state**. Weaver cannot tell why: a `kubeadm reset` by hand, a failed or older uninstall, a rebuilt VM with surviving state, or the wrong kubeconfig all look the same, so moving the state automatically would be a guess. The error names the recorded and live cluster IDs and tells the operator to review and clean up the state by hand. The hint lists the files to move or remove and the paths they live in. 3. **Cluster-side lock.** A Lease per component per cluster, in the component's namespace, in both modes, built on `leaderelection` (section 5.5). -4. **External state is a local cache.** Almost all of it can be re-derived from the cluster, and the cluster wins when they disagree. Storing state in the cluster is a follow-up with its own design (RBAC, ownership). +4. **All state is a last-known cache; there is no desired state.** Weaver stores only what it last observed. Reality — the cluster *and* the host — is the source of truth, and every field is re-derivable by reading it: cluster state from the cluster (Helm releases, CRs, pods), host state from the host (tc, nft, sysctl, mounts). Operator intent lives in reality too, not in the cache (for example a consensus `desiredRunState` is a field on a cluster CR), so weaver does not maintain a separate desired state. A remote-Git / GitOps desired state is a possible future, explicitly out of scope here. + + Reconciliation (reality vs cache) therefore has **three** outcomes, not two: + + | Result | Action | + |---|---| + | reality matches the cache | proceed | + | reality is **confirmed different** (resource gone, or a value changed) | drift — stop and ask the operator to overwrite the cache or investigate | + | reality is **unobservable** (API down, RBAC denied, timeout) | error and stop — never assume "gone" (this is the missing-vs-unobservable distinction from #1193) | + + The unobservable-is-an-error rule applies only to commands that must read the cluster (`api` / `host+api`). The offline `host`-need recovery and uninstall path never reaches reconciliation, so a down cluster does not block it (section 5.3). Storing state in the cluster is a separate follow-up with its own design (RBAC, ownership). 5. **Garbage collection.** Nothing is deleted automatically, since a cluster that is only unreachable for a moment must not lose its state. A later `solo-provisioner state prune` lists clusters that are unreachable or whose ID no longer exists, and asks before deleting. 6. **Older installs.** No separate backfill. The gate checks binaries against the catalog checksum and writes the provision record (cluster ID, server address and CA fingerprint) on its first successful pass (section 5.4). 7. **macOS.** There is no provisioned host on darwin, so `hosts/local` is never created and every cluster is `external`. Provisioned-only commands fail with a message that they configure a Linux host. State, including `provisioner.yaml`, lives in `~/.solo-weaver/state`. @@ -550,3 +561,7 @@ Solo (`hiero-ledger/solo`) solves part of this problem. We do not adopt its code **What happens on a machine with no cluster, or on macOS?** With no provisioned install and no `--kubeconfig`, a gated command stops at rule C.2 with a hint to run `kube cluster install` or pass `--kubeconfig`. On macOS there is no provisioned host, so `hosts/local` is never created and every cluster is `external` (decision 7 in section 12). **How does a command get gated?** It declares a need with `SetProvenanceNeed(cmd, need)`; the shared pre-run reads it (inherited from a parent command if set there) and runs the gate after migrations. Commands that declare nothing default to `NeedNone` and are not gated. A gated command must not also skip the global pre-run checks. The need vocabulary and what each checks is in section 5.3. + +**Does weaver keep a desired state?** No. All state is a last-known cache; reality (cluster + host) is the source of truth, and intent lives in reality (for example a cluster CR), not in a separate desired-state store. Drift is reality not matching the cache. See decision 4 for the three reconcile outcomes (match, confirmed-drift, unobservable-error). + +**What if reality can't be observed during reconcile?** It is an error — weaver stops and never treats "can't observe" as "gone", so a transient outage can't make the cache drop real state. This is separate from the offline `host`-need path (uninstall, recovery), which does not read the cluster and so is not blocked when the API is down. From e01457e6a6e6902af17063fd3588931fc1d06226 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Fri, 16 Oct 2026 23:38:42 +1100 Subject: [PATCH 19/21] docs(design): update startup migration split to three scopes Match the implementation (PR #1298): section 8.1 and 5.0 now describe ScopeStartupFilesystem (pre-gate, state only), ScopeStartupHost (post-gate, host-system, host-verified/offline) and ScopeStartupCluster (post-gate, cluster API, provisioned + api need). Add the pre-run ordering, the version-record-after- full-set rule, and that self-install runs filesystem + host migrations only. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 9ce4ac62..cf7ccb3d 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -141,23 +141,23 @@ The full sequence on every invocation: 1. **Load `provisioner.yaml`** (file-reader tier; no host or cluster chosen yet). Gives the recorded state-model version the migration runner gates on. The read tolerates both the pre-migration location and `provisioner.yaml`, because step 3 creates it. 2. **Take the startup-migrations lock** on the state directory (#1235). -3. **Run the filesystem startup migrations** (`ScopeStartup`): `unified-state`, the per-component split (#1231), then `state-hosts-layout` (section 8). Pure filesystem, idempotent, no cluster. This creates `hosts/local/` and `provisioner.yaml`, so it must finish before the gate can read them. +3. **Run the filesystem startup migrations** (`ScopeStartupFilesystem`): `unified-state`, the per-component split (#1231), then `state-hosts-layout` (section 8). Pure filesystem, idempotent, no cluster. This creates `hosts/local/` and `provisioner.yaml`, so it must finish before the gate can read them. 4. **Release the migration lock.** 5. **Run the provenance gate** (file-reader tier only). Reads `provisioner.yaml`, `hosts/local/host.yaml` and the software records, recomputes binary checksums against the embedded catalog (no hash is persisted, section 5.4), and reads the live `ClusterIdentity` when the need requires the API. On its first successful pass the trust-on-first-use provision record is written through the normal state path (not inside the gate, which stays read-only). -6. **Run the post-gate cluster migrations** (`ScopeStartupCluster`, section 8) — only when the gate resolved `provisioned`. These may touch the cluster, because the target is now proven ours. +6. **Run the post-gate migrations** (section 8.1) — only when the gate resolved `provisioned`: `ScopeStartupHost` (host-system work, needs only a verified host), then `ScopeStartupCluster` (cluster work, for `api` / `host+api` needs that verified the live cluster). 7. **Run the command body.** Only now do the state manager, RSL and reality open, the per-component `flock`s get taken, and the cluster-side lease (section 5.5) is acquired. **Why the gate is pinned in the middle.** Two dependencies point in opposite directions, so there is no single point where all migrations can run on one side of the gate: ``` filesystem migrations ──(gate reads their output)──▶ gate ──(cluster work needs a verified target)──▶ cluster migrations - (ScopeStartup) (step 5) (ScopeStartupCluster) + (ScopeStartupFilesystem) (step 5) (ScopeStartupHost, ScopeStartupCluster) ``` - The gate **depends on** the filesystem migrations: it reads `hosts/local/host.yaml`, which `state-hosts-layout` creates. A gate running before that migration would look at the old `state/` root and misclassify a provisioned host as "nothing recorded". - The cluster migrations **depend on** the gate: they must not touch a cluster until the gate has proved it is ours and which one. -So the gate cannot move earlier (its state dependency) and cluster-touching work cannot stay before it (the safety it exists to provide). Splitting the startup migrations around the gate is the only order that satisfies both. This is why `ScopeStartup` **must stay filesystem-only** — see section 8. +So the gate cannot move earlier (its state dependency) and cluster-touching work cannot stay before it (the safety it exists to provide). Splitting the startup migrations around the gate is the only order that satisfies both. This is why the pre-gate scope (`ScopeStartupFilesystem`) **must stay filesystem-only** — see section 8.1. ### 5.1 Evidence @@ -422,13 +422,17 @@ Add a startup migration, `state-hosts-layout`, registered in `RegisterMigrations ### 8.1 Startup migrations split around the gate -The provenance gate (section 5.0) runs between two groups of startup migrations, so the startup scope is split by what a migration touches. +The provenance gate (section 5.0) runs between groups of startup migrations, so the single `ScopeStartup` is split three ways by what a migration touches and what it needs verified: -- **`ScopeStartup` — filesystem only, before the gate.** These reshape on-disk state and never read or write the cluster: `unified-state`, the per-component split (#1231), `state-hosts-layout`, and the existing host-level migrations (crio socket drop-in, network nft/shaper units, sysctl tuning, shell completion, legacy binary). The gate reads their output, so they must run first. This is an invariant: **a `ScopeStartup` migration must not take a kube client.** That is what lets the whole pre-gate phase have no dependency on the gate's verdict. +- **`ScopeStartupFilesystem` — on-disk state, before the gate.** Reshapes weaver's state and never reads the cluster or mutates the host system: `unified-state`, the per-component split (#1231), `state-hosts-layout`, `helm-release-schema-v2`, `mgmt-ports-v1`. The gate reads their output, so they run first. Invariant: **a filesystem migration must not take a kube client.** This is what lets the whole pre-gate phase have no dependency on the gate's verdict. -- **`ScopeStartupCluster` — cluster-touching, after the gate.** A new scope for startup migrations that read or write the cluster. They run only when the gate resolved `provisioned`, against the resolved client, and are skipped in external mode (weaver touches nothing it did not create). The three Cilium migrations (`cilium-acceleration`, `cilium-agent-restart`, `cilium-host-legacy-routing`), today registered as `ScopeStartup`, move here: they reconfigure `kube-system/cilium-config` and the agent, which is cluster work on the cluster this host provisioned. +- **`ScopeStartupHost` — host-system work, after the gate.** systemd units, sysctl, host files, binaries: `legacy-binary`, `crio-socket-dropin`, the network nft/shaper units, `shell-completion`, `sysctl-network-tuning`. These need only a verified host (P.1), not the cluster, so they run after the gate resolves `provisioned` and work even when the API is down. -This split is not optional. A cluster-touching migration left in `ScopeStartup` would run before the gate has proved the target is ours — exactly the wrong-cluster accident the gate prevents — and once external mode exists it could reconfigure Cilium on a cluster weaver does not own. The rule for any future migration is the same: if it needs the cluster, it is `ScopeStartupCluster`, not `ScopeStartup`. +- **`ScopeStartupCluster` — cluster-touching, after the gate.** Reads or writes the cluster: the three Cilium migrations (`cilium-acceleration`, `cilium-agent-restart`, `cilium-host-legacy-routing`), which reconfigure `kube-system/cilium-config` and the agent. They run only when the gate resolved `provisioned` and the command's need verified the live cluster (`api` / `host+api`), against the resolved client, and are skipped in external mode (weaver touches nothing it did not create). + +Ordering in the root pre-run: filesystem migrations → gate → host migrations → cluster migrations. The running version is recorded only after the full provisioned + cluster set has run, so a host-only invocation cannot mark cluster boundary migrations done before they run. `solo-provisioner install` (self-install) skips the pre-run, so it runs the filesystem and host migrations explicitly; it has no cluster, so it runs no cluster migrations. + +This split is not optional. A cluster-touching migration left in a pre-gate scope would run before the gate has proved the target is ours — the wrong-cluster accident the gate prevents — and once external mode exists it could reconfigure Cilium on a cluster weaver does not own. The rule for any future migration: filesystem if it only reshapes state, `ScopeStartupHost` if it touches the host system, `ScopeStartupCluster` if it needs the cluster. Risks to settle: @@ -548,7 +552,7 @@ Solo (`hiero-ledger/solo`) solves part of this problem. We do not adopt its code **Why does the provenance gate run *after* migrations, not before?** Because the gate reads `hosts/local/host.yaml`, and the `state-hosts-layout` migration is what creates that file and moves state into it. A gate running first would read the old `state/` root and misclassify a provisioned host as "nothing recorded". So the filesystem migrations run first, then the gate (sections 5.0 and 8.1). -**Then can a startup migration talk to the cluster?** No — not in `ScopeStartup`. That scope is filesystem-only, precisely so the whole pre-gate phase has no dependency on the gate's verdict. A migration that reads or writes the cluster runs in `ScopeStartupCluster`, after the gate, and only when the target resolved `provisioned`. The three Cilium migrations move there (section 8.1). The rule for any new migration: if it needs the cluster, it is `ScopeStartupCluster`. +**Then can a startup migration talk to the cluster?** Not before the gate. The pre-gate scope `ScopeStartupFilesystem` is filesystem-only, precisely so the whole pre-gate phase has no dependency on the gate's verdict. Work that touches the host system runs in `ScopeStartupHost` and work that reads or writes the cluster runs in `ScopeStartupCluster` — both after the gate, only when the target resolved `provisioned`. The three Cilium migrations are `ScopeStartupCluster` (section 8.1). The rule for any new migration: filesystem if it only reshapes state, host if it touches the host system, cluster if it needs the cluster. **Is the gate a security boundary?** No. It prevents *accidents* — running against the wrong cluster, or against state left by another one. A user with root can fake any of its evidence. It is about provenance and state-safety, not authorization. From 1eee335f8da2a2c512693478ce50942077932cf3 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Fri, 16 Oct 2026 23:42:03 +1100 Subject: [PATCH 20/21] docs(design): add the story breakdown (section 16) Add a living story breakdown derived from the delivery plan (section 11) and the spike: five milestones with dependencies, macOS-vs-VM testability, MVP flags, the critical path (per-component state work gates the state layout), and what can proceed in parallel now. Signed-off-by: Lenin Mehedy --- docs/dev/bring-your-own-cluster.md | 59 ++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index cf7ccb3d..64e2edaf 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -569,3 +569,62 @@ Solo (`hiero-ledger/solo`) solves part of this problem. We do not adopt its code **Does weaver keep a desired state?** No. All state is a last-known cache; reality (cluster + host) is the source of truth, and intent lives in reality (for example a cluster CR), not in a separate desired-state store. Drift is reality not matching the cache. See decision 4 for the three reconcile outcomes (match, confirmed-drift, unobservable-error). **What if reality can't be observed during reconcile?** It is an error — weaver stops and never treats "can't observe" as "gone", so a transient outage can't make the cache drop real state. This is separate from the offline `host`-need path (uninstall, recovery), which does not read the cluster and so is not blocked when the API is down. + +## 16. Story breakdown + +Derived from the delivery plan (section 11) and the spike (design PR #1288, code PR #1298). Grouped into milestones with dependencies and where each is testable (macOS vs the UTM VM). This is a living plan — update it as stories are created and linked to the epic. + +### Milestone 1 - Provenance gate foundation (mostly spiked; macOS-testable) + +| # | Story | Scope | Depends on | +|---|---|---|---| +| 1 | Cluster resolver + root flags | `kube.ResolveCluster`, `--kubeconfig` / `--context` / `--cluster-name`; one place resolves the target. Update `global-flags.md`. | - | +| 2 | Provenance gate core | `internal/provenance` (Evaluate outcome table, Need, Outcome, allowlist, rule constants) + `common.RunProvenanceGate` wired into `block node install`. Replace the spike's placeholder collector with real evidence (kubelet/CRI-O running, `/proc` exe, catalog checksum). | 1 | +| 3 | Startup migration scope split | `ScopeStartupFilesystem/Host/Cluster`, re-registration, phased pre-run around the gate, self-install wiring (section 8.1). | 2 | + +Stories 2-3 exist in the spike; this milestone productionizes them. Gate decision logic is unit-tested on macOS; real host/cluster evidence and the host/cluster migration phases need the VM. + +### Milestone 2 - Per-cluster state layout (critical path; gated on #1231 / #1235 / #1245) + +| # | Story | Scope | Depends on | +|---|---|---|---| +| 4 | `provisioner.yaml` + path resolver | binary-level file (cli + daemon sections), single path resolver all callers use. | #1231/#1235 | +| 5 | `hosts/local/` + `host.yaml` + layout migration | identity + provision record; `state-hosts-layout` migration; gate reads the real provision record. | 4, #1245 | +| 6 | `clusters//` + `index.yaml` + host-bound guard | cluster-scoped component files, cluster index, the section 6.2 writer guard. | 5 | + +This chain gates everything cluster-aware; it cannot start until the per-component state work merges. + +### Milestone 3 - Coordination and registration + +| # | Story | Scope | Depends on | +|---|---|---|---| +| 7 | Cluster-side lease | `internal/kube/lease` (leaderelection), taken by mutating commands; fencing. | 1 | +| 8 | `kube cluster connect` / `list` / `info` | registration, alias binding, read-only views. | 6 | + +### Milestone 4 - External mode, ownership, drift + +| # | Story | Scope | Depends on | MVP? | +|---|---|---|---|---| +| 9 | External-mode gating + workflow skips | experimental feature flag; skip host steps for external targets. | 2, 6 | later | +| 10 | StorageClass storage path (block node) | PVC-only, no hostPath, for external clusters. | 9 | later | +| 11 | Resource ownership | `managed-by: weaver` labels + ownerReferences; `weaver-remote-config` audit log (lease-protected); uninstall deletes only labeled. | 6, 7 | MVP | +| 12 | Drift / reconcile (3 outcomes) | match / confirmed-drift / unobservable-error; integrate with #1193. | 6 | later | + +### Milestone 5 - Rollout and CI + +| # | Story | Scope | Depends on | +|---|---|---|---| +| 13 | Assign `Need` to all commands | per-command section 5.3 mapping (today only block node install is wired). | 2 | +| 14 | CI Kind lane + test split | external-mode Kind lane, privilege-based split, KVM path-filter lane. | 9, 10 | + +### Later / out of scope (separate epic or backlog) + +- Multi-VM from macOS (`--host`, SSH delegation) - section 9. +- In-cluster state storage - section 13. +- `state prune`, `state adopt` / discard, stale-state cleanup commands - section 13. + +### Sequencing notes + +- Critical path: #1231 / #1235 / #1245 -> story 4 -> 5 -> 6 -> (8, 9, 11, 12). Nothing cluster-state-aware starts until the per-component state work lands. +- Can run now, in parallel, on the spike branch: stories 1-3 and 7 (the lease needs only the resolver, not the state work; see section 11.2). Story 13 is incremental once story 2 is in. +- Correctness-bearing MVP: milestone 1 plus resource ownership (11) and the three-outcome reconcile (12). External mode (9-10) and the Kind CI lane (14) are the payoff and can follow. From b0260fcfa064864fd6a9243003c63ac1719906a5 Mon Sep 17 00:00:00 2001 From: Lenin Mehedy Date: Sat, 17 Oct 2026 00:07:55 +1100 Subject: [PATCH 21/21] docs(design): cluster migrations run in external mode, ownership-gated External mode is Kubernetes-API work, so ScopeStartupCluster migrations run for any verified, reachable cluster (api / host+api needs) in both provisioned and external mode, not provisioned-only. Because the scope now runs in external mode, each cluster migration's Applies() must be ownership-aware and act only on resources weaver created; the Cilium migrations stay effectively provisioned-only through that, preserving "external touches nothing it did not create" (section 7). Update section 8.1, the 5.0 startup order, and the FAQ. Host migrations remain provisioned-only. Signed-off-by: Lenin Mehedy (cherry picked from commit 4450e931d9277c0a35f9b0fab6437ae7d5b3fc87) --- docs/dev/bring-your-own-cluster.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/dev/bring-your-own-cluster.md b/docs/dev/bring-your-own-cluster.md index 64e2edaf..f9bbb0c7 100644 --- a/docs/dev/bring-your-own-cluster.md +++ b/docs/dev/bring-your-own-cluster.md @@ -144,7 +144,7 @@ The full sequence on every invocation: 3. **Run the filesystem startup migrations** (`ScopeStartupFilesystem`): `unified-state`, the per-component split (#1231), then `state-hosts-layout` (section 8). Pure filesystem, idempotent, no cluster. This creates `hosts/local/` and `provisioner.yaml`, so it must finish before the gate can read them. 4. **Release the migration lock.** 5. **Run the provenance gate** (file-reader tier only). Reads `provisioner.yaml`, `hosts/local/host.yaml` and the software records, recomputes binary checksums against the embedded catalog (no hash is persisted, section 5.4), and reads the live `ClusterIdentity` when the need requires the API. On its first successful pass the trust-on-first-use provision record is written through the normal state path (not inside the gate, which stays read-only). -6. **Run the post-gate migrations** (section 8.1) — only when the gate resolved `provisioned`: `ScopeStartupHost` (host-system work, needs only a verified host), then `ScopeStartupCluster` (cluster work, for `api` / `host+api` needs that verified the live cluster). +6. **Run the post-gate migrations** (section 8.1): `ScopeStartupHost` (host-system work) only when the gate resolved `provisioned`, then `ScopeStartupCluster` (cluster work) whenever an `api` / `host+api` need verified a live, reachable cluster — provisioned or external. 7. **Run the command body.** Only now do the state manager, RSL and reality open, the per-component `flock`s get taken, and the cluster-side lease (section 5.5) is acquired. **Why the gate is pinned in the middle.** Two dependencies point in opposite directions, so there is no single point where all migrations can run on one side of the gate: @@ -428,7 +428,7 @@ The provenance gate (section 5.0) runs between groups of startup migrations, so - **`ScopeStartupHost` — host-system work, after the gate.** systemd units, sysctl, host files, binaries: `legacy-binary`, `crio-socket-dropin`, the network nft/shaper units, `shell-completion`, `sysctl-network-tuning`. These need only a verified host (P.1), not the cluster, so they run after the gate resolves `provisioned` and work even when the API is down. -- **`ScopeStartupCluster` — cluster-touching, after the gate.** Reads or writes the cluster: the three Cilium migrations (`cilium-acceleration`, `cilium-agent-restart`, `cilium-host-legacy-routing`), which reconfigure `kube-system/cilium-config` and the agent. They run only when the gate resolved `provisioned` and the command's need verified the live cluster (`api` / `host+api`), against the resolved client, and are skipped in external mode (weaver touches nothing it did not create). +- **`ScopeStartupCluster` — cluster-touching, after the gate.** Reads or writes the cluster — for example the three Cilium migrations (`cilium-acceleration`, `cilium-agent-restart`, `cilium-host-legacy-routing`), which reconfigure `kube-system/cilium-config` and the agent. They run whenever the command's need verified a live, reachable cluster (`api` / `host+api`), against the resolved client, in **both** provisioned and external mode — external mode is Kubernetes-API work, so pending migrations for weaver's own cluster components must run there too before the command proceeds. Because this scope now runs in external mode, each cluster migration's `Applies()` must be **ownership-aware** and act only on resources weaver created; weaver must never reconfigure a cluster it does not own. The Cilium migrations target host-provisioned infra, so their `Applies()` keeps them effectively provisioned-only, satisfying "external mode touches nothing it did not create" (section 7). Ordering in the root pre-run: filesystem migrations → gate → host migrations → cluster migrations. The running version is recorded only after the full provisioned + cluster set has run, so a host-only invocation cannot mark cluster boundary migrations done before they run. `solo-provisioner install` (self-install) skips the pre-run, so it runs the filesystem and host migrations explicitly; it has no cluster, so it runs no cluster migrations. @@ -552,7 +552,7 @@ Solo (`hiero-ledger/solo`) solves part of this problem. We do not adopt its code **Why does the provenance gate run *after* migrations, not before?** Because the gate reads `hosts/local/host.yaml`, and the `state-hosts-layout` migration is what creates that file and moves state into it. A gate running first would read the old `state/` root and misclassify a provisioned host as "nothing recorded". So the filesystem migrations run first, then the gate (sections 5.0 and 8.1). -**Then can a startup migration talk to the cluster?** Not before the gate. The pre-gate scope `ScopeStartupFilesystem` is filesystem-only, precisely so the whole pre-gate phase has no dependency on the gate's verdict. Work that touches the host system runs in `ScopeStartupHost` and work that reads or writes the cluster runs in `ScopeStartupCluster` — both after the gate, only when the target resolved `provisioned`. The three Cilium migrations are `ScopeStartupCluster` (section 8.1). The rule for any new migration: filesystem if it only reshapes state, host if it touches the host system, cluster if it needs the cluster. +**Then can a startup migration talk to the cluster?** Not before the gate. The pre-gate scope `ScopeStartupFilesystem` is filesystem-only, precisely so the whole pre-gate phase has no dependency on the gate's verdict. Work that touches the host system runs in `ScopeStartupHost` (after the gate, provisioned only), and work that reads or writes the cluster runs in `ScopeStartupCluster` (after the gate, for any verified cluster — provisioned or external, since external mode is API work). The three Cilium migrations are `ScopeStartupCluster` but stay effectively provisioned-only through an ownership-aware `Applies()`, so weaver never reconfigures a cluster it does not own (section 8.1). The rule for any new migration: filesystem if it only reshapes state, host if it touches the host system, cluster if it needs the cluster. **Is the gate a security boundary?** No. It prevents *accidents* — running against the wrong cluster, or against state left by another one. A user with root can fake any of its evidence. It is about provenance and state-safety, not authorization.