[leninmehedy]

Design doc for #1287. No code changes.

IMPORTANT: This external cluster support is for provisioner test only feature. We don't encourage anyone to use it in their CI or cluster

Summary

• Strict identity gate before any state or reality check: a cluster is only trusted if we can prove we installed it on this machine (catalog checksum, running processes, admin.conf, IP).
• Bring your own cluster: --kubeconfig, --context, --cluster-name, and an explicit solo-provisioner kube cluster connect (with list and info). External mode only talks to the Kubernetes API.
• State grouped by host and cluster: hosts/local (with a host.yaml identity file), clusters/<uid>, and a global provisioner.yaml. It builds on the per-component state files in refactor(state): split state.yaml into per-component files with a migration #1231 / refactor(state): split state.yaml into per-component files with a migration #1235 and state: split machine.yaml by owner and flush it only from commands that mutate it #1245.
• Cluster-side lock: a Lease per component per cluster, in both modes, built on client-go leaderelection.
• Local development with Kind, including macOS (~/.solo-weaver/state).
• CI impact: a Kind lane for API-level UAT, with privileged and KVM runners kept only where needed.
• Prior art from hiero-ledger/solo.

Please review

• Section 5: Identity checks
• Section 5.5: the lease design.

Refs #1287

[swirlds-automation]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[brunodam]

given solo-provisioner is a tool to be run in a single node cluster, should we avoid the complexity of running with multiple clusters? I see the benefit in trying to parallelise tests by using multiple clusters, but I wonder how worth it those tests would be if they are all sharing the same state file.

[leninmehedy]

given solo-provisioner is a tool to be run in a single node cluster

This assumption is now relaxed a bit in the sense that someone can deploy resources in remote/external/Kind clusters with no access to the underlying bare-metal (gated under --experimental feature). This is primarily for various testing scenarios in local-dev/CI/CITR so on. We are adding proper external cluster support (with locks and in-cluster audit records) rather than leaving a risky loophole (currently available) or closing it altogether.

if they are all sharing the same state file

When we use multiple clusters, the state files will be separate for each cluster (state managedment will be refactored to support this). So they are not sharing the same state file.

/opt/solo/weaver/state/

└── clusters/
    ├── <cluser-1-uid>/        
    │   ├── cluster.yaml           # cluster-scoped components only
    │   ├── blocknode.yaml
    │   ├── consensus.yaml
    │   └── teleport.yaml
    ├── <cluser-2-uid>/        
     │   ├── cluster.yaml           # cluster-scoped components only
    │   ├── blocknode.yaml.    # cluster-2 has blocknode only
    └── index.yaml                 # complete view of every known cluster

Please let me know if you see any other issues

[alex-au]

Thanks Lenin for another great design, a few thoughts:

• Identity gate (Sections 5 and 8): The gate needs persisted state (binary hashes, cluster ID, etc.) for verification, but it's supposed to run before state loading. Could we clarify the startup order, including migrations?

• Verification failures (Sections 5.2, 5.3): What happens if we can't determine the cluster identity because the API is unreachable? Also, could a failed identity check block recovery or uninstall when the local kubelet is broken?

• External state (Section 12.3): Can we really treat all external state as a disposable cache? Some fields may represent desired state or information we can't recover from the cluster. We should probably keep the same distinction between missing and unobservable resources that we introduced in Out-of-band drift detection framework (reality-vs-persisted diff + report) #1193.

• Resource ownership (Section 7): Cluster identity tells us which cluster we're connected to, but not which resources Weaver owns. How do we handle existing Helm releases or other resources that weren't created by Weaver, especially during uninstall?

[leninmehedy]

Thanks @alex-au for the feedback. Below are my responses to your questions.

• Identity gate (Sections 5 and 8): The gate needs persisted state (binary hashes, cluster ID, etc.) for verification, but it's supposed to run before state loading. Could we clarify the startup order, including migrations?

Good question. I have renamed Identity -> Provenance since Identity term is overloaded.

1. Provenance gate doesn't need to go through BLL/RSL. BLL/RSL needs a verified cluster/host. so gate needs to run before a verified target (HostIdentity, or ClusterIdentity) is determined for the given command. Each command would define it's need (host vs k8s api or host+api) and based on that the gate would ensure the need is met before we go into the intent handling by BLL.

2. For the sort order I had to do a small coding spike (feat(provenance): add cluster provenance gate and wire block node install #1298) to see where we get blocked in the startup order. What I found is that the start-up migrations (ScopeStartup) are mixed with filesystem + cluster/host level migrations. We need to define new types - ScopeStartupFilesystem, ScopeStartupCluster, ScopeStartupHost. So the Provenance Gate needs to run after the filesystem startup migrations (ScopeStartupFilesystem), but before the cluster/host level migrations.

flowchart TD
    cmd["Command, e.g. block node install<br/>declares need: host+api"] --> prerun["Shared pre-run<br/>root PersistentPreRunE"]
    prerun --> fsmig["Filesystem migrations<br/>ScopeStartupFilesystem"]
    fsmig --> rpg["RunProvenanceGate<br/>common helper"]
    rpg --> collect
    subgraph prov ["internal/provenance"]
        collect["defaultCollector.Collect<br/>reads host + cluster into Input"] --> gate["Gate(need, Input)<br/>need policy check"]
        gate --> eval["Evaluate(Input)<br/>P.1 then branch A/B/C"]
        eval --> allow["Allowlist (modes.go)<br/>narrow only"]
    end
    gate -. policy stop .-> stop
    eval -. rule stop .-> stop
    allow -->|stop / not allowed| stop["Stop: doctor message + hint"]
    allow -->|external| clustermig["Cluster migrations<br/>ScopeStartupCluster (api / host+api)"]
    allow -->|provisioned| hostmig["Host migrations<br/>ScopeStartupHost"]
    hostmig --> clustermig
    clustermig --> bll["BLL handles the intent<br/>command proceeds"]

Loading

I have updated the design doc based on my current findings/understanding. We may discover few more ambiguity as we implement, but it is hard to predict all such ambiguity unless we get into the trench.

One more thought, this design doc gives us a north star, but it is not a full proof solution doc yet. Rather, we need to keep it as a live design doc and update as we identify new blockers. I have added a FAQ section so that we can track various questions and decisions/answers that helped us to evolve it.

---

• Verification failures (Sections 5.2, 5.3): What happens if we can't determine the cluster identity because the API is unreachable? Also, could a failed identity check block recovery or uninstall when the local kubelet is broken?

Current thought is that we trigger error and stop, just keep it simple in the first MVP. We then let the user determine what to do since we cannot recover from such error automatically.

Scenario	Action
reality matches cache	proceed
reality confirmed different (resource gone, or value changed)	drift → stop and ask the operator (overwrite cache, or investigate)
reality unobservable (API down, RBAC denied, timeout)	error and stop — never assume “gone”

Commands that need to observe the cluster (api / host+api) would block if API is down. It must not block the offline recovery/uninstall commands (need = 'host') — those don’t observe the cluster, so a down API shouldn’t stop them.

---

• External state (Section 12.3): Can we really treat all external state as a disposable cache? Some fields may represent desired state or information we can't recover from the cluster. We should probably keep the same distinction between missing and unobservable resources that we introduced in Out-of-band drift detection framework (reality-vs-persisted diff + report) #1193.

I don't have a concrete idea on how to differentiate desired and cached state, it's complicated. Question would be where should we keep the desired state, ideally in a git repository (GitOps style) to track changes, but that's an overkill for solo-provisioner's current usecase.

Therefore, the provisioner state was originally designed as a cached state since that's easier to reason about, but with the tradeoff of keeping it in sync. Idea was that if all resources are managed through solo-provisioner, we can sync the cached state as our last known state. So the kubernetes cluster is the reality and it must match with our cached state (our last known state). Thus it helps us to detect the out-of-band drift and therefore we can ask user if we should overwrite cached state or stop for manual investigation.

state cache = weaver’s last-known-observed state
reality = the cluster and the host
drift = reality ≠ cache

Having said, that we can still support desired state in a remote Git repository later if we want, unless other GitOps solutions are not enough.

Please let me know if I missed anything.

---

• Resource ownership (Section 7): Cluster identity tells us which cluster we're connected to, but not which resources Weaver owns. How do we handle existing Helm releases or other resources that weren't created by Weaver, especially during uninstall?

Each cluster will have a new configmap (weaver-remote-config) where we shall log who did what and when (similar to how solo-cli handles remote clusters). We shall also use labels (app.kubernetes.io/managed-by: weaver), plus ownerReferences where applicable — that’s the ownership source of truth, written with the resource itself.

[alex-au]

Thanks Lenin for clearing up all of my feedbacks.

One small concern about 4th item - resource ownership: Helm already uses app.kubernetes.io/managed-by: Helm to track its resources. Not sure setting it to weaver would interfere with Helm? Perhaps we can consider using a separate weaver specific label and keep Helm's own ownership metadata intact.