Edge clusters: shared-library extraction + spec + implementation

[schmidt-scaled]

Summary

Two commits implementing the edge-clusters feature (see docs/edge_clusters_analysis.md and docs/edge_clusters_spec.md):

1. simplyblock_lib — sbcli-agnostic infrastructure extracted from core/web so edge (and future services) reuse it without duplication: task lease/claim + TaskRunner poll-loop base, PollingService/PerItemSupervisor monitor skeletons, v2 API scaffolding (typed scalars, creation response, access-log middleware), events/units/secrets helpers. Reference conversions: tasks_runner_fdb_backup, device_monitor, health_check_service. Includes a fix: a successful lease claim/refresh now stamps the caller's task copy so a follow-up full-object write can't clobber the committed owner.

2. simplyblock_edge — spdk-only 1-2 node edge clusters managed by the same centralized CP over exactly two channels (edge k8s API + SPDK JSON-RPC, no snode agent): edge cluster = Cluster record with cluster_type=edge; deterministic bdev-stack planner (1 partition = aio / 2 = raid1 / 3+ = raid5f locally; cross-node raid1 mirror via nvme-tcp leg; lazy lvstore); volume CRUD + connect; node/cluster status derivation (unreachable = mgmt-plane verdict, DOWN sticky, returned nodes reassembled by task before ONLINE); EdgeMonitor + EdgeTaskRunner services; per-cluster k8s clients + 2-vCPU SPDK pod template; v2 routers /clusters/{id}/edge-nodes + /edge-volumes.

Testing

• Unit tier: 1270 passed locally (151 new: lib + edge, incl. API router tests).
• Integration tier: 15 new FDB tests (lease CAS semantics, runner end-to-end, full edge lifecycle: outage → degraded → reassembly → active). Not yet executed — no Docker on the dev box; this PR exists to run them in CI.
• ruff clean; mypy clean for all new/touched files.

Deferred (spec §10)

raid5f rebuild/grow fork-capability check, takeover/failback, CSI integration, CLI command group, operator CRD, auth scaling fixes (per-request cluster-secret scan).

🤖 Generated with Claude Code

[mxsrc]

Okay, I have a few points, with no particular order:

1. restart_node will restart without checking the node status. If that node (A) is the secondary of a failed node (B), this will trigger the rebuild of elvs_B from the copy, even though A is the leader. The minimal fix of this would be the restart refusing on nodes that lead stores they don't own.

2. Edge{Node,Partition}.STATUS_REMOVED are never assigned anywhere. Are they simply introduced as a provision for a future change?

3. delete_lvol returned errors are not checked. This should be made explicit if intentional.

4. add_edge_node uses threads rather than task-based async execution, leading to issues with execution reliability. This is the same issue with the regular storagenode add, as well as cluster start/shutdown/activate/expand, so probably out of scope here, but good to be aware of. Probably this should be reworked, but given the prevalence int he code base we might defer it until reworking this with proper async execution.

One thing that may be considered critical though is the lack of cleanup on failed add-node calls, they simply silently go missing along with the spawned thread.

5. Concurrent add_edge_node calls may violate the MAX_EDGE_NODES limit.

6. k8s_token entry and storage (along with k8s_ca_cert) are not hanlded according to the mandated secret handling.

7. _reassemble_node calls _publish_volume → _ensure_crypto_stack for every
   volume in a loop with no per-volume isolation. an exception from one crypto
   volume's KMS lookup aborts the whole reassembly.

8. k8s.py's CA-cert temp-file cache never evicts stale files on rotation. (_ca_files dict, k8s.py:22-37) today this is a minor leak of public data. Since this is adjacent to certificate data this risks developing into an actual problem

9. I'd like to avoid the added top-level edge_e2e directory, this might be something we can fold in with e2e as e2e/{hyperscale,edge

10. The added API shape is non restful and contradicts spec and analysis. I'd like to propose this shape instead:

POST   /clusters/                                  {..., cluster_type: "edge", k8s_api_url, k8s_token, k8s_ca_cert, k8s_namespace}
GET    /clusters/{id}/                              -> cluster_type now in the response
GET    /clusters/{id}/edge/storage-nodes
POST   /clusters/{id}/edge/storage-nodes
GET    /clusters/{id}/edge/storage-nodes/{node_id}
POST   /clusters/{id}/edge/storage-nodes/{node_id}/shutdown
POST   /clusters/{id}/edge/storage-nodes/{node_id}/restart
POST   /clusters/{id}/edge/storage-nodes/{node_id}/devices
PUT    /clusters/{id}/edge/storage-nodes/{node_id}/devices
POST   /clusters/{id}/edge/storage-nodes/{node_id}/devices/remove
POST   /clusters/{id}/edge/storage-nodes/{node_id}/devices/restart
GET    /clusters/{id}/edge/volumes
POST   /clusters/{id}/edge/volumes
GET    /clusters/{id}/edge/volumes/{vol_id}
PUT    /clusters/{id}/edge/volumes/{vol_id}
DELETE /clusters/{id}/edge/volumes/{vol_id}
GET    /clusters/{id}/edge/volumes/{vol_id}/connect

Folding the creation into the existing create would also inherit the name-locking avoiding reintroducing the name duplication.
The cluster type should also be introduced to the ClusterDTO.