fix(containerlab): give ns30/ns40 two attachments, not one shared

Rebased onto the updated fix/vrf-shared-per-vpc-2-cnibgp (which
picked up my own rebase-and-reconcile of that branch after
fix/vrf-shared-per-vpc-1-core moved out from under it) after that
branch's history moved out from under this one -- same ripple as
#306 -> #307 -> #311 -> #312 -> #313.

No conflicts: this PR only touches containerlab docs/scripts/
manifests, none of which overlapped with the Go source changes further
up the stack. Applied cleanly.

Verification: task lint, task build (all 8 binaries) pass on the
rebased tree. No Go source changed in this PR, so task test:unit is
unaffected. tests/e2e and the containerlab lab itself not run in this
sandbox (no Kind cluster / root), same caveat as every PR in this
stack.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: privateip

deploy/containerlab/README.md

@@ -138,12 +138,15 @@ deploy/containerlab/
 │   ├── galactic-router/         # galactic-router DaemonSet + BGP CRs (dfw, iad, sjc)
 │   ├── galactic-control/iad/    # galactic-router RR + BGP CRs (iad-control)
 │   └── tenants/                 # test VPCs — one shared base/ (Namespace + netshoot
-│       ├── base/                 # Deployment), each tenant patching its namespace,
-│       ├── ns50/                 # default-network annotation, and (ns30/ns40) replica
-│       ├── ns10/                 # count; per-site dirs hold each site's NAD(s):
-│       ├── ns20/                 #   ns50 (IPv4, 3-site), ns10 (IPv6-only, 3-site),
-│       ├── ns30/                 #   ns20 (dual-stack, 3-site), ns30 (IPv6-only,
-│       └── ns40/                 #   dfw only, 2 pods), ns40 (IPv4-only, iad only, 2 pods)
+│       ├── base/                 # Deployment), each tenant patching its namespace and
+│       ├── ns50/                 # default-network annotation; per-site dirs hold each
+│       ├── ns10/                 # site's NAD(s): ns50 (IPv4, 3-site), ns10 (IPv6-only,
+│       ├── ns20/                 # 3-site), ns20 (dual-stack, 3-site), ns30 (IPv6-only,
+│       ├── ns30/                 # dfw only, 2 attachments), ns40 (IPv4-only, iad only,
+│       └── ns40/                 # 2 attachments) — ns30/ns40's two attachments are each
+│                                  # their own NAD+Deployment (distinct vpcattachment,
+│                                  # same vpc), not one NAD scaled to replicas: 2 — see
+│                                  # docs/tenants.md for why.
 ├── node_files/
 │   ├── dfw/     config.yaml
 │   ├── iad/     config.yaml

deploy/containerlab/docs/tenants.md

@@ -19,32 +19,54 @@ reachability across sites.
 
 They differ only in scope and addressing:
 
-| VPC    | Sites                     | Address families | VRF interface    | Notes |
-|--------|---------------------------|-------------------|-------------------|-------|
-| `ns50` | dfw, sjc, iad (3-site)    | IPv4 + public IPv6 ptp | `G000000050050V` | Also defines a `public` NAD with an IPv6 IPAM pool for external-connectivity testing. |
-| `ns10` | dfw, sjc, iad (3-site)    | IPv6-only (fd20 ULA) | `G000000010010V` | No `ipv4_subnet` at all. |
-| `ns20` | dfw, sjc, iad (3-site)    | Dual-stack (fd20 ULA + IPv4) | `G000000020020V` | Both families active; exercises the dual-stack IPAM path. |
-| `ns30` | dfw only, 2 replicas      | IPv6-only (fd20 ULA) | `G000000030030V` | Both pods land on `dfw-worker` and share one VRF — same-node connectivity, no cross-site hop. |
-| `ns40` | iad only, 2 replicas      | IPv4-only | `G000000040040V` | Both pods land on `iad-worker` (not `iad-worker-control`, which is tainted for the route-reflector role) and share one VRF. |
-
-The VRF interface name follows `G<vpc, zero-padded to 9><vpcattachment,
-zero-padded to 3>V` on every worker — e.g. `ns20` (`vpc="20"`,
-`vpcattachment="20"`) is `G000000020020V`.
+| VPC    | Sites                        | Address families | VRF interface    | Notes |
+|--------|------------------------------|-------------------|-------------------|-------|
+| `ns50` | dfw, sjc, iad (3-site)       | IPv4 + public IPv6 ptp | `G000000050V` | Also defines a `public` NAD with an IPv6 IPAM pool for external-connectivity testing. |
+| `ns10` | dfw, sjc, iad (3-site)       | IPv6-only (fd20 ULA) | `G000000010V` | No `ipv4_subnet` at all. |
+| `ns20` | dfw, sjc, iad (3-site)       | Dual-stack (fd20 ULA + IPv4) | `G000000020V` | Both families active; exercises the dual-stack IPAM path. |
+| `ns30` | dfw only, 2 attachments      | IPv6-only (fd20 ULA) | `G000000030V` | Two distinct attachments (`private`/`private-b`, distinct `vpcattachment` values, same `vpc`), each its own single-replica Deployment, both land on `dfw-worker` and share one VRF — same-node connectivity, no cross-site hop. |
+| `ns40` | iad only, 2 attachments      | IPv4-only | `G000000040V` | Two distinct attachments (`private`/`private-b`), each its own single-replica Deployment, both land on `iad-worker` (not `iad-worker-control`, which is tainted for the route-reflector role) and share one VRF. |
+
+The VRF interface name follows `G<vpc, zero-padded to 9>V` on every worker —
+e.g. `ns20` (`vpc="20"`) is `G000000020V`. Unlike the host/guest veth
+interface names (`G<vpc><vpcattachment>H`/`G<vpc><vpcattachment>G`, still
+per-attachment), the VRF carries no `vpcattachment` segment: it's shared by
+every attachment on that VPC on a given node, not created fresh per
+attachment — see [ns30's `vpcattachment` note](#ns30ns40-use-two-distinct-vpcattachment-values)
+below for why `ns30`/`ns40` deliberately use two attachments instead of one.
+
+### `ns30`/`ns40` use two distinct `vpcattachment` values
+
+`ns30` and `ns40` each define **two** NADs (`private` and `private-b`) with
+the same `vpc` but different `vpcattachment` values, each backing its own
+single-replica Deployment — not one NAD scaled to `replicas: 2`. This is
+required, not a style choice: `galactic-cni` derives its host/guest veth
+interface names from `(vpc, vpcAttachment)` alone, so two pods sharing one
+`vpcattachment` on the same node would collide on that name, and
+`internal/cni/veth`'s "stale veth" self-heal would delete whichever pod's
+veth got there second — silently breaking it without either pod's own CNI
+ADD ever reporting a failure. Two distinct `vpcattachment` values avoid the
+collision while still landing both pods in the one shared VRF their common
+`vpc` gets on this node (see the VRF interface table above).
 
 ### SRv6 USID Argument allocation
 
 Each site's tenant node advertises its own `/56` SRv6 locator block into the
 fabric. The low hextet of a pod's USID is `(Function << 12) | Argument`
 (`uFMT 48+16`, `internal/plumbing/ebpf/uformat`): `Function` is the constant
 `0xE` (`FunctionEndDT46`) for every plain L3 VRF attachment, and `Argument` is
-a 12-bit value `galactic-router` allocates per-node as the lowest unused slot
-in `[0x001, 0xFFF]` among that node's existing `BGPVRFInstance` CRDs
-(`allocateArgument`, `internal/cnibgp/bgp.go`) — **not** a decode of the NAD's
-`vpc`/`vpcattachment` values. Concretely, expect hextets in the
-`0xe001`–`0xefff` range; the exact value depends on allocation order (`ns50`
-is provisioned first in `task deploy`, then `ns10`, `ns20`, `ns30`, `ns40` in
-that order, each consuming the next free slot on each node it lands on).
-Always confirm the live value rather than trusting a table:
+a 12-bit value `galactic-router` allocates per **(VPC, node)** — not per
+attachment — as the lowest unused slot in `[0x001, 0xFFF]` among that node's
+existing `BGPVRFInstance` CRDs (`allocateArgument`, `internal/cnibgp/bgp.go`)
+— **not** a decode of the NAD's `vpc`/`vpcattachment` values. Every attachment
+sharing a VPC on a node (`ns30`'s and `ns40`'s two apiece) resolves to the
+same `BGPVRFInstance` and therefore the same Argument, so each VPC still
+consumes exactly one slot per node regardless of how many attachments land on
+it. Concretely, expect hextets in the `0xe001`–`0xefff` range; the exact
+value depends on allocation order (`ns50` is provisioned first in `task
+deploy`, then `ns10`, `ns20`, `ns30`, `ns40` in that order, each consuming the
+next free slot on each node it lands on). Always confirm the live value
+rather than trusting a table:
 
 ```bash
 docker exec dfw-control-plane kubectl get bgpvrfinstances -A
@@ -63,14 +85,19 @@ docker exec dfw-control-plane kubectl get bgpvrfinstances -A
 | dfw  | `ns20` | `fd20:20:ff01::/48`   | `172.21.1.0/24`   | —                   |
 | sjc  | `ns20` | `fd20:20:ff02::/48`   | `172.21.20.0/24`  | —                   |
 | iad  | `ns20` | `fd20:20:ff03::/48`   | `172.21.10.0/24`  | —                   |
-| dfw  | `ns30` | `fd20:30:ff01::/48`   | none               | —                   |
-| iad  | `ns40` | none                   | `172.40.10.0/24`  | —                   |
+| dfw  | `ns30` (`private`)   | `fd20:30:ff01::/48`   | none               | —                   |
+| dfw  | `ns30` (`private-b`) | `fd20:30:ff02::/48`   | none               | —                   |
+| iad  | `ns40` (`private`)   | none                   | `172.40.10.0/24`  | —                   |
+| iad  | `ns40` (`private-b`) | none                   | `172.40.20.0/24`  | —                   |
 
 `ns50`'s IPv4 pools and `ns20`'s are deliberately from distinct `/16` blocks
 (`172.20.0.0/16` vs. `172.21.0.0/16`) so the two VPCs' addressing never
-overlaps; `ns40`'s `172.40.0.0/16` is separate again. The IPv6 pools for
-`ns10`/`ns20`/`ns30` share the `fd20` ULA prefix, distinguished by the second
-hextet (`10`/`20`/`30`).
+overlaps; `ns40`'s `172.40.0.0/16` is separate again, split further into a
+`.10.0/24`/`.20.0/24` pair between its two attachments (same pattern `ns20`
+uses per-site). The IPv6 pools for `ns10`/`ns20`/`ns30` share the `fd20` ULA
+prefix, distinguished by the second hextet (`10`/`20`/`30`); `ns30`'s two
+attachments split further into `ff01`/`ff02` (same pattern `ns10`/`ns20` use
+per-site).
 
 ## Prerequisites
 
@@ -119,14 +146,17 @@ docker exec iad-control-plane kubectl get pods -n <namespace> -o wide
 ```
 
 For the 3-site VPCs (`ns50`, `ns10`, `ns20`), expect one `Running` pod per
-site. For the single-site VPCs, expect **two** `Running` pods, both on the
-one site's worker (`dfw-worker` for `ns30`, `iad-worker` for `ns40` — not
+site. For the single-site VPCs, expect **two** `Running` pods — one per
+attachment (`private`/`private-b`, distinct Deployments) — both on the one
+site's worker (`dfw-worker` for `ns30`, `iad-worker` for `ns40` — not
 `iad-worker-control`, which carries the route-reflector taint).
 
 ### Inspect a pod's VPC interface
 
 Every pod's VPC address lands on `eth0` (the `default-network` annotation
-replaces the pod's primary interface — see [Overview](#overview)):
+replaces the pod's primary interface — see [Overview](#overview)). For the
+3-site VPCs, `-l app=private` alone finds the (only) pod; for `ns30`/`ns40`,
+add `-l app=private-b` for the second attachment's pod:
 
 ```bash
 POD=$(docker exec dfw-control-plane kubectl get pods -n <namespace> -l app=private -o jsonpath='{.items[0].metadata.name}')
@@ -155,14 +185,17 @@ task verify:ns40   # same-node pod-to-pod, IPv4 (iad)
 `task verify` (via `task verify:scenarios`) runs all five. Use the scripts as
 the reference for how to resolve pod names/addresses by hand (`lib.sh`'s
 `pod_name`/`pod_ip4`/`pod_ip6`/`ping_pod` helpers) if you need to reproduce a
-step manually while debugging — e.g. to ping the `ns30` pods on dfw directly:
+step manually while debugging — e.g. to ping the `ns30` pods on dfw directly.
+`ns30`'s two pods are two distinct attachments/labels, not two replicas of
+one, so each is resolved by its own `pod_name` call rather than `pod_names`:
 
 ```bash
 source scripts/lib.sh
 NODE=$(control_plane dfw)
-mapfile -t PODS < <(pod_names "${NODE}" ns30 | tr ' ' '\n')
-IP=$(pod_ip6 "${NODE}" ns30 "${PODS[1]}")
-ping_pod "${NODE}" ns30 "${PODS[0]}" -6 "${IP}"
+POD_A=$(pod_name "${NODE}" ns30 app=private)
+POD_B=$(pod_name "${NODE}" ns30 app=private-b)
+IP_B=$(pod_ip6 "${NODE}" ns30 "${POD_B}")
+ping_pod "${NODE}" ns30 "${POD_A}" -6 "${IP_B}"
 ```
 
 ## Troubleshooting
@@ -187,9 +220,11 @@ task deploy:ns50   # or whichever VPC's pods aren't getting IPs
 
 ### BGPAdvertisements not created
 
-The CNI creates a `BGPAdvertisement` CRD per pod on attach. The 3-site VPCs
-get one advertisement per site; the single-site VPCs get **two** (one per
-pod, both on the same site):
+The CNI creates a `BGPAdvertisement` CRD per attachment on attach. The 3-site
+VPCs get one advertisement per site; the single-site VPCs get **two** (one
+per attachment, both on the same site) — but only **one** `BGPVRFInstance`
+each, since that CRD is shared by every attachment on a VPC/node rather than
+created per attachment (see [SRv6 USID Argument allocation](#srv6-usid-argument-allocation)):
 
 ```bash
 docker exec dfw-control-plane kubectl get bgpadvertisements -n galactic-system
@@ -223,12 +258,12 @@ docker exec dfw-worker dmesg | grep galactic
    interface name from the [Overview](#overview) table:
 
    ```bash
-   docker exec dfw-worker ip -4 route show table G000000050050V   # ns50
-   docker exec dfw-worker ip -6 route show table G000000010010V   # ns10
-   docker exec dfw-worker ip -6 route show table G000000020020V   # ns20 (IPv6 leg)
-   docker exec dfw-worker ip -4 route show table G000000020020V   # ns20 (IPv4 leg)
-   docker exec dfw-worker ip -6 route show table G000000030030V   # ns30
-   docker exec iad-worker ip -4 route show table G000000040040V   # ns40
+   docker exec dfw-worker ip -4 route show table G000000050V   # ns50
+   docker exec dfw-worker ip -6 route show table G000000010V   # ns10
+   docker exec dfw-worker ip -6 route show table G000000020V   # ns20 (IPv6 leg)
+   docker exec dfw-worker ip -4 route show table G000000020V   # ns20 (IPv4 leg)
+   docker exec dfw-worker ip -6 route show table G000000030V   # ns30 (both attachments, shared VRF)
+   docker exec iad-worker ip -4 route show table G000000040V   # ns40 (both attachments, shared VRF)
    ```
 
    An empty table for a family the VPC doesn't use (e.g. IPv4 on `ns10`/`ns30`,

deploy/containerlab/resources/tenants/ns30/base/kustomization.yaml

@@ -1,5 +1,14 @@
-# ns30 is single-site (dfw only, see ../dfw/) — 2 replicas land two pods on
-# the same worker, attached to the same VPC/NAD, for same-node same-VPC
+# ns30 is single-site (dfw only, see ../dfw/). This is the first of its two
+# attachments — ../dfw/pod-b.yaml is the second, on its own NAD
+# (ns30/private-b). They deliberately use two distinct vpcattachment values
+# under the same vpc rather than one NAD scaled to replicas: 2: galactic-cni
+# derives its host/guest veth interface names from (vpc, vpcAttachment)
+# alone (internal/plumbing/intf), so two pods sharing one vpcAttachment on
+# the same node collide on that name — internal/cni/veth's "stale veth"
+# self-heal would delete whichever pod's veth got there second, silently
+# breaking it (see the incident this fixture layout replaced). Two distinct
+# vpcattachments avoids that collision and lands both pods in one shared VRF
+# (internal/plumbing/vrf is keyed by vpc alone) for same-node, same-VPC
 # pod-to-pod connectivity testing rather than cross-site.
 namespace: ns30
 resources:
@@ -18,6 +27,3 @@ patches:
           metadata:
             annotations:
               v1.multus-cni.io/default-network: ns30/private
-replicas:
-  - name: netshoot
-    count: 2

deploy/containerlab/resources/tenants/ns30/dfw/kustomization.yaml

@@ -1,3 +1,5 @@
 resources:
   - ../base
   - nad.yaml
+  - pod-b.yaml
+  - nad-b.yaml

deploy/containerlab/resources/tenants/ns30/dfw/nad-b.yaml

@@ -0,0 +1,31 @@
+---
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: private-b
+  namespace: ns30
+spec:
+  config: |-
+    {
+      "cniVersion": "1.0.0",
+      "name": "private-b",
+      "plugins": [
+        {
+          "type": "galactic-cni",
+          "vpc": "30",
+          "vpcattachment": "31",
+          "namespace": "galactic-system",
+          "ipam": {
+            "type": "galactic-ipam",
+            "ipv6_subnet": "fd20:30:ff02::/48",
+            "address_families": ["ipv6"]
+          }
+        },
+        {
+          "type": "galactic-bgp",
+          "vpc": "30",
+          "vpcattachment": "31",
+          "namespace": "galactic-system"
+        }
+      ]
+    }

deploy/containerlab/resources/tenants/ns30/dfw/pod-b.yaml

@@ -0,0 +1,38 @@
+# The second of ns30's two attachments — see ../base/kustomization.yaml's
+# doc comment for why this is a standalone Deployment on its own NAD
+# (nad-b.yaml, vpcattachment "31") rather than a second replica of ../base's
+# Deployment. Mirrors ../../base/pod.yaml (the shared tenant Deployment
+# template) directly rather than composing it through kustomize, since a
+# second copy needs a different name/label/annotation than kustomize's
+# nameSuffix + patch ordering can reliably guarantee.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: netshoot-b
+  namespace: ns30
+  labels:
+    app: private-b
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: private-b
+  template:
+    metadata:
+      labels:
+        app: private-b
+      annotations:
+        v1.multus-cni.io/default-network: ns30/private-b
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                  - key: node-role.kubernetes.io/control-plane
+                    operator: DoesNotExist
+      containers:
+        - name: netshoot
+          image: nicolaka/netshoot:latest
+          imagePullPolicy: IfNotPresent
+          command: ["sleep", "infinity"]

deploy/containerlab/resources/tenants/ns40/base/kustomization.yaml

@@ -1,7 +1,13 @@
-# ns40 is single-site (iad only, see ../iad/) — 2 replicas land two pods on
-# iad-worker (the only untainted worker in that cluster), attached to the
-# same VPC/NAD, for same-node same-VPC pod-to-pod connectivity testing
-# rather than cross-site.
+# ns40 is single-site (iad only, see ../iad/), landing on iad-worker (the
+# only untainted worker in that cluster). This is the first of its two
+# attachments — ../iad/pod-b.yaml is the second, on its own NAD
+# (ns40/private-b). They deliberately use two distinct vpcattachment values
+# under the same vpc rather than one NAD scaled to replicas: 2 — see
+# ../../ns30/base/kustomization.yaml's doc comment for why (galactic-cni's
+# host/guest veth naming collides when two pods share one vpcattachment on
+# the same node). Two distinct vpcattachments avoids that collision and
+# lands both pods in one shared VRF for same-node, same-VPC pod-to-pod
+# connectivity testing rather than cross-site.
 namespace: ns40
 resources:
   - ../../base
@@ -19,6 +25,3 @@ patches:
           metadata:
             annotations:
               v1.multus-cni.io/default-network: ns40/private
-replicas:
-  - name: netshoot
-    count: 2

deploy/containerlab/resources/tenants/ns40/iad/kustomization.yaml

@@ -1,3 +1,5 @@
 resources:
   - ../base
   - nad.yaml
+  - pod-b.yaml
+  - nad-b.yaml

deploy/containerlab/resources/tenants/ns40/iad/nad-b.yaml

@@ -0,0 +1,31 @@
+---
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: private-b
+  namespace: ns40
+spec:
+  config: |-
+    {
+      "cniVersion": "1.0.0",
+      "name": "private-b",
+      "plugins": [
+        {
+          "type": "galactic-cni",
+          "vpc": "40",
+          "vpcattachment": "41",
+          "namespace": "galactic-system",
+          "ipam": {
+            "type": "galactic-ipam",
+            "ipv4_subnet": "172.40.20.0/24",
+            "address_families": ["ipv4"]
+          }
+        },
+        {
+          "type": "galactic-bgp",
+          "vpc": "40",
+          "vpcattachment": "41",
+          "namespace": "galactic-system"
+        }
+      ]
+    }