Replace L2 broadcast with sync via CRD

.dockerignore

@@ -0,0 +1,11 @@
+# SPDX-FileCopyrightText: 2026 SAP SE or an SAP affiliate company
+#
+# SPDX-License-Identifier: Apache-2.0
+
+.git
+go-pmtud
+build/
+.superpowers/
+lab/
+.github/
+*.md

.gitignore

@@ -1,4 +1,9 @@
-/main
+/go-pmtud
 /build/
 
+.vscode/
 .idea
+.pi/
+.superpowers/
+.serena
+.vscode/

CONTRIBUTING.md

@@ -0,0 +1,95 @@
+<!--
+SPDX-FileCopyrightText: 2026 SAP SE or an SAP affiliate company
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Contributing
+
+## Requirements
+
+- Go 1.26+
+- [controller-gen](https://pkg.go.dev/sigs.k8s.io/controller-tools/cmd/controller-gen) — for regenerating CRD manifests and deepcopy code
+- [setup-envtest](https://pkg.go.dev/sigs.k8s.io/controller-runtime/tools/setup-envtest) — for running component tests against a real API server binary
+
+`make prepare-static-check` installs all static analysis tools. `make install-setup-envtest` installs setup-envtest.
+
+## Running tests
+
+```sh
+make check
+```
+
+This runs `generate`, all unit and component tests (with coverage), and the full static analysis suite.
+
+### Unit tests only
+
+```sh
+go test ./...
+```
+
+### Component tests (envtest)
+
+Component tests verify a single component against a real dependency spun up in-process. The CRD backend tests (`internal/relay/crd`) use [envtest](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest), which starts a real Kubernetes API server binary but requires no cluster. `make check` handles this automatically via `setup-envtest`. To run them in isolation:
+
+```sh
+KUBEBUILDER_ASSETS=$(setup-envtest use 1.35 -p path) go test ./internal/relay/crd/...
+```
+
+If `KUBEBUILDER_ASSETS` is not set, the test binary prints a notice and exits cleanly (not a failure).
+
+### Integration tests (kind)
+
+Integration tests verify that go-pmtud works correctly inside a real multi-node cluster, including actual NFLOG capture, TUN injection, and CRD relay across nodes. They are not yet automated. To test manually with [kind](https://kind.sigs.k8s.io/):
+
+1. Create a multi-node cluster: `kind create cluster --config kind-config.yaml` (use a config with at least two worker nodes)
+2. Install the CRD: `kubectl apply -f crd/`
+3. Build and load the image: `docker build -t go-pmtud . && kind load docker-image go-pmtud`
+4. Deploy as a DaemonSet with `--relay-backend=crd` and the appropriate iptables init container
+5. Trigger an ICMP frag-needed packet (e.g. send a large UDP packet with DF set toward a destination with a lower MTU path)
+6. Verify the packet appears on other nodes via `kubectl logs` or by inspecting the route cache MTU entry on the target node
+
+### End-to-end tests (real cluster)
+
+End-to-end testing in a real cluster (production or staging) is performed manually before releases. It follows the same steps as the kind integration test but validates behaviour under real ECMP traffic patterns. No scripted harness exists yet.
+
+To confirm a relay is working, check the route cache on a node that should have received the relayed packet:
+
+```sh
+ip route get <destination-IP>
+# Expected: cache entry with a reduced mtu value
+```
+
+## Code generation
+
+CRD manifests, deepcopy functions, and apply-configuration types are generated from the types in `api/v1alpha1`:
+
+```sh
+make generate
+```
+
+Run this after any change to the API types and commit the generated files.
+
+## Static analysis
+
+```sh
+make static-check
+```
+
+Runs golangci-lint, shellcheck, typos, and license-header checks. Individual targets:
+
+| Target | Tool |
+|---|---|
+| `run-golangci-lint` | golangci-lint |
+| `run-shellcheck` | shellcheck |
+| `run-typos` | typos spell checker |
+| `check-license-headers` | addlicense + reuse |
+| `check-dependency-licenses` | go-licence-detector |
+
+## License headers
+
+All `.go` files require an SPDX license header. To add or fix headers:
+
+```sh
+make license-headers
+```

Dockerfile

@@ -2,17 +2,19 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-FROM golang:1.25-alpine AS builder
+FROM golang:1.26-alpine AS builder
 
 WORKDIR /go/src/github.com/sapcc/go-pmtud
-ADD go.mod go.sum ./
+COPY go.mod go.sum ./
 RUN go mod download
-ADD . .
-RUN go build -v -o /go-pmtud cmd/go-pmtud/main.go
+COPY . .
+RUN CGO_ENABLED=0 go build -v -o /go-pmtud cmd/go-pmtud/main.go
 
-FROM ubuntu:noble
+FROM alpine:latest AS certs
+RUN apk add --no-cache ca-certificates
+
+FROM scratch
 LABEL source_repository="https://github.com/sapcc/go-pmtud"
-RUN apt-get update && apt-get install -y \
-    iptables iproute2 \
-    && rm -rf /var/lib/apt/lists/*
+COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=builder /go-pmtud /go-pmtud
+ENTRYPOINT ["/go-pmtud"]

README.md

@@ -25,87 +25,122 @@ More details in this blog post by Cloudflare: [Path MTU discovery in practice](h
 
 go-pmtud replicates ICMP Destination Unreachable packets to all nodes in same Kubernetes cluster, so that the sender gets awareness that it has to use smaller packets for a particular destination.
 
-## Concept
+## Architecture
 
-1. ICMP Destination Unreachable message (type 3 code 4 message) packets are filtered and sent to specific NFlog group by iptables.
+```
+iptables (NFLOG group 33)
+        │
+        ▼
+  nflog controller          ← captures ICMP type 3 code 4 packets
+        │
+        ▼
+  relay backend             ← distributes packets to all (on other node)
+        │
+        ▼
+  TUN device (pmtud0)       ← injects replicated packets into the network stack
+```
 
-2. go-pmtud replicates ICMP packets to all nodes in same Kubernetes cluster, so that the sender pod gets awareness that it has to use smaller packets for particular destination.
+Each node runs go-pmtud as a DaemonSet. When an ICMP frag-needed packet arrives, iptables redirects it to an NFLOG group. go-pmtud reads it, replicates it via the configured relay backend, and re-injects it on every other node through a TUN device.
 
-```
-Exec into the pod:
+## Relay Backends
 
-# ip route get 192.168.100.10
-192.168.100.10 via 192.100.0.1 dev eth0 src 192.100.0.50
-    cache  expires 484sec mtu 9000  <<<< connection is failing
+go-pmtud supports two relay backends, selected with `--relay-backend`.
 
-# ip route get 192.168.100.10
-192.168.100.10 via 192.100.0.1 dev eth0 src 192.100.0.50
-    cache  expires 484sec mtu 8996  <<<< correct MTU information, connection is working
-```
+### `crd` (default)
 
-## Build
+Relay packets are stored as `PMTUNodeRelay` Kubernetes CRD objects. Every node watches the API server for new objects and injects those originating from other nodes. Expired objects are garbage-collected by each node for its own packets.
 
-Build from source:
+This backend requires no direct network connectivity between nodes — it uses the Kubernetes API server as the transport. It does require the `PMTUNodeRelay` CRD to be installed in the cluster:
 
-```
-go mod download
-go build -v -o /go-pmtud cmd/go-pmtud/main.go
+```sh
+kubectl apply -f crd/pmtud.cloud.sap_pmtunoderelays.yaml
 ```
 
-Build a Docker image:
+Relay objects are namespaced; set the namespace via `--relay-namespace` or the `POD_NAMESPACE` environment variable.
 
-```
-docker build -t go-pmtud .
-```
+### `udp`
+
+Relay packets are sent directly to a peer list over UDP. This is the original replication mechanism. It requires all node IPs to be provided via the node controller and needs UDP port `4390` (configurable) to be open between nodes.
+
+## CLI Options
+
+| Flag | Default | Description |
+|---|---|---|
+| `--nodename` | | Node hostname, used as a metric label |
+| `--relay-backend` | `crd` | Relay backend: `crd` or `udp` |
+| `--relay-namespace` | `$POD_NAMESPACE` | Namespace for CRD relay objects (`crd` backend only) |
+| `--relay-gc-interval` | `60s` | How often expired CRD relay objects are garbage-collected |
+| `--replication-port` | `4390` | UDP port for packet replication (`udp` backend only) |
+| `--nflog_group` | `33` | NFLOG group number |
+| `--ttl` | `1` | TTL of re-injected ICMP packets |
+| `--ignore-networks` | | Comma-separated CIDRs — packets from these sources are not relayed |
+| `--metrics_port` | `:30040` | Prometheus metrics endpoint |
+| `--health_port` | `:30041` | Healthz endpoint |
+| `--kube_context` | | Kubeconfig context to use |
+
+All flags can also be set via environment variables prefixed with `PMTUD_` (e.g. `PMTUD_NODENAME`).
 
-## go-pmtud options
+## Metrics
 
-Following options are available:
+Prometheus metrics are exposed on `--metrics_port` (default `:30040`). Every metric carries a `node` label. The three stages of the relay pipeline each map to exactly one counter:
 
-1. peers - resend ICMP frag-needed packets to this peer list.
-2. iface - interface that listens for ICMP packets and resends them to other peers.
-3. nodename - node hostname, used for metric label.
-4. nflog-group - NFLOG group, set to 33 in our case.
-5. metrics-port - Port for Prometheus metrics (30040 by default).
-6. ttl - TTL of replicated ICMP packets.
-7. ignore-networks - Do not resend ICMP frag-needed packets originated from specified networks
+| Metric | Labels | Meaning |
+|---|---|---|
+| `go_pmtud_recv_packets_total` | `node`, `source_ip` | ICMP frag-needed packets **captured from the kernel** (nflog) on this node. Capture only. |
+| `go_pmtud_relay_send_total` | `node`, `result` | Relay send attempts by outcome: `created`, `deduplicated`, or `error` (`crd` backend). |
+| `go_pmtud_injected_packets_total` | `node`, `source` | Packets **received from a peer** and injected via the TUN device. `source` is the relaying peer (node name for `crd`, peer IP for `udp`). |
 
-If `iface` is empty, it finds out the outgoing interface based on the default route. 
+Supporting metrics:
 
-## Example - go-pmtud Daemonset
+| Metric | Labels | Meaning |
+|---|---|---|
+| `go_pmtud_sent_packets_total` | `node` | Packets sent to peers (`udp` backend, per successful send). |
+| `go_pmtud_sent_packets_peer` | `node`, `peer` | Packets sent, per peer (`udp` backend). |
+| `go_pmtud_sent_error_peer_total` | `node`, `peer` | Send errors, per peer (`udp` backend). |
+| `go_pmtud_error_total` | `node` | General error counter. |
+| `go_pmtud_callback_duration_seconds` | `node` | Histogram of nflog callback duration. |
 
-go-pmtud can run as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), [example](https://github.com/sapcc/helm-charts/blob/master/system/go-pmtud).
+Useful queries:
 
-Example values.yaml:
+```promql
+# ICMP frag-needed captured cluster-wide, per minute
+sum(rate(go_pmtud_recv_packets_total[1m])) * 60
 
+# CR creation rate (actual etcd writes) — crd backend
+sum(rate(go_pmtud_relay_send_total{result="created"}[1m])) * 60
+
+# Deduplication ratio — how many sends collapsed onto an existing CR
+sum(rate(go_pmtud_relay_send_total{result="deduplicated"}[1m]))
+  / sum(rate(go_pmtud_relay_send_total{result=~"created|deduplicated"}[1m]))
 ```
-images:
-  iptables:
-    repository: sapcc/iptables
-    tag: v20191226161919
-  pmtud:
-    repository: sapcc/go-pmtud
-    tag: latest
-
-iptables:
-  nflogGroup: 33
-  ignoreSourceNetworks: 192.168.100.0/24
-
-pmtud:
-  ttl: 10
-  metricsPort: 30040
-  interface: eth0
-  peers: 192.168.100.2, 192.168.100.3, 192.168.100.4, 192.168.100.5, 192.100.0.50
+
+## Build
+
+```sh
+go mod download
+go build -o go-pmtud ./cmd/go-pmtud
 ```
 
-## Example - iptables and NFlog
+Docker image:
 
-There is an iptables rule on each node that redirects ICMP Destination Unreachable` packets to NFlog group nr. 33:
+```sh
+docker build -t go-pmtud .
+```
 
-`iptables -t raw -D PREROUTING -i <interface> -p icmp -m icmp --icmp-type 3/4 --j NFLOG --nflog-group 33`
+## iptables and NFlog
 
-Important: we need ignore packets from summarized source networks of all nodes in the local cluster to avoid re-sending loops. Use `ignore-networks` option for this. 
-This means a node will not re-send already retransmitted ICMP messages. It will only resend messages that are usually originated by routers on the path. 
+Each node needs an iptables rule that redirects ICMP Destination Unreachable packets to the NFLOG group. The rule **must** exclude the `pmtud0` TUN interface to prevent replication loops:
+
+```sh
+iptables -t raw -A PREROUTING -p icmp -m icmp --icmp-type 3/4 ! -i pmtud0 -j NFLOG --nflog-group 33
+```
+
+Optionally use `--ignore-networks` to suppress packets from known infrastructure networks (e.g. node subnets) as an additional safety layer.
+
+## Example — DaemonSet
+
+go-pmtud is designed to run as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/). A Helm chart is available at [sapcc/helm-charts](https://github.com/sapcc/helm-charts/blob/master/system/go-pmtud).
 
 ## License
-This project is licensed under the Apache2 License - see the [LICENSE](LICENSE) file for details
+
+This project is licensed under the Apache 2.0 License — see the [LICENSE](LICENSE) file for details.

REUSE.toml

@@ -28,3 +28,12 @@ path = [
 ]
 SPDX-FileCopyrightText = "SAP SE or an SAP affiliate company"
 SPDX-License-Identifier = "Apache-2.0"
+
+[[annotations]]
+path = [
+  "api/v1alpha1/zz_generated.deepcopy.go",
+  "config/rbac/role.yaml",
+  "crd/pmtud.cloud.sap_pmtunoderelays.yaml",
+]
+SPDX-FileCopyrightText = "SAP SE or an SAP affiliate company"
+SPDX-License-Identifier = "Apache-2.0"

api/v1alpha1/groupversion_info.go

@@ -0,0 +1,17 @@
+// SPDX-FileCopyrightText: 2026 SAP SE or an SAP affiliate company
+// SPDX-License-Identifier: Apache-2.0
+
+// +kubebuilder:object:generate=true
+// +groupName=pmtud.cloud.sap
+package v1alpha1
+
+import (
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	"sigs.k8s.io/controller-runtime/pkg/scheme"
+)
+
+var (
+	GroupVersion  = schema.GroupVersion{Group: "pmtud.cloud.sap", Version: "v1alpha1"}
+	SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
+	AddToScheme   = SchemeBuilder.AddToScheme
+)