Skip to content

Replace L2 broadcast with sync via CRD - #76

Closed
rkschamer wants to merge 88 commits into
sapcc:masterfrom
rkschamer:d053727/eliminate-l2-via-crds
Closed

Replace L2 broadcast with sync via CRD #76
rkschamer wants to merge 88 commits into
sapcc:masterfrom
rkschamer:d053727/eliminate-l2-via-crds

Conversation

@rkschamer

@rkschamer rkschamer commented Aug 19, 2026

Copy link
Copy Markdown

Summary

#69 delivers ICMP 3/4 messages using UDP across L2 boundaries. This requires every node in the cluster have an open port (e.g. 4390). This PR is eliminating this drawback by using CRD/CRs to exchange the received ICMP 3/4 messages between nodes, and hence used the API server to exchange the information.

The PR also abstracts the used relay (udp or crd) so that different relays can be used, depending on the use case.

What changes

New Relay interface (internal/relay)

type Relay interface {
    Send(ctx context.Context, pkt RelayPacket) error
    Start(ctx context.Context, inject func(payload []byte) error) error
}

Send is called from the NFLOG hot path; Start runs the receive loop (manager.Runnable). Capture (NFLOG) and injection (TUN pmtud0) stay shared across backends.

Shared TUN injector

createTUN + configureTUNNetlink extracted from internal/receiver into internal/relay/tun_linux.go as an Injector. One owner of the TUN fd; avoids double-open and centralises the loop-prevention contract (! -i pmtud0).

UDP backend (refactor, identical behavior)

Existing UDP path moved under the Relay interface. All existing tests preserved.

CRD backend

  • Resource: PMTUNodeRelay (namespaced, CRD generated by controller-gen from api/v1alpha1/pmtunoderelay_types.go).
  • Object name: <srcNode>--<sha256(payload)[:8]> — deterministic; duplicate events collapse to one object (AlreadyExists is a no-op).
  • Spec: sourceNode, payload (base64 raw ICMP), expiresAt (RFC 3339).
  • Delivery: broadcast — every daemon pod injects; peers skip objects where sourceNode == own NODE_NAME.
  • GC: delete-after-inject (happy path) + TTL sweep on --relay-gc-interval ticker (default 60 s). Any daemon pod sweeps any expired object — no leader election needed.
  • RBAC: namespaced Role (create/get/list/watch/delete on pmtunoderelays), no ClusterRole.

Backend selection

--relay-backend=udp|crd, default crd. --replication-port is rejected at startup if backend is not udp.

Why namespaced and broadcast

  • Namespaced over cluster-scoped: least privilege; Role instead of ClusterRole; scoped list/watch/GC.
  • Broadcast over targeted: util.CalcSrcDst yields the inner sender IP — a pod IP in Calico BGP clusters, not a node IP. Reverse lookup to a node would require a pod→node watch. Broadcast mirrors proven UDP semantics; PMTU events are rare (cold paths), so targeting is a premature optimisation.

Testing

Concern Unit Kind E2E
Relay interface contract
UDP backend (unchanged)
CRD create/watch/inject/delete ✅ (envtest)
CRD GC of expired objects
Kernel PMTU cache update
Loop prevention (! -i pmtud0)

Non-Goals

  • Running multiple backends simultaneously.
  • Targeted CRD delivery / pod→node resolution (deferred).
  • Helm chart changes (separate repo, follow-up).
  • IPv6 or encryption changes.

defo89 and others added 30 commits May 1, 2026 15:58
Two Kind clusters in separate Docker networks connected by a router
container with MTU 1500 transit path. Simulates the real-world scenario
of ICMP fragmentation-needed generation and go-pmtud UDP replication
across L3 boundaries.

Includes setup/teardown scripts, DaemonSet manifest, podinfo workload,
traffic generation, tcpdump observation helpers, and e2e validation.
Add --relay-backend flag (udp/crd, default udp), --relay-namespace flag with
POD_NAMESPACE env var fallback, and --relay-gc-interval flag (default 60s).
Validate backend value and resolve namespace early in preRunRootCmd; fail fast
if CRD backend selected without resolvable namespace. Implements spec section
'Runtime backend selection' and 'CRD backend namespace resolution'.
Add kubebuilder RBAC marker in doc.go to declare that the relay package
requires permissions to get, list, watch, create, and delete pmtunoderelays
resources. Running 'make generate' produces the corresponding ClusterRole
with the pmtunoderelays resource included.
Add --relay-backend flag and POD_NAMESPACE env var to daemonset for runtime
backend selection (udp or crd). Conditionally install PMTUNodeRelay CRD when
crd backend is used. Document 4-namespace netns fast-path in README explaining
how CRD backend works transparently across namespace boundaries. Add test-backends
Makefile target and test script for validating both relay implementations.

Changes:
- lab/manifests/pmtud-daemonset.yaml: Add --relay-backend and POD_NAMESPACE vars
- lab/manifests/crd.yaml: New, CRD definition for lab manifests
- lab/manifests/pmtud-daemonset-crd.yaml: Variant with crd backend hardcoded
- lab/scripts/deploy-pmtud.sh: Conditional CRD deployment on RELAY_BACKEND env
- lab/scripts/test-relay-backends.sh: New, validates both UDP and CRD backends
- lab/README.md: Document relay backends, 4-namespace fast-path, usage examples
- lab/Makefile: Add test-backends target

Backward compatible: default remains UDP backend.
…kage

Register v1alpha1.AddToScheme, build relay backend via relay.New with Deps,
set on nflog controller, and add relay.Runnable to manager. Remove UDP
receiver and internal/receiver package (content moved to relay). Build and
tests pass clean.
- Runs E2E tests nightly (02:00 UTC) or on manual trigger
- Label-gated: runs on pull_request when 'e2e' label is present
- Loops over both UDP and CRD relay backends with pod verification
- Asserts both backends successfully pass E2E tests
- Includes diagnostic collection on failure and cleanup
fix(relay): remove consumer-side CR delete; source-node-scoped TTL GC
chore(repo): untrack build binary + sdd scratch, add .dockerignore
feat(crd): add field validation + printer columns
fix(rbac): grant pmtunoderelays in lab ClusterRole
ci(e2e): add concurrency, job timeout, real build step
Deploy go-pmtud CRD from crd/ (generated by controller-gen via make generate)
instead of maintaining a manual copy in lab/manifests/. Eliminates duplication
risk and keeps the source of truth in one place.
Replace ~785 lines of bash lab provisioning + test scripts with a single
Go Ginkgo/Gomega e2e suite. Kind clusters via the Kind Go API (eliminates
the ~/.kube/config merge-corruption failure class); docker network/exec
plumbing behind thin wrappers (no docker SDK); client-go typed assertions;
Eventually-based convergence polling; udp+crd backend matrix in one run.
Gated behind a //go:build e2e tag so `go test ./...` stays green.
@rkschamer
rkschamer requested a review from defo89 August 20, 2026 13:20
@rkschamer
rkschamer marked this pull request as ready for review August 20, 2026 13:22
@rkschamer rkschamer changed the title [WIP] Replace L2 broadcast with sync via CRD Replace L2 broadcast with sync via CRD Aug 20, 2026
@rkschamer rkschamer closed this by deleting the head repository Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants