Replace L2 broadcast with sync via CRD - #76
Closed
rkschamer wants to merge 88 commits into
Closed
Conversation
Two Kind clusters in separate Docker networks connected by a router container with MTU 1500 transit path. Simulates the real-world scenario of ICMP fragmentation-needed generation and go-pmtud UDP replication across L3 boundaries. Includes setup/teardown scripts, DaemonSet manifest, podinfo workload, traffic generation, tcpdump observation helpers, and e2e validation.
Add --relay-backend flag (udp/crd, default udp), --relay-namespace flag with POD_NAMESPACE env var fallback, and --relay-gc-interval flag (default 60s). Validate backend value and resolve namespace early in preRunRootCmd; fail fast if CRD backend selected without resolvable namespace. Implements spec section 'Runtime backend selection' and 'CRD backend namespace resolution'.
Add kubebuilder RBAC marker in doc.go to declare that the relay package requires permissions to get, list, watch, create, and delete pmtunoderelays resources. Running 'make generate' produces the corresponding ClusterRole with the pmtunoderelays resource included.
Add --relay-backend flag and POD_NAMESPACE env var to daemonset for runtime backend selection (udp or crd). Conditionally install PMTUNodeRelay CRD when crd backend is used. Document 4-namespace netns fast-path in README explaining how CRD backend works transparently across namespace boundaries. Add test-backends Makefile target and test script for validating both relay implementations. Changes: - lab/manifests/pmtud-daemonset.yaml: Add --relay-backend and POD_NAMESPACE vars - lab/manifests/crd.yaml: New, CRD definition for lab manifests - lab/manifests/pmtud-daemonset-crd.yaml: Variant with crd backend hardcoded - lab/scripts/deploy-pmtud.sh: Conditional CRD deployment on RELAY_BACKEND env - lab/scripts/test-relay-backends.sh: New, validates both UDP and CRD backends - lab/README.md: Document relay backends, 4-namespace fast-path, usage examples - lab/Makefile: Add test-backends target Backward compatible: default remains UDP backend.
…kage Register v1alpha1.AddToScheme, build relay backend via relay.New with Deps, set on nflog controller, and add relay.Runnable to manager. Remove UDP receiver and internal/receiver package (content moved to relay). Build and tests pass clean.
- Runs E2E tests nightly (02:00 UTC) or on manual trigger - Label-gated: runs on pull_request when 'e2e' label is present - Loops over both UDP and CRD relay backends with pod verification - Asserts both backends successfully pass E2E tests - Includes diagnostic collection on failure and cleanup
fix(relay): remove consumer-side CR delete; source-node-scoped TTL GC chore(repo): untrack build binary + sdd scratch, add .dockerignore
feat(crd): add field validation + printer columns fix(rbac): grant pmtunoderelays in lab ClusterRole ci(e2e): add concurrency, job timeout, real build step
Deploy go-pmtud CRD from crd/ (generated by controller-gen via make generate) instead of maintaining a manual copy in lab/manifests/. Eliminates duplication risk and keeps the source of truth in one place.
Replace ~785 lines of bash lab provisioning + test scripts with a single Go Ginkgo/Gomega e2e suite. Kind clusters via the Kind Go API (eliminates the ~/.kube/config merge-corruption failure class); docker network/exec plumbing behind thin wrappers (no docker SDK); client-go typed assertions; Eventually-based convergence polling; udp+crd backend matrix in one run. Gated behind a //go:build e2e tag so `go test ./...` stays green.
rkschamer
marked this pull request as ready for review
August 20, 2026 13:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#69 delivers ICMP 3/4 messages using UDP across L2 boundaries. This requires every node in the cluster have an open port (e.g. 4390). This PR is eliminating this drawback by using CRD/CRs to exchange the received ICMP 3/4 messages between nodes, and hence used the API server to exchange the information.
The PR also abstracts the used relay (udp or crd) so that different relays can be used, depending on the use case.
What changes
New
Relayinterface (internal/relay)Sendis called from the NFLOG hot path;Startruns the receive loop (manager.Runnable). Capture (NFLOG) and injection (TUNpmtud0) stay shared across backends.Shared TUN injector
createTUN+configureTUNNetlinkextracted frominternal/receiverintointernal/relay/tun_linux.goas anInjector. One owner of the TUN fd; avoids double-open and centralises the loop-prevention contract (! -i pmtud0).UDP backend (refactor, identical behavior)
Existing UDP path moved under the
Relayinterface. All existing tests preserved.CRD backend
PMTUNodeRelay(namespaced, CRD generated by controller-gen fromapi/v1alpha1/pmtunoderelay_types.go).<srcNode>--<sha256(payload)[:8]>— deterministic; duplicate events collapse to one object (AlreadyExistsis a no-op).sourceNode,payload(base64 raw ICMP),expiresAt(RFC 3339).sourceNode == own NODE_NAME.--relay-gc-intervalticker (default 60 s). Any daemon pod sweeps any expired object — no leader election needed.Role(create/get/list/watch/deleteonpmtunoderelays), noClusterRole.Backend selection
--relay-backend=udp|crd, defaultcrd.--replication-portis rejected at startup if backend is notudp.Why namespaced and broadcast
Roleinstead ofClusterRole; scoped list/watch/GC.util.CalcSrcDstyields the inner sender IP — a pod IP in Calico BGP clusters, not a node IP. Reverse lookup to a node would require a pod→node watch. Broadcast mirrors proven UDP semantics; PMTU events are rare (cold paths), so targeting is a premature optimisation.Testing
! -i pmtud0)Non-Goals