Skip to content

feat(replication): replace L2 broadcast with UDP unicast - #69

Open
defo89 wants to merge 3 commits into
masterfrom
feature/eliminate-l2-dependency
Open

feat(replication): replace L2 broadcast with UDP unicast#69
defo89 wants to merge 3 commits into
masterfrom
feature/eliminate-l2-dependency

Conversation

@defo89

@defo89 defo89 commented May 1, 2026

Copy link
Copy Markdown
Member

Summary

Eliminates the Layer 2 adjacency requirement for ICMP fragmentation-needed packet replication between Kubernetes cluster nodes. Replaces raw Ethernet frame broadcasting (ARP + MAC resolution) with UDP unicast.

Changes

  • Removed: internal/arp/ package, L2 config fields, --iface_names/--iface_mtu/--arp-timeout-seconds/--node-timeout-minutes flags
  • Added: UDP sender (persistent socket with WriteTo), UDP receiver with TUN device injection, internal/packet/ parser
  • Added: --replication-port (default 4390), --ignore-networks flags
  • Refactored: Node reconciler stores peer IPs directly (no MAC resolution)
  • Security: Receiver validates UDP source IP against known peer list before TUN injection
  • Loop prevention: TUN device named pmtud0 deterministically; iptables NFLOG rule must exclude it (! -i pmtud0)

Key Design Decisions

  1. TUN device for injection — forces packets through kernel receive path (ip_input → icmp_rcv → icmp_unreach) to correctly update PMTU cache
  2. Peer IP validation — rejects packets from unknown sources to prevent PMTU poisoning
  3. Interface exclusion for loop prevention! -i pmtud0 in iptables rule is the primary mechanism; peer IP filtering + --ignore-networks provide defense-in-depth

Required iptables rule

iptables -t raw -A PREROUTING -p icmp -m icmp --icmp-type 3/4 ! -i pmtud0 -j NFLOG --nflog-group 33

Testing

  • go build ./...
  • go test ./...
  • go vet ./...

@defo89
defo89 force-pushed the feature/eliminate-l2-dependency branch 4 times, most recently from 53ffa61 to bb8a559 Compare May 1, 2026 12:54
@defo89
defo89 force-pushed the feature/eliminate-l2-dependency branch 3 times, most recently from 5687854 to e9adf6d Compare May 1, 2026 17:26
Two Kind clusters in separate Docker networks connected by a router
container with MTU 1500 transit path. Simulates the real-world scenario
of ICMP fragmentation-needed generation and go-pmtud UDP replication
across L3 boundaries.

Includes setup/teardown scripts, DaemonSet manifest, podinfo workload,
traffic generation, tcpdump observation helpers, and e2e validation.
@defo89
defo89 force-pushed the feature/eliminate-l2-dependency branch from e9adf6d to 6eb7525 Compare May 1, 2026 19:51
@github-actions

github-actions Bot commented May 1, 2026

Copy link
Copy Markdown

Merging this branch will increase overall coverage

Impacted Packages Coverage Δ 🤖
github.com/sapcc/go-pmtud/internal/arp 0.00% (ø)
github.com/sapcc/go-pmtud/internal/cmd 0.00% (ø)
github.com/sapcc/go-pmtud/internal/config 0.00% (ø)
github.com/sapcc/go-pmtud/internal/metrics 0.00% (ø)
github.com/sapcc/go-pmtud/internal/nflog 9.20% (+9.20%) 👍
github.com/sapcc/go-pmtud/internal/node 0.00% (ø)
github.com/sapcc/go-pmtud/internal/packet 80.65% (+80.65%) 🌟
github.com/sapcc/go-pmtud/internal/receiver 6.98% (+6.98%) 👍
github.com/sapcc/go-pmtud/internal/util 0.00% (ø)

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/sapcc/go-pmtud/internal/arp/resolve.go 0.00% (ø) 0 (-29) 0 0 (-29)
github.com/sapcc/go-pmtud/internal/cmd/command.go 0.00% (ø) 0 (-63) 0 0 (-63)
github.com/sapcc/go-pmtud/internal/config/config.go 0.00% (ø) 0 0 0
github.com/sapcc/go-pmtud/internal/metrics/metrics.go 0.00% (ø) 0 0 0
github.com/sapcc/go-pmtud/internal/nflog/controller.go 9.20% (+9.20%) 174 (+174) 16 (+16) 158 (+158) 👍
github.com/sapcc/go-pmtud/internal/node/controller.go 0.00% (ø) 0 (-31) 0 0 (-31)
github.com/sapcc/go-pmtud/internal/node/reconciler.go 0.00% (ø) 0 0 0
github.com/sapcc/go-pmtud/internal/packet/packet.go 80.65% (+80.65%) 31 (+31) 25 (+25) 6 (+6) 🌟
github.com/sapcc/go-pmtud/internal/receiver/peer.go 100.00% (+100.00%) 12 (+12) 12 (+12) 0 🌟
github.com/sapcc/go-pmtud/internal/receiver/receiver.go 0.00% (ø) 142 (+142) 0 142 (+142)
github.com/sapcc/go-pmtud/internal/receiver/receiver_other.go 0.00% (ø) 0 0 0
github.com/sapcc/go-pmtud/internal/receiver/tun_linux.go 0.00% (ø) 18 (+18) 0 18 (+18)
github.com/sapcc/go-pmtud/internal/util/interface.go 0.00% (ø) 41 (-15) 0 41 (-15)

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

  • github.com/sapcc/go-pmtud/internal/nflog/controller_test.go
  • github.com/sapcc/go-pmtud/internal/receiver/receiver_test.go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants