Run a local DERP relay on the NixOS box - #191
Merged
Merged
Conversation
Every VM is on the tailnet but behind per-VM NAT, and both physical machines are behind T-Mobile 5G CGNAT, so Tailscale can never connect them directly and relays all VM-to-VM traffic through a cloud DERP — up over the slow 5G upload and back, at ~1 MB/s. Run derper on the always-on NixOS machine so the relay stays on the LAN: the VMs prefer it at home (lowest latency) and fall back to the cloud DERPs when away. The host does not join the tailnet; derper only forwards already-encrypted WireGuard packets, so it can't read the traffic or reach the VMs. Self-signed and pinned by SHA256 in the DERP map, so no domain or ACME cert is needed. Also disable Wi-Fi power saving, which added ~60-110 ms to every LAN round-trip — enough that Tailscale rated the local relay no faster than the cloud one and refused to use it; off, the hop is ~5 ms and the local relay wins. The relay's IP is pinned so its self-signed cert doesn't drift. The gateway (Arcadyan TMO-G4AR) exposes no DHCP settings, so rather than a reservation the box holds a static 192.168.12.10 on its Wi-Fi connection, below the gateway's pool. README documents that plus the rest: reading the pin derper logs, adding the DERP-map region to the tailnet policy, and verifying. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wku9K54eBg5KJnXDfpvkQ
derper stores its own private key in the file named by -c, and defaults that path (/var/lib/derper/derper.key) only when it runs as root. Under DynamicUser it's unprivileged, so without -c it exits at once with "-c <config path> not specified" and systemd's restart limiter latches. Point -c at the state directory so it creates and reuses the key there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Wku9K54eBg5KJnXDfpvkQ
This was referenced Jul 20, 2026
Measured from both Linux VMs: the relay works, but they mostly aren't using it. `derper`'s LAN-side STUN lets them discover LAN-local mapped endpoints, the per-VM NATs hole-punch, and they hold a direct path (192.168.12.130:39424 <-> 192.168.12.10:41641). So step 4's "should report via DERP(home)" read a success as a failure. Also: the NixOS box's LAN hop is ~1 ms, not ~5 ms (p50 1.0, p90 1.5), while the macOS host shows the same Wi-Fi power-saving tail this config turns off on NixOS (p50 ~4 ms, p90 33 ms, p99 74 ms) with no equivalent knob available. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GNWf7bQzqqVN7GUSEtbDMG
The static 192.168.12.10 lived only on the Wi-Fi connection profile, set once by hand with `nmcli con mod`. Moving the box to Ethernet therefore dropped it, and DHCP handed out 192.168.12.184 instead. That is enough to take the whole tailnet down between the VMs, not just slow it: the DERP map pins the relay at .10, so `netcheck` falls back to a cloud DERP, LAN STUN goes with it, and without LAN-side endpoints to hole-punch with the VMs lose every path to each other. Declare the address instead, as a NetworkManager profile bound to whatever Ethernet device is present. On Ethernet there is no PSK to keep out of the repo, which is what forced the imperative step before. Wi-Fi keeps its own profile and stays a DHCP fallback. `ensureProfiles` only reloads NetworkManager, which will not drop an active connection to adopt a new profile, so bring the profile up in `postStart` — otherwise a switch would not take effect until reboot. Assisted-by: Claude:opus-5
The branch had grown to ~150 lines, most of it prose that told the same CGNAT-and-hairpin story three times (README, the derper comment, and the commit messages). Keep the story once, in the README, and leave each Nix block a short "why" that points there. Drop `networking.networkmanager.wifi.powersave = false`. Its comment said it was load-bearing on the Wi-Fi fallback, but on that fallback the box is on DHCP rather than 192.168.12.10, so the pinned relay is unreachable regardless of power saving; the line no longer did what it claimed. (Its comment also said the LAN hop was ~5 ms where the README and 06d07df said ~1 ms.) In the README, cut the Ethernet-vs-Wi-Fi ping measurements, which were a lab note rather than setup documentation, and fold the one-time Wi-Fi profile cleanup into the drift paragraph at the end instead of making it a numbered setup step. The DERP-map JSON and the verification commands stay, since the tailnet policy is state that lives nowhere else. No functional change other than the powersave line: `derper`'s ExecStart and the `lan-wired` profile evaluate identically. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every remaining line was checked against the derper, Tailscale, and NetworkManager sources. Dropped as redundant with defaults: - `-a :443` (derper's default listen address) - `autoconnect = true` (NetworkManager's default) - `autoconnect-priority = 100`: the generated "Wired connection 1" profile has priority -999, so a user profile at the default 0 already outranks it, and NetworkManager doesn't generate one at all when a compatible profile exists - `ipv6.method = "auto"`: what NetworkManager assumes when the section is absent - `after`/`wants = [ "network-online.target" ]`: derper binds wildcard addresses and does no outbound work at startup (no ACME), so it has nothing to wait for; the nixpkgs module has neither - `description` and `Restart`, which were cosmetic and optional - the `postStart` re-activation hook: changes to the profile are rare, and "reboot or `nmcli connection up lan-wired`" is enough `connection.id` stays only because the NixOS option requires it; NetworkManager itself would fall back to the file name. Fold the README's dedicated DERP section into the NixOS section as the manual steps that follow a rebuild, and state plainly that a VM away from home currently cannot reach the VMs at home: their home DERP is the LAN-only relay, which the away node has no path to. The old text claimed the cloud DERPs covered that case, and they don't. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Binding :443 as the dynamic user needed CAP_NET_BIND_SERVICE. Moving to an unprivileged port drops the capability at the cost of one explicit `-a :8443` flag and a `"DERPPort": 8443` field in the tailnet DERP map. With -certmode manual, derper serves TLS on any port, so nothing else changes; STUN was already on an unprivileged port. `-http-port -1` goes from optional to required: without the capability, the default port-80 listener would fail to bind, and derper treats that as fatal. derper's logged DERP-map JSON carries only the name, region, hostname, and cert pin, so the README now says to add the port by hand. Deploying this needs the policy file updated at the same time; until both agree, the VMs can't reach the relay. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.