[meshnet] Manage system-level kernel and link tunables for packet performance - #734
[meshnet] Manage system-level kernel and link tunables for packet performance#734kraney wants to merge 1 commit into
Conversation
e5e0afb to
6863cec
Compare
…mance
Implement kernel sysctl tuning, gRPC stream multiplexing, and persistent TAP device
packet I/O in meshnetd for high-density, high-throughput network meshes.
Reason for Change:
- Default Linux link queue lengths, netdev backlogs, multicast membership limits, and
strict rp_filter settings cause packet drops during netdev creation bursts and multi-link tests.
- IPv6 DAD (Duplicate Address Detection) and Router Solicitations introduce 1-2s readiness
delays per netdev when launching virtual topologies.
Key Changes:
- System & Kernel Tuning (`utils/wireutil/sys_tune.go`):
- Sets `RLIMIT_NOFILE` to 1,048,576 open file descriptors.
- Sets `netdev_max_backlog` to 10,000 for device queue bursts.
- Sets OS send/recv max & default buffers (`rmem_max`, `wmem_max`, `rmem_default`, `wmem_default`) to 16 MB.
- Sets ARP/neighbor GC thresholds (`gc_thresh1`=1024, `gc_thresh2`=4096, `gc_thresh3`=8192).
- Sets `igmp_max_memberships` to 10,000 and `mld_max_msf` to 4,096 to prevent multicast group netdev drops.
- Sets `rp_filter` to 2 (Loose mode) to allow asymmetric routing across multi-link meshes.
- Disables IPv6 DAD (`accept_dad`=0) and RS floods (`router_solicitations`=0) for instant link readiness.
- Sets `ipv6/route/max_size` to 1,048,576 for large routing tables.
- Configured link `txqueuelen` (default 10,000) across TAP, vEth, and vxLAN interface creation.
- All tunables are overridable via environment variables without recompilation.
Environment Variable │ Default Value │ Configures
───────────────────────────────┼───────────────────────────────┼─────────────────────────────────────────────────────
LINK_TXQUEUELEN │ 10000 │ Interface txqueuelen for TAP, vEth, and vxLAN links
RLIMIT_NOFILE │ 1048576 │ Process open file descriptor limit (unix.Setrlimit)
NETDEV_MAX_BACKLOG │ 10000 │ /proc/sys/net/core/netdev_max_backlog
RMEM_MAX │ 16777216 │ /proc/sys/net/core/rmem_max (16 MB)
WMEM_MAX │ 16777216 │ /proc/sys/net/core/wmem_max (16 MB)
RMEM_DEFAULT │ 16777216 │ /proc/sys/net/core/rmem_default (16 MB)
WMEM_DEFAULT │ 16777216 │ /proc/sys/net/core/wmem_default (16 MB)
ARP_GC_THRESH1 │ 1024 │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh1
ARP_GC_THRESH2 │ 4096 │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh2
ARP_GC_THRESH3 │ 8192 │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh3
Sysctl / Limit │ Env Var Override │ High-Density Default
───────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────
igmp_max_memberships │ IGMP_MAX_MEMBERSHIPS │ 10000
mld_max_msf │ MLD_MAX_MSF │ 4096
rp_filter │ RP_FILTER │ 2 (loose)
accept_dad │ IPV6_ACCEPT_DAD │ 0 (disabled)
router_solicitations │ IPV6_ROUTER_SOLICITATIONS │ 0 (disabled)
max_size (IPv6 route) │ IPV6_ROUTE_MAX_SIZE │ 1048576
netdev_max_backlog │ NETDEV_MAX_BACKLOG │ 10000
rmem_max / wmem_max │ RMEM_MAX / WMEM_MAX │ 16777216 (16 MB)
gc_thresh1..3 │ ARP_GC_THRESH1..3 │ 1024 / 4096 / 8192
RLIMIT_NOFILE │ RLIMIT_NOFILE │ 1048576
Defaults are chosen with the intent of making custom tuning unnecessary for all but the
most demanding topologies.
6863cec to
af4f95a
Compare
| } | ||
| defer cni.Cleanup() | ||
|
|
||
| wireutil.TuneSystem() |
There was a problem hiding this comment.
Can we move all of this to documentation instead (explaining why we are changing these things?) I think these kind of settings are really up to the user to implement - we should not be forcing tuning settings in code like this. We could make changes to cloudbuild/internal.pkr.hcl and cloudbuild/external.pkr.hcl if there are changes that will improve performance measurably in the VM images.
I don't think many of these changes are relevant or helping: ARP/ND cache GC thresh doesn't do much, rp_filter is not a performance knob, ipv6/route/max_size should not be changed, default/router_solicitations , accept_dad, mld_max_msf, and igmp_max_memberships don't do much in the host OS and are not going to change performance. RLIMIT_NOFILE seems like its changing something for a limit we've never hit?
The txqueuelen, netdev_max_backlog, and [wr]mem_(default|max) I can imagine are going to affect performance. I think setting the initial window size to 16MB is a little excessive, but increasing the max seems OK.
There was a problem hiding this comment.
I may have gotten overzealous trying to clear the last packet drops. Let me revisit


Note: this PR stacks on top of #733 . Please merge that one first (which should make the shared commits disappear from this PR.) The incremental change of this PR starts with sha:b3c915457042d606dd0dd9bb659ed1ea619af81e.