Skip to content

[meshnet] Manage system-level kernel and link tunables for packet performance - #734

Draft
kraney wants to merge 1 commit into
openconfig:mainfrom
kraney:meshnet-tunables
Draft

[meshnet] Manage system-level kernel and link tunables for packet performance#734
kraney wants to merge 1 commit into
openconfig:mainfrom
kraney:meshnet-tunables

Conversation

@kraney

@kraney kraney commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Note: this PR stacks on top of #733 . Please merge that one first (which should make the shared commits disappear from this PR.) The incremental change of this PR starts with sha:b3c915457042d606dd0dd9bb659ed1ea619af81e.

    Implement kernel sysctl tuning in meshnetd for high-density, high-throughput network meshes.

    Reason for Change:
    - Default Linux link queue lengths, netdev backlogs, multicast membership limits, and
      strict rp_filter settings cause packet drops during netdev creation bursts and multi-link tests.
    - IPv6 DAD (Duplicate Address Detection) and Router Solicitations introduce 1-2s readiness
      delays per netdev when launching virtual topologies.

    Key Changes:
    - System & Kernel Tuning (`utils/wireutil/sys_tune.go`):
      - Sets `RLIMIT_NOFILE` to 1,048,576 open file descriptors.
      - Sets `netdev_max_backlog` to 10,000 for device queue bursts.
      - Sets OS send/recv max & default buffers (`rmem_max`, `wmem_max`, `rmem_default`, `wmem_default`) to 16 MB.
      - Sets ARP/neighbor GC thresholds (`gc_thresh1`=1024, `gc_thresh2`=4096, `gc_thresh3`=8192).
      - Sets `igmp_max_memberships` to 10,000 and `mld_max_msf` to 4,096 to prevent multicast group netdev drops.
      - Sets `rp_filter` to 2 (Loose mode) to allow asymmetric routing across multi-link meshes.
      - Disables IPv6 DAD (`accept_dad`=0) and RS floods (`router_solicitations`=0) for instant link readiness.
      - Sets `ipv6/route/max_size` to 1,048,576 for large routing tables.
      - Configured link `txqueuelen` (default 10,000) across TAP, vEth, and vxLAN interface creation.
      - All tunables are overridable via environment variables without recompilation.

@kraney

kraney commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

FWIW, I created a throughput test comparing before & after for grpcwire. ("Before" being before #731 , "after" being after #734 , including the full set of improvements.)

Before:
image

After:
image

…mance

    Implement kernel sysctl tuning, gRPC stream multiplexing, and persistent TAP device
    packet I/O in meshnetd for high-density, high-throughput network meshes.

    Reason for Change:
    - Default Linux link queue lengths, netdev backlogs, multicast membership limits, and
      strict rp_filter settings cause packet drops during netdev creation bursts and multi-link tests.
    - IPv6 DAD (Duplicate Address Detection) and Router Solicitations introduce 1-2s readiness
      delays per netdev when launching virtual topologies.

    Key Changes:
    - System & Kernel Tuning (`utils/wireutil/sys_tune.go`):
      - Sets `RLIMIT_NOFILE` to 1,048,576 open file descriptors.
      - Sets `netdev_max_backlog` to 10,000 for device queue bursts.
      - Sets OS send/recv max & default buffers (`rmem_max`, `wmem_max`, `rmem_default`, `wmem_default`) to 16 MB.
      - Sets ARP/neighbor GC thresholds (`gc_thresh1`=1024, `gc_thresh2`=4096, `gc_thresh3`=8192).
      - Sets `igmp_max_memberships` to 10,000 and `mld_max_msf` to 4,096 to prevent multicast group netdev drops.
      - Sets `rp_filter` to 2 (Loose mode) to allow asymmetric routing across multi-link meshes.
      - Disables IPv6 DAD (`accept_dad`=0) and RS floods (`router_solicitations`=0) for instant link readiness.
      - Sets `ipv6/route/max_size` to 1,048,576 for large routing tables.
      - Configured link `txqueuelen` (default 10,000) across TAP, vEth, and vxLAN interface creation.
      - All tunables are overridable via environment variables without recompilation.

   Environment Variable          │ Default Value                 │ Configures
  ───────────────────────────────┼───────────────────────────────┼─────────────────────────────────────────────────────
   LINK_TXQUEUELEN               │ 10000                         │ Interface txqueuelen for TAP, vEth, and vxLAN links
   RLIMIT_NOFILE                 │ 1048576                       │ Process open file descriptor limit (unix.Setrlimit)
   NETDEV_MAX_BACKLOG            │ 10000                         │ /proc/sys/net/core/netdev_max_backlog
   RMEM_MAX                      │ 16777216                      │ /proc/sys/net/core/rmem_max (16 MB)
   WMEM_MAX                      │ 16777216                      │ /proc/sys/net/core/wmem_max (16 MB)
   RMEM_DEFAULT                  │ 16777216                      │ /proc/sys/net/core/rmem_default (16 MB)
   WMEM_DEFAULT                  │ 16777216                      │ /proc/sys/net/core/wmem_default (16 MB)
   ARP_GC_THRESH1                │ 1024                          │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh1
   ARP_GC_THRESH2                │ 4096                          │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh2
   ARP_GC_THRESH3                │ 8192                          │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh3

   Sysctl / Limit                        │ Env Var Override                     │ High-Density Default
  ───────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────
   igmp_max_memberships                  │ IGMP_MAX_MEMBERSHIPS                 │ 10000
   mld_max_msf                           │ MLD_MAX_MSF                          │ 4096
   rp_filter                             │ RP_FILTER                            │ 2 (loose)
   accept_dad                            │ IPV6_ACCEPT_DAD                      │ 0 (disabled)
   router_solicitations                  │ IPV6_ROUTER_SOLICITATIONS            │ 0 (disabled)
   max_size (IPv6 route)                 │ IPV6_ROUTE_MAX_SIZE                  │ 1048576
   netdev_max_backlog                    │ NETDEV_MAX_BACKLOG                   │ 10000
   rmem_max / wmem_max                   │ RMEM_MAX / WMEM_MAX                  │ 16777216 (16 MB)
   gc_thresh1..3                         │ ARP_GC_THRESH1..3                    │ 1024 / 4096 / 8192
   RLIMIT_NOFILE                         │ RLIMIT_NOFILE                        │ 1048576

Defaults are chosen with the intent of making custom tuning unnecessary for all but the
most demanding topologies.
}
defer cni.Cleanup()

wireutil.TuneSystem()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move all of this to documentation instead (explaining why we are changing these things?) I think these kind of settings are really up to the user to implement - we should not be forcing tuning settings in code like this. We could make changes to cloudbuild/internal.pkr.hcl and cloudbuild/external.pkr.hcl if there are changes that will improve performance measurably in the VM images.

I don't think many of these changes are relevant or helping: ARP/ND cache GC thresh doesn't do much, rp_filter is not a performance knob, ipv6/route/max_size should not be changed, default/router_solicitations , accept_dad, mld_max_msf, and igmp_max_memberships don't do much in the host OS and are not going to change performance. RLIMIT_NOFILE seems like its changing something for a limit we've never hit?

The txqueuelen, netdev_max_backlog, and [wr]mem_(default|max) I can imagine are going to affect performance. I think setting the initial window size to 16MB is a little excessive, but increasing the max seems OK.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may have gotten overzealous trying to clear the last packet drops. Let me revisit

@kraney
kraney marked this pull request as draft August 17, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants