[meshnet] multiplex tcp streams for a node pair - #733
Merged
Conversation
kraney
force-pushed
the
meshnet-multiplex
branch
4 times, most recently
from
August 11, 2026 01:14
cf0cddd to
61df5fe
Compare
1. Created nodeStreamManager (stream_manager.go):
• Keyed by nodeStreamKey{ topoNs string, peerIP string }.
• Opens only one gRPC connection (grpc.Dial) and only one SendToStream bidirectional streaming RPC per (peerIP,
topoNs) pair.
• Includes reference counting (GetOrCreateStream / ReleaseStream). When the last wire for a topology targeting a
peer node is removed, the shared stream and gRPC connection close gracefully.
• Features a high-capacity buffered channel (10,000 packet queue) and dedicated sender worker loop with automatic
reconnect logic.
2. Updated TAP Reader Threads (grpcwire.go):
• Modified RecvFrmLocalPodThread so TAP readers no longer execute individual grpc.Dial or SendToStream calls.
• Each TAP reader now acquires the shared topology stream via nodeStream := streamMgr.GetOrCreateStream(wire.
TopoNamespace, wire.PeerNodeIP) and multiplexes packets into nodeStream.Send(payload).
3. Per-Topology Isolation:
• Topology topo-A and topology topo-B running between the same pair of physical nodes maintain completely
separate gRPC connections and streams.
The base design had a lot of single-interface RPCs to update k8s and remote meshnets. For large numbers of links it costs a lot of serial round trip wait time. This batches things and pipelines them so that we avoid a lot of unnecessary overhead delay at start time.
kraney
force-pushed
the
meshnet-multiplex
branch
from
August 11, 2026 15:32
61df5fe to
c86997a
Compare
bstoll
reviewed
Aug 11, 2026
### 1. Unused getEnvString in sys_tune.go
• Removed the unused private helper sys_tune.go.
──────
### 2. Remove gwire_recon.go
• Removed the obsolete single-wire gwire_recon.go function, as all updates are now handled by gwire_recon.go.
• Updated the gwire_recon.go doc comment.
──────
### 3. Namespace Isolation in Status Batch Updates
• Problem: gwire_recon.go used updates[0].wire.TopoNamespace for the entire batch, which caused cross-namespace collisions when multiple
topologies ran on the same cluster.
• Fix: Updated gwire_recon.go to group updates by statusGroupKey{nodeName, topoNs} and delegate each group to gwire_recon.go.
• Added unit test gwire_recon_test.go in gwire_recon_test.go to verify concurrent multi-namespace status updates.
──────
### 4. Unused s.conn and s.mu in stream_manager.go
• s.conn was assigned in run() but never read elsewhere because the active connection lifecycle is managed directly by drainAndSend(ctx,
cancel, conn, stream) (which closes conn in its defer block upon termination).
• Removed the unused conn and mu fields from stream_manager.go along with the redundant mutex locking.
──────
### 5. Additional Dead Code Cleanup
• **grpcwire.go**:
• Removed unused (w *wireMap) Delete method.
• Simplified grpcwire.go to delegate to grpcwire.go, eliminating duplicated locking and update logic.
• **gwire_map.go**:
• Removed unused (w *wireMap) DeleteWoLock method.
• **handler.go**:
• Removed commented-out legacy call in handler.go.
bstoll
approved these changes
Aug 12, 2026
#### 1. Status Queue Deadlocks & Ghost Wires
• gwire_recon.go:
• Replaced the scalar flushSignal chan struct{} in gwire_recon.go with a slice (flushSignals []chan struct{}) so all flush requests
in a batch are tracked and closed without overwrites.
• Added wire.mu.Lock() / wire.mu.Unlock() to gwire_recon.go to eliminate data races against concurrent peer updates.
• Added gwire_recon.go to gwire_recon.go before calling deleteGRPCWireStatus.
• Added in-memory presence checks via grpcwire.go in gwire_recon.go and gwire_recon.go to prevent writing deleted wires back into
K8s.
• Handled apierrors.IsNotFound(err) gracefully in gwire_recon.go.
#### 2. Leaked TAP File Descriptors on Wire Deletion
• gwire_map.go & grpcwire.go:
• Added wires.CloseAndRemoveHandle(key) to close the underlying *os.File and delete it from wires.handles.
• Updated gwire_map.go to avoid deleting the handle prematurely before grpcwire.go closes it.
• Updated grpcwire.go and gwire_map.go to close and remove TAP file descriptors.
#### 3. Goroutine & Buffer Leak on Teardown
• grpcwire.go:
• Made packet channel writes in grpcwire.go select on <-wire.StopC. If teardown occurs, the reader immediately returns the buffer
to packetPool and exits cleanly.
#### 4. Controller Batch Error Handling & Bounds Checking
• controller.go:
• Added nil checks for batchResp and slice bounds validation against chunkLinks.
• Captured individual remote link creation failures and returned an aggregate errors.Join(batchErrs...) from controller.go so the
reconciler retries when remote wire setup fails.
#### 5. Duplicate Packet Receive Goroutines & Stream Refcount Leaks
• gwire_rpc_handlers.go & handler.go:
• Updated gwire_rpc_handlers.go to return (wire *GRPCWire, created bool, err error).
• Updated handler.go and handler.go to only spawn grpcwire.go when created == true.
#### 6. Throttled Reconnection on Send Errors
• stream_manager.go:
• Added a backoff delay in stream_manager.go after stream_manager.go returns due to connection drops, preventing 100% CPU spin
loops while continuing to respect stopChan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: this PR stacks on top of #732 , please merge that one first. The incremental changes start at sha ceb9511
Also:
The base design had a lot of single-interface RPCs
to update k8s and remote meshnets. For large numbers
of links it costs a lot of serial round trip wait time.
This batches things and pipelines them so that we avoid
a lot of unnecessary overhead delay at start time.