Skip to content

Fix infinite blocking in TrySendMultipartMessage at low bandwidth - #1157

Open
VincentVbs wants to merge 1 commit into
zeromq:masterfrom
VincentVbs:fix-1127-TrySendMultipartMessage-With-Low-Network-Speeds
Open

VincentVbs wants to merge 1 commit into
zeromq:masterfrom
VincentVbs:fix-1127-TrySendMultipartMessage-With-Low-Network-Speeds

Conversation

@VincentVbs

Copy link
Copy Markdown

Apply timeout to all frames in TrySendMultipartMessage, not just the first one. Previously, only the first frame used TrySendFrame with timeout, while subsequent frames used blocking SendMoreFrame/SendFrame methods. This caused infinite hangs when network bandwidth was extremely low (< 10 kbit/s).

The fix ensures all frames respect the timeout parameter, preventing infinite blocking and allowing graceful failure when send operations cannot complete.

Added three unit tests to validate the fix:

  • TrySendMultipartMessage_AppliesTimeoutToAllFrames
  • TrySendMultipartMessage_FailsGracefullyOnMiddleFrameTimeout
  • TrySendMultipartMessage_FailsGracefullyOnLastFrameTimeout

Fixes #1127

Apply timeout to all frames in TrySendMultipartMessage, not just the first one.
Previously, only the first frame used TrySendFrame with timeout, while subsequent
frames used blocking SendMoreFrame/SendFrame methods. This caused infinite hangs
when network bandwidth was extremely low (< 10 kbit/s).

The fix ensures all frames respect the timeout parameter, preventing infinite
blocking and allowing graceful failure when send operations cannot complete.

Added three unit tests to validate the fix:
- TrySendMultipartMessage_AppliesTimeoutToAllFrames
- TrySendMultipartMessage_FailsGracefullyOnMiddleFrameTimeout
- TrySendMultipartMessage_FailsGracefullyOnLastFrameTimeout

Fixes zeromq#1127

@drewnoakes drewnoakes left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks great too.

Comment thread src/NetMQ/OutgoingSocketExtensions.cs

return true;
// Send the last frame with timeout (no 'more' flag)
return TrySendFrame(socket, timeout, message.Last.Buffer, message.Last.MessageSize, false);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return TrySendFrame(socket, timeout, message.Last.Buffer, message.Last.MessageSize, false);
return TrySendFrame(socket, timeout, message.Last.Buffer, message.Last.MessageSize, more: false);

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates TrySendMultipartMessage so timeout-aware sends are used for every frame of a NetMQMessage, addressing the reported hang when multipart sends encounter extreme backpressure. It also adds focused unit tests around timeout handling for successful sends and failures on later frames.

Changes:

  • Reworked TrySendMultipartMessage(TimeSpan, NetMQMessage) to call TrySendFrame for every frame instead of mixing one timed send with later blocking sends.
  • Removed the old special-case path that timed only the first frame and blocked on the rest.
  • Added three tests covering timeout propagation and failure behavior on middle/last frames.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
src/NetMQ/OutgoingSocketExtensions.cs Changes multipart send logic so all NetMQMessage frames use the timeout-aware send path.
src/NetMQ.Tests/OutgoingSocketExtensionsTests.cs Adds unit tests intended to validate timeout handling across all frames and timeout failures on later frames.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +463 to +469
bool sentSuccessfully = TrySendFrame(socket, timeout, message[i].Buffer, message[i].MessageSize, true);
if (!sentSuccessfully)
return false;
}

for (int i = 1; i < message.FrameCount - 1; i++)
{
socket.SendMoreFrame(message[i].Buffer, message[i].MessageSize);
}

socket.SendFrame(message.Last.Buffer, message.Last.MessageSize);

return true;
// Send the last frame with timeout (no 'more' flag)
return TrySendFrame(socket, timeout, message.Last.Buffer, message.Last.MessageSize, false);
Comment on lines +362 to +363
// CRITICAL: ALL frames must respect the timeout parameter
Assert.Equal(timeout, actualTimeout);
Comment on lines +399 to +400
// Verify timeout is applied to all frames
Assert.Equal(timeout, actualTimeout);

var socket = new MockOutgoingSocket((ref Msg msg, TimeSpan actualTimeout, bool more) =>
{
Assert.Equal(timeout, actualTimeout);
@drewnoakes

Copy link
Copy Markdown
Member

Looks like there's a test failure. I like where this is going.

The active test run was aborted. Reason: Test host process crashed : Unhandled exception.
System.Net.Sockets.SocketException (22): Invalid argument
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, Boolean disconnectOnFailure, String callerName)
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketOptionErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.SetSocketOption(SocketOptionLevel optionLevel, SocketOptionName optionName, Int32 optionValue, Boolean silent)
   at NetMQ.Core.Transports.Tcp.TcpConnector.OutCompleted(SocketError socketError, Int32 bytesTransferred) in /_/src/NetMQ/Core/Transports/Tcp/TcpConnector.cs:line 240
   at NetMQ.Core.Utils.Proactor.Loop() in /_/src/NetMQ/Core/Utils/Proactor.cs:line 130
   at System.Threading.Thread.StartCallback()

@drewnoakes

Copy link
Copy Markdown
Member

@VincentVbs are you interested in continuing with this PR?

@VincentVbs

Copy link
Copy Markdown
Author

Thanks, @drewnoakes, for following up with me and for giving me the opportunity to contribute to this project.
Unfortunately, even though I spent extra time on the PR, I’m not able to do more today.
I did not anticipate running the unit tests on my machine, and the fix I initially proposed, which seemed trivial, turned out to be unfounded after reviewing the failing unit test.
In the end, I wasn’t able to reproduce the bug, but I have the feeling that the issue might no longer exist in the latest sources.
It would be useful to check exactly which version the user experienced the infinite loop with.
I did some tests using a Debian virtual machine and a tool to slow down communication, but I admit I wasn’t systematic enough and ended up getting exhausted.
I’m also not sure whether using a VM is the right approach to recommend.
Sorry for the noise this may have caused in the project.
Maybe I should have asked for guidance before working on the code.
How should I come back to zmq for a future contribution?
Thank you for taking the time to read this.

@drewnoakes

Copy link
Copy Markdown
Member

No problem at all @VincentVbs, it's great that you added to the project. Even incomplete PRs can help move things along.

Maybe I should have asked for guidance before working on the code.

Some projects ask contributors to do that, but generally I think the main reason to do that is to protect your own time against working on something that the project won't accept. Sometimes it's hard to know whether it's acceptable before the investigation is finished though. Don't worry about it.

How should I come back to zmq for a future contribution?

Just work on anything that seems interesting to you. You can discuss beforehand if you like. We're happy to have any contribution here on NetMQ. Other projects in the zeromq org may have different requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RouterSocket TrySendMultipartMessage SendMultipartMessage Hangs Infinitely With Low Network Speeds

4 participants