Skip to content

Prevent lingering MJPEG streams by switching to mode=single when stopping monitor streams #5029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
IgorA100 wants to merge 1 commit into
base: master
Choose a base branch
from
+6 −2
Draft

Prevent lingering MJPEG streams by switching to mode=single when stopping monitor streams #5029

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions web/js/MonitorStream.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -734,8 +734,12 @@ function MonitorStream(monitorData) {
this.mediaStream = this.audioTrack = this.videoTrack = null;

if (-1 !== this.activePlayer.indexOf('zms')) {
// Icon: My current thought is to just tell zms to stop. Don't go to single.
if (this.started) this.streamCommand(CMD_STOP);
// Stop the active MJPEG stream first. After that, switch the image to mode=single
// so the last frame remains visible while the MJPEG connection is fully released.
if (this.started || wasStarting) this.streamCommand(CMD_STOP);
const src = stream.src;
stream.src = "";
stream.src = src.replace(/([?&])mode=[^&]*/, '$1mode=single');

@connortechnology connortechnology Jul 31, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. You just killed zms. This is stop(), not kill().
stop() means stop streaming, wait for further commands.
Ideally we should be able at this time to tell it to switch to the other monitor and stream from there without relaunching another process but I don't know if that has ever been tried.

Also, we have seen that sometimes zms doesn't get SIGPIPE, so it might linger, preventing the mode=single from happening. You would at least need to remove the connkey. You don't need a connkey if mode=single because it will just exit after the single jpeg.

@IgorA100 IgorA100 Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we have seen that sometimes zms doesn't get SIGPIPE, so it might linger,

I'm currently trying to resolve a problem, and it seems there really is a problem with ZMS stopping even when sending a STOP command.
I have a camera with an H.265 stream, go2RTC and rtsp2Web enabled, and the player selection in the monitor settings is set to Auto. We're trying to watch it in Firefox with the player selection set to Automatic, and eventually have to switch to viewing using ZMS. If the player loop is 5 seconds, then between the 4th and 5th seconds, the player cycle reaches ZMS, and then switches to another monitor. At this point, sometimes (probably after the 10th attempt), the stream doesn't stop correctly, and network traffic continues. Something else is happening in the browser, as after about an hour, memory consumption increases, significantly increasing the load on the SDD on the workstation where the browser is running.
If the loop is set to 10 seconds, the problem doesn't seem to occur yet, as the STOP command likely works more correctly.

@connortechnology connortechnology Aug 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but let's start with the first thing I see that is wrong: CMD_STOP will not in and of itself kill zms. You want CMD_QUIT.

5s is interesting. I wonder if the signal is coming before we start listening for commands... and so we miss it? Maybe.

@IgorA100 IgorA100 Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the signal is coming before we start listening for commands... and so we miss it? Maybe.

It seems to me that this is exactly what is happening. Neither CMD_QUIT, nor mode=single, nor deleting conkey, nor clearing src helps. I've been trying to solve this problem at the JS level for several days now, but so far without success. I can't 100% determine the cause of the problem.

@connortechnology connortechnology Aug 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@IgorA100 @connortechnology I went digging on this. Isaac's hunch is right — the command does get missed — but the reason it's permanently missed is on the JS side, and it explains why none of CMD_QUIT / mode=single / clearing connkey / clearing src made any difference.

The command isn't lost, the process is

web/js/MonitorStream.js:1383-1394, the failure branch of getStreamCmdResponse:

} else {
  if (!this.started) return;
  console.error(respObj.message);
  // Try to reload the image stream.
  ...
  this.streamCmdParms.connkey = this.statusCmdParms.connkey = this.connKey = this.genConnKey();
  src = src.replace(/connkey=\d+/i, 'connkey='+this.connKey);
  stream.src = ''; stream.src = src;
}

On any stream-command error we mint a new connkey and reload the src. And ajaxError() (web/includes/functions.php:1794) returns HTTP 200 with result: 'Error', so these land in jQuery's .done(), not .fail() — every one of them takes this path:

  • Socket zms-NNNNNNs.sock does not exist
  • Timed out waiting for msg
  • socket_bind / socket_sendto failures

The moment that fires, the still-running zms's connkey is gone from JS. CMD_STOP, CMD_QUIT, mode=single, clearing src — all of them are now addressed to the new connkey. That's why nothing Igor tried helped: the fix is being applied to the wrong process. The orphan can only be stopped by SIGPIPE, which we already know is unreliable.

Why the error fires in the first place

closeComms() (src/zm_stream.cpp:410) deliberately never unlinks zms-NNNNNNs.sock; the only thing that removes it is the unlink() before bind() in a later zms that draws the same connkey. So dead socket files accumulate in PATH_SOCKS forever.

web/ajax/stream.php:91 is while (!file_exists($remSockFile) ...). A file left by a dead zms satisfies that immediately, so PHP skips the wait entirely and socket_sendto gets ECONNREFUSED → error → orphan. genConnKey() is Math.random()*999999, and a 5-second cycle burns ~720 connkeys an hour, so a collision is likely within the first hour or two and near-certain over an evening. That matches "roughly the 10th attempt" and "worse during multi-hour looped viewing" better than a pure startup race does.

Two smaller notes on that loop: it's 1000 × usleep(1000) = 1 second, while the comment right above it says "Pi can take up to 3 seconds for zms to start up." And zms itself is clean here — openComms() binds at zm_monitorstream.cpp:532, before the command thread starts at :611, so datagrams that arrive early are buffered by the kernel rather than dropped.

Also: kill() silently disables stop()

MonitorStream.js:848-855 sets this.started = false and then calls this.stop() — which early-returns on !this.started (:712). So for the zms path clearInterval(this.statusCmdTimer) and clearInterval(this.streamCmdTimer) never run, and activePlayer is never cleared. A pair of orphaned intervals per cycle. (The beforeunload listener accumulation that used to compound this is already handled by manageEventListener since #5018.)

One more, found on the way

StreamBase::lock_fd is initialised to 0 (src/zm_stream.h:204) while every other use treats negative as "no lock held" — openComms() sets -1 on failure, closeComms() guards on lock_fd >= 0. So a StreamBase destructed without openComms() having succeeded calls close(0) and closes stdin. It needs connkey > 0 plus a runStream() that returns before openComms(), and MonitorStream::runStream() has two of those: the STREAM_SINGLE branch and the !monitor branch. mode=single URLs still carry a connkey (:307), so both are reachable.

zms exits shortly after and does little in between, so the practical impact today is small — but it's closing a descriptor the class doesn't own, and once fd 0 is free the next open() silently lands on it.

On this PR

I don't think mode=single is wrong — it does release the browser-side MJPEG connection, which is real. But it can't help the case Igor is actually hitting, because by then JS is talking to a different connkey than the leaking zms. Worth fixing the ordering first:

  1. Don't regenerate the connkey on error without dealing with the old zms first. This is the actual root cause and it's a design call — @connortechnology, is reusing the connkey on retry acceptable once the socket file is a reliable liveness signal?
  2. Make closeComms() unlink the socket, so file_exists() means something.
  3. Fix the kill()/stop() ordering.
  4. Initialise lock_fd to -1.

I have 2, 3 and 4 written up as small independent commits (4 with a regression test; full suite passes at 117 cases / 1788 assertions), happy to open them separately. 1 I'd rather agree on first.

@connortechnology connortechnology Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@IgorA100 I think we found it. #5038 has the fix — it would be very helpful if you could test it against the setup you described (H.265 + go2rtc + rtsp2Web, player on Auto, 5 second loop).

Short version: you were right that the command was being missed, and right that it wasn't fixable from the JS stop path. The reason nothing you tried worked is that by the time you sent CMD_QUIT / mode=single / cleared the connkey, the JS was already addressing a different connkey than the zms that was leaking. The fix was being applied to the wrong process.

What happens on your 5 second loop:

  1. A stream command to zms doesn't get a reply within ZM_WEB_AJAX_TIMEOUT/2 (5s by default).
  2. stream.php hits the select() timeout. ajaxError is commented out there, so it carries on to socket_recvfrom() on a socket it just set non-blocking. That returns false, and false == 0 under switch's loose comparison, so it lands on case 0 and reports 'No data to read from socket'.
  3. ajaxError() returns HTTP 200 with result: 'Error', so it arrives in jQuery's .done(), not .fail().
  4. getStreamCmdResponse()'s error branch mints a new connkey and reloads the src.
  5. The original zms is still alive and streaming, but now unreachable — nothing addresses its connkey any more. Only SIGPIPE can stop it, and we already know that's unreliable.

So a merely slow zms was being reported exactly like a dead one, then orphaned. That matches what you saw: fine at a 10s loop, failing intermittently at 5s, and getting worse over hours as the orphans accumulate.

The fix classifies each failure in stream.php (no_socket / timeout / transient / invalid) and only restarts the stream for no_socket. A timeout now just retries on the next poll. And before the connkey is ever replaced, CMD_QUIT goes to the old one first, so a process we're about to lose track of is asked to exit.

What's worth checking on your setup:

  • Does zms still accumulate over a long looped session? ps aux | grep zms after an hour or two of 5s cycling.
  • Does the browser memory growth stop? That was the symptom loading your workstation's SSD.
  • $PATH_SOCKS should also stop filling with zms-*s.sock files, from fix: unlink the zms command socket on exit #5034.

On this PR (#5029): mode=single isn't wrong — it does release the browser-side MJPEG connection. But it couldn't fix what you were chasing, because the leaking zms was already unaddressable by then. Worth re-testing on top of #5038 to see whether it still helps once the orphaning is gone.

Note I haven't tested #5038 against a live install — it's unit tested only, so your real-world check is the one that matters.

@IgorA100 IgorA100 Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@connortechnology There are actually many more things that need to be checked. I spent over five days, many hours, experimenting. When cycling through players and then stopping streams, JS would sometimes regularly send packets via this.streamCmdQuery(). But even if I stopped sending CMD_QUERY, the ZMS process wasn't killed, and I don't think this should happen; it's very dangerous.
I'll definitely check out #5038 and let you know.

@IgorA100 IgorA100 Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@connortechnology
I tried to compile ZM from the branch https://github.com/connortechnology/ZoneMinder/tree/fix-connkey-regeneration which, as I understand, includes #5038 but the problem remained. After several cycles, the zms process sending the stream to the browser remains, and the browser receives it.

} else if (-1 !== this.activePlayer.indexOf('go2rtc')) {
if (!(stream.wsState === WebSocket.CLOSED && stream.pcState === WebSocket.CLOSED)) {
try {
Expand Down
Loading