Skip to content

fix(cli): don't raise from the worker signal handler so SIGTERM always drains - #6738

Open
RGB-loop wants to merge 1 commit into
livekit:mainfrom
RGB-loop:fix/worker-sigterm-graceful-shutdown
Open

fix(cli): don't raise from the worker signal handler so SIGTERM always drains#6738
RGB-loop wants to merge 1 commit into
livekit:mainfrom
RGB-loop:fix/worker-sigterm-graceful-shutdown

Conversation

@RGB-loop

@RGB-loop RGB-loop commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Fixes #6724

The start-mode signal handler raised _ExitCli directly. CPython delivers signal handlers on the main thread at an arbitrary bytecode boundary, so when the event loop was executing a task — e.g. a user request_fnc doing blocking I/O inside a job-request task — the exception surfaced inside that task instead of at the run_until_complete boundary. The task died with an unretrieved _ExitCli, the worker never entered the drain/aclose path, and it kept accepting new dispatches until SIGKILL. Because the handler latches, the first (and in container environments, only) SIGTERM was wasted.

Changes:

  • The handler now schedules the exit on the loop via call_soon_threadsafe — the same signal-safe pattern the TCP/console path in this module already uses — instead of raising.
  • run_until_complete waits on either the main task or the exit future, leaving server.run alive so the existing drain/aclose sequence still has a live worker. A second signal still force-exits.

Added a subprocess regression test (tests/test_cli_sigterm.py) that SIGTERMs a worker while its event loop is blocked by a synchronous call inside a task. On the previous code the worker never shuts down (test times out after 30s); with this change it drains and exits cleanly

…s drains

The start-mode signal handler raised _ExitCli directly. CPython delivers
signal handlers on the main thread at an arbitrary bytecode boundary, so
when the event loop was executing a task — e.g. a user request_fnc doing
blocking I/O inside a job-request task — the exception surfaced inside
that task instead of at the run_until_complete boundary. The task died
with an unretrieved _ExitCli, the worker never entered the drain/aclose
path, and it kept accepting new dispatches until SIGKILL. Because the
handler latches, the first (and in container environments, only) SIGTERM
was wasted.

Schedule the exit on the loop via call_soon_threadsafe instead, the same
signal-safe pattern the TCP/console path in this module already uses.
run_until_complete now waits on either the main task or the exit future,
leaving server.run alive so the existing drain/aclose sequence still has
a live worker. Second signal still force-exits.

Added a subprocess regression test that SIGTERMs a worker while its event
loop is blocked by a synchronous call inside a task; on the previous code
the worker never shuts down.

Fixes livekit#6724

Generated with AI

Co-Authored-By: AI <ai@example.com>
@RGB-loop
RGB-loop requested a review from a team as a code owner August 7, 2026 06:45
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


jingyan seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIGTERM does not shut down workers when request_fnc blocks the event loop

2 participants