…adlocks
Segmented video reversal runs a multiprocessing Pool while the parent
process still holds an OPEN PyAV input container (the 'with av.open(src)'
that wraps the whole function body). On Linux the default 'fork' start
method duplicates the parent mid-flight: a forked child can inherit an
FFmpeg/PyAV (or tqdm) lock in a locked state with no owner left to
release it, and the pool then deadlocks on its own queue semaphore.
This matches the reported failures exactly:
- renders hanging in 'Reversing large file by cutting it in segments'
once every 10-20 runs (jeertmans#562)
- a worker stuck in multiprocessing/queues.py get() -> semlock
__enter__, only surfaced as a KeyboardInterrupt stack trace (jeertmans#569)
Both are probabilistic (the fork must race a held lock), which is why
small projects never hit it and larger ones hit it intermittently.
Create the pool from an explicit 'spawn' context instead: freshly
spawned interpreters cannot inherit held locks, and this matches the
default start method on macOS and Windows — Linux was the odd one out.
The worker function (reverse_video_file_in_one_chunk) is a module-level
function taking picklable Path arguments, so it is already
spawn-compatible; the cost of a few interpreter startups is noise next
to the video re-encoding they perform.
Adds a regression test driving the segmented path (segment duration
forced below the test asset's length) and asserting a decodable,
non-empty output.
Fixes jeertmans#562
Fixes jeertmans#569
Description
Segmented video reversal runs a
multiprocessing.Poolwhile the parent process still holds an open PyAV input container (thewith av.open(src)wrapping the whole body ofreverse_video_file). On Linux the defaultforkstart method duplicates the parent mid-flight: a forked child can inherit an FFmpeg/PyAV (or tqdm) lock in a locked state with no owner left to release it, and the pool then deadlocks on its own queue semaphore.This matches the reported failures exactly:
Reversing large file by cutting it in segments([BUG] Rendering stalled in 'Reversing large file by cutting it in segment' #562)multiprocessing/queues.py get() → semlock.__enter__, only surfaced after Ctrl-C as aKeyboardInterruptstack trace ([BUG] Concurrency bug when reversing videos #569 — the trace in that issue is precisely a worker blocked on the inherited queue rlock)Both are probabilistic — the fork has to race a held lock — which is why small projects never hit it and larger ones (more segments → more workers → more forks) hit it once every 10–20 renders, as reported.
Fix: create the pool from an explicit
spawncontext. Freshly spawned interpreters cannot inherit held locks, and this matches the default start method on macOS and Windows — Linux was the odd one out.reverse_video_file_in_one_chunkis a module-level function taking picklablePatharguments, so it is already spawn-compatible; a few interpreter startups are noise next to the video re-encoding the workers perform.Testing
pytest tests/test_utils.py— 3 passed.ruff check/ruff format --checkclean on the touched files.Fixes #562
Fixes #569