Skip to content

Skip partial merge when using ffmpeg concat demuxer (#946) - #947

Open
outhud wants to merge 1 commit into
nilaoda:mainfrom
outhud:fix/skip-partial-merge-with-concat-demuxer
Open

outhud wants to merge 1 commit into
nilaoda:mainfrom
outhud:fix/skip-partial-merge-with-concat-demuxer

Conversation

@outhud

@outhud outhud commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Fixes #946.

Problem

Downloading a stream with >=1800 segments always runs PartialCombineMultipleFiles before the ffmpeg merge, which byte-splices every 100 segments into a single T####.ts intermediate. ffmpeg then reads each intermediate as one continuous stream and cannot see the timestamp resets that occur inside it, so the merged file keeps all of its data but ends up with a broken timeline — the reporter's 2140-segment / 142-minute movie muxes to 955 MB but reports ~33 minutes, with a flood of Non-monotonic DTS warnings.

This happens even with --use-ffmpeg-concat-demuxer, so there is currently no option combination that produces a correct file for long content whose segments contain timestamp resets (movies, TV recordings).

Why the partial merge is not needed here

Partial merge exists to protect the concat protocol path, which opens every segment at once (#338, #89) and puts every file name on the command line.

The concat demuxer has neither limit: MergeUtil.MergeByFFmpeg writes the file list to a temp file (-f concat -safe 0 -i <list>) and ffmpeg opens the segments one at a time. In that mode the partial merge is pure downside — it corrupts timestamps, doubles peak disk usage, and adds a full extra copy pass over every segment.

Change

Skip the partial merge when --use-ffmpeg-concat-demuxer is used. The threshold and the reasoning now live next to PartialCombineMultipleFiles as MergeUtil.ShouldPartialMerge. Behaviour of the concat protocol path is completely unchanged: no new options, no default changes, no resource-string changes.

Verification

Locally served 1900-segment HLS playlist whose segments each restart at PTS 0 (playlist total 1944.12s):

build --use-ffmpeg-concat-demuxer partial merge ffprobe duration
v0.6.0-beta yes ran 19.56s + Non-monotonic DTS flood
this PR yes skipped 1944.14s
v0.6.0-beta no ran 1900.04s
this PR no ran 1900.04s (identical)

Possible follow-ups (not in this PR)

  • Fall back to the concat demuxer automatically when ffmpeg reports "Too many open files" (today MergeByFFmpeg only warns).
  • Consider whether >=1800 segments should prefer the concat demuxer by default rather than partial merge.

Downloading a stream with >=1800 segments always ran
PartialCombineMultipleFiles first, which byte-splices every 100 segments
into a single T####.ts intermediate. ffmpeg then reads each intermediate
as one continuous stream and cannot see the timestamp resets that occur
inside it, so the merged file keeps all its data but gets a broken
timeline (reported as ~33 minutes for a 142-minute movie, with a flood of
"Non-monotonic DTS" warnings).

Partial merge only exists to protect the concat *protocol* path, which
opens every segment at once (nilaoda#338, nilaoda#89) and puts every file name on the
command line. The concat *demuxer* has neither limit: MergeByFFmpeg
writes the file list to a temp file and ffmpeg opens the segments one at
a time. There it is pure downside - it corrupts timestamps, doubles peak
disk usage and adds a full extra copy pass.

So skip it when --use-ffmpeg-concat-demuxer is used. Behaviour of the
concat protocol path is unchanged.

Verified on a locally served 1900-segment HLS playlist whose segments
each restart at PTS 0:

  before, --use-ffmpeg-concat-demuxer:  19.56s   (expected 1944.12s)
  after,  --use-ffmpeg-concat-demuxer:  1944.14s
  before/after, without the flag:       1900.04s (identical)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

分片数 ≥1800 触发 partial merge 会损坏时间轴,合并后时长严重偏短(即使已启用 --use-ffmpeg-concat-demuxer)

1 participant