feat(mp4): defragment hybrid files with a progressive part - #561
Open
nchitkara-xai wants to merge 2 commits into
Open
feat(mp4): defragment hybrid files with a progressive part#561nchitkara-xai wants to merge 2 commits into
nchitkara-xai wants to merge 2 commits into
Conversation
nchitkara-xai
force-pushed
the
defragment-hybrid
branch
from
September 1, 2026 02:38
9dcc61d to
addadb3
Compare
nchitkara-xai
marked this pull request as ready for review
September 1, 2026 16:49
nchitkara-xai
force-pushed
the
defragment-hybrid
branch
3 times, most recently
from
September 5, 2026 03:38
b603735 to
262d9e0
Compare
A fragment whose tfdt re-declares an earlier decode time supersedes the earlier samples of its track (a retransmission): the fragment appearing later in the file wins, whole re-sent fragments disappear, superseded tails are trimmed at sample granularity, and the timeline continues from the declaring fragment, whether or not the re-sent bytes are identical. Non-overlapping input converts byte-identically to before. Ambiguous overlaps keep failing closed instead of being guessed at: an overlap that starts inside a sample (unless it only shrinks earlier tfdt-gap padding), any abandoned time range that no surviving later fragment declares again (whether the superseded fragment was trimmed or dropped whole, and with voided declarations not counting as coverage), and overlapping files whose fragments use absolute base data offsets are rejected, so no declared content is ever silently dropped. The payload size bound applies to the surviving samples, so re-sent declarations larger than the input file do not reject a file that resolves cleanly. The work is linear in the number of fragments. Coverage checking merges each track's surviving declaration windows once and binary searches them. dropTrackSamplesFrom walked track.frags backwards until a record ended at or below the cut; records voided by earlier cuts sit above later live records in collection order and never satisfy that test once the cut walks below them, so each later cut rescanned the whole voided tail: n one-sample fragments followed by n-1 fragments each rewinding one sample took O(n^2). A parallel live slice keeps the records with kept > 0; their sample ranges tile the samples in order, so a cut pops the suffix with firstSample >= keep and trims at most the record straddling keep. The firstSample+kept <= keep guard stays so a cut landing exactly on a record boundary inside gap padding does not advance its cutoff. Gap padding is tracked as a stack indexed by sample so truncation pops instead of scanning. Each record and padding entry leaves once, making every cut amortized O(1).
A fragmented file may carry progressive samples in its moov sample tables before the first fragment. These samples now come first in the converted output with their chunk structure preserved, and the fragment samples are appended to the same tracks. Per-track chunk order follows the sample tables even when stco offsets are not monotone, and the tracks' chunk runs are interleaved by their input file position. Hostile sample tables are validated before any count-proportional allocation, and the progressive payload counts toward the total payload bound. The progressive part takes part in overlap resolution like a fragment: a later fragment that re-declares its decode times supersedes the progressive samples under the same later-declarer-wins, sample-boundary, and coverage rules, so no declared content is ever silently dropped. A track with a progressive prefix starts at decode time zero, anchors the empty-edit alignment of later-starting fragment-only tracks, and gets its edit list validated like any other track. The sample table expansion lives on the box types as SttsBox.SampleDurations, CttsBox.CompositionTimeOffsets, and StssBox.SampleIsSync, each returning one value per sample after validating the entries against the declared sample count.
nchitkara-xai
force-pushed
the
defragment-hybrid
branch
from
September 7, 2026 04:57
262d9e0 to
1717dd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #560 (the first commit is #560); please review the last commit only.
Problem
Hybrid files carry a progressive part in the moov sample tables followed by
fragments (a recorder that finalizes periodically, or a progressive file
extended by appending fragments). #557 rejects them; converting requires
dropping the hybrid track via track selection.
Fix
The moov sample tables are expanded into the same sample list the fragments
feed (stts/ctts/stsc/stsz/stss/stco/co64, with hostile-table guards so
declared counts a file cannot physically hold error out before any
count-proportional allocation). Per-track chunk order follows the sample
tables unconditionally; across tracks, chunk runs are interleaved by input
file position, so non-monotonic chunk offsets (legal, if unusual) cannot
corrupt the sample-to-bytes mapping.
The progressive part counts as each track's earliest declaration and flows
through the overlap machinery from #560 unchanged: a later fragment may
supersede progressive samples under the same later-declarer-wins and
no-silently-dropped-ranges rules, and mid-sample cuts reject. A hybrid track
starts at decode time 0 by construction and anchors the cross-track
alignment of #557 for later-starting fragment-only tracks. Purely
progressive input (no fragments) still rejects as not fragmented.
Tests
Hybrid round-trips with multi-entry stsc, sync-sample subsets, and negative
composition offsets; non-monotonic chunk offsets round-trip byte-correct;
fragments superseding progressive samples (covered resolves, uncovered and
mid-sample reject); hybrid track anchoring a fragment-only track's empty
edit; hostile stts/stsc/stsz/ctts tables. All fragmented testdata files
convert byte-identically to #560 output.
go test ./...,go vet,gofmt,golangci-lintpass.