Skip to content

cuda: don't flush CUDA feature extractors twice when a thread pool is present - #1553

Open
BardieJoensen wants to merge 1 commit into
Netflix:masterfrom
BardieJoensen:cuda-fix-threaded-double-flush
Open

cuda: don't flush CUDA feature extractors twice when a thread pool is present#1553
BardieJoensen wants to merge 1 commit into
Netflix:masterfrom
BardieJoensen:cuda-fix-threaded-double-flush

Conversation

@BardieJoensen

Copy link
Copy Markdown
Contributor

Any CUDA-enabled run with --threads N currently aborts at flush time:

libvmaf WARNING feature "VMAF_integer_feature_motion2_score" cannot be overwritten at index 47
libvmaf ERROR context could not be synchronized

The error message points at CUDA, but no CUDA call fails (CUDA_LAUNCH_BLOCKING=1 surfaces nothing, compute-sanitizer is clean). The actual problem is in the flush logic. motion_cuda is the only extractor declared with both flags:

.flags = VMAF_FEATURE_EXTRACTOR_TEMPORAL | VMAF_FEATURE_EXTRACTOR_CUDA,

When a thread pool exists, flush_context() flushes it twice: once in flush_context_threaded(), which walks every TEMPORAL extractor, and again in the #ifdef HAVE_CUDA block, which walks every CUDA extractor. The second flush re-appends the final motion2 score at the same picture index, feature_vector_append() rejects it (that's the warning), and the -EINVAL is OR-ed into the same err as the cuCtxSynchronize() calls, producing the misleading context error.

Instrumenting flush_fex_cuda() with an fprintf on entry and return shows it directly (12-frame clip, indexes 0..11):

[dbg] flush enter s->index=11
[dbg] flush append ret=0 -> flush returns 1      <- first flush: done, no error
[dbg] flush enter s->index=11                    <- second flush, from the HAVE_CUDA block
libvmaf WARNING feature "VMAF_integer_feature_motion2_score" cannot be overwritten at index 11
[dbg] flush append ret=-22 -> flush returns -22
libvmaf ERROR context could not be synchronized

The fix skips CUDA extractors in the temporal flush loop and leaves them to the HAVE_CUDA block, which matches what the non-threaded path already does (its non-CUDA loop excludes them the same way).

Verified on an RTX 5060 Ti (driver 595.71.05, CUDA 12.9 and 13.3 builds, master @ 0f9912e), 1080p y4m pairs: all six combinations of --threads 0/1/4 x default/--gpumask 0 now complete, where before every --threads >= 1 run aborted with exit 234. On the 8-bit pair all six produce identical scores (mean 63.9282). On a 10-bit pair they only fully converge with the 16bpc motion stride fix (#1552) applied on top — that bug reads out of bounds, so its effect on the score varies with allocation layout; with both fixes, all six combinations give the same mean (63.5439).

#1538 addresses the same symptom by destroying the thread pool in vmaf_cuda_import_state(). That also resolves the abort, but it silently disables threading whenever CUDA state is imported — including --gpumask 0 runs, where feature extraction happens on the CPU and does benefit from the pool. With the double flush fixed the pool can stay; threaded and single-threaded runs produce identical scores in all the combinations above.

A possible follow-up (not in this PR): separating the extractor-flush err from the CUDA sync err in flush_context() would stop feature-collector errors from being reported as "context could not be synchronized" — that conflation is what made this bug (and apparently #1538) hard to triage in the first place.

motion_cuda is declared TEMPORAL | CUDA. With a thread pool present,
flush_context_threaded() flushes every TEMPORAL extractor and the
HAVE_CUDA block in flush_context() then flushes every CUDA extractor,
so motion_cuda gets flushed twice. The second flush re-appends the
final motion2 score at the same picture index, which
feature_vector_append() rejects, and the resulting -EINVAL is folded
into the same err as the cuCtxSynchronize() calls, so any CUDA run
with --threads N aborts with a misleading "context could not be
synchronized" even though no CUDA call failed.

Skip CUDA extractors in the temporal flush loop and leave them to the
HAVE_CUDA block, which is the only thing that flushes them on the
non-threaded path already. Verified across --threads 0/1/4 x
default/--gpumask 0: every combination now completes, where before
any --threads >= 1 run aborted with exit 234. On 8-bit input all six
combinations produce identical scores; 10-bit needs the 16bpc motion
stride fix (submitted separately) on top for full score equality,
since that bug's out-of-bounds reads vary with allocation layout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant