diff --git a/AGENTS.md b/AGENTS.md index aa6160f081..8db8414474 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,6 +25,10 @@ panic path, and fix a crash before building on top of it. The cross-app standard uses `unsafe_code = "deny"` with `#[allow(unsafe_code)]` only on its FFI modules, a `// SAFETY:` comment on every `unsafe` block, a safe `Result`-returning public API, and the pure-Rust decoder as the tested fallback ([ADR 0001](docs/adr/0001-platform-ffi.md)). + *Proposed, not in effect* ([ADR 0002](docs/adr/0002-optional-vendor-sdk-helpers.md), #383): an + opt-in vendor SDK feature may add one module that maps its helper process's frames read-only, + under the same rules; such a format has no pure-Rust decoder, and without the feature it is + reported as unsupported. 4. **Every input-derived number is hostile.** Media files, project and interchange files, presets, fonts, CLI / MCP / control-channel parameters and UI state are untrusted. Index and slice with `get()` (or validate bounds once, up front, for a hot loop); slice strings only at char boundaries; use @@ -122,7 +126,20 @@ reproduce an image (for example, point lists traced from someone else's icon). See [§0](#0-never-crash). It outranks every other engineering rule. -### 3.2 The rest +### 3.2 Optional vendor SDK decoders (proposed, not in effect) + +*Proposed in [ADR 0002](docs/adr/0002-optional-vendor-sdk-helpers.md) for #383; until the owners +accept it, the rules above stand unchanged.* A camera format with no public specification may be +decoded through its vendor's SDK only as an optional Cargo feature that is off by default and never +enabled by CI or official builds, built only against an SDK the user supplies (nothing of it +downloaded, committed or bundled), with the vendor's code running in a separate helper process the +editor talks to over a versioned protocol whose replies it treats as hostile input. Its `unsafe` is +at most one shared-memory module in `crates/platform` under ADR 0001's rules; the default build, +its gates and its tests are unchanged, and the format is still recognised there with a message +naming the feature. The adapter is original code; SDK-enabled builds are not distributed by the +project. + +### 3.3 The rest **Invariant — move identity uniqueness:** A move plan lists each clip identity at most once. The core move operation rejects repeated ids before mutation or id allocation; placing one identity diff --git a/Cargo.lock b/Cargo.lock index 4b8908302f..d18f6152c7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2221,6 +2221,7 @@ dependencies = [ "png", "rayon", "serde", + "serde_json", "thiserror 2.0.21", ] @@ -2276,6 +2277,7 @@ dependencies = [ name = "filmcraft-platform" version = "0.13.0" dependencies = [ + "cc", "filmcraft-bitstream", "filmcraft-codecs", "filmcraft-color", @@ -2289,6 +2291,7 @@ dependencies = [ "filmcraft-render", "filmcraft-testkit", "filmcraft-time", + "libc", "libloading", "log", "objc2-core-foundation", diff --git a/Cargo.toml b/Cargo.toml index 380d806eac..7d4a9f92ce 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -154,6 +154,11 @@ opt-level = 3 [profile.dev.package.filmcraft-vp9] opt-level = 3 +# OS media bindings and the RED helper's client: a 6K camera RAW frame is converted on the +# playback path (20 ms a frame at opt-level 1, a few at 3). +[profile.dev.package.filmcraft-platform] +opt-level = 3 + [profile.release] lto = "thin" codegen-units = 1 diff --git a/apps/filmcraft-cli/Cargo.toml b/apps/filmcraft-cli/Cargo.toml index 218b7f7d72..e9ae963e49 100644 --- a/apps/filmcraft-cli/Cargo.toml +++ b/apps/filmcraft-cli/Cargo.toml @@ -14,6 +14,7 @@ path = "src/main.rs" [features] default = [] +red-sdk = ["filmcraft-platform/red-sdk"] # Speech-to-text (Text panel ▸ Transcribe) with Whisper on the CPU, and model downloads. whisper = ["filmcraft-engine/whisper", "filmcraft-engine/speech-download"] diff --git a/apps/filmcraft-cli/src/probe.rs b/apps/filmcraft-cli/src/probe.rs index c5f97ba170..e30640ac40 100644 --- a/apps/filmcraft-cli/src/probe.rs +++ b/apps/filmcraft-cli/src/probe.rs @@ -153,6 +153,14 @@ pub fn probe(path: &str, image_sequence: bool) -> Result { }); return Ok(v); } + // Native SDK sources need the local reader, and camera RAW files must not be read whole. + if filmcraft_codecs::r3d::is_r3d(path) { + let reader: filmcraft_media::SharedReader = + Arc::new(filmcraft_media::reader::FileReader::open(std::path::Path::new(path)).map_err(|e| format!("{path}: {e}"))?); + let src = filmcraft_media::reader::open_reader(path, reader, &filmcraft_codecs::reader_openers(), &filmcraft_codecs::openers()) + .map_err(|e| format!("{path}: {e}"))?; + return serde_json::to_value(src.info()).map_err(|e| e.to_string()); + } let bytes: Arc<[u8]> = std::fs::read(path).map_err(|e| format!("{path}: {e}"))?.into(); let name = std::path::Path::new(path).file_name().map(|s| s.to_string_lossy().to_string()).unwrap_or_else(|| path.to_string()); let head = &bytes[..bytes.len().min(70_000)]; diff --git a/apps/filmcraft/Cargo.toml b/apps/filmcraft/Cargo.toml index 9e3f1cba8d..dd75353dcf 100644 --- a/apps/filmcraft/Cargo.toml +++ b/apps/filmcraft/Cargo.toml @@ -12,6 +12,7 @@ workspace = true [features] default = [] +red-sdk = ["filmcraft-platform/red-sdk"] # Speech-to-text (Text panel ▸ Transcribe) with Whisper on the CPU, and model downloads. whisper = ["filmcraft-engine/whisper", "filmcraft-engine/speech-download"] # Natural Text to Speech voices (Kokoro-82M on the CPU, downloaded on first use). diff --git a/crates/codecs/src/gop.rs b/crates/codecs/src/gop.rs index 240aeb5db9..43d53cd481 100644 --- a/crates/codecs/src/gop.rs +++ b/crates/codecs/src/gop.rs @@ -556,7 +556,7 @@ impl GopCache { /// container's clean aperture and then its display rotation are applied. fn finish(&self, mut f: VideoFrame) -> VideoFrame { if let Some(c) = self.explicit_color - && !matches!(f.data, filmcraft_frame::PixelData::Rgba8(_) | filmcraft_frame::PixelData::RgbaF32(_)) + && !matches!(f.data, filmcraft_frame::PixelData::Rgba8(_) | filmcraft_frame::PixelData::RgbaF32(_) | filmcraft_frame::PixelData::RgbaF16(_)) { f.color = c; } diff --git a/crates/codecs/src/lib.rs b/crates/codecs/src/lib.rs index f3ae47b72b..73c9e2a85b 100644 --- a/crates/codecs/src/lib.rs +++ b/crates/codecs/src/lib.rs @@ -32,6 +32,7 @@ pub mod mp4; pub mod mpeg; pub mod mxf; pub mod ogg; +pub mod r3d; mod stream_color; pub mod video; mod webm_alpha; @@ -134,13 +135,14 @@ pub fn software_video_decoder(entry: &filmcraft_isobmff::SampleEntry) -> Result< /// Openers for the engine's media pool (MP4/MOV, Matroska/WebM, MXF, Ogg Opus/Vorbis, MPEG TS/PS and /// MPEG-1/2 video elementary streams, APV raw bitstreams, YUV4MPEG2, standalone audio). pub fn openers() -> Vec { - vec![mp4::opener, mkv::opener, mxf::opener, ogg::opener, mpeg::opener, apv::opener, avi::opener, y4m::opener, audio::opener] + vec![r3d::opener, mp4::opener, mkv::opener, mxf::opener, ogg::opener, mpeg::opener, apv::opener, avi::opener, y4m::opener, audio::opener] } fn reader_registry() -> &'static RwLock> { static R: std::sync::OnceLock>> = std::sync::OnceLock::new(); R.get_or_init(|| { RwLock::new(vec![ + r3d::reader_opener, mp4::reader_opener, mkv::reader_opener, mxf::reader_opener, diff --git a/crates/codecs/src/r3d.rs b/crates/codecs/src/r3d.rs new file mode 100644 index 0000000000..ba80e04585 --- /dev/null +++ b/crates/codecs/src/r3d.rs @@ -0,0 +1,52 @@ +//! Recognise R3D imports even in builds without the optional native SDK backend. +use filmcraft_media::{MediaError, Result, SharedReader, SharedSource}; +use std::{path::Path, sync::Arc}; + +pub fn is_r3d(name: &str) -> bool { + Path::new(name).extension().is_some_and(|s| s.eq_ignore_ascii_case("r3d")) +} + +fn unavailable() -> Result { + Err(MediaError::Unsupported( + "R3D / Nikon ZR R3D NE requires a native FilmCraft build with the red-sdk feature, RED SDK 9.2.1, and a local file. See docs/r3d-ne.md.".into(), + )) +} + +pub fn opener(name: &str, _: Arc<[u8]>) -> Option> { + is_r3d(name).then(unavailable) +} + +pub fn reader_opener(name: &str, _: &[u8], _: &SharedReader) -> Option> { + is_r3d(name).then(unavailable) +} + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn disabled_backend_does_not_read_a_large_raw_file_into_memory() { + struct Large; + impl filmcraft_media::ByteReader for Large { + fn len(&self) -> u64 { + 10 * 1024 * 1024 * 1024 + } + fn read_at(&self, _: u64, buf: &mut [u8]) -> std::io::Result<()> { + assert!(buf.len() <= filmcraft_media::reader::HEAD_LEN); + buf.fill(0); + Ok(()) + } + } + let result = filmcraft_media::reader::open_reader("DSC_0001.R3D", Arc::new(Large), &[reader_opener], &[opener]); + assert!(matches!(result, Err(MediaError::Unsupported(message)) if message.contains("red-sdk"))); + } + #[test] + fn recognises_case_without_claiming_other_files() { + for name in ["DSC_0001.R3D", "DSC_0002.r3d", "a.R3d"] { + assert!(is_r3d(name)); + assert!(opener(name, Arc::from([])).unwrap().is_err()); + } + for name in ["a.mov", "a.nev", "a.r3d.mov", "r3d", ""] { + assert!(!is_r3d(name)); + } + } +} diff --git a/crates/codecs/src/webm_alpha.rs b/crates/codecs/src/webm_alpha.rs index 12750b21fc..2595446150 100644 --- a/crates/codecs/src/webm_alpha.rs +++ b/crates/codecs/src/webm_alpha.rs @@ -233,7 +233,7 @@ pub(crate) fn set_alpha(frame: &mut VideoFrame, a: &VideoFrame) { p[3] = rescale(v, 8) as u8; } } - PixelData::RgbaF32(_) => {} + PixelData::RgbaF32(_) | PixelData::RgbaF16(_) => {} } } diff --git a/crates/color/README.md b/crates/color/README.md index e78774322e..28b2f51814 100644 --- a/crates/color/README.md +++ b/crates/color/README.md @@ -40,6 +40,7 @@ for limited- and full-range files. | DJI D-Log | D-Gamut | DJI, *White Paper on D-Log and D-Gamut of DJI Cinema Color System* (2017) | CV | y ≤ 0.14: x = (y − 0.0929)/6.025; else x = (10^(3.89616y − 2.27752) − 0.0108)/0.9892 | | Fujifilm F-Log | F-Gamut (= BT.2020 primaries) | Fujifilm, *GFX ETERNA Technical Guide – White Paper* v1.01 (2026), F-Log conversion formula (same as the F-Log data sheet) | CV | y ≥ cut2: x = 10^((y − d)/c)/a − b/a; else x = (y − f)/e; a 0.555556, b 0.009468, c 0.344676, d 0.790453, e 8.735631, f 0.092864, cut1 0.00089, cut2 0.100537775223865 | | Fujifilm F-Log2 / F-Log2 C | F-Gamut / F-Gamut C (R 0.7347, 0.2653; G 0.0263, 0.9737; B 0.1173, −0.0224) | same document, F-Log2 C / F-Log2 conversion formula | CV | same form; a 5.555556, b 0.064829, c 0.245281, d 0.384316, e 8.799461, f 0.092864, cut1 0.000889, cut2 0.100686685370811 | +| RED Log3G10 | REDWideGamutRGB | RED, *White Paper on REDWideGamutRGB and Log3G10* (Rev B, 2017; shipped with the R3D SDK) | CV (full-scale float) | y < 0: x = y/15.1927 − 0.01; else x = (10^(y/0.224282) − 1)/155.975327 − 0.01 (18 % → 1/3, 184.32 → 1.0) | The encoders are the exact inverses given in the same documents. DJI **D-Log M** is not implemented: DJI has not published its formula (only conversion LUTs), and we do not derive curves @@ -60,6 +61,7 @@ Gamut primaries (all D65 white, x/y): | ARRI Wide Gamut 3 | 0.6840, 0.3130 | 0.2210, 0.8480 | 0.0861, −0.1020 | ARRI Log C document | | ARRI Wide Gamut 4 | 0.7347, 0.2653 | 0.1424, 0.8576 | 0.0991, −0.0308 | ARRI LogC4 specification | | D-Gamut | 0.71, 0.31 | 0.21, 0.88 | 0.09, −0.08 | DJI white paper | +| REDWideGamutRGB | 0.780308, 0.304253 | 0.121595, 1.493994 | 0.095612, −0.084589 | RED white paper | Matrices are derived from the chromaticities (SMPTE RP 177 normalised primary matrix); tests check the result against the published BT.709 RGB→XYZ and BT.2087 BT.709→BT.2020 matrices. diff --git a/crates/color/src/log.rs b/crates/color/src/log.rs index ec5967518b..9feed7de88 100644 --- a/crates/color/src/log.rs +++ b/crates/color/src/log.rs @@ -46,10 +46,12 @@ pub enum LogCurve { FLog, /// Fujifilm F-Log2 (also the curve of F-Log2 C). FLog2, + /// RED Log3G10 (18 % grey at 1/3, 10 stops above grey to 1.0). + Log3G10, } impl LogCurve { - pub const ALL: [LogCurve; 10] = [ + pub const ALL: [LogCurve; 11] = [ LogCurve::SLog3, LogCurve::VLog, LogCurve::CLog2, @@ -60,6 +62,7 @@ impl LogCurve { LogCurve::DLog, LogCurve::FLog, LogCurve::FLog2, + LogCurve::Log3G10, ]; pub fn domain(self) -> Domain { @@ -81,6 +84,7 @@ impl LogCurve { LogCurve::DLog => "DJI D-Log", LogCurve::FLog => "Fujifilm F-Log", LogCurve::FLog2 => "Fujifilm F-Log2", + LogCurve::Log3G10 => "RED Log3G10", } } @@ -97,6 +101,7 @@ impl LogCurve { LogCurve::DLog => dlog_encode(x), LogCurve::FLog => fujifilm_encode(&FLOG, x), LogCurve::FLog2 => fujifilm_encode(&FLOG2, x), + LogCurve::Log3G10 => log3g10_encode(x), } } @@ -113,6 +118,7 @@ impl LogCurve { LogCurve::DLog => dlog_decode(y), LogCurve::FLog => fujifilm_decode(&FLOG, y), LogCurve::FLog2 => fujifilm_decode(&FLOG2, y), + LogCurve::Log3G10 => log3g10_decode(y), } } @@ -311,6 +317,22 @@ fn fujifilm_decode(k: &Fujifilm, y: f64) -> f64 { if y >= k.cut2 { 10f64.powf((y - k.d) / k.c) / k.a - k.b / k.a } else { (y - k.f) / k.e } } +// RED, "White Paper on REDWideGamutRGB and Log3G10" (Rev B, 2017): a = 0.224282, b = 155.975327, +// offset c = 0.01, and the linear extension below −c with the curve's gradient at zero, g = 15.1927. +// The signal is the full-scale float value (18 % grey → 1/3, 184.32 → 1.0). +const LOG3G10_A: f64 = 0.224_282; +const LOG3G10_B: f64 = 155.975_327; +const LOG3G10_C: f64 = 0.01; +const LOG3G10_G: f64 = 15.1927; + +pub fn log3g10_encode(x: f64) -> f64 { + let x = x + LOG3G10_C; + if x < 0.0 { x * LOG3G10_G } else { LOG3G10_A * (x * LOG3G10_B + 1.0).log10() } +} +pub fn log3g10_decode(y: f64) -> f64 { + if y < 0.0 { y / LOG3G10_G - LOG3G10_C } else { (10f64.powf(y / LOG3G10_A) - 1.0) / LOG3G10_B - LOG3G10_C } +} + #[cfg(test)] mod tests { use super::*; @@ -376,6 +398,14 @@ mod tests { // the decoding cut is where the encoding's two pieces meet (within the published rounding) close(LogCurve::FLog.encode(0.000_89), 0.100_537_775_223_865, 2e-4, "F-Log cut"); close(LogCurve::FLog2.encode(0.000_889), 0.100_686_685_370_811, 2e-4, "F-Log2 cut"); + // RED Log3G10 mapping table: −0.01 → 0, 0 → 0.091551, 0.18 → 1/3, 1.0 → 0.493449, 184.322 → 1.0. + close(LogCurve::Log3G10.encode(-0.01), 0.0, 1e-12, "Log3G10 -1%"); + close(LogCurve::Log3G10.encode(0.0), 0.091_551, 1e-6, "Log3G10 0"); + close(LogCurve::Log3G10.encode(0.18), 1.0 / 3.0, 1e-6, "Log3G10 18%"); + close(LogCurve::Log3G10.encode(1.0), 0.493_449, 1e-6, "Log3G10 100%"); + close(LogCurve::Log3G10.encode(184.322), 1.0, 1e-6, "Log3G10 peak"); + close(LogCurve::Log3G10.decode(1.0), 184.322, 0.001, "Log3G10 peak linear"); + close(LogCurve::Log3G10.decode(-0.1), -0.1 / 15.1927 - 0.01, 1e-12, "Log3G10 linear toe"); } #[test] diff --git a/crates/color/src/spaces.rs b/crates/color/src/spaces.rs index e4a994df1c..214c97a478 100644 --- a/crates/color/src/spaces.rs +++ b/crates/color/src/spaces.rs @@ -36,6 +36,8 @@ pub enum Gamut { DGamut, /// Fujifilm F-Gamut C (F-Log2 C). F-Gamut itself has the BT.2020 primaries. FGamutC, + /// REDWideGamutRGB. + Rwg, } impl Gamut { @@ -54,6 +56,7 @@ impl Gamut { Gamut::Awg4 => [[0.7347, 0.2653], [0.1424, 0.8576], [0.0991, -0.0308]], Gamut::DGamut => [[0.71, 0.31], [0.21, 0.88], [0.09, -0.08]], Gamut::FGamutC => [[0.7347, 0.2653], [0.0263, 0.9737], [0.1173, -0.0224]], + Gamut::Rwg => [[0.780_308, 0.304_253], [0.121_595, 1.493_994], [0.095_612, -0.084_589]], } } @@ -174,12 +177,14 @@ pub enum ColorSpace { FLog2FGamut, /// Fujifilm F-Log2 C / F-Gamut C. FLog2CFGamutC, + /// RED Log3G10 in REDWideGamutRGB (the R3D SDK's IPP2 grading output, also RED's log exports). + Log3G10Rwg, /// Linear light with BT.709 primaries (EXR-style float sources). Linear709, } impl ColorSpace { - pub const ALL: [ColorSpace; 19] = [ + pub const ALL: [ColorSpace; 20] = [ ColorSpace::Rec709, ColorSpace::Srgb, ColorSpace::Rec2020, @@ -198,6 +203,7 @@ impl ColorSpace { ColorSpace::FLogFGamut, ColorSpace::FLog2FGamut, ColorSpace::FLog2CFGamutC, + ColorSpace::Log3G10Rwg, ColorSpace::Linear709, ]; @@ -222,6 +228,7 @@ impl ColorSpace { ColorSpace::FLogFGamut => "flog-fgamut", ColorSpace::FLog2FGamut => "flog2-fgamut", ColorSpace::FLog2CFGamutC => "flog2c-fgamutc", + ColorSpace::Log3G10Rwg => "log3g10-rwg", ColorSpace::Linear709 => "linear709", } } @@ -246,6 +253,7 @@ impl ColorSpace { ColorSpace::FLogFGamut => "Fujifilm F-Log / F-Gamut", ColorSpace::FLog2FGamut => "Fujifilm F-Log2 / F-Gamut", ColorSpace::FLog2CFGamutC => "Fujifilm F-Log2 C / F-Gamut C", + ColorSpace::Log3G10Rwg => "RED Log3G10 / REDWideGamutRGB", ColorSpace::Linear709 => "Linear (Rec. 709 primaries)", } } @@ -272,6 +280,7 @@ impl ColorSpace { ColorSpace::LogC4Awg4 => Gamut::Awg4, ColorSpace::DLogDGamut => Gamut::DGamut, ColorSpace::FLog2CFGamutC => Gamut::FGamutC, + ColorSpace::Log3G10Rwg => Gamut::Rwg, } } @@ -292,6 +301,7 @@ impl ColorSpace { ColorSpace::DLogDGamut => Curve::Log(LogCurve::DLog), ColorSpace::FLogFGamut => Curve::Log(LogCurve::FLog), ColorSpace::FLog2FGamut | ColorSpace::FLog2CFGamutC => Curve::Log(LogCurve::FLog2), + ColorSpace::Log3G10Rwg => Curve::Log(LogCurve::Log3G10), } } diff --git a/crates/engine/src/camera_raw.rs b/crates/engine/src/camera_raw.rs new file mode 100644 index 0000000000..c9ed9ae46c --- /dev/null +++ b/crates/engine/src/camera_raw.rs @@ -0,0 +1,648 @@ +//! Camera RAW development: Clip ▸ Source Settings… (`clip.sourceSettings`) and File ▸ Project +//! Settings ▸ Camera RAW (`file.projectSettings.cameraRaw`), modelled on a RED Camera Raw panel. +//! +//! A RAW clip decodes with the project's Camera RAW settings (Decode Using ▸ Project, what a clip +//! without settings of its own does) or with its own (Decode Using ▸ Clip). Both are +//! [`RawDevelop`]s: decode quality, colour science (IPP2), colour space and gamma curve, output +//! tone map, highlight rolloff and HDR peak, chroma noise reduction, flashing pixel adjustment, +//! the clip's metadata curves, creative LUT and CDL, and ISO, colour temperature, tint and +//! exposure adjustment. The decoder hands over linear light in the developed primaries, and the +//! compositor interprets it through [`filmcraft_project::Project::color_space_override`], so +//! nothing about the colour pipeline is stored per clip. + +use filmcraft_media::{ + DecodeUsing, Hundredths, RAW_EXPOSURE_RANGE, RAW_HDR_PEAK_RANGE, RAW_ISO_VALUES, RAW_KELVIN_RANGE, RAW_TINT_RANGE, RawClipInfo, RawColorSpace, RawDevelop, + RawFlashing, RawGamma, RawOutput, RawQuality, RawRollOff, RawToneMap, +}; +use filmcraft_project::{ItemId, ItemKind, MediaClip}; +use serde_json::{Value, json}; + +use crate::commands::bad; +use crate::project_tools::{media_of, targets}; +use crate::{EngineError, Result, Session}; + +/// A named choice parameter: absent is None, anything but one of the names is an error. +pub(crate) fn choice_p(p: &Value, key: &str, all: &[T], id: fn(T) -> &'static str, cmd: &str) -> Result> { + match p.get(key) { + None | Some(Value::Null) => Ok(None), + Some(v) => { + let names: Vec<&str> = all.iter().map(|t| id(*t)).collect(); + let name = v.as_str().ok_or_else(|| bad(cmd, format!("{key} must be one of {}", names.join(", "))))?; + all.iter() + .copied() + .find(|t| id(*t).eq_ignore_ascii_case(name)) + .map(Some) + .ok_or_else(|| bad(cmd, format!("{key} must be one of {}, not `{name}`", names.join(", ")))) + } + } +} + +/// A setting with a recorded value: absent leaves it alone, `null` sets it back to the clip's +/// recorded value (as shot), anything else sets it. +#[derive(Clone, Copy)] +enum Adjust { + Keep, + AsShot, + Set(T), +} + +impl Adjust { + fn apply(&self, field: &mut Option) { + match self { + Adjust::Keep => {} + Adjust::AsShot => *field = None, + Adjust::Set(v) => *field = Some(*v), + } + } + fn given(&self) -> bool { + !matches!(self, Adjust::Keep) + } +} + +fn adjust_p(p: &Value, key: &str, cmd: &str, what: &str, parse: impl Fn(&Value) -> Option) -> Result> { + match p.get(key) { + None => Ok(Adjust::Keep), + Some(Value::Null) => Ok(Adjust::AsShot), + Some(v) => parse(v).map(Adjust::Set).ok_or_else(|| bad(cmd, format!("{key} must be {what}, or null for as recorded"))), + } +} + +fn flag_p(p: &Value, key: &str, cmd: &str) -> Result> { + match p.get(key) { + None | Some(Value::Null) => Ok(None), + Some(Value::Bool(b)) => Ok(Some(*b)), + Some(_) => Err(bad(cmd, format!("{key} must be true or false"))), + } +} + +/// The development changes one call asks for (every field optional). +struct Change { + decode_using: Option, + /// Some(None) = the project's decode quality. + quality: Option>, + color_space: Option, + gamma: Option, + tone_map: Option, + rolloff: Option, + hdr_peak: Adjust, + chroma: Adjust, + flashing: Adjust, + metadata_curves: Option, + creative_lut: Option, + cdl: Option, + as_shot: bool, + iso: Adjust, + kelvin: Adjust, + tint: Adjust, + exposure: Adjust, +} + +impl Change { + fn parse(p: &Value, cmd: &str) -> Result { + let quality = match p.get("quality") { + None | Some(Value::Null) => None, + Some(Value::String(s)) if s.eq_ignore_ascii_case("project") => Some(None), + Some(_) => choice_p(p, "quality", &RawQuality::ALL, RawQuality::id, cmd)?.map(Some), + }; + // the first version's single output list maps onto colour space and gamma + let output = choice_p(p, "output", &RawOutput::ALL, RawOutput::id, cmd)?.map(RawOutput::space_gamma); + let whole = |v: &Value| v.as_u64().filter(|n| *n <= u64::from(u32::MAX)).map(|n| n as u32); + let hundredths = |v: &Value| v.as_f64().and_then(Hundredths::from_f64); + Ok(Change { + decode_using: choice_p(p, "decodeUsing", &DecodeUsing::ALL, DecodeUsing::id, cmd)?, + quality, + color_space: choice_p(p, "colorSpace", &RawColorSpace::ALL, RawColorSpace::id, cmd)?.or(output.map(|o| o.0)), + gamma: choice_p(p, "gamma", &RawGamma::ALL, RawGamma::id, cmd)?.or(output.map(|o| o.1)), + tone_map: choice_p(p, "toneMap", &RawToneMap::ALL, RawToneMap::id, cmd)?, + rolloff: choice_p(p, "rolloff", &RawRollOff::ALL, RawRollOff::id, cmd)?, + hdr_peak: adjust_p(p, "hdrPeakNits", cmd, "a peak in cd/m²", whole)?, + chroma: adjust_p(p, "chromaNoiseReduction", cmd, "true or false", Value::as_bool)?, + flashing: match p.get("flashingPixel") { + None => Adjust::Keep, + Some(Value::Null) => Adjust::AsShot, + Some(_) => choice_p(p, "flashingPixel", &RawFlashing::ALL, RawFlashing::id, cmd)?.map_or(Adjust::AsShot, Adjust::Set), + }, + metadata_curves: flag_p(p, "metadataCurves", cmd)?, + creative_lut: flag_p(p, "creativeLut", cmd)?, + cdl: flag_p(p, "cdl", cmd)?, + as_shot: flag_p(p, "asShot", cmd)?.unwrap_or(false), + iso: adjust_p(p, "iso", cmd, "one of the decoder's ISO values", whole)?, + kelvin: adjust_p(p, "kelvin", cmd, "a colour temperature in Kelvin", whole)?, + tint: adjust_p(p, "tint", cmd, "a tint from -100 to 100", hundredths)?, + exposure: adjust_p(p, "exposure", cmd, "an exposure adjustment in stops", hundredths)?, + }) + } + + /// A development setting is given (anything but Decode Using and the decode quality). + fn develops(&self) -> bool { + self.color_space.is_some() + || self.gamma.is_some() + || self.tone_map.is_some() + || self.rolloff.is_some() + || self.hdr_peak.given() + || self.chroma.given() + || self.flashing.given() + || self.metadata_curves.is_some() + || self.creative_lut.is_some() + || self.cdl.is_some() + || self.as_shot + || self.iso.given() + || self.kelvin.given() + || self.tint.given() + || self.exposure.given() + } + + fn any(&self) -> bool { + self.develops() || self.decode_using.is_some() || self.quality.is_some() + } + + /// Apply the development settings (not Decode Using or the quality) to `d`. A colour space + /// or gamma given alone that does not go with the other one brings the other one along + /// (REDWideGamutRGB → Log3G10, ST 2084 → Rec. 2020…); both given must fit together. + fn develop(&self, d: &mut RawDevelop) { + if let Some(c) = self.color_space { + d.color_space = c; + } + if let Some(g) = self.gamma { + d.gamma = g; + } + if !RawDevelop::supports(d.color_space, d.gamma) { + match (self.color_space.is_some(), self.gamma.is_some()) { + (true, false) => d.gamma = RawDevelop::gamma_for(d.color_space), + (false, true) => d.color_space = RawDevelop::space_for(d.gamma), + _ => {} // refused by validate + } + } + if let Some(t) = self.tone_map { + d.tone_map = t; + } + if let Some(r) = self.rolloff { + d.rolloff = r; + } + self.hdr_peak.apply(&mut d.hdr_peak_nits); + self.chroma.apply(&mut d.chroma_noise_reduction); + self.flashing.apply(&mut d.flashing_pixel); + if let Some(v) = self.metadata_curves { + d.metadata_curves = v; + } + if let Some(v) = self.creative_lut { + d.creative_lut = v; + } + if let Some(v) = self.cdl { + d.cdl = v; + } + if self.as_shot { + *d = d.with_as_shot(); + } + self.iso.apply(&mut d.iso); + self.kelvin.apply(&mut d.kelvin); + self.tint.apply(&mut d.tint); + self.exposure.apply(&mut d.exposure); + } +} + +/// A clip's own settings after `change`, under the project's `camera_raw`: Decode Using is the +/// one given, else Clip when a development setting is given, else what it was. A clip that +/// turns to Clip starts from the project's settings; one that decodes with the project's keeps +/// only its decode quality (None when it has none). +fn changed(camera_raw: &RawDevelop, own: Option, change: &Change) -> Option { + let was = own.map_or(DecodeUsing::Project, |d| d.decode_using); + let using = change.decode_using.unwrap_or(if change.develops() { DecodeUsing::Clip } else { was }); + let quality = change.quality.unwrap_or(own.and_then(|d| d.quality)); + match using { + DecodeUsing::Project => quality.map(|q| RawDevelop { decode_using: DecodeUsing::Project, quality: Some(q), ..Default::default() }), + DecodeUsing::Clip => { + let mut d = match own { + Some(d) if d.decode_using == DecodeUsing::Clip => d, + _ => RawDevelop { decode_using: DecodeUsing::Clip, ..*camera_raw }, + }; + d.quality = quality; + change.develop(&mut d); + Some(d) + } + } +} + +/// The settings that reproduce what the clip records (or its RMD sidecar says), as its decoder +/// reported it (`info`): Decode Using ▸ Clip, everything as recorded, the LUT and CDL as the clip +/// asks for them, and its recorded colour space and gamma when they are ones offered here (else +/// the ones in effect). The decode quality stays. +fn recorded(current: &RawDevelop, info: Option<&RawClipInfo>) -> RawDevelop { + let (space, gamma) = match info.and_then(|i| i.color_space.zip(i.gamma)) { + Some((s, g)) if RawDevelop::supports(s, g) => (s, g), + _ => (current.color_space, current.gamma), + }; + RawDevelop { + decode_using: DecodeUsing::Clip, + quality: current.quality, + color_space: space, + gamma, + creative_lut: info.is_some_and(|i| i.lut && i.lut_enabled), + cdl: info.is_some_and(|i| i.cdl_enabled), + ..Default::default() + } +} + +/// The merge key every live preview of these clips shares, until a commit or a revert ends it. +fn merge_key(ids: &[ItemId]) -> String { + format!("clip.sourceSettings:{}", ids.iter().map(|i| i.0.to_string()).collect::>().join(",")) +} + +/// `clip.sourceSettings`: the Camera Raw panel of camera RAW clips (Clip ▸ Source Settings…). +/// Without parameters it reports the controls (`settings`, each with its id, kind, label, value, +/// options, whether it is enabled and why not, and the panel column it sits in) and their +/// values; with them it sets them on every selected RAW clip (one undo step) and reports the +/// result. Values outside the decoder's ranges, and a colour space and gamma that do not go +/// together, are refused before anything changes. +/// +/// - `decodeUsing: clip | project`: the clip's own settings, or the project's Camera RAW +/// settings. A development setting given for a clip that uses the project's gives it settings +/// of its own, starting from the project's; `decodeUsing: project` drops them. +/// - `quality: project | fullPremium | halfPremium | halfGood | quarterGood | eighthGood | +/// sixteenthGood`: the decode quality (the largest decode a preview uses). +/// - `colorSpace`, `gamma` (and the first version's `output`), `toneMap`, `rolloff`, +/// `hdrPeakNits`, `chromaNoiseReduction`, `flashingPixel`, `metadataCurves`, `creativeLut`, +/// `cdl`, `iso`, `kelvin`, `tint`, `exposure` (`null`: as recorded; `asShot: true` puts the +/// four RAW adjustments back). +/// - `useChanges: true` copies the first clip's settings (after this call's changes) to every +/// other clip in `items`; `useSettings: true` puts every clip back to what it records (or its +/// RMD sidecar says); both in the same undo step as the changes. +/// - `updateRmd: true` writes every clip's settings into its RMD sidecar beside the clip +/// (creating or updating it; not an undo step: it changes a file), after the changes; the +/// decoder then reads them as the clip's recorded settings. +/// +/// Live preview, as the dialog does it: `preview: true` applies the call at once (the open decoder +/// develops the current frame with it) as a merged undo step, so every preview of the same clips +/// until the dialog closes is one step; `commit: true` (with or without settings, which it applies +/// first) ends that step; `revert: true` takes the whole step back, leaving the project as it was +/// before the first preview. The result's `preview` says whether such a step is open; a revert +/// reports `reverted`. An undo or any other edit in between ends the step as well. Other media +/// report that they have no source settings. +pub(crate) fn source_settings(s: &mut Session, p: &Value) -> Result { + const CMD: &str = "clip.sourceSettings"; + let mut ids: Vec = Vec::new(); + for i in targets(s, p) { + if let Some((root, _)) = media_of(&s.project, i) + && !ids.contains(&root) + { + ids.push(root); + } + } + // the first RAW clip speaks for the selection; a mixed selection develops its RAW clips only + let id = *ids + .iter() + .find(|i| media_of(&s.project, **i).is_some_and(|(_, m)| crate::media_pool::is_camera_raw(m))) + .or_else(|| ids.first()) + .ok_or_else(|| EngineError::Other("select a clip".into()))?; + let (_, m) = media_of(&s.project, id).ok_or_else(|| EngineError::Other("select a clip".into()))?; + let codec = m.info.video.as_ref().map(|v| v.codec.clone()).or_else(|| m.info.audio().map(|a| a.codec.clone())).unwrap_or_default(); + let container = m.info.container.clone(); + let developable = crate::media_pool::is_camera_raw(m); + let change = Change::parse(p, CMD)?; + let flag = |key: &str| flag_p(p, key, CMD).map(|v| v.unwrap_or(false)); + let (preview, commit, revert) = (flag("preview")?, flag("commit")?, flag("revert")?); + let (use_changes, use_settings, update_rmd) = (flag("useChanges")?, flag("useSettings")?, flag("updateRmd")?); + let key = merge_key(&ids); + let changing = change.any() || use_changes || use_settings; + if revert && (changing || preview || commit || update_rmd) { + return Err(bad(CMD, "revert takes no settings, preview, commit or RMD update")); + } + let reverted = revert && s.revert_merged(&key); + if !developable { + let name = if codec.is_empty() { "This clip".to_string() } else { codec.to_uppercase() }; + if changing || update_rmd { + return Err(bad(CMD, format!("{name} has no source settings"))); + } + return Ok(json!({ + "item": id.0, + "items": ids.iter().map(|i| i.0).collect::>(), + "codec": codec, + "container": container, + "settings": [], + "message": format!("{name} has no source settings controls in FilmCraft. Supported camera RAW clips use their recorded settings; use timeline color effects for grading."), + })); + } + if changing { + let camera_raw = s.project.settings.camera_raw; + let raw_ids: Vec = ids.iter().copied().filter(|i| media_of(&s.project, *i).is_some_and(|(_, m)| crate::media_pool::is_camera_raw(m))).collect(); + // what each clip records, for Use Settings (a clip whose decoder is not open has nothing + // to report: its LUT and CDL stay off, its colour space and gamma as they are) + let infos: Vec> = + raw_ids.iter().map(|i| if use_settings { s.source(*i).and_then(|src| src.raw_clip_info()) } else { None }).collect(); + // every clip's resulting settings, worked out and checked before anything changes + let mut next: Vec<(ItemId, Option)> = Vec::new(); + for (i, info) in raw_ids.iter().zip(&infos) { + let Some((_, m)) = media_of(&s.project, *i) else { continue }; + let mut own = changed(&camera_raw, m.interpret.develop, &change); + if use_settings { + let current = RawDevelop::effective(&camera_raw, own.as_ref()); + own = Some(recorded(¤t, info.as_ref())); + } + next.push((*i, own)); + } + if use_changes && let Some(first) = next.first().map(|n| n.1) { + for n in next.iter_mut().skip(1) { + n.1 = first; + } + } + for (_, own) in &next { + RawDevelop::effective(&camera_raw, own.as_ref()).validate().map_err(|e| bad(CMD, e))?; + } + let body = |pr: &mut filmcraft_project::Project, _: &mut crate::EditorState| { + for (i, own) in &next { + let Some(ItemKind::Media(m)) = pr.item_mut(*i).map(|it| &mut it.kind) else { continue }; + set_own(m, &camera_raw, *own); + } + Ok(()) + }; + if preview || commit { + // a live preview, or the dialog's OK after one: the same undo step as the previews + s.edit_merged("Source Settings", &key, body)?; + } else { + s.edit("Source Settings", body)?; + } + } + if commit { + s.end_merged(&key); + } + let mut written = Vec::new(); + let mut rmd_errors: Vec = Vec::new(); + if update_rmd { + let mut errors = Vec::new(); + for i in &ids { + let Some((_, m)) = media_of(&s.project, *i) else { continue }; + let Some(d) = crate::media_pool::develop_of(&s.project.settings.camera_raw, m) else { continue }; + let name = s.project.item(*i).map(|it| it.name.clone()).unwrap_or_default(); + match s.source(*i).ok_or_else(|| "its media is not open".to_string()).and_then(|src| src.write_raw_sidecar(&d).map_err(|e| e.to_string())) { + Ok(path) => written.push(json!({"item": i.0, "path": path})), + Err(e) => errors.push(format!("{name}: {e}")), + } + } + if written.is_empty() && !errors.is_empty() { + return Err(EngineError::Other(format!("RMD not written: {}", errors.join("; ")))); + } + rmd_errors = errors; + } + let mut r = report(s, id, &ids)?; + r["preview"] = json!(s.history.merge_key.as_deref() == Some(key.as_str())); + r["reverted"] = json!(reverted); + if update_rmd { + r["rmdWritten"] = json!(written); + r["rmdErrors"] = json!(rmd_errors); + } + Ok(r) +} + +/// Store a clip's own settings and bring what depends on them in line: a colour space +/// interpretation the first version of Source Settings stored for its log output (or one that +/// the development now decides) gives way to the derived one, and the stream reports the linear +/// light the decoder hands over. +fn set_own(m: &mut MediaClip, camera_raw: &RawDevelop, own: Option) { + let before = RawDevelop::effective(camera_raw, m.interpret.develop.as_ref()); + m.interpret.develop = own; + let after = RawDevelop::effective(camera_raw, m.interpret.develop.as_ref()); + sync_derived(m, &before, &after); +} + +/// Bring what depends on a clip's development in line after it changed from `before` to `after` +/// (its own settings, or the project's for a clip that follows them). +fn sync_derived(m: &mut MediaClip, before: &RawDevelop, after: &RawDevelop) { + let output_changed = (before.color_space, before.gamma) != (after.color_space, after.gamma); + if let Some(cs) = m.interpret.color_space + && RawDevelop::INTERPRETATIONS.contains(&cs) + && (output_changed || cs == filmcraft_color::ColorSpace::Log3G10Rwg) + { + m.interpret.color_space = None; + } + if let Some(v) = m.info.video.as_mut() { + v.color.transfer = filmcraft_color::Transfer::Linear; + v.hdr = (after.gamma == RawGamma::St2084) + .then_some(after.hdr_peak_nits) + .flatten() + .map(|nits| filmcraft_color::HdrMetadata { mastering_max_nits: Some(nits as f32), ..Default::default() }); + } +} + +fn options(all: &[T], id: fn(T) -> &'static str, label: fn(T) -> &'static str) -> Vec { + all.iter().map(|t| json!({"id": id(*t), "label": label(*t)})).collect() +} + +/// The controls of the panel for develop `d` (the settings in effect), the clip's own `own` +/// (None for the project's panel), what the clip records (`info`, `as_shot`) and whether its +/// development controls follow the project (`follows`). Columns 1–3 as the panel lays them out. +fn controls( + d: &RawDevelop, + own: Option>, + info: Option<&RawClipInfo>, + as_shot: Option, + follows: bool, +) -> Vec { + let clip_panel = own.is_some(); + let on = !follows; + let display = d.gamma.is_display(); + let shot = |f: fn(&filmcraft_media::RawAsShot) -> f64| as_shot.as_ref().map(f); + let not_display = "Applies to a display gamma curve (BT.1886, sRGB, ST 2084, Hybrid Log-Gamma)."; + let linear_note = "The linear output carries no look: choose a display or log gamma curve to apply it."; + let mut v = Vec::new(); + if clip_panel { + let quality = own.flatten().and_then(|o| o.quality).map_or("project", RawQuality::id); + let mut q = vec![json!({"id": "project", "label": "Use project setting"})]; + q.extend(options(&RawQuality::ALL, RawQuality::id, RawQuality::label)); + v.push(json!({"id": "quality", "kind": "choice", "label": "Decode Quality", "column": 1, "value": quality, "enabled": true, "options": q, + "note": "The largest decode a preview uses; an export and a monitor at 100 % always decode at full resolution, premium."})); + v.push(json!({"id": "decodeUsing", "kind": "choice", "label": "Decode Using", "column": 1, "value": d.decode_using.id(), "enabled": true, + "options": options(&DecodeUsing::ALL, DecodeUsing::id, DecodeUsing::label)})); + } else { + v.push(json!({"id": "quality", "kind": "choice", "label": "Decode Quality", "column": 1, "value": d.quality().id(), "enabled": true, + "options": options(&RawQuality::ALL, RawQuality::id, RawQuality::label), + "note": "The largest decode a preview uses; an export and a monitor at 100 % always decode at full resolution, premium."})); + } + v.push(json!({"id": "colorScience", "kind": "fixed", "label": "Color Science", "column": 1, "value": "ipp2", "enabled": false, + "options": [{"id": "ipp2", "label": "IPP2"}], "note": "Nikon R3D NE is developed with RED IPP2 only."})); + let spaces: Vec = RawColorSpace::ALL.iter().map(|c| json!({"id": c.id(), "label": c.label()})).collect(); + let gammas: Vec = RawGamma::ALL.iter().map(|g| json!({"id": g.id(), "label": g.label(), "fits": RawDevelop::supports(d.color_space, *g)})).collect(); + v.push(json!({"id": "colorSpace", "kind": "choice", "label": "Color Space", "column": 1, "value": d.color_space.id(), "enabled": on, "options": spaces})); + v.push(json!({"id": "gamma", "kind": "choice", "label": "Gamma Curve", "column": 1, "value": d.gamma.id(), "enabled": on, "options": gammas, + "note": "Display curves go with Rec. 709, Rec. 2020 and P3 D65, Log3G10 with REDWideGamutRGB, ST 2084 and Hybrid Log-Gamma with Rec. 2020; linear with any."})); + if clip_panel { + let hdrx = info.is_some_and(|i| i.hdrx); + let why = if hdrx { + "HDRx blending is not available in FilmCraft yet." + } else { + "This clip has no HDRx track (Nikon R3D NE records none): there is nothing to blend." + }; + v.push(json!({"id": "blendType", "kind": "fixed", "label": "Blend Type", "column": 1, "value": "none", "enabled": false, + "options": [{"id": "none", "label": "None"}], "note": why})); + v.push(json!({"id": "blendBias", "kind": "fixed", "label": "Blend Bias", "column": 1, "value": 0.0, "enabled": false, "note": why})); + } + let grades = d.grades_apply(); + v.push(json!({"id": "metadataCurves", "kind": "bool", "label": "Apply Metadata Curves", "column": 1, "value": d.metadata_curves, "enabled": on && grades, + "recorded": info.map(|i| i.curves), "note": if grades { "The luma and RGB curves the clip records." } else { linear_note }})); + let has_lut = info.is_none_or(|i| i.lut); + v.push(json!({"id": "creativeLut", "kind": "bool", "label": "Apply Creative LUT", "column": 1, "value": d.creative_lut, "enabled": on && grades && has_lut, "available": has_lut, + "recorded": info.map(|i| i.lut_enabled), + "note": if !has_lut { "This clip carries no creative LUT." } else if grades { "The 3D LUT the clip (or its RMD) carries." } else { linear_note }})); + let cdl = info.map(|i| i.cdl); + v.push( + json!({"id": "cdl", "kind": "bool", "label": "Apply CDL", "column": 1, "value": d.cdl, "enabled": on && grades, "recorded": info.map(|i| i.cdl_enabled), + "cdl": cdl.map(|c| json!({"saturation": c[0], "slope": [c[1], c[4], c[7]], "offset": [c[2], c[5], c[8]], "power": [c[3], c[6], c[9]]})), + "note": if grades { "The ASC CDL the clip (or its RMD) records." } else { linear_note }}), + ); + v.push(json!({"id": "iso", "kind": "iso", "label": "ISO", "column": 2, "value": d.iso, "asShot": shot(|a| f64::from(a.iso)), "enabled": on, + "options": RAW_ISO_VALUES, "default": 800})); + v.push(json!({"id": "toneMap", "kind": "choice", "label": "Output Tone Map", "column": 2, "value": d.tone_map.id(), "enabled": on && display, + "recorded": info.and_then(|i| i.tone_map).map(RawToneMap::id), + "options": options(&RawToneMap::ALL, RawToneMap::id, RawToneMap::label), "note": if display { "" } else { not_display }})); + v.push(json!({"id": "rolloff", "kind": "choice", "label": "Highlight Roll Off", "column": 2, "value": d.rolloff.id(), "enabled": on && display, + "recorded": info.and_then(|i| i.rolloff).map(RawRollOff::id), + "options": options(&RawRollOff::ALL, RawRollOff::id, RawRollOff::label), "note": if display { "" } else { not_display }})); + let pq = d.gamma == RawGamma::St2084; + let (lo, hi) = info.map(|i| i.hdr_peak_range).filter(|r| r.0 <= r.1 && r.1 > 0).unwrap_or((*RAW_HDR_PEAK_RANGE.start(), *RAW_HDR_PEAK_RANGE.end())); + v.push(json!({"id": "hdrPeakNits", "kind": "number", "label": "HDR Peak Nits", "column": 2, "value": d.hdr_peak_nits, + "asShot": info.map(|i| f64::from(i.hdr_peak_nits.unwrap_or(i.hdr_peak_default))), "enabled": on && pq, + "min": lo.max(*RAW_HDR_PEAK_RANGE.start()), "max": hi.min(*RAW_HDR_PEAK_RANGE.end()), "step": 10, "decimals": 0, "suffix": " cd/m²", "default": 1000, + "note": if pq { "" } else { "Applies to the ST 2084 (PQ) gamma curve only." }})); + v.push(json!({"id": "chromaNoiseReduction", "kind": "bool", "label": "Chroma Noise Reduction", "column": 2, "value": d.chroma_noise_reduction, + "recorded": info.map(|i| i.chroma_noise_reduction), "enabled": on, "note": "Full and half resolution premium decodes."})); + let mut fpa = vec![json!({"id": "asRecorded", "label": "As recorded"})]; + fpa.extend(options(&RawFlashing::ALL, RawFlashing::id, RawFlashing::label)); + v.push(json!({"id": "flashingPixel", "kind": "choice", "label": "Flashing Pixel Adjust", "column": 2, "value": d.flashing_pixel.map_or("asRecorded", RawFlashing::id), + "recorded": info.map(|i| i.flashing_pixel.id()), "enabled": on, "options": fpa, "note": "Full and half resolution premium decodes."})); + v.push( + json!({"id": "kelvin", "kind": "number", "label": "Color Temp", "column": 3, "value": d.kelvin, "asShot": shot(|a| f64::from(a.kelvin)), "enabled": on, + "min": RAW_KELVIN_RANGE.start(), "max": RAW_KELVIN_RANGE.end(), "step": 50, "decimals": 0, "suffix": " K", "default": 5600}), + ); + v.push(json!({"id": "tint", "kind": "number", "label": "Tint", "column": 3, "value": d.tint, "asShot": shot(|a| f64::from(a.tint)), "enabled": on, + "min": RAW_TINT_RANGE.start(), "max": RAW_TINT_RANGE.end(), "step": 1, "decimals": 2, "suffix": "", "default": 0})); + v.push(json!({"id": "exposure", "kind": "number", "label": "Exposure Adjust", "column": 3, "value": d.exposure, "asShot": shot(|a| f64::from(a.exposure)), + "enabled": on, "min": RAW_EXPOSURE_RANGE.start(), "max": RAW_EXPOSURE_RANGE.end(), "step": 0.1, "decimals": 2, "suffix": " stops", "default": 0})); + v +} + +/// The report of `clip.sourceSettings` for clip `id` of the selection `ids`. +fn report(s: &Session, id: ItemId, ids: &[ItemId]) -> Result { + let (_, m) = media_of(&s.project, id).ok_or_else(|| EngineError::Other("select a clip".into()))?; + let camera_raw = s.project.settings.camera_raw; + let d = RawDevelop::effective(&camera_raw, m.interpret.develop.as_ref()); + let follows = d.decode_using == DecodeUsing::Project; + // the clip's recorded values, from the open decoder (an unopened or offline clip has none yet) + let src = s.source(id); + let as_shot = src.as_ref().and_then(|src| src.raw_as_shot()); + let info = src.as_ref().and_then(|src| src.raw_clip_info()); + let codec = m.info.video.as_ref().map(|v| v.codec.clone()).unwrap_or_default(); + let rmd_note = match info.as_ref().and_then(|i| i.rmd.clone()) { + Some(path) => format!("The clip's RMD sidecar is in effect: its settings are the clip's recorded settings ({path})."), + None => "No RMD sidecar: the clip's recorded settings are the camera's.".to_string(), + }; + Ok(json!({ + "item": id.0, + "items": ids.iter().map(|i| i.0).collect::>(), + "codec": codec, + "container": m.info.container, + "develop": d, + "own": m.interpret.develop, + "project": camera_raw, + "asShot": as_shot, + "clip": info, + "rmd": info.as_ref().and_then(|i| i.rmd.clone()), + "rmdNote": rmd_note, + "colorSpace": s.project.color_space_override(m).map(|c| c.id()), + "settings": controls(&d, Some(m.interpret.develop), info.as_ref(), as_shot, follows), + "message": if follows { + "This clip decodes with the project's Camera RAW settings (File > Project Settings > Camera RAW). Choose Decode Using > Clip to give it settings of its own." + } else { + "RED IPP2 development of the RAW picture. ISO, colour temperature, tint and exposure are applied by the SDK before development (as shot unless set); the output tone map and highlight roll-off shape a display gamma curve. Proxies keep the look they were made with." + }, + })) +} + +/// `file.projectSettings.cameraRaw`: the project's Camera RAW settings, which every RAW clip +/// that decodes with the project's settings (Source Settings ▸ Decode Using ▸ Project) develops +/// with. The same parameters as `clip.sourceSettings` without Decode Using (`quality` is the +/// project's decode quality); one undo step. Without parameters it reports them. `standard: true` +/// puts back the settings a new project starts with (Rec. 709, BT.1886, the SDK's tone map and +/// rolloff, as shot). +pub(crate) fn project_settings(s: &mut Session, p: &Value) -> Result { + const CMD: &str = "file.projectSettings.cameraRaw"; + let change = Change::parse(p, CMD)?; + if change.decode_using.is_some() { + return Err(bad(CMD, "decodeUsing is a clip setting (clip.sourceSettings)")); + } + if change.quality == Some(None) { + return Err(bad(CMD, "the project's quality is one of the decode qualities, not `project`")); + } + let standard = flag_p(p, "standard", CMD)?.unwrap_or(false); + let mut d = if standard { RawDevelop::STANDARD } else { s.project.settings.camera_raw }; + if let Some(q) = change.quality { + d.quality = q; + } + change.develop(&mut d); + d.decode_using = DecodeUsing::Clip; + d.validate().map_err(|e| bad(CMD, e))?; + if d != s.project.settings.camera_raw { + s.edit("Project Settings", |pr, _| { + let old = std::mem::replace(&mut pr.settings.camera_raw, d); + // Clips that follow the project (Decode Using ▸ Project) take its new output with it. + for it in pr.items.values_mut() { + if let ItemKind::Media(m) = &mut it.kind + && m.is_camera_raw() + { + let (before, after) = (RawDevelop::effective(&old, m.interpret.develop.as_ref()), RawDevelop::effective(&d, m.interpret.develop.as_ref())); + if before != after { + sync_derived(m, &before, &after); + } + } + } + Ok(()) + })?; + } + let d = s.project.settings.camera_raw; + let follow = s + .project + .items + .values() + .filter(|it| matches!(&it.kind, ItemKind::Media(m) if m.is_camera_raw() && RawDevelop::effective(&d, m.interpret.develop.as_ref()).decode_using == DecodeUsing::Project)) + .count(); + Ok(json!({ + "develop": d, + "standard": d == RawDevelop::STANDARD, + "clips": follow, + "settings": controls(&d, None, None, None, false), + "message": "Camera RAW clips develop with these settings unless they have settings of their own (Clip > Source Settings > Decode Using > Clip).", + })) +} + +/// Every text the Camera Raw panels show from these commands (labels, option labels, notes and +/// messages, in each state that changes them), for the interface's translation catalogs. +pub fn panel_texts() -> Vec { + let mut out: Vec = Vec::new(); + let mut add = |s: &str| { + if !s.is_empty() && !out.iter().any(|o| o == s) { + out.push(s.to_string()); + } + }; + let pq = RawDevelop { color_space: RawColorSpace::Rec2020, gamma: RawGamma::St2084, ..Default::default() }; + let no_lut = RawClipInfo::default(); + let lut = RawClipInfo { lut: true, hdrx: true, ..Default::default() }; + for d in [RawDevelop::default(), RawDevelop::STANDARD, pq] { + for (own, info) in [(None, None), (Some(None), None), (Some(None), Some(&no_lut)), (Some(None), Some(&lut))] { + for st in controls(&d, own, info, None, false) { + for k in ["label", "note"] { + add(st[k].as_str().unwrap_or_default()); + } + for o in st["options"].as_array().into_iter().flatten() { + add(o["label"].as_str().unwrap_or_default()); + } + } + } + } + for m in [ + "This clip decodes with the project's Camera RAW settings (File > Project Settings > Camera RAW). Choose Decode Using > Clip to give it settings of its own.", + "RED IPP2 development of the RAW picture. ISO, colour temperature, tint and exposure are applied by the SDK before development (as shot unless set); the output tone map and highlight roll-off shape a display gamma curve. Proxies keep the look they were made with.", + "Camera RAW clips develop with these settings unless they have settings of their own (Clip > Source Settings > Decode Using > Clip).", + "No RMD sidecar: the clip's recorded settings are the camera's.", + ] { + add(m); + } + out +} diff --git a/crates/engine/src/camera_raw_tests.rs b/crates/engine/src/camera_raw_tests.rs new file mode 100644 index 0000000000..d01bb6c559 --- /dev/null +++ b/crates/engine/src/camera_raw_tests.rs @@ -0,0 +1,516 @@ +//! Tests of the Camera Raw panel (`clip.sourceSettings`) and the project's Camera RAW settings +//! (`file.projectSettings.cameraRaw`), on hand-made R3D NE items: the commands work on the +//! project, so the RED SDK is optional; a stand-in decoder plays the RMD sidecar. + +use std::sync::{Arc, Mutex}; + +use filmcraft_frame::{AudioBuffer, VideoFrame}; +use filmcraft_media::{ + DecodeUsing, FrameRequest, Hundredths, MediaError, MediaInfo, MediaSource, RAW_ISO_VALUES, RawAsShot, RawClipInfo, RawColorSpace, RawDevelop, RawGamma, + RawQuality, RawRollOff, RawToneMap, +}; +use filmcraft_project::{ItemId, ItemKind, MediaClip}; +use serde_json::{Value, json}; + +use crate::Session; +use crate::media_test_util::tmp_dir; + +fn demo() -> Session { + let mut s = Session::default(); + s.execute("file.openDemoProject", json!({})).unwrap(); + s +} + +fn item_named(s: &Session, name: &str) -> ItemId { + s.project.items.values().find(|i| i.name == name).map(|i| i.id).unwrap_or_else(|| panic!("no item {name}")) +} + +/// A copy of the demo's MP4 item as an R3D NE clip at `path` (no decoder: the file does not exist). +fn add_r3d(s: &mut Session, name: &str, path: &str) -> ItemId { + let ocean = item_named(s, "Ocean_Sunset.mp4"); + let mut p = (*s.project).clone(); + let mut clip = p.item(ocean).unwrap().as_media().unwrap().clone(); + clip.media = filmcraft_project::MediaRef::File { path: path.into() }; + clip.info.video.as_mut().unwrap().codec = "R3D NE".into(); + clip.info.video.as_mut().unwrap().color.transfer = filmcraft_color::Transfer::Linear; + let id = p.add_item(name, filmcraft_project::Label::Iris, ItemKind::Media(clip), None); + s.project = Arc::new(p); + id +} + +fn media(s: &Session, id: ItemId) -> MediaClip { + s.project.item(id).unwrap().as_media().unwrap().clone() +} + +fn setting(r: &Value, id: &str) -> Value { + r["settings"].as_array().unwrap().iter().find(|x| x["id"] == id).cloned().unwrap_or_else(|| panic!("no setting {id} in {r}")) +} + +/// The panel's controls in their three columns, as a RED Camera Raw panel lays them out; a new +/// project's clip decodes with the project's settings (IPP2, Rec. 709, BT.1886), whose +/// development controls it shows disabled until Decode Using is Clip; Blend Type and Bias are +/// disabled with a reason. +#[test] +fn the_panel_reports_three_columns_and_a_new_projects_standard() { + let mut s = demo(); + let id = add_r3d(&mut s, "DSC_0001.R3D", "/footage/DSC_0001.R3D"); + s.execute("project.select", json!({"items": [id.0]})).unwrap(); + let r = s.execute("clip.sourceSettings", json!({})).unwrap(); + let column = + |c: u64| r["settings"].as_array().unwrap().iter().filter(|x| x["column"] == c).map(|x| x["id"].as_str().unwrap().to_string()).collect::>(); + assert_eq!(column(1), ["quality", "decodeUsing", "colorScience", "colorSpace", "gamma", "blendType", "blendBias", "metadataCurves", "creativeLut", "cdl"]); + assert_eq!(column(2), ["iso", "toneMap", "rolloff", "hdrPeakNits", "chromaNoiseReduction", "flashingPixel"]); + assert_eq!(column(3), ["kelvin", "tint", "exposure"]); + assert_eq!(r["develop"]["decodeUsing"], "project", "{r}"); + assert_eq!((r["develop"]["colorSpace"].as_str(), r["develop"]["gamma"].as_str()), (Some("rec709"), Some("bt1886"))); + assert_eq!((r["develop"]["toneMap"].as_str(), r["develop"]["rolloff"].as_str()), (Some("medium"), Some("soft"))); + assert_eq!(setting(&r, "quality")["value"], "project"); + assert_eq!(setting(&r, "colorScience")["options"][0]["label"], "IPP2"); + for id in ["colorSpace", "gamma", "iso", "kelvin", "toneMap", "cdl"] { + assert_eq!(setting(&r, id)["enabled"], false, "{id} follows the project"); + } + for id in ["quality", "decodeUsing"] { + assert_eq!(setting(&r, id)["enabled"], true, "{id}"); + } + let blend = setting(&r, "blendType"); + assert_eq!((blend["enabled"].as_bool(), blend["kind"].as_str()), (Some(false), Some("fixed"))); + assert!(blend["note"].as_str().unwrap().contains("HDRx"), "{blend}"); + assert_eq!(media(&s, id).interpret.develop, None, "reading changes nothing"); + // Decode Using ▸ Clip: the clip starts from the project's settings, now editable + let r = s.execute("clip.sourceSettings", json!({"decodeUsing": "clip"})).unwrap(); + assert_eq!(media(&s, id).interpret.develop, Some(RawDevelop { decode_using: DecodeUsing::Clip, ..RawDevelop::STANDARD })); + assert_eq!((setting(&r, "gamma")["enabled"].as_bool(), setting(&r, "toneMap")["enabled"].as_bool()), (Some(true), Some(true))); + assert_eq!(setting(&r, "hdrPeakNits")["enabled"], false, "HDR peak is for ST 2084 only"); + // only the combinations the compositor interprets fit + let gamma = setting(&r, "gamma"); + let fits: Vec<&str> = gamma["options"].as_array().unwrap().iter().filter(|o| o["fits"] == true).map(|o| o["id"].as_str().unwrap()).collect(); + assert_eq!(fits, ["linear", "bt1886", "srgb"]); +} + +/// Colour space and gamma are separate lists; one given alone brings a fitting other along, both +/// given must fit. The compositor's interpretation follows (nothing is stored for it), HDR +/// peaks reach the tone mapping, and hostile values change nothing. +#[test] +fn colour_space_and_gamma_develop_and_the_interpretation_follows() { + let mut s = demo(); + let id = add_r3d(&mut s, "DSC_0001.R3D", "/footage/DSC_0001.R3D"); + s.execute("project.select", json!({"items": [id.0]})).unwrap(); + let effective = |s: &mut Session| s.execute("media.colorInfo", json!({"item": id.0})).unwrap()["effective"].as_str().map(str::to_string); + let d = |s: &Session| media(s, id).interpret.develop.unwrap(); + s.execute("clip.sourceSettings", json!({"colorSpace": "redWideGamutRgb"})).unwrap(); + assert_eq!((d(&s).color_space, d(&s).gamma, d(&s).decode_using), (RawColorSpace::RedWideGamutRgb, RawGamma::Log3G10, DecodeUsing::Clip)); + assert_eq!(effective(&mut s).as_deref(), Some("log3g10-rwg")); + assert_eq!(media(&s, id).interpret.color_space, None, "derived, not stored"); + s.execute("clip.sourceSettings", json!({"gamma": "linear"})).unwrap(); + assert_eq!((d(&s).color_space, d(&s).gamma), (RawColorSpace::RedWideGamutRgb, RawGamma::Linear), "linear goes with any space"); + s.execute("clip.sourceSettings", json!({"gamma": "st2084", "hdrPeakNits": 1000})).unwrap(); + assert_eq!((d(&s).color_space, d(&s).hdr_peak_nits), (RawColorSpace::Rec2020, Some(1000))); + assert_eq!(effective(&mut s).as_deref(), Some("rec2100-pq")); + let peak = media(&s, id).info.video.unwrap().hdr.and_then(|h| h.mastering_max_nits); + assert_eq!(peak, Some(1000.0), "the tone mapping knows the peak"); + s.execute("clip.sourceSettings", json!({"colorSpace": "p3d65"})).unwrap(); + assert_eq!((d(&s).gamma, effective(&mut s).as_deref()), (RawGamma::Bt1886, Some("p3d65"))); + assert_eq!(media(&s, id).info.video.unwrap().hdr, None); + s.execute("clip.sourceSettings", json!({"colorSpace": "rec709", "gamma": "srgb"})).unwrap(); + // Rec. 709 frames say what they are: no interpretation (the fast path) + let info = s.execute("media.colorInfo", json!({"item": id.0})).unwrap(); + assert_eq!((info["override"].clone(), info["effective"].as_str()), (Value::Null, Some("linear709")), "{info}"); + // the first version's `output` still works + s.execute("clip.sourceSettings", json!({"output": "log3g10"})).unwrap(); + assert_eq!((d(&s).color_space, d(&s).gamma), (RawColorSpace::RedWideGamutRgb, RawGamma::Log3G10)); + // a person's own Interpret Footage choice wins until the output changes + s.execute("clip.interpretFootage", json!({"items": [id.0], "colorSpace": "rec2020"})).unwrap(); + assert_eq!(effective(&mut s).as_deref(), Some("rec2020")); + s.execute("clip.sourceSettings", json!({"toneMap": "high"})).unwrap(); + assert_eq!(effective(&mut s).as_deref(), Some("rec2020")); + s.execute("clip.sourceSettings", json!({"gamma": "bt1886", "colorSpace": "rec709"})).unwrap(); + assert_eq!(effective(&mut s).as_deref(), Some("linear709"), "a new output clears an interpretation that conflicts with it"); + // processing options + s.execute( + "clip.sourceSettings", + json!({"chromaNoiseReduction": false, "flashingPixel": "strong", "metadataCurves": false, "cdl": true, "quality": "halfGood"}), + ) + .unwrap(); + let got = d(&s); + assert_eq!( + (got.chroma_noise_reduction, got.flashing_pixel.map(|f| f.id()), got.metadata_curves, got.cdl, got.quality), + (Some(false), Some("strong"), false, true, Some(RawQuality::HalfGood)) + ); + s.execute("clip.sourceSettings", json!({"chromaNoiseReduction": null, "flashingPixel": null, "quality": "project"})).unwrap(); + let got = d(&s); + assert_eq!((got.chroma_noise_reduction, got.flashing_pixel, got.quality), (None, None, None), "back to as recorded"); + // hostile values are refused and change nothing + let before = media(&s, id); + let rev = s.revision; + for bad in [ + json!({"colorSpace": "redWideGamutRgb", "gamma": "bt1886"}), + json!({"colorSpace": "rec709", "gamma": "hlg"}), + json!({"gamma": "rec2020"}), + json!({"colorSpace": 1}), + json!({"output": "rec2020"}), + json!({"toneMap": "ultra"}), + json!({"rolloff": {}}), + json!({"hdrPeakNits": 99}), + json!({"hdrPeakNits": 10_001}), + json!({"hdrPeakNits": -5}), + json!({"hdrPeakNits": "1000"}), + json!({"chromaNoiseReduction": "yes"}), + json!({"flashingPixel": "blinding"}), + json!({"metadataCurves": 1}), + json!({"quality": "eighth"}), + json!({"quality": 2}), + json!({"decodeUsing": "camera"}), + json!({"useChanges": "yes"}), + json!({"updateRmd": 1}), + ] { + assert!(s.execute("clip.sourceSettings", bad.clone()).is_err(), "{bad}"); + } + assert_eq!((s.revision, media(&s, id)), (rev, before)); + // the project round-trips with the settings + let text = serde_json::to_string(&*s.project).unwrap(); + let back: filmcraft_project::Project = serde_json::from_str(&text).unwrap(); + assert_eq!(back.item(id).unwrap().as_media().unwrap().interpret.develop, media(&s, id).interpret.develop); + // an MP4 has no source settings + let ocean = item_named(&s, "Ocean_Sunset.mp4"); + s.execute("project.select", json!({"items": [ocean.0]})).unwrap(); + for p in [json!({"gamma": "bt1886"}), json!({"iso": 800}), json!({"useSettings": true}), json!({"updateRmd": true})] { + assert!(s.execute("clip.sourceSettings", p.clone()).is_err(), "{p}"); + } + assert!(s.execute("clip.sourceSettings", json!({})).unwrap()["settings"].as_array().unwrap().is_empty()); +} + +/// ISO, colour temperature, tint and exposure: as shot until set, set in one undo step, put back +/// one at a time (`null`) or together (`asShot`), refused outside the decoder's ranges. +#[test] +fn raw_adjustments_are_set_reset_and_range_checked() { + let mut s = demo(); + let id = add_r3d(&mut s, "DSC_0002.R3D", "/footage/DSC_0002.R3D"); + s.execute("project.select", json!({"items": [id.0]})).unwrap(); + let r = s.execute("clip.sourceSettings", json!({})).unwrap(); + let iso = setting(&r, "iso"); + assert_eq!((iso["kind"].as_str(), iso["value"].clone(), iso["asShot"].clone()), (Some("iso"), Value::Null, Value::Null), "{iso}"); + assert_eq!(iso["options"], json!(RAW_ISO_VALUES)); + let kelvin = setting(&r, "kelvin"); + assert_eq!((kelvin["label"].as_str(), kelvin["min"].as_u64(), kelvin["max"].as_u64()), (Some("Color Temp"), Some(1700), Some(100_000))); + assert_eq!(setting(&r, "exposure")["label"], "Exposure Adjust"); + let rev = s.revision; + let r = s.execute("clip.sourceSettings", json!({"iso": 3200, "kelvin": 4300, "tint": -12.5, "exposure": 0.333})).unwrap(); + assert_eq!((r["develop"]["exposure"].clone(), r["develop"]["tint"].clone()), (json!(0.33), json!(-12.5)), "{r}"); + let d = media(&s, id).interpret.develop.unwrap(); + assert_eq!( + (d.iso, d.kelvin, d.tint, d.exposure, d.decode_using), + (Some(3200), Some(4300), Some(Hundredths(-1250)), Some(Hundredths(33)), DecodeUsing::Clip) + ); + assert!(s.revision > rev); + assert_eq!( + crate::media_pool::media_key(&media(&s, id), &s.project.settings.camera_raw), + "file:/footage/DSC_0002.R3D|develop:rec709/bt1886/medium/soft/iso3200/k4300/t-1250/e33", + "the pool key carries them" + ); + s.execute("clip.sourceSettings", json!({"tint": null})).unwrap(); + assert_eq!(media(&s, id).interpret.develop.unwrap().tint, None); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(media(&s, id).interpret.develop.unwrap().tint, Some(Hundredths(-1250)), "one undo step per change"); + s.execute("edit.redo", json!({})).unwrap(); + s.execute("clip.sourceSettings", json!({"asShot": true})).unwrap(); + assert!(!media(&s, id).interpret.develop.unwrap().adjusts_raw()); + let before = media(&s, id); + let rev = s.revision; + for bad in [ + json!({"iso": 1250}), + json!({"iso": 0}), + json!({"iso": "800"}), + json!({"iso": 800.5}), + json!({"kelvin": 1699}), + json!({"kelvin": 100_001}), + json!({"kelvin": 5600.5}), + json!({"tint": 100.01}), + json!({"tint": "warm"}), + json!({"exposure": 8.01}), + json!({"exposure": 1e300}), + json!({"asShot": "yes"}), + json!({"iso": 3200, "kelvin": 5}), + ] { + assert!(s.execute("clip.sourceSettings", bad.clone()).is_err(), "{bad}"); + } + assert_eq!((s.revision, media(&s, id)), (rev, before), "a refused request changes nothing, not even the valid part"); + s.execute("clip.sourceSettings", json!({"iso": 25600, "kelvin": 100_000, "tint": 100, "exposure": -8})).unwrap(); + let d = media(&s, id).interpret.develop.unwrap(); + assert_eq!((d.iso, d.kelvin, d.tint, d.exposure), (Some(25600), Some(100_000), Some(Hundredths(10_000)), Some(Hundredths(-800)))); +} + +/// The project's Camera RAW settings: a new project's standard, an old project's scene-linear +/// Rec. 709 (its clips keep their picture), every clip under Decode Using ▸ Project following a +/// change (one undo step) while clips with settings of their own keep them. +#[test] +fn project_settings_are_the_default_of_clips_that_decode_using_project() { + let mut s = demo(); + assert_eq!(s.project.settings.camera_raw, RawDevelop::STANDARD, "a new project starts from the standard"); + let a = add_r3d(&mut s, "A.R3D", "/footage/A.R3D"); + let b = add_r3d(&mut s, "B.R3D", "/footage/B.R3D"); + s.execute("clip.sourceSettings", json!({"items": [b.0], "gamma": "srgb"})).unwrap(); + let key = |s: &Session, id: ItemId| crate::media_pool::media_key(&media(s, id), &s.project.settings.camera_raw); + assert_eq!(key(&s, a), "file:/footage/A.R3D|develop:rec709/bt1886/medium/soft"); + let r = s.execute("file.projectSettings.cameraRaw", json!({})).unwrap(); + assert_eq!((r["standard"].as_bool(), r["clips"].as_u64()), (Some(true), Some(1)), "{r}"); + assert!(r["settings"].as_array().unwrap().iter().all(|c| c["id"] != "decodeUsing"), "Decode Using is a clip setting"); + let steps = s.history.undo.len(); + s.execute("file.projectSettings.cameraRaw", json!({"colorSpace": "redWideGamutRgb", "quality": "quarterGood", "iso": 1600})).unwrap(); + assert_eq!(s.history.undo.len(), steps + 1); + let cr = s.project.settings.camera_raw; + assert_eq!((cr.color_space, cr.gamma, cr.quality, cr.iso), (RawColorSpace::RedWideGamutRgb, RawGamma::Log3G10, Some(RawQuality::QuarterGood), Some(1600))); + assert_eq!(key(&s, a), "file:/footage/A.R3D|develop:redWideGamutRgb/log3g10/medium/soft/q:quarterGood/iso1600", "A follows the project"); + assert_eq!(s.execute("media.colorInfo", json!({"item": a.0})).unwrap()["effective"], "log3g10-rwg"); + assert_eq!(media(&s, b).interpret.develop.unwrap().gamma, RawGamma::Srgb, "B keeps its own"); + assert!(key(&s, b).contains("rec709/srgb"), "{}", key(&s, b)); + // the clip's own decode quality wins over the project's, Decode Using aside + s.execute("clip.sourceSettings", json!({"items": [a.0], "quality": "fullPremium"})).unwrap(); + assert_eq!(media(&s, a).interpret.develop.map(|d| (d.decode_using, d.quality)), Some((DecodeUsing::Project, Some(RawQuality::FullPremium)))); + assert!(key(&s, a).contains("q:fullPremium"), "{}", key(&s, a)); + s.execute("edit.undo", json!({})).unwrap(); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(s.project.settings.camera_raw, RawDevelop::STANDARD, "one undo step"); + // a development setting on a clip that follows the project gives it settings of its own + s.execute("clip.sourceSettings", json!({"items": [a.0], "kelvin": 3000})).unwrap(); + let own = media(&s, a).interpret.develop.unwrap(); + assert_eq!((own.decode_using, own.gamma, own.kelvin), (DecodeUsing::Clip, RawGamma::Bt1886, Some(3000))); + // back to the project's drops them + s.execute("clip.sourceSettings", json!({"items": [a.0], "decodeUsing": "project", "kelvin": 3200})).unwrap(); + assert_eq!(media(&s, a).interpret.develop, None, "Decode Using ▸ Project: nothing of its own"); + // the project command refuses clip settings and hostile values, changing nothing + let rev = s.revision; + for bad in [ + json!({"decodeUsing": "clip"}), + json!({"quality": "project"}), + json!({"gamma": "log3g10", "colorSpace": "rec709"}), + json!({"iso": 1250}), + json!({"standard": 1}), + ] { + assert!(s.execute("file.projectSettings.cameraRaw", bad.clone()).is_err(), "{bad}"); + } + assert_eq!(s.revision, rev); + s.execute("file.projectSettings.cameraRaw", json!({"gamma": "linear"})).unwrap(); + s.execute("file.projectSettings.cameraRaw", json!({"standard": true})).unwrap(); + assert_eq!(s.project.settings.camera_raw, RawDevelop::STANDARD); +} + +/// A project change of output reaches the clips that follow it as it does a clip's own: ST 2084 +/// gives them HDR metadata with the project's peak (what tone mapping reads), and leaving it +/// takes the metadata away. A clip with settings of its own keeps what it had. +#[test] +fn project_settings_refresh_the_hdr_metadata_of_clips_that_follow_them() { + let mut s = demo(); + let a = add_r3d(&mut s, "A.R3D", "/footage/A.R3D"); + let b = add_r3d(&mut s, "B.R3D", "/footage/B.R3D"); + s.execute("clip.sourceSettings", json!({"items": [b.0], "gamma": "srgb"})).unwrap(); + let peak = |s: &Session, id: ItemId| media(s, id).info.video.unwrap().hdr.and_then(|h| h.mastering_max_nits); + s.execute("file.projectSettings.cameraRaw", json!({"gamma": "st2084", "hdrPeakNits": 1000})).unwrap(); + assert_eq!((peak(&s, a), peak(&s, b)), (Some(1000.0), None)); + assert_eq!(s.execute("media.colorInfo", json!({"item": a.0})).unwrap()["effective"], "rec2100-pq"); + s.execute("file.projectSettings.cameraRaw", json!({"hdrPeakNits": 4000})).unwrap(); + assert_eq!(peak(&s, a), Some(4000.0)); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(peak(&s, a), Some(1000.0), "undo puts the metadata back with the settings"); + s.execute("file.projectSettings.cameraRaw", json!({"standard": true})).unwrap(); + assert_eq!(peak(&s, a), None); +} + +/// A project saved before Camera RAW settings existed opens with scene-linear Rec. 709 as its +/// project settings (what its clips developed with), and a clip's old `output` still reads. +#[test] +fn an_old_project_keeps_what_it_had() { + let mut s = demo(); + let a = add_r3d(&mut s, "A.R3D", "/footage/A.R3D"); + let b = add_r3d(&mut s, "B.R3D", "/footage/B.R3D"); + let mut doc = serde_json::to_value(&*s.project).unwrap(); + doc["settings"].as_object_mut().unwrap().remove("camera_raw"); + let items = doc["items"].as_object_mut().unwrap(); + for (k, v) in items.iter_mut() { + if *k == b.0.to_string() { + v["kind"]["Media"]["interpret"]["develop"] = json!({"output": "rec709", "toneMap": "high", "rolloff": "soft", "iso": 800}); + } + } + let old: filmcraft_project::Project = serde_json::from_value(doc).unwrap(); + assert_eq!(old.settings.camera_raw, RawDevelop::default()); + let da = old.develop_of(old.item(a).unwrap().as_media().unwrap()).unwrap(); + assert_eq!((da.color_space, da.gamma, da.decode_using), (RawColorSpace::Rec709, RawGamma::Linear, DecodeUsing::Project), "as before: scene-linear"); + let db = old.develop_of(old.item(b).unwrap().as_media().unwrap()).unwrap(); + assert_eq!( + (db.decode_using, db.gamma, db.tone_map, db.rolloff, db.iso), + (DecodeUsing::Clip, RawGamma::Bt1886, RawToneMap::High, RawRollOff::Soft, Some(800)), + "the old output and settings" + ); + assert_eq!(crate::media_pool::media_key(old.item(a).unwrap().as_media().unwrap(), &old.settings.camera_raw), "file:/footage/A.R3D"); +} + +/// Use Changes copies the shown clip's settings to every selected clip, and a live preview of it +/// stays one undo step with the dialog's other changes. +#[test] +fn use_changes_applies_one_clips_settings_to_every_selected_clip_in_one_step() { + let mut s = demo(); + let ids: Vec = (0..3).map(|i| add_r3d(&mut s, &format!("C{i}.R3D"), &format!("/footage/C{i}.R3D"))).collect(); + let ocean = item_named(&s, "Ocean_Sunset.mp4"); + s.execute("clip.sourceSettings", json!({"items": [ids[2].0], "iso": 400, "gamma": "srgb"})).unwrap(); + let items: Vec = std::iter::once(ids[0].0).chain(ids.iter().skip(1).map(|i| i.0)).chain([ocean.0]).collect(); + let steps = s.history.undo.len(); + s.execute("clip.sourceSettings", json!({"items": [ids[0].0], "kelvin": 3000, "exposure": 0.5, "preview": true})).unwrap(); + let r = s.execute("clip.sourceSettings", json!({"items": items, "useChanges": true, "preview": true})).unwrap(); + assert_eq!(r["item"], ids[0].0, "the first RAW clip speaks for the selection"); + let first = media(&s, ids[0]).interpret.develop.unwrap(); + assert_eq!((first.kelvin, first.exposure), (Some(3000), Some(Hundredths(50)))); + for id in &ids { + assert_eq!(media(&s, *id).interpret.develop, Some(first), "{id:?}"); + } + assert_eq!(media(&s, ocean).interpret.develop, None, "the MP4 is skipped"); + s.execute("clip.sourceSettings", json!({"items": items, "commit": true})).unwrap(); + // the dialog's previews were on the first clip's merge key, Use Changes on the selection's: + // two steps at most, and undoing them restores every clip + assert!(s.history.undo.len() <= steps + 2); + while s.history.undo.len() > steps { + s.execute("edit.undo", json!({})).unwrap(); + } + assert_eq!(media(&s, ids[1]).interpret.develop, None); + assert_eq!(media(&s, ids[2]).interpret.develop.unwrap().iso, Some(400)); + // without a preview: one ordinary undo step for every clip + s.execute("clip.sourceSettings", json!({"items": [ids[0].0], "tint": 5})).unwrap(); + let steps = s.history.undo.len(); + s.execute("clip.sourceSettings", json!({"items": items, "useChanges": true})).unwrap(); + assert_eq!(s.history.undo.len(), steps + 1); + assert!(ids.iter().all(|i| media(&s, *i).interpret.develop.unwrap().tint == Some(Hundredths(500)))); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(media(&s, ids[2]).interpret.develop.unwrap().iso, Some(400), "one undo restores them all"); + assert_eq!(media(&s, ids[1]).interpret.develop, None); +} + +/// A stand-in RAW decoder: reports recorded settings and an RMD path in a temp folder, and writes +/// the settings it is given there as JSON (the RED SDK writes XML; the real-clip test on the +/// helper checks that). +struct Sidecar { + info: MediaInfo, + dir: std::path::PathBuf, + written: Mutex>, +} +impl MediaSource for Sidecar { + fn info(&self) -> &MediaInfo { + &self.info + } + fn video_frame(&self, _: FrameRequest) -> filmcraft_media::Result> { + Err(MediaError::NoStream("video")) + } + fn audio(&self, _: i64, _: usize, _: u32) -> filmcraft_media::Result { + Err(MediaError::NoStream("audio")) + } + fn set_develop(&self, _: &RawDevelop) -> filmcraft_media::Result<()> { + Ok(()) + } + fn raw_as_shot(&self) -> Option { + let w = *self.written.lock().unwrap(); + Some(RawAsShot { iso: w.and_then(|d| d.iso).unwrap_or(800), kelvin: 5600.0, tint: 0.0, exposure: 0.0 }) + } + fn raw_clip_info(&self) -> Option { + let path = self.dir.join("DSC_0009.RMD").to_string_lossy().into_owned(); + let present = self.written.lock().unwrap().is_some(); + Some(RawClipInfo { + color_space: Some(RawColorSpace::Rec709), + gamma: Some(RawGamma::Bt1886), + lut: true, + lut_enabled: true, + cdl_enabled: false, + rmd: present.then(|| path.clone()), + rmd_path: path, + ..Default::default() + }) + } + fn write_raw_sidecar(&self, develop: &RawDevelop) -> filmcraft_media::Result { + let path = self.dir.join("DSC_0009.RMD"); + std::fs::write(&path, serde_json::to_vec(develop).unwrap()).map_err(|e| MediaError::Io(e.to_string()))?; + *self.written.lock().unwrap() = Some(*develop); + Ok(path.to_string_lossy().into_owned()) + } +} + +/// Update RMD writes the clip's settings beside it (only when asked; never an undo step) and the +/// panel then says the RMD is in effect; Use Settings puts the clip back to what it records. +#[test] +fn update_rmd_writes_the_sidecar_and_use_settings_reverts_to_the_recorded() { + let dir = tmp_dir("camera-raw-rmd"); + let mut s = demo(); + let path = dir.join("DSC_0009.R3D").to_string_lossy().into_owned(); + let id = add_r3d(&mut s, "DSC_0009.R3D", &path); + let src = Arc::new(Sidecar { info: media(&s, id).info, dir: dir.clone(), written: Mutex::default() }); + s.media.insert_keyed(id, String::new(), src.clone()); + s.execute("project.select", json!({"items": [id.0]})).unwrap(); + let r = s.execute("clip.sourceSettings", json!({})).unwrap(); + assert_eq!(r["rmd"], Value::Null); + assert!(r["rmdNote"].as_str().unwrap().starts_with("No RMD"), "{r}"); + assert_eq!(setting(&r, "creativeLut")["enabled"], false, "follows the project"); + assert!(!dir.join("DSC_0009.RMD").exists(), "nothing is written without the action"); + s.execute("clip.sourceSettings", json!({"iso": 1600, "gamma": "srgb", "creativeLut": false})).unwrap(); + let steps = s.history.undo.len(); + let r = s.execute("clip.sourceSettings", json!({"updateRmd": true})).unwrap(); + assert_eq!(s.history.undo.len(), steps, "writing a file is not an undo step"); + assert_eq!(r["rmdWritten"][0]["path"].as_str(), Some(dir.join("DSC_0009.RMD").to_string_lossy().as_ref()), "{r}"); + let written: RawDevelop = serde_json::from_slice(&std::fs::read(dir.join("DSC_0009.RMD")).unwrap()).unwrap(); + assert_eq!((written.iso, written.gamma), (Some(1600), RawGamma::Srgb), "the settings in effect"); + assert!(r["rmd"].is_string() && r["rmdNote"].as_str().unwrap().contains("RMD sidecar is in effect"), "{r}"); + assert_eq!(setting(&r, "iso")["asShot"], json!(1600.0), "the decoder now records the RMD's ISO"); + // Use Settings: back to the clip's recorded settings, the LUT on as the clip asks + s.execute("clip.sourceSettings", json!({"exposure": 1.0, "toneMap": "low"})).unwrap(); + let steps = s.history.undo.len(); + s.execute("clip.sourceSettings", json!({"useSettings": true})).unwrap(); + assert_eq!(s.history.undo.len(), steps + 1); + let d = media(&s, id).interpret.develop.unwrap(); + assert_eq!( + (d.decode_using, d.iso, d.exposure, d.tone_map, d.creative_lut, d.gamma), + (DecodeUsing::Clip, None, None, RawToneMap::Clip, true, RawGamma::Bt1886), + "as recorded" + ); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(media(&s, id).interpret.develop.unwrap().exposure, Some(Hundredths(100))); + // a clip whose decoder cannot write is an error naming it, and changes nothing + let other = add_r3d(&mut s, "Gone.R3D", "/nowhere/Gone.R3D"); + let e = s.execute("clip.sourceSettings", json!({"items": [other.0], "updateRmd": true})).unwrap_err().to_string(); + assert!(e.contains("RMD not written") && e.contains("Gone.R3D"), "{e}"); + let _ = std::fs::remove_dir_all(dir); +} + +/// The live preview flow is the dialog's: previews merge into one undo step, a commit ends it, a +/// revert takes it back, hostile previews change nothing. +#[test] +fn previews_coalesce_into_one_undo_step_and_revert() { + let mut s = demo(); + let id = add_r3d(&mut s, "DSC_0004.R3D", "/footage/DSC_0004.R3D"); + s.execute("project.select", json!({"items": [id.0]})).unwrap(); + let develop = |s: &Session| media(s, id).interpret.develop; + let steps = |s: &Session| s.history.undo.len(); + s.execute("clip.sourceSettings", json!({"iso": 800})).unwrap(); + let opened = (develop(&s), steps(&s)); + for iso in [1600, 3200, 6400] { + s.execute("clip.sourceSettings", json!({"items": [id.0], "iso": iso, "preview": true})).unwrap(); + assert_eq!((develop(&s).unwrap().iso, steps(&s)), (Some(iso), opened.1 + 1), "previews share one undo step"); + } + s.execute("clip.sourceSettings", json!({"items": [id.0], "gamma": "log3g10", "decodeUsing": "clip", "preview": true})).unwrap(); + for bad in [json!({"iso": 1250, "preview": true}), json!({"gamma": "hlg", "colorSpace": "p3d65", "preview": true}), json!({"iso": 800, "preview": "yes"})] { + assert!(s.execute("clip.sourceSettings", bad.clone()).is_err(), "{bad}"); + } + assert_eq!((develop(&s).unwrap().gamma, steps(&s)), (RawGamma::Log3G10, opened.1 + 1)); + let r = s.execute("clip.sourceSettings", json!({"items": [id.0], "kelvin": 4300, "commit": true})).unwrap(); + assert_eq!((r["preview"].clone(), develop(&s).unwrap().kelvin, steps(&s)), (json!(false), Some(4300), opened.1 + 1), "{r}"); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!((develop(&s), steps(&s)), opened, "one Undo takes the whole dialog back"); + let undone = s.history.redo.len(); + let before = (develop(&s), steps(&s)); + s.execute("clip.sourceSettings", json!({"items": [id.0], "decodeUsing": "project", "preview": true})).unwrap(); + assert_eq!(develop(&s), None); + let r = s.execute("clip.sourceSettings", json!({"items": [id.0], "revert": true})).unwrap(); + assert_eq!(r["reverted"], true); + assert_eq!((develop(&s), steps(&s)), before, "Cancel leaves no trace"); + // nor takes away the Redo of what was undone before the preview + assert_eq!(s.history.redo.len(), undone, "the redo steps are back"); + s.execute("edit.redo", json!({})).unwrap(); + assert_eq!(develop(&s).unwrap().kelvin, Some(4300), "Redo brings the committed dialog back"); + s.execute("edit.undo", json!({})).unwrap(); + for bad in [json!({"revert": true, "iso": 800}), json!({"revert": true, "updateRmd": true}), json!({"commit": 1})] { + assert!(s.execute("clip.sourceSettings", bad.clone()).is_err(), "{bad}"); + } +} diff --git a/crates/engine/src/color.rs b/crates/engine/src/color.rs index d25270ad56..6fd0c8bf5b 100644 --- a/crates/engine/src/color.rs +++ b/crates/engine/src/color.rs @@ -361,7 +361,7 @@ fn color_info(s: &mut Session, p: &Value) -> Result { let id = item_p(p, "item").ok_or_else(|| bad("media.colorInfo", "need `item`"))?; let Some(ItemKind::Media(m)) = s.project.item(id).map(|i| &i.kind) else { return Err(bad("media.colorInfo", "not a media item")) }; let detected = m.info.video.as_ref().map(|v| ColorSpace::from_info(&v.color)); - let effective = m.interpret.color_space.or(detected); + let effective = s.project.color_space_override(m).or(detected); Ok(json!({ "detected": detected.map(|c| c.id()), "detectedLabel": detected.map(|c| c.label()), diff --git a/crates/engine/src/demo.rs b/crates/engine/src/demo.rs index 18fdf3cca4..c0ddd1b8c3 100644 --- a/crates/engine/src/demo.rs +++ b/crates/engine/src/demo.rs @@ -32,7 +32,7 @@ pub fn add_generator(p: &mut Project, pool: &MediaPool, src: GeneratorSource, na proxy: None, identity: None, }; - let key = crate::media_pool::media_key(&clip); + let key = crate::media_pool::media_key(&clip, &p.settings.camera_raw); let id = p.add_item(name, label, ItemKind::Media(clip), bin); pool.insert_keyed(id, key, Arc::new(src.with_name(name))); id diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index 5b70393d89..bb717e2ab5 100644 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -12,6 +12,7 @@ pub mod aaf_omf; pub mod autosave; +pub mod camera_raw; pub mod captions; pub mod clip_ops; pub mod color; @@ -233,6 +234,9 @@ pub struct History { /// Key of the last [`Session::edit_merged`] step: a continuous gesture (a fader drag) with the /// same key folds into that one undo step. pub merge_key: Option, + /// The redo steps the first edit of that merged step cleared, given back when the step is + /// reverted ([`Session::revert_merged`]): cancelling a preview after an Undo keeps the Redo. + pub merge_redo: Vec<(String, Arc)>, } impl History { @@ -900,6 +904,7 @@ impl Session { } self.history.redo.clear(); self.history.merge_key = None; + self.history.merge_redo.clear(); self.state = st; self.drop_stale_gap(); self.bump(); @@ -911,8 +916,10 @@ impl Session { pub fn edit_merged(&mut self, label: &str, key: &str, f: impl FnOnce(&mut Project, &mut EditorState) -> Result) -> Result { let merge = self.history.merge_key.as_deref() == Some(key) && self.history.undo.last().is_some_and(|u| u.0 == label); if !merge { + let redo = self.history.redo.clone(); let r = self.edit(label, f)?; self.history.merge_key = Some(key.to_string()); + self.history.merge_redo = redo; return Ok(r); } let mut p = (*self.project).clone(); @@ -997,6 +1004,37 @@ impl Session { n } + /// Take back the merged step that `key` is building (a dialog's live preview, cancelled): the + /// project returns to how it was before its first [`Session::edit_merged`], and neither undo + /// nor redo keeps it. Nothing happens, and false is returned, when the last edit was not that + /// step (another edit, an undo or a commit ended it). + pub fn revert_merged(&mut self, key: &str) -> bool { + if self.history.merge_key.as_deref() != Some(key) { + return false; + } + let Some((_, prev)) = self.history.undo.pop() else { + self.history.merge_key = None; + return false; + }; + self.project = prev; + self.history.merge_key = None; + self.history.redo = std::mem::take(&mut self.history.merge_redo); + self.fix_state(); + self.bump(); + true + } + + /// End the merged step that `key` is building (a dialog's live preview, confirmed): it stays + /// one undo step, and the next edit with the same key starts another. Returns whether such a + /// step was open. + pub fn end_merged(&mut self, key: &str) -> bool { + let open = self.history.merge_key.as_deref() == Some(key); + if open { + self.history.merge_key = None; + } + open + } + /// Edit the active sequence with the edit-algebra context. pub fn edit_sequence(&mut self, label: &str, f: impl FnOnce(&mut Sequence, &mut EditCtx, &mut EditorState) -> Result) -> Result { self.edit_sequence_as(label, None, f) @@ -1276,6 +1314,8 @@ mod autosave_tests; #[cfg(test)] mod avi_import_tests; #[cfg(test)] +mod camera_raw_tests; +#[cfg(test)] mod clip_ops_tests; #[cfg(test)] mod color_tests; diff --git a/crates/engine/src/media_pool.rs b/crates/engine/src/media_pool.rs index d3bc86e875..4541e023c8 100644 --- a/crates/engine/src/media_pool.rs +++ b/crates/engine/src/media_pool.rs @@ -14,14 +14,14 @@ //! - Paths that came from a document (the project file, a template, an interchange file) are //! opened through [`crate::doc_path`] ([`MediaPool::follow`]); a refused one renders as missing. -use std::collections::{BTreeSet, HashMap}; +use std::collections::{BTreeSet, HashMap, HashSet}; use std::path::PathBuf; use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Mutex, PoisonError, RwLock}; +use std::sync::{Arc, Condvar, Mutex, PoisonError, RwLock}; use filmcraft_frame::{AudioBuffer, VideoFrame}; use filmcraft_media::generators::GeneratorSource; -use filmcraft_media::{FrameRequest, MediaError, MediaInfo, MediaSource, Opener, SharedSource}; +use filmcraft_media::{FrameRequest, MediaError, MediaInfo, MediaSource, Opener, RawDevelop, SharedSource}; use filmcraft_project::{ItemId, MediaClip, MediaRef, Project}; use filmcraft_render::offline::OfflineReason; @@ -54,6 +54,10 @@ pub struct MediaPool { use_proxies: AtomicBool, /// Per item: video frames that failed in a row in the monitors (see [`Self::monitor_source`]). failures: Arc>>, + /// Items a thread is opening right now, and the wake-up when one finishes (see + /// [`Self::claim_open`]). + opening: Mutex>, + opened: Condvar, /// Openers tried before the built-in ones (MP4/MOV + codecs register here). pub openers: RwLock>, /// Media and proxy paths that came from a document, with that document's path (see @@ -69,6 +73,8 @@ impl Default for MediaPool { proxies: RwLock::new(HashMap::new()), offline: Arc::new(RwLock::new(HashMap::new())), use_proxies: AtomicBool::new(false), + opening: Mutex::new(HashSet::new()), + opened: Condvar::new(), failures: Arc::default(), openers: RwLock::new(filmcraft_codecs::openers()), stored: RwLock::default(), @@ -80,17 +86,52 @@ impl Default for MediaPool { /// MP3 and other byte-opened formats, or an unsupported container). pub const PROBE_WHOLE_FILE_MAX: u64 = 64 * 1024 * 1024; +/// Whether a clip is camera RAW with development settings (today: R3D NE through the RED SDK). +pub(crate) fn is_camera_raw(m: &MediaClip) -> bool { + m.info.video.as_ref().is_some_and(|v| v.codec == "R3D NE") || matches!(&m.media, MediaRef::File { path } if filmcraft_codecs::r3d::is_r3d(path)) +} + +/// The development settings a camera RAW clip decodes with: its own or, under Decode Using ▸ +/// Project (or with none of its own), the project's Camera RAW settings `camera_raw` +/// ([`RawDevelop::effective`]). Decode Using itself is left at Clip: the two develop the same +/// picture. None for media that is not camera RAW. +pub fn develop_of(camera_raw: &RawDevelop, m: &MediaClip) -> Option { + is_camera_raw(m).then(|| RawDevelop { decode_using: filmcraft_media::DecodeUsing::Clip, ..RawDevelop::effective(camera_raw, m.interpret.develop.as_ref()) }) +} + /// Cache key of a media clip's full-resolution reference ("" matches anything: sources inserted /// without a key). A generator's key holds its parameters, so a Color Matte whose color changes -/// (or comes back with undo) is generated again. -pub fn media_key(m: &MediaClip) -> String { +/// (or comes back with undo) is generated again. A camera RAW file's key carries the development +/// settings it decodes with ([`develop_of`], under the project's Camera RAW settings +/// `camera_raw`) when they are not the default, so a change reaches the open decoder +/// ([`MediaPool::resolve`] hands it the new settings rather than matching the stale key). +pub fn media_key(m: &MediaClip, camera_raw: &RawDevelop) -> String { match &m.media { MediaRef::File { path } if m.offline => format!("offline:{path}"), - MediaRef::File { path } => format!("file:{path}"), + MediaRef::File { path } => match develop_of(camera_raw, m).filter(|d| *d != RawDevelop::default()) { + Some(d) => format!("file:{path}|develop:{}", d.key()), + None => format!("file:{path}"), + }, MediaRef::Generator(g) => format!("generator:{g:?}"), } } +/// The file part of a media key: the key without its development settings. +fn key_file(key: &str) -> &str { + key.split("|develop:").next().unwrap_or(key) +} + +/// Hand the development settings a clip decodes with ([`develop_of`]) to an open source. A source +/// without such settings (every non-RAW decoder) is left alone; a RAW decoder that refuses them +/// logs why and keeps decoding as it did. +fn apply_develop(src: &SharedSource, m: &MediaClip, camera_raw: &RawDevelop) { + let Some(develop) = develop_of(camera_raw, m) else { return }; + match src.set_develop(&develop) { + Ok(()) | Err(MediaError::Unsupported(_)) => {} + Err(e) => log::warn!("source settings not applied: {}: {e}", m.info.name), + } +} + /// Open the image sequence that `first` (a numbered still) starts: its frames from that number to /// the highest present one, played at `rate`. Frames are listed with [`Services::list_dir`], or /// found by probing consecutive numbers (stopping after 100 missing in a row) when the host cannot @@ -318,12 +359,29 @@ impl MediaPool { { return Some(s); } - let key = media_key(m); - if let Some((k, s)) = self.sources.read().unwrap_or_else(PoisonError::into_inner).get(&item) - && (k.is_empty() || *k == key) - { - return Some(s.clone()); + let key = media_key(m, &p.settings.camera_raw); + let cached = self.sources.read().unwrap_or_else(PoisonError::into_inner).get(&item).cloned(); + if let Some((k, s)) = cached { + if k.is_empty() || k == key { + return Some(s); + } + // The same file with other Source Settings: the open decoder takes them. A second + // decoder process (and its GPU memory) is never started for a setting change. + if !k.is_empty() && key_file(&k) == key_file(&key) && key_file(&key).starts_with("file:") { + apply_develop(&s, m, &p.settings.camera_raw); + let mut sources = self.sources.write().unwrap_or_else(PoisonError::into_inner); + if sources.get(&item).is_some_and(|(_, cur)| Arc::ptr_eq(cur, &s)) { + sources.insert(item, (key, s.clone())); + } + return Some(s); + } } + // One open per item at a time: the frame workers all miss together at start-up, and a + // source whose decoder costs a process or GPU memory must not be opened six times. + let _opening = match self.claim_open(item, &key) { + Claimed::Mine(guard) => guard, + Claimed::Opened(s) => return Some(s), + }; let src: SharedSource = match &m.media { MediaRef::Generator(g) => { let v = m.info.video.as_ref(); @@ -343,6 +401,9 @@ impl MediaPool { }) { Ok(s) => { self.offline.write().unwrap_or_else(PoisonError::into_inner).remove(&item); + if develop_of(&p.settings.camera_raw, m).is_some_and(|d| d != RawDevelop::default()) { + apply_develop(&s, m, &p.settings.camera_raw); + } self.unsupported_as_slate(item, path, s) } Err(_) if filmcraft_media::pending::is_set() => { @@ -361,6 +422,23 @@ impl MediaPool { Some(src) } + /// Claim the opening of `item` for this thread, or wait for the thread that is opening it + /// and take its source. A thread that waited but found no source (the opener gave up, or + /// the host is still fetching the bytes) claims the open itself. + fn claim_open(&self, item: ItemId, key: &str) -> Claimed<'_> { + let mut opening = self.opening.lock().unwrap_or_else(PoisonError::into_inner); + while opening.contains(&item) { + opening = self.opened.wait(opening).unwrap_or_else(PoisonError::into_inner); + if let Some((k, s)) = self.sources.read().unwrap_or_else(PoisonError::into_inner).get(&item) + && (k.is_empty() || *k == key) + { + return Claimed::Opened(s.clone()); + } + } + opening.insert(item); + Claimed::Mine(OpenGuard { pool: self, item }) + } + fn set_offline(&self, item: ItemId, reason: OfflineReason, path: &str, error: &str) { self.offline.write().unwrap_or_else(PoisonError::into_inner).insert(item, OfflineStatus { reason, path: path.to_string(), error: error.to_string() }); } @@ -518,6 +596,21 @@ impl MediaSource for UnsupportedAsSlate { fn info(&self) -> &MediaInfo { self.inner.info() } + fn frame_delivery(&self) -> filmcraft_media::FrameDelivery { + self.inner.frame_delivery() + } + fn set_develop(&self, develop: &RawDevelop) -> filmcraft_media::Result<()> { + self.inner.set_develop(develop) + } + fn raw_as_shot(&self) -> Option { + self.inner.raw_as_shot() + } + fn raw_clip_info(&self) -> Option { + self.inner.raw_clip_info() + } + fn write_raw_sidecar(&self, develop: &RawDevelop) -> filmcraft_media::Result { + self.inner.write_raw_sidecar(develop) + } fn video_frame(&self, req: FrameRequest) -> filmcraft_media::Result> { match self.inner.video_frame(req) { Err(MediaError::Unsupported(why)) => { @@ -538,6 +631,26 @@ impl MediaSource for UnsupportedAsSlate { } } +/// The outcome of [`MediaPool::claim_open`]. +enum Claimed<'a> { + /// This thread opens the item; the guard releases the claim however the open ends. + Mine(OpenGuard<'a>), + /// Another thread opened it meanwhile. + Opened(SharedSource), +} + +struct OpenGuard<'a> { + pool: &'a MediaPool, + item: ItemId, +} + +impl Drop for OpenGuard<'_> { + fn drop(&mut self) { + self.pool.opening.lock().unwrap_or_else(PoisonError::into_inner).remove(&self.item); + self.pool.opened.notify_all(); + } +} + /// Consecutive failed video frames after which a monitor treats a source as damaged. A single /// failure (one bad slice, a read hiccup) and the decoder's retry of it stay as they were; three /// in a row (an eighth of a second at 24 fps) mean the file itself is damaged. @@ -558,6 +671,21 @@ impl MediaSource for DamagedAsSlate { fn info(&self) -> &MediaInfo { self.inner.info() } + fn frame_delivery(&self) -> filmcraft_media::FrameDelivery { + self.inner.frame_delivery() + } + fn set_develop(&self, develop: &RawDevelop) -> filmcraft_media::Result<()> { + self.inner.set_develop(develop) + } + fn raw_as_shot(&self) -> Option { + self.inner.raw_as_shot() + } + fn raw_clip_info(&self) -> Option { + self.inner.raw_clip_info() + } + fn write_raw_sidecar(&self, develop: &RawDevelop) -> filmcraft_media::Result { + self.inner.write_raw_sidecar(develop) + } fn video_frame(&self, req: FrameRequest) -> filmcraft_media::Result> { let r = self.inner.video_frame(req); let mut failures = self.failures.lock().unwrap_or_else(PoisonError::into_inner); @@ -600,6 +728,10 @@ impl MediaSource for ProxySource { fn info(&self) -> &MediaInfo { &self.info } + /// The proxy decodes the pictures, so its delivery is what playback sees. + fn frame_delivery(&self) -> filmcraft_media::FrameDelivery { + self.proxy.frame_delivery() + } fn video_frame(&self, req: FrameRequest) -> filmcraft_media::Result> { let full = self.info.video.as_ref().ok_or(MediaError::NoStream("video"))?; let pv = self.proxy.info().video.as_ref().map(|v| (v.width, v.height)).unwrap_or((full.width, full.height)); @@ -690,6 +822,106 @@ mod tests { } } + /// A stand-in RAW decoder: records the development settings it is given. + struct Developable { + info: MediaInfo, + develops: Mutex>, + } + impl MediaSource for Developable { + fn info(&self) -> &MediaInfo { + &self.info + } + fn video_frame(&self, _: FrameRequest) -> filmcraft_media::Result> { + Err(MediaError::NoStream("video")) + } + fn audio(&self, _: i64, _: usize, _: u32) -> filmcraft_media::Result { + Err(MediaError::NoStream("audio")) + } + fn set_develop(&self, develop: &RawDevelop) -> filmcraft_media::Result<()> { + self.develops.lock().unwrap().push(*develop); + Ok(()) + } + } + + /// Clip ▸ Source Settings… reach the open RAW decoder through the pool: a change hands the new + /// settings to the cached source instead of opening a second decoder, and a source without + /// such settings is left alone. + #[test] + fn source_settings_reach_the_open_source_without_reopening() { + use filmcraft_media::{RawGamma, RawToneMap}; + let mut project = Project::new("raw"); + // a project from before Camera RAW settings: its clips develop scene-linear Rec. 709 + project.settings.camera_raw = RawDevelop::default(); + let info = MediaInfo { + name: "DSC_0001.R3D".into(), + kind: filmcraft_media::MediaKind::Movie, + duration: filmcraft_time::Tick(filmcraft_time::TICKS_PER_SECOND), + video: Some(filmcraft_media::VideoStreamInfo { + width: 16, + height: 9, + frame_rate: filmcraft_time::FrameRate::FPS_24, + par: (1, 1), + codec: "R3D NE".into(), + pixel_format: String::new(), + color: filmcraft_color::ColorInfo::SRGB_FULL, + has_alpha: false, + bitrate: None, + hdr: None, + }), + audio_streams: vec![], + container: "R3D NE".into(), + start_timecode: None, + file_size: None, + }; + let clip = MediaClip { + media: MediaRef::File { path: "/footage/DSC_0001.R3D".into() }, + info: info.clone(), + interpret: Default::default(), + mark_in: None, + mark_out: None, + markers: vec![], + offline: false, + proxy: None, + identity: None, + }; + let id = project.add_item("DSC_0001.R3D", filmcraft_project::Label::Iris, ItemKind::Media(clip), None); + let pool = MediaPool::default(); + let raw = Arc::new(Developable { info, develops: Mutex::default() }); + pool.insert_file(id, "/footage/DSC_0001.R3D", raw.clone()); + let develops = |raw: &Developable| raw.develops.lock().unwrap().clone(); + // default settings: the cached source is used as it is + assert!(pool.source_for(&project, id, &crate::FsServices).is_some()); + assert_eq!(develops(&raw), Vec::::new()); + let set = |project: &mut Project, d: Option| { + if let Some(ItemKind::Media(m)) = project.item_mut(id).map(|i| &mut i.kind) { + m.interpret.develop = d; + } + }; + let rec709 = RawDevelop { gamma: RawGamma::Bt1886, tone_map: RawToneMap::Medium, ..Default::default() }; + set(&mut project, Some(rec709)); + assert!(pool.source_for(&project, id, &crate::FsServices).is_some()); + assert_eq!(develops(&raw), vec![rec709], "the open source took the new settings"); + assert_eq!(pool.open_sources(), 1, "no second decoder"); + // the same settings again: nothing to do + assert!(pool.source_for(&project, id, &crate::FsServices).is_some()); + assert_eq!(develops(&raw).len(), 1); + // back to the default: the source is told so + set(&mut project, None); + assert!(pool.source_for(&project, id, &crate::FsServices).is_some()); + assert_eq!(develops(&raw), vec![rec709, RawDevelop::default()]); + assert_eq!(media_key(project.item(id).unwrap().as_media().unwrap(), &project.settings.camera_raw), "file:/footage/DSC_0001.R3D"); + set(&mut project, Some(rec709)); + assert_eq!( + media_key(project.item(id).unwrap().as_media().unwrap(), &project.settings.camera_raw), + "file:/footage/DSC_0001.R3D|develop:rec709/bt1886/medium/clip" + ); + assert_eq!(key_file("file:/a.R3D|develop:rec709/bt1886/medium/clip"), "file:/a.R3D"); + // a source without development settings answers Unsupported and is left alone + let plain = Arc::new(SlateSource::new(raw.info.clone(), "plain", OfflineReason::Missing)); + assert!(matches!(plain.set_develop(&rec709), Err(MediaError::Unsupported(_)))); + apply_develop(&(plain as SharedSource), project.item(id).unwrap().as_media().unwrap(), &project.settings.camera_raw); + } + #[test] fn open_file_prefers_the_hosts_reader() { let wav = filmcraft_media::wav::write_wav16(&[0.0, 0.5, -0.5, 0.25], 2, 48_000); @@ -759,6 +991,61 @@ mod tests { assert!(pool.offline_status(item).is_none()); } + /// The frame workers all miss the cache together at start-up: the item is opened once and + /// the others wait for that source, instead of each starting a decoder of its own (some + /// decoders cost a process or a gigabyte of GPU memory per open). + #[test] + fn concurrent_cold_misses_open_an_item_once() { + use std::sync::atomic::{AtomicUsize, Ordering}; + struct Slow(Arc<[u8]>, AtomicUsize); + impl Services for Slow { + fn read_file(&self, path: &str) -> std::io::Result> { + Err(std::io::Error::other(format!("{path}: read whole"))) + } + fn write_file(&self, _path: &str, _data: &[u8]) -> std::io::Result<()> { + Ok(()) + } + fn reader(&self, _path: &str) -> Option> { + self.1.fetch_add(1, Ordering::Relaxed); + std::thread::sleep(std::time::Duration::from_millis(150)); + Some(Ok(Arc::new(filmcraft_media::reader::MemReader(self.0.clone())))) + } + } + let wav: Arc<[u8]> = filmcraft_media::wav::write_wav16(&[0.0, 0.5, -0.5, 0.25], 2, 48_000).into(); + let services = Slow(wav.clone(), AtomicUsize::new(0)); + let info = MediaPool::default().open_file("a.wav", &ReaderOnly(wav)).unwrap().info().clone(); + let mut project = Project::new("single-flight"); + let item = project.add_item( + "a.wav", + filmcraft_project::Label::Caribbean, + ItemKind::Media(MediaClip { + media: MediaRef::File { path: "a.wav".into() }, + info, + interpret: Default::default(), + mark_in: None, + mark_out: None, + markers: vec![], + offline: false, + proxy: None, + identity: None, + }), + None, + ); + let pool = MediaPool::default(); + let sources: Vec<_> = std::thread::scope(|scope| { + let handles: Vec<_> = (0..6).map(|_| scope.spawn(|| pool.source_for(&project, item, &services))).collect(); + handles.into_iter().map(|h| h.join().unwrap()).collect() + }); + assert_eq!(services.1.load(Ordering::Relaxed), 1, "opened once"); + let first = sources[0].as_ref().unwrap(); + assert!(sources.iter().all(|s| s.as_ref().is_some_and(|s| Arc::ptr_eq(s, first)))); + assert_eq!(pool.open_sources(), 1); + // the claim is released: a later miss (the file relinked) opens again + pool.clear(); + assert!(pool.source_for(&project, item, &services).is_some()); + assert_eq!(services.1.load(Ordering::Relaxed), 2); + } + /// A source whose video decodes (`OK`) or fails as its mode says, switchable at run time. struct Flaky(MediaInfo, std::sync::atomic::AtomicU8); const OK: u8 = 0; diff --git a/crates/engine/src/panels.rs b/crates/engine/src/panels.rs index 6246e7a315..d751c0f40e 100644 --- a/crates/engine/src/panels.rs +++ b/crates/engine/src/panels.rs @@ -222,7 +222,7 @@ pub fn metadata_fields(p: &Project, id: ItemId) -> Vec { add("Clip", "Video Info", format!("{} x {} ({:.4})", v.width, v.height, v.par.0 as f64 / v.par.1.max(1) as f64), false); add("Clip", "Video Codec", v.codec.clone(), false); let detected = filmcraft_color::ColorSpace::from_info(&v.color); - let cs = media.and_then(|m| m.interpret.color_space).unwrap_or(detected); + let cs = media.and_then(|m| p.color_space_override(m)).unwrap_or(detected); add("Clip", "Color Space", cs.label().to_string(), false); } if let Some(a) = media.and_then(|m| m.info.audio()) { diff --git a/crates/engine/src/project_panel.rs b/crates/engine/src/project_panel.rs index 735f5b281a..3c00ace996 100644 --- a/crates/engine/src/project_panel.rs +++ b/crates/engine/src/project_panel.rs @@ -402,7 +402,7 @@ pub fn cell(p: &Project, it: &ProjectItem, column: &str) -> (String, SortKey) { ), "Audio Codec" => text(media.and_then(|m| m.info.audio()).map(|a| a.codec.clone()).unwrap_or_default()), "Color Space" => text(match (media.and_then(|m| m.info.video.as_ref().map(|v| (m, v))), &it.kind) { - (Some((m, v)), _) => m.interpret.color_space.unwrap_or_else(|| filmcraft_color::ColorSpace::from_info(&v.color)).label().to_string(), + (Some((m, v)), _) => p.color_space_override(m).unwrap_or_else(|| filmcraft_color::ColorSpace::from_info(&v.color)).label().to_string(), (None, ItemKind::Sequence(q)) => q.settings.color.working.label().to_string(), _ => String::new(), }), diff --git a/crates/engine/src/project_tools.rs b/crates/engine/src/project_tools.rs index a7959bac06..e19c0fb436 100644 --- a/crates/engine/src/project_tools.rs +++ b/crates/engine/src/project_tools.rs @@ -11,12 +11,12 @@ //! | `file.exportSelectionProject` | File ▸ Export ▸ Selection as FilmCraft Project… | //! | `file.exportAle` | File ▸ Export ▸ Avid Log Exchange… | //! | `file.mediaPropertiesFile` / `file.mediaProperties` | File ▸ Get Media File Properties for ▸ File… / Selection… (⇧⌘H) | -//! | `file.projectSettings.general` / `.scratchDisks` | File ▸ Project Settings ▸ General… / Scratch Disks… | +//! | `file.projectSettings.general` / `.scratchDisks` / `.cameraRaw` | File ▸ Project Settings ▸ General… / Scratch Disks… / Camera RAW… (`camera_raw`) | //! | `edit.find` / `edit.findNext` | Edit ▸ Find… (⌘F) / Find Next | //! | `edit.editOriginal` | Edit ▸ Edit Original (⌘E): the frontend opens the files (regular files with a media extension only) | //! | `file.reveal` | Project panel ▸ Reveal in Finder / Show in Explorer: the frontend reveals the media file | //! | `clip.editOffline` | Clip ▸ Edit Offline… | -//! | `clip.sourceSettings` | Clip ▸ Source Settings… | +//! | `clip.sourceSettings` | Clip ▸ Source Settings… (the Camera Raw panel, `camera_raw`) | //! | `clip.restoreCaptionsFromSource` | Clip ▸ Restore Captions from Source Clip (always disabled: no embedded caption reader) | //! | `clip.updateMetadata` | Clip ▸ Update Metadata… (XMP sidecars) | //! | `clip.generateAudioWaveform` | Clip ▸ Generate Audio Waveform (the frontend rebuilds its peaks) | @@ -160,6 +160,15 @@ pub(crate) fn commands() -> Vec { always, project_settings_scratch, ), + spec( + "file.projectSettings.cameraRaw", + "Camera RAW…", + &["File", "Project Settings"], + None, + r#"{"quality":"fullPremium|halfPremium|halfGood|quarterGood|eighthGood|sixteenthGood"?,"colorSpace":"rec709|rec2020|p3d65|redWideGamutRgb"?,"gamma":"linear|bt1886|srgb|log3g10|st2084|hlg"?,"toneMap":"clip|none|low|medium|high"?,"rolloff":"clip|none|hard|medium|soft|verySoft"?,"hdrPeakNits":n|null?,"chromaNoiseReduction":bool|null?,"flashingPixel":"off|mild|medium|strong"|null?,"metadataCurves":bool?,"creativeLut":bool?,"cdl":bool?,"iso":n|null?,"kelvin":n|null?,"tint":f64|null?,"exposure":f64|null?,"asShot":bool?,"standard":bool?}"#, + always, + crate::camera_raw::project_settings, + ), // ---------------- Edit ---------------- spec("edit.find", "Find…", &["Edit"], Some("Cmd+F"), FIND_PARAMS, always, find), spec("edit.findNext", "Find Next", &["Edit"], None, "{}", has_find, find_next), @@ -176,7 +185,15 @@ pub(crate) fn commands() -> Vec { has_offline_selection, edit_offline, ), - query("clip.sourceSettings", "Source Settings…", &["Clip"], None, r#"{"item":id?}"#, has_media_selection, source_settings), + spec( + "clip.sourceSettings", + "Source Settings…", + &["Clip"], + None, + r#"{"item":id?,"items":[id]?,"decodeUsing":"clip|project"?,"quality":"project|fullPremium|halfPremium|halfGood|quarterGood|eighthGood|sixteenthGood"?,"colorSpace":"rec709|rec2020|p3d65|redWideGamutRgb"?,"gamma":"linear|bt1886|srgb|log3g10|st2084|hlg"?,"output":"linearRec709|rec709|log3g10"?,"toneMap":"clip|none|low|medium|high"?,"rolloff":"clip|none|hard|medium|soft|verySoft"?,"hdrPeakNits":n|null?,"chromaNoiseReduction":bool|null?,"flashingPixel":"off|mild|medium|strong"|null?,"metadataCurves":bool?,"creativeLut":bool?,"cdl":bool?,"iso":n|null?,"kelvin":n|null?,"tint":f64|null?,"exposure":f64|null?,"asShot":bool?,"useChanges":bool?,"useSettings":bool?,"updateRmd":bool?,"preview":bool?,"commit":bool?,"revert":bool?}"#, + has_media_selection, + crate::camera_raw::source_settings, + ), spec("clip.restoreCaptionsFromSource", "Restore Captions from Source Clip", &["Clip"], None, "{}", no_embedded_captions, |_, _| { Err(EngineError::Other(NO_CAPTIONS.into())) }), @@ -239,6 +256,7 @@ pub(crate) fn apply_layout(v: &mut Vec) { "file.projectSettings.general", "file.projectSettings.scratchDisks", "project.ingestSettings", + "file.projectSettings.cameraRaw", ], ), (At::After("edit.deselectAll"), vec!["edit.find", "edit.findNext"]), @@ -348,7 +366,7 @@ fn has_find(s: &Session) -> std::result::Result<(), String> { /// Project items the command works on: `items`, else the Project panel selection, else the media /// of the selected timeline clips. -fn targets(s: &Session, p: &Value) -> Vec { +pub(crate) fn targets(s: &Session, p: &Value) -> Vec { if let Some(a) = p.get("items").and_then(Value::as_array) { return a.iter().filter_map(|v| v.as_u64().map(ItemId)).filter(|i| s.project.items.contains_key(i)).collect(); } @@ -372,7 +390,7 @@ fn targets(s: &Session, p: &Value) -> Vec { } /// The media clip behind an item (subclips resolve to their parent). -fn media_of(p: &Project, id: ItemId) -> Option<(ItemId, &filmcraft_project::MediaClip)> { +pub(crate) fn media_of(p: &Project, id: ItemId) -> Option<(ItemId, &filmcraft_project::MediaClip)> { p.resolve_media(id).map(|(root, media, _)| (root, media)) } @@ -1161,19 +1179,6 @@ fn edit_offline(s: &mut Session, p: &Value) -> Result { Ok(json!({"item": id.0, "name": it.name, "metadata": it.metadata})) } -fn source_settings(s: &mut Session, p: &Value) -> Result { - let id = targets(s, p).into_iter().find(|i| media_of(&s.project, *i).is_some()).ok_or_else(|| EngineError::Other("select a clip".into()))?; - let (_, m) = media_of(&s.project, id).ok_or_else(|| EngineError::Other("select a clip".into()))?; - let codec = m.info.video.as_ref().map(|v| v.codec.clone()).or_else(|| m.info.audio().map(|a| a.codec.clone())).unwrap_or_default(); - Ok(json!({ - "item": id.0, - "codec": codec, - "container": m.info.container, - "settings": [], - "message": format!("{} has no source settings in FilmCraft. Source settings apply to camera raw formats (R3D, ARRIRAW, CinemaDNG, ProRes RAW…), which FilmCraft does not decode.", if codec.is_empty() { "This clip".to_string() } else { codec.to_uppercase() }), - })) -} - fn xml_escape(s: &str) -> String { s.replace('&', "&").replace('<', "<").replace('>', ">").replace('"', """) } diff --git a/crates/engine/src/project_tools_tests.rs b/crates/engine/src/project_tools_tests.rs index 64965166c5..0715b83196 100644 --- a/crates/engine/src/project_tools_tests.rs +++ b/crates/engine/src/project_tools_tests.rs @@ -56,7 +56,10 @@ fn file_edit_clip_menu_order_and_shortcuts_match_premiere() { ); assert_order(&menu_ids(&["File", "Export"]), &["file.exportEdl", "file.exportSelectionProject", "file.exportAle", "file.exportOtio", "file.exportFcp7Xml"]); assert_eq!(menu_ids(&["File", "Get Media File Properties for"]), vec!["file.mediaPropertiesFile", "file.mediaProperties"]); - assert_eq!(menu_ids(&["File", "Project Settings"]), vec!["file.projectSettings.general", "file.projectSettings.scratchDisks", "project.ingestSettings"]); + assert_eq!( + menu_ids(&["File", "Project Settings"]), + vec!["file.projectSettings.general", "file.projectSettings.scratchDisks", "project.ingestSettings", "file.projectSettings.cameraRaw"] + ); assert_order(&menu_ids(&["Edit"]), &["edit.deselectAll", "edit.find", "edit.findNext", "edit.consolidateDuplicates", "edit.editOriginal"]); let clip: Vec<&str> = crate::commands::command_specs().iter().filter(|c| c.menu.first() == Some(&"Clip")).map(|c| c.id).collect(); assert_order( @@ -384,6 +387,8 @@ fn edit_original_update_metadata_and_clip_items() { assert!(d.is_enabled("clip.sourceSettings")); let ss = d.execute("clip.sourceSettings", json!({})).unwrap(); assert!(ss["message"].as_str().unwrap().contains("no source settings")); + assert!(ss["settings"].as_array().unwrap().is_empty()); + assert!(d.execute("clip.sourceSettings", json!({"output": "rec709"})).is_err(), "an MP4 has nothing to develop"); assert!(!d.is_enabled("clip.restoreCaptionsFromSource")); let w = d.execute("clip.generateAudioWaveform", json!({})).unwrap(); assert_eq!(w["items"], json!([ocean.0])); diff --git a/crates/format/src/lib.rs b/crates/format/src/lib.rs index b77968dd17..24c7f1709a 100644 --- a/crates/format/src/lib.rs +++ b/crates/format/src/lib.rs @@ -39,6 +39,9 @@ //! would drop them when saving. //! - **v15**: project notes (`Project::notes`, Window ▸ Project Notes). No-op step; older builds //! would drop them when saving. +//! - **v16**: Camera RAW (`ProjectSettings::camera_raw`; Source Settings colour space, gamma, decode +//! quality, Decode Using and processing options). No-op step; older builds would drop the +//! settings when saving and develop the clips differently. #![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable))] @@ -59,8 +62,10 @@ pub type Migration = fn(Value) -> Result; /// `MIGRATIONS[i]` upgrades schema `i + 1` to `i + 2`. Append one function per schema bump; never /// edit a shipped one. -pub const MIGRATIONS: &[Migration] = - &[v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7, v7_to_v8, v8_to_v9, v9_to_v10, v10_to_v11, v11_to_v12, v12_to_v13, v13_to_v14, v14_to_v15]; +pub const MIGRATIONS: &[Migration] = &[ + v1_to_v2, v2_to_v3, v3_to_v4, v4_to_v5, v5_to_v6, v6_to_v7, v7_to_v8, v8_to_v9, v9_to_v10, v10_to_v11, v11_to_v12, v12_to_v13, v13_to_v14, v14_to_v15, + v15_to_v16, +]; /// The schema version this build writes (and the newest it reads). pub const SCHEMA_VERSION: u32 = MIGRATIONS.len() as u32 + 1; @@ -233,11 +238,22 @@ fn validate_loaded(mut loaded: Loaded) -> Result { // Out-of-range audio stream data is repaired, not refused: failing the load would lose the // whole project over one bad index. A media item keeps its first `MAX_AUDIO_STREAMS` streams; a // clip pointing past that plays stream 0. + // Camera RAW settings a RAW source would refuse at its first frame are repaired here, not + // refused: the project's fall back to what a project from before them reads as, a clip's own + // give way to the project's. + if let Err(e) = loaded.project.settings.camera_raw.validate() { + log::warn!("Project Settings > Camera RAW: {e}; using the defaults"); + loaded.project.settings.camera_raw = filmcraft_media::RawDevelop::default(); + } let max = filmcraft_media::MAX_AUDIO_STREAMS; for item in loaded.project.items.values_mut() { let name = item.name.clone(); match &mut item.kind { filmcraft_project::ItemKind::Media(m) => { + if let Some(Err(e)) = m.interpret.develop.as_ref().map(filmcraft_media::RawDevelop::validate) { + log::warn!("media `{name}`: Camera RAW settings: {e}; it follows the project's"); + m.interpret.develop = None; + } if let Some(video) = &m.info.video { filmcraft_project::validate_frame_size(video.width, video.height).map_err(|e| FormatError::Corrupt(format!("media `{name}`: {e}")))?; } @@ -392,13 +408,20 @@ fn v14_to_v15(doc: Value) -> Result { Ok(doc) } +/// v15 → v16: Camera RAW (project settings and per-clip colour space, gamma, decode quality and +/// options). Existing data needs no change: a project without Camera RAW settings reads as the +/// scene-linear Rec. 709 its clips developed with, and a clip's old `output` still reads. +fn v15_to_v16(doc: Value) -> Result { + Ok(doc) +} + #[cfg(test)] mod tests { use super::*; #[test] fn schema_version_matches_table() { - assert_eq!(SCHEMA_VERSION, 15); + assert_eq!(SCHEMA_VERSION, 16); } #[test] @@ -479,6 +502,46 @@ mod tests { } } + /// Camera RAW settings a RAW source would refuse (a gamma that does not go with the colour + /// space, a hand edit) are repaired when the project opens, not when a frame is asked for. + #[test] + fn invalid_camera_raw_settings_are_repaired_on_load() { + use filmcraft_media::{RawColorSpace, RawDevelop, RawGamma}; + let bad = RawDevelop { color_space: RawColorSpace::Rec709, gamma: RawGamma::Log3G10, ..RawDevelop::default() }; + assert!(bad.validate().is_err()); + let mut p = Project::new("Camera RAW"); + p.settings.camera_raw = bad; + let clip = filmcraft_project::MediaClip { + media: filmcraft_project::MediaRef::File { path: "a.R3D".into() }, + info: filmcraft_media::MediaInfo { + name: "a.R3D".into(), + kind: filmcraft_media::MediaKind::Movie, + duration: filmcraft_time::Tick(0), + video: None, + audio_streams: vec![], + container: "r3d".into(), + start_timecode: None, + file_size: None, + }, + interpret: filmcraft_project::Interpretation { develop: Some(bad), ..Default::default() }, + mark_in: None, + mark_out: None, + markers: vec![], + offline: false, + proxy: None, + identity: None, + }; + let id = p.add_item("a.R3D", filmcraft_project::Label::Caribbean, filmcraft_project::ItemKind::Media(clip), None); + let loaded = decode(&encode(&p, false)).unwrap(); + assert_eq!(loaded.project.settings.camera_raw, RawDevelop::default()); + let filmcraft_project::ItemKind::Media(m) = &loaded.project.items[&id].kind else { panic!("not media") }; + assert_eq!(m.interpret.develop, None, "the clip follows the project's settings"); + // valid settings are kept as they are + let mut p = Project::new("Camera RAW"); + p.settings.camera_raw = RawDevelop::STANDARD; + assert_eq!(decode(&encode(&p, false)).unwrap().project.settings.camera_raw, RawDevelop::STANDARD); + } + #[test] fn oversized_embedded_luts_are_dropped_on_load() { let mut p = Project::new("LUT limit"); diff --git a/crates/frame/src/lib.rs b/crates/frame/src/lib.rs index 9c8e643a2e..4bf94d270b 100644 --- a/crates/frame/src/lib.rs +++ b/crates/frame/src/lib.rs @@ -1,8 +1,10 @@ //! Video frames and audio buffers. //! //! - [`VideoFrame`]: decoded or rendered pictures. Planar Y'CbCr (8 or 16-bit containers) straight -//! from decoders, sRGB-encoded RGBA8 from stills/generators, or linear premultiplied RGBA f32 in the -//! compositor. Pixel data is `Arc`-shared so caches, monitors and export share frames for free. +//! from decoders, sRGB-encoded RGBA8 from stills/generators, linear premultiplied RGBA f32 in the +//! compositor, or the same as IEEE halves from a decoder that produces them (camera RAW), which +//! the GPU uploads as they are. Pixel data is `Arc`-shared so caches, monitors and export share +//! frames for free. //! - [`AudioBuffer`]: planar f32 samples. #![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable))] @@ -48,6 +50,10 @@ pub enum PixelData { Rgba8(Arc>), /// Premultiplied, linear-light RGBA f32 (compositor working format). RgbaF32(Arc>), + /// Premultiplied, linear-light RGBA as little-endian IEEE halves, 8 bytes per pixel: what a + /// decoder that produces half floats hands over, and the GPU's own texel layout + /// (`Rgba16Float`), so a 6K frame is neither widened to f32 nor converted back for upload. + RgbaF16(Arc>), /// 8-bit planar Y'CbCr (optionally with an alpha plane). Yuv8 { planes: [Arc>; 3], chroma: Chroma, alpha: Option>> }, /// 9–16-bit planar Y'CbCr stored in u16 (`bits` significant bits). @@ -107,6 +113,11 @@ impl VideoFrame { debug_assert_eq!(data.len(), (width * height * 4) as usize); Self { width, height, data: PixelData::RgbaF32(Arc::new(data)), color: ColorInfo::SRGB_FULL, par: (1, 1), pts: Tick::ZERO } } + /// A frame of little-endian IEEE halves (see [`PixelData::RgbaF16`]). + pub fn rgba_f16(width: u32, height: u32, data: Vec) -> Self { + debug_assert_eq!(data.len(), (width * height * 8) as usize); + Self { width, height, data: PixelData::RgbaF16(Arc::new(data)), color: ColorInfo::SRGB_FULL, par: (1, 1), pts: Tick::ZERO } + } pub fn transparent_f32(width: u32, height: u32) -> Self { Self::rgba_f32(width, height, vec![0.0; (width * height * 4) as usize]) } @@ -119,6 +130,7 @@ impl VideoFrame { match &self.data { PixelData::Rgba8(_) => "RGBA 8-bit".into(), PixelData::RgbaF32(_) => "RGBA 32-bit float".into(), + PixelData::RgbaF16(_) => "RGBA 16-bit float".into(), PixelData::Yuv8 { chroma, .. } => format!("YUV {} 8-bit", chroma.label()), PixelData::Yuv16 { chroma, bits, .. } => format!("YUV {} {bits}-bit", chroma.label()), } @@ -129,6 +141,7 @@ impl VideoFrame { match &self.data { PixelData::Rgba8(d) => d.len(), PixelData::RgbaF32(d) => d.len() * 4, + PixelData::RgbaF16(d) => d.len(), PixelData::Yuv8 { planes, alpha, .. } => planes.iter().map(|p| p.len()).sum::() + alpha.as_ref().map_or(0, |a| a.len()), PixelData::Yuv16 { planes, alpha, .. } => (planes.iter().map(|p| p.len()).sum::() + alpha.as_ref().map_or(0, |a| a.len())) * 2, } @@ -175,7 +188,7 @@ impl VideoFrame { alpha: alpha.as_ref().map(|a| Arc::new(box_plane(a, w, h, ow, oh, n))), } } - PixelData::Rgba8(_) | PixelData::RgbaF32(_) => return None, + PixelData::Rgba8(_) | PixelData::RgbaF32(_) | PixelData::RgbaF16(_) => return None, }; Some(VideoFrame { width: ow as u32, height: oh as u32, data, color: self.color, par: self.par, pts: self.pts }) } @@ -188,7 +201,7 @@ impl VideoFrame { let (w, h) = (self.width as usize, self.height as usize); let (sx, sy) = match &self.data { PixelData::Yuv8 { chroma, .. } | PixelData::Yuv16 { chroma, .. } => chroma.shifts(), - PixelData::Rgba8(_) | PixelData::RgbaF32(_) => (0, 0), + PixelData::Rgba8(_) | PixelData::RgbaF32(_) | PixelData::RgbaF16(_) => (0, 0), }; let (mx, my) = ((1usize << sx) - 1, (1usize << sy) - 1); let (x, y) = (region.x.min(w) & !mx, region.y.min(h) & !my); @@ -202,6 +215,7 @@ impl VideoFrame { let data = match &self.data { PixelData::Rgba8(d) => PixelData::Rgba8(Arc::new(crop_plane(d, w, h, 4, r))), PixelData::RgbaF32(d) => PixelData::RgbaF32(Arc::new(crop_plane(d, w, h, 4, r))), + PixelData::RgbaF16(d) => PixelData::RgbaF16(Arc::new(crop_plane(d, w, h, 8, r))), PixelData::Yuv8 { planes, chroma, alpha } => { let (cw, ch) = (w.div_ceil(1 << sx), h.div_ceil(1 << sy)); let (cx, cy, cr_w, cr_h) = chroma_rect(cw, ch); @@ -248,6 +262,7 @@ impl VideoFrame { let data = match &self.data { PixelData::Rgba8(d) => PixelData::Rgba8(Arc::new(rotate_plane(d, w, h, 4, q))), PixelData::RgbaF32(d) => PixelData::RgbaF32(Arc::new(rotate_plane(d, w, h, 4, q))), + PixelData::RgbaF16(d) => PixelData::RgbaF16(Arc::new(rotate_plane(d, w, h, 8, q))), PixelData::Yuv8 { planes, chroma, alpha } => { let (c, chroma) = rotate_chroma(planes, *chroma, w, h, q); PixelData::Yuv8 { @@ -338,6 +353,22 @@ impl VideoFrame { } return; } + if let PixelData::RgbaF16(d) = &self.data + && n == 1 + { + out.par_chunks_mut(r.w * 4).enumerate().for_each(|(ry, row)| { + let start = ((r.y + ry) * w + r.x) * 8; + match d.get(start..start + r.w * 8) { + Some(src) => { + for (o, s) in row.iter_mut().zip(src.as_chunks::<2>().0) { + *o = half_to_f32(u16::from_le_bytes(*s)); + } + } + None => row.fill(0.0), + } + }); + return; + } // Encoded (0..1, quantised to 12 bits) → linear lookup for this frame's transfer. let info = self.color; let lin: Vec = if decode.is_some() { Vec::new() } else { (0..4096).map(|i| to_linear(i as f32 / 4095.0, info.transfer)).collect() }; @@ -347,6 +378,30 @@ impl VideoFrame { }; let inv = 1.0 / (n as f32 * n as f32); match &self.data { + PixelData::RgbaF16(d) => { + // the f32 accumulation below, sample by sample, so the two formats agree bit for bit + let sample = |i: usize| d.get(i..i + 2).map_or(0.0, |b| half_to_f32(u16::from_le_bytes([b[0], b[1]]))); + out.par_chunks_mut(r.w * 4).enumerate().for_each(|(ry, row)| { + let oy = r.y + ry; + for rx in 0..r.w { + let ox = r.x + rx; + let mut acc = [0f32; 4]; + for dy in 0..n { + let y = (oy * n + dy).min(h - 1); + for dx in 0..n { + let x = (ox * n + dx).min(w - 1); + let i = (y * w + x) * 8; + for k in 0..4 { + acc[k] += sample(i + k * 2); + } + } + } + for k in 0..4 { + row[rx * 4 + k] = acc[k] * inv; + } + } + }); + } PixelData::RgbaF32(d) => { out.par_chunks_mut(r.w * 4).enumerate().for_each(|(ry, row)| { let oy = r.y + ry; @@ -665,6 +720,69 @@ fn box_plane + TryFrom + Send + Sync + Default>(src: &[ out } +/// IEEE half → f32: the exact value of every bit pattern (subnormals, infinities and NaNs +/// included), with a branch-light path for normal numbers and zeros. +#[inline(always)] +pub fn half_to_f32(h: u16) -> f32 { + let u = u32::from(h); + let sign = (u & 0x8000) << 16; + let abs = u & 0x7fff; + if (0x0400..0x7c00).contains(&abs) { + // rebias the exponent (127 - 15 = 112) and widen the mantissa + return f32::from_bits(sign | ((abs + (112 << 10)) << 13)); + } + if abs == 0 { + return f32::from_bits(sign); + } + let exp = abs >> 10; + let mant = abs & 0x03ff; + let bits = if exp == 0 { + // subnormal: mant × 2⁻²⁴ = (1 + frac) × 2^(b−24), b the mantissa's highest set bit + let b = 31 - mant.leading_zeros(); + let frac = (mant ^ (1 << b)) << (23 - b); + sign | ((103 + b) << 23) | frac + } else { + sign | 0x7f80_0000 | (mant << 13) + }; + f32::from_bits(bits) +} + +/// f32 → IEEE half, rounded to nearest even; beyond the half range (±65504) an infinity, a NaN +/// stays a NaN. The inverse of [`half_to_f32`] for every value a half holds. +pub fn f32_to_half(v: f32) -> u16 { + let x = v.to_bits(); + let sign = ((x >> 16) & 0x8000) as u16; + let exp = ((x >> 23) & 0xff) as i32; + let mant = x & 0x7f_ffff; + if exp == 0xff { + return sign | 0x7c00 | if mant != 0 { 0x200 } else { 0 }; + } + let e = exp - 127 + 15; + if e >= 0x1f { + return sign | 0x7c00; + } + if e <= 0 { + if e < -10 { + return sign; + } + // a subnormal: the mantissa with its implicit bit at the half's scale, ties to even + let full = mant | 0x80_0000; + let shift = (14 - e) as u32; // 14..=24 + let m = full >> shift; + let rest = full & ((1 << shift) - 1); + let tie = 1 << (shift - 1); + let up = rest > tie || (rest == tie && m & 1 == 1); + return sign | (m + u32::from(up)) as u16; + } + let m = mant >> 13; + let round_bits = mant & 0x1fff; + let mut h = (sign as u32) | ((e as u32) << 10) | m; + if round_bits > 0x1000 || (round_bits == 0x1000 && (m & 1) == 1) { + h += 1; + } + h as u16 +} + /// The `r` part of a `w`×`h` plane of `n`-element pixels (`r` lies inside the plane). A plane too /// short for its size gives a blank result, as in [`rotate_plane`]. fn crop_plane(src: &[T], w: usize, h: usize, n: usize, r: Region) -> Vec { @@ -932,10 +1050,23 @@ mod tests { let c16: Vec = (0..w * h).map(|i| (64 + i * 7 % 896) as u16).collect(); let a16: Vec = (0..w * h).map(|i| (alpha(i % w, i / w) as u16) * 4).collect(); let c422: Vec = (0..cw * h).map(|i| (64 + i * 7 % 896) as u16).collect(); + // halves: a spread of normal values, zeros and a subnormal, with the alpha of the others + let halves: Vec = (0..w * h) + .flat_map(|i| { + let a = match alpha(i % w, i / w) { + 0 => 0x0000, + 77 => 0x34d5, + _ => 0x3c00, + }; + [0x3000 + (i * 7 % 0x0c00) as u16, if i % 13 == 0 { 0x0001 } else { 0x3800 + (i * 5 % 0x0400) as u16 }, (i * 29 % 0x3c00) as u16, a] + }) + .collect(); + let f16s: Vec = halves.iter().flat_map(|v| v.to_le_bytes()).collect(); let mk = |data| VideoFrame { width: w as u32, height: h as u32, data, color: ColorInfo::REC709, par: (1, 1), pts: Tick::ZERO }; vec![ ("rgba8", mk(PixelData::Rgba8(Arc::new(rgba8)))), ("rgba f32", mk(PixelData::RgbaF32(Arc::new(f32s)))), + ("rgba f16", mk(PixelData::RgbaF16(Arc::new(f16s)))), ( "yuv8 4:2:0 + alpha", mk(PixelData::Yuv8 { @@ -970,6 +1101,55 @@ mod tests { ] } + #[test] + fn halves_convert_like_the_same_values_as_f32() { + // every bit pattern, against the arithmetic definition + for bits in 0..=u16::MAX { + let (sign, exp, mant) = (if bits & 0x8000 == 0 { 1.0f32 } else { -1.0 }, (bits >> 10) & 31, bits & 1023); + let want = match exp { + 0 => sign * mant as f32 * 2.0f32.powi(-24), + 31 if mant == 0 => sign * f32::INFINITY, + 31 => f32::NAN, + _ => sign * (1.0 + mant as f32 / 1024.0) * 2.0f32.powi(exp as i32 - 15), + }; + let got = half_to_f32(bits); + assert!(got.to_bits() == want.to_bits() || (got.is_nan() && want.is_nan()), "half {bits:04x}: {got} vs {want}"); + if !got.is_nan() { + assert_eq!(f32_to_half(got), bits, "{got} back to a half"); + } + } + // halfway between two halves rounds to the even one, subnormals included + let tiny = 2.0f32.powi(-25); // half the smallest subnormal + assert_eq!([f32_to_half(tiny), f32_to_half(3.0 * tiny), f32_to_half(5.0 * tiny)], [0, 2, 2]); + assert_eq!(f32_to_half(tiny * 1.0001), 1, "above the tie rounds up"); + assert_eq!([f32_to_half(1.0 + 2.0f32.powi(-11)), f32_to_half(1.0 + 3.0 * 2.0f32.powi(-11))], [0x3c00, 0x3c02]); + // an f16 frame converts, crops and rotates exactly like the f32 frame of its values + let frames = sample_frames_sized(41, 19); + let f16 = frames.iter().find(|(n, _)| *n == "rgba f16").map(|(_, f)| f.clone()).unwrap(); + let PixelData::RgbaF16(d) = &f16.data else { panic!() }; + let f32s: Vec = d.as_chunks::<2>().0.iter().map(|b| half_to_f32(u16::from_le_bytes(*b))).collect(); + let f32 = VideoFrame { data: PixelData::RgbaF32(Arc::new(f32s)), ..f16.clone() }; + assert_eq!(f16.byte_size(), 41 * 19 * 8); + assert_eq!(f16.format_label(), "RGBA 16-bit float"); + for n in [1usize, 2, 3] { + let (a, b) = (f16.to_linear_f32_decimated(n), f32.to_linear_f32_decimated(n)); + assert_eq!((a.0, a.1), (b.0, b.1)); + assert!(a.2.iter().zip(&b.2).all(|(x, y)| x.to_bits() == y.to_bits()), "n={n}"); + } + let r = Region { x: 3, y: 2, w: 20, h: 9 }; + assert_eq!(f16.cropped(r).to_linear_f32(), f32.cropped(r).to_linear_f32()); + for q in 1..4 { + let (a, b) = (f16.rotated(q), f32.rotated(q)); + assert_eq!((a.width, a.height, a.par), (b.width, b.height, b.par)); + assert_eq!(a.to_linear_f32(), b.to_linear_f32(), "quarter turns {q}"); + } + assert!(f16.box_decimated(2).is_none(), "RGBA frames are not box-decimated"); + // short data converts to transparent, never panics + let short = VideoFrame { data: PixelData::RgbaF16(Arc::new(vec![0x3c; 10])), ..f16.clone() }; + assert!(short.to_linear_f32().iter().all(|v| *v == 0.0)); + assert_eq!(short.to_linear_f32_decimated(2).2.len(), 20 * 9 * 4); + } + #[test] fn a_region_is_exactly_the_crop_of_the_full_conversion() { for (name, f) in sample_frames() { diff --git a/crates/frame/src/pool.rs b/crates/frame/src/pool.rs index b9952398a7..2ac7a976c9 100644 --- a/crates/frame/src/pool.rs +++ b/crates/frame/src/pool.rs @@ -18,8 +18,10 @@ use std::sync::{Arc, Mutex, PoisonError}; use crate::{PixelData, VideoFrame}; -/// Bytes of idle buffers kept per sample type; a buffer that does not fit is freed. -const MAX_BYTES: usize = 192 << 20; +/// Bytes of idle buffers kept per sample type; a buffer that does not fit is freed. Sized for +/// 6K camera RAW at half resolution: five 31 MB helper replies in flight and a couple of 41 MB +/// half-float frames between the plan cache evicting one and the next decode taking it. +const MAX_BYTES: usize = 320 << 20; /// Smaller buffers are not worth keeping (the allocator recycles those well). const MIN_BYTES: usize = 64 << 10; @@ -102,6 +104,12 @@ pub fn take_u16(len: usize) -> Vec { take(&U16, len) } +/// Keep a byte buffer nobody needs any more for the next [`take_u8`]: a decoder's own scratch, +/// such as a reply read from a helper process, that never became a frame's plane. +pub fn recycle_u8(buf: Vec) { + U8.lock().unwrap_or_else(PoisonError::into_inner).give(buf, MAX_BYTES); +} + /// A buffer of exactly `len` floats whose contents are unspecified (whatever it held last, or zeros /// when none was idle): the caller must write every element before it reads any. Handing it back /// ([`recycle_f32`]) is optional; a buffer that is dropped is simply freed. @@ -133,9 +141,17 @@ pub fn recycle_f32(buf: Vec) { /// [`take_u16`] unless something else still holds the frame or a plane. pub fn recycle(frame: Arc) { let Ok(f) = Arc::try_unwrap(frame) else { return }; - match f.data { + recycle_pixels(f.data); +} + +/// [`recycle`] for a frame's pixels on their own: whoever holds the last reference to a plane +/// hands it back (a GPU upload cache dropping a frame the plan cache evicted earlier, or the +/// other way round); a plane something else still shows is theirs. +pub fn recycle_pixels(data: PixelData) { + match data { PixelData::Rgba8(d) => give(&U8, d), PixelData::RgbaF32(_) => {} + PixelData::RgbaF16(d) => give(&U8, d), PixelData::Yuv8 { planes, alpha, .. } => planes.into_iter().chain(alpha).for_each(|p| give(&U8, p)), PixelData::Yuv16 { planes, alpha, .. } => planes.into_iter().chain(alpha).for_each(|p| give(&U16, p)), } diff --git a/crates/gpu/src/lib.rs b/crates/gpu/src/lib.rs index 4f15697dd7..d6f50ed8b2 100644 --- a/crates/gpu/src/lib.rs +++ b/crates/gpu/src/lib.rs @@ -109,74 +109,24 @@ fn tex_entry(binding: u32) -> wgpu::BindGroupLayoutEntry { } } -/// f32 → IEEE half (round to nearest even), no dependency needed. +/// f32 → IEEE half (round to nearest even): [`filmcraft_frame::f32_to_half`]. +#[inline] pub fn f32_to_f16(v: f32) -> u16 { - let x = v.to_bits(); - let sign = ((x >> 16) & 0x8000) as u16; - let exp = ((x >> 23) & 0xff) as i32; - let mant = x & 0x7f_ffff; - if exp == 0xff { - return sign | 0x7c00 | if mant != 0 { 0x200 } else { 0 }; - } - let e = exp - 127 + 15; - if e >= 0x1f { - return sign | 0x7c00; - } - if e <= 0 { - if e < -10 { - return sign; - } - let m = (mant | 0x80_0000) >> (1 - e); - let round = (m >> 12) & 1; - return sign | (((m >> 13) + round) as u16); - } - let m = mant >> 13; - let round_bits = mant & 0x1fff; - let mut h = (sign as u32) | ((e as u32) << 10) | m; - if round_bits > 0x1000 || (round_bits == 0x1000 && (m & 1) == 1) { - h += 1; - } - h as u16 + filmcraft_frame::f32_to_half(v) } -/// IEEE half → f32 (the inverse of [`f32_to_f16`]), for reading `Rgba16Float` textures back. +/// IEEE half → f32 (the inverse of [`f32_to_f16`]), for reading `Rgba16Float` textures back: +/// [`filmcraft_frame::half_to_f32`]. +#[inline] pub fn f16_to_f32(h: u16) -> f32 { - let sign = ((h & 0x8000) as u32) << 16; - let exp = ((h >> 10) & 0x1f) as u32; - let mant = (h & 0x03ff) as u32; - let bits = match exp { - 0 => { - if mant == 0 { - sign - } else { - // Half subnormal: value = mant × 2⁻²⁴ = (1 + frac) × 2^(b−24) with b the mantissa's - // highest set bit — renormalise into an f32. - let b = 31 - mant.leading_zeros(); - let frac = (mant ^ (1 << b)) << (23 - b); - sign | ((103 + b) << 23) | frac - } - } - 0x1f => sign | 0x7f80_0000 | (mant << 13), - _ => sign | ((exp + 112) << 23) | (mant << 13), - }; - f32::from_bits(bits) + filmcraft_frame::half_to_f32(h) } -/// [`f16_to_f32`] with a branch-light path for normal numbers and zeros (the bulk of read-back -/// data); subnormals, infinities and NaNs take the general path. Same result for every input. +/// [`f16_to_f32`]; [`filmcraft_frame::half_to_f32`] already takes a branch-light path for normal +/// numbers and zeros (the bulk of read-back data). #[inline(always)] pub fn f16_to_f32_fast(h: u16) -> f32 { - let u = u32::from(h); - let sign = (u & 0x8000) << 16; - let abs = u & 0x7fff; - if (0x0400..0x7c00).contains(&abs) { - // rebias the exponent (127 - 15 = 112) and widen the mantissa - f32::from_bits(sign | ((abs + (112 << 10)) << 13)) - } else if abs == 0 { - f32::from_bits(sign) - } else { - f16_to_f32(h) - } + filmcraft_frame::half_to_f32(h) } /// Half-float texel data for one frame (one byte vector per plane), converted off the UI thread. @@ -249,7 +199,8 @@ pub fn prepare_frame(f: &VideoFrame) -> Option { PixelData::Yuv16 { planes, bits, alpha, .. } => { Some(Prepared { planes: planes.iter().chain(alpha.as_ref()).map(|p| codes_to_f16_bytes(p, *bits)).collect() }) } - PixelData::Rgba8(_) | PixelData::Yuv8 { .. } => None, + // halves are the texel layout already + PixelData::Rgba8(_) | PixelData::RgbaF16(_) | PixelData::Yuv8 { .. } => None, } } @@ -482,6 +433,7 @@ impl GpuCompositor { let id = match &f.data { PixelData::Rgba8(d) => Arc::as_ptr(d) as *const u8 as usize, PixelData::RgbaF32(d) => Arc::as_ptr(d) as *const u8 as usize, + PixelData::RgbaF16(d) => Arc::as_ptr(d) as *const u8 as usize, PixelData::Yuv8 { planes, .. } => Arc::as_ptr(&planes[0]) as *const u8 as usize, PixelData::Yuv16 { planes, .. } => Arc::as_ptr(&planes[0]) as *const u8 as usize, }; @@ -527,6 +479,18 @@ impl GpuCompositor { _pixels: f.data.clone(), } } + PixelData::RgbaF16(d) => { + let v = self.plane_texture(w, h, wgpu::TextureFormat::Rgba16Float, d, 8); + Uploaded { + views: [v, dummy(), dummy(), dummy()], + kind: 1, + code_scale: 1.0, + alpha_scale: 0.0, + chroma: (w, h), + last_used: self.clock, + _pixels: f.data.clone(), + } + } PixelData::Yuv8 { planes, chroma, alpha } => { let (sx, sy) = chroma.shifts(); let (cw, ch) = (w.div_ceil(1 << sx), h.div_ceil(1 << sy)); @@ -581,7 +545,10 @@ impl GpuCompositor { let mut v: Vec<(u64, (usize, u32, u32))> = self.uploads.iter().map(|(k, u)| (u.last_used, *k)).collect(); v.sort_unstable(); for (_, k) in v.into_iter().take(self.uploads.len() - 24) { - self.uploads.remove(&k); + // the last holder of a decoded frame's planes hands them back to the pool + if let Some(up) = self.uploads.remove(&k) { + filmcraft_frame::pool::recycle_pixels(up._pixels); + } } } key diff --git a/crates/media/Cargo.toml b/crates/media/Cargo.toml index 3259e502f4..53be0059f7 100644 --- a/crates/media/Cargo.toml +++ b/crates/media/Cargo.toml @@ -21,6 +21,7 @@ image = { version = "0.25", default-features = false, features = ["png", "jpeg", [dev-dependencies] filmcraft-testkit = { workspace = true } +serde_json = { workspace = true } filmcraft-dng = { workspace = true, features = ["writer"] } # APNG fixtures for the animated-image tests (the image crate does not encode APNG) png = "0.18" diff --git a/crates/media/src/cancel.rs b/crates/media/src/cancel.rs index 5c2b5f28b3..a8d39d0b3f 100644 --- a/crates/media/src/cancel.rs +++ b/crates/media/src/cancel.rs @@ -17,6 +17,11 @@ //! [`with_background`] marks work nobody waits on interactively (thumbnails of the Project panel, //! the Media Browser and the timeline): sources should not keep expensive state, such as a //! frame-threaded 4K decoder, for it once it is done. +//! +//! [`with_still`] marks a single frame a person waits for with playback stopped (a scrub, a click +//! on the ruler, a paused monitor): its latency matters, not throughput. A source with a fast +//! path for one frame (R3D: the CPU decoder answers a reduced frame in a fraction of the GPU +//! pipeline's latency) may take it. use std::cell::{Cell, RefCell}; use std::sync::Arc; @@ -29,6 +34,26 @@ thread_local! { static CATCH_UP: Cell> = const { Cell::new(None) }; static DRAFT: Cell = const { Cell::new(false) }; static BACKGROUND: Cell = const { Cell::new(false) }; + static STILL: Cell = const { Cell::new(false) }; +} + +/// Run `f` with the still-frame hint `on` (restoring the previous one however `f` ends): one +/// frame someone waits for while playback is stopped. +pub fn with_still(on: bool, f: impl FnOnce() -> R) -> R { + struct Restore(bool); + impl Drop for Restore { + fn drop(&mut self) { + STILL.with(|c| c.set(self.0)); + } + } + let _restore = Restore(STILL.with(|c| c.replace(on))); + f() +} + +/// Whether the work running on this thread is a still frame (off unless inside +/// [`with_still`]`(true, …)`). +pub fn still() -> bool { + STILL.with(Cell::get) } /// Run `f` as background work when `on` (restoring the previous hint after): a thumbnail, not a @@ -97,19 +122,21 @@ pub fn cancelled() -> bool { } /// The hints of the work running on this thread ([`with_cancel`], [`with_catch_up`], -/// [`with_draft`], [`with_background`]), to carry them to another thread doing part of that work. +/// [`with_draft`], [`with_background`], [`with_still`]), to carry them to another thread doing +/// part of that work. #[derive(Clone, Default)] pub struct Context { cancel: Option>, catch_up: Option, draft: bool, background: bool, + still: bool, } impl Context { /// This thread's hints. pub fn current() -> Self { - Context { cancel: CURRENT.with(|c| c.borrow().clone()), catch_up: catch_up(), draft: draft(), background: background() } + Context { cancel: CURRENT.with(|c| c.borrow().clone()), catch_up: catch_up(), draft: draft(), background: background(), still: still() } } /// Run `f` with these hints, restoring this thread's however `f` ends (a pool thread must @@ -123,6 +150,7 @@ impl Context { CATCH_UP.with(|c| c.set(prev.catch_up)); DRAFT.with(|c| c.set(prev.draft)); BACKGROUND.with(|c| c.set(prev.background)); + STILL.with(|c| c.set(prev.still)); } } let _restore = Restore(Context::current()); @@ -130,6 +158,7 @@ impl Context { CATCH_UP.with(|c| c.set(self.catch_up)); DRAFT.with(|c| c.set(self.draft)); BACKGROUND.with(|c| c.set(self.background)); + STILL.with(|c| c.set(self.still)); f() } } @@ -141,15 +170,15 @@ mod tests { #[test] fn context_carries_the_hints_to_another_thread_and_restores_its_own() { let flag = Arc::new(AtomicBool::new(true)); - let ctx = with_cancel(&flag, || with_draft(true, || with_background(true, || with_catch_up(Some(Tick(5)), Context::current)))); + let ctx = with_cancel(&flag, || with_draft(true, || with_background(true, || with_still(true, || with_catch_up(Some(Tick(5)), Context::current))))); std::thread::scope(|s| { s.spawn(|| { - let seen = with_draft(false, || ctx.clone().run(|| (cancelled(), draft(), background(), catch_up()))); - assert_eq!(seen, (true, true, true, Some(Tick(5)))); + let seen = with_draft(false, || ctx.clone().run(|| (cancelled(), draft(), background(), still(), catch_up()))); + assert_eq!(seen, (true, true, true, true, Some(Tick(5)))); // a panic inside leaves this thread's own hints in place let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| ctx.clone().run(|| panic!("job")))); assert!(r.is_err()); - assert_eq!((cancelled(), draft(), background(), catch_up()), (false, false, false, None)); + assert_eq!((cancelled(), draft(), background(), still(), catch_up()), (false, false, false, false, None)); }); }); } @@ -188,6 +217,19 @@ mod tests { assert!(!background()); } + #[test] + fn still_is_off_by_default_scoped_and_survives_a_panic() { + assert!(!still()); + with_still(true, || { + assert!(still()); + with_still(false, || assert!(!still())); + assert!(still()); + }); + assert!(!still()); + assert!(std::panic::catch_unwind(|| with_still(true, || panic!("boom"))).is_err()); + assert!(!still()); + } + #[test] fn a_panic_inside_background_work_does_not_leave_the_thread_marked() { let r = std::panic::catch_unwind(|| with_background(true, || panic!("boom"))); diff --git a/crates/media/src/lib.rs b/crates/media/src/lib.rs index ea8c4a7d7f..9681a65bb9 100644 --- a/crates/media/src/lib.rs +++ b/crates/media/src/lib.rs @@ -16,6 +16,7 @@ pub mod digits; pub mod dpx; pub mod generators; pub mod pending; +pub mod raw; pub mod reader; pub mod sequence; pub mod still; @@ -32,6 +33,7 @@ use serde::{Deserialize, Serialize}; pub use cache::FrameCache; pub use generators::{DemoScene, Generator}; +pub use raw::*; pub use reader::{ByteReader, ReaderOpener, SharedReader}; #[derive(Debug, thiserror::Error)] @@ -133,6 +135,35 @@ pub struct VideoStreamInfo { pub hdr: Option, } +/// The pixel aspect ratio of a lens squeeze a camera recorded as a number (`1.33`, `1.5`, `2.0`, +/// the RED SDK's `pixel_aspect_ratio`): the squeeze to two decimals as an exact ratio (1.33 is +/// 133:100, 1.5 is 3:2, 1.8 is 9:5), except a value within 0.0005 of 4/3, which is 4:3 (a +/// camera that stores 1.3333 means the 4:3 pixel). `None` for no squeeze (1.0) and for a value +/// that is not a usable ratio (beyond 8:1 either way, as `filmcraft_project::checked_par` +/// refuses). +pub fn lens_squeeze_par(factor: f64) -> Option<(u32, u32)> { + if !factor.is_finite() || factor <= 0.0 { + return None; + } + if (factor - 4.0 / 3.0).abs() < 0.0005 { + return Some((4, 3)); + } + let hundredths = (factor * 100.0).round(); + if !(1.0..=f64::from(u32::MAX)).contains(&hundredths) { + return None; + } + let n = hundredths as u32; + if !(13..=800).contains(&n) { + return None; + } + let (mut a, mut b) = (n, 100u32); + while b != 0 { + (a, b) = (b, a % b); + } + let g = a.max(1); + Some((n / g, 100 / g)).filter(|p| *p != (1, 1)) +} + /// Most audio streams a source reports (the count comes from the file: never trusted). pub const MAX_AUDIO_STREAMS: usize = 64; @@ -208,10 +239,65 @@ impl FrameRequest { } } +/// How a source delivers video frames, for playback scheduling ([`MediaSource::frame_delivery`]). +/// +/// Playback asks for every frame while the source keeps up. A source that cannot (a 6K camera +/// RAW decoder at 25 fps under a 59.94 fps sequence) gets every frame requested, delivers each +/// one after the playhead has passed it, and shows nothing. With `independent` set, playback +/// asks only for the frames the source can finish in time, evenly spaced and early enough. +#[derive(Clone, Copy, Debug, Default, PartialEq)] +pub struct FrameDelivery { + /// Frames decode independently of each other (camera RAW through a pipelined decoder, + /// intra-only codecs): leaving a frame out saves its whole cost. Off (the default) for GOP + /// codecs, whose decoder decodes every picture on its way to a later one, so skipping saves + /// nothing and playback must not try. + pub independent: bool, + /// Frames per second the source has been delivering lately, when it measures that itself (a + /// decoder process answering several requests at once knows its own throughput). `None`: + /// unknown; playback measures what it gets. + pub rate: Option, + /// `rate` was measured within the last couple of seconds. A source that measures itself at + /// saturation keeps its last reading through a lull, and a sparse playback plan is a lull: + /// playback may then try for more than a stale reading allows. + pub fresh: bool, +} + pub trait MediaSource: Send + Sync { fn info(&self) -> &MediaInfo; /// The frame displayed at media time `req.time`. fn video_frame(&self, req: FrameRequest) -> Result>; + /// How this source delivers frames (see [`FrameDelivery`]); the default says nothing, which + /// playback treats as a GOP decoder. Wrappers forward it to the source that decodes. + fn frame_delivery(&self) -> FrameDelivery { + FrameDelivery::default() + } + /// Develop the RAW picture with `develop` from now on (Clip ▸ Source Settings…): the next + /// frames come out in that output. Only camera RAW decoders implement it; every other source + /// (and a proxy, whose pictures were developed when it was made) answers + /// [`MediaError::Unsupported`], which callers treat as "no such setting", not a failure. + /// Wrappers forward it to the source that decodes. + fn set_develop(&self, develop: &RawDevelop) -> Result<()> { + let _ = develop; + Err(MediaError::Unsupported("this media has no RAW development settings".into())) + } + /// The development values recorded in a RAW clip (what [`set_develop`](Self::set_develop)'s + /// None fields mean): only camera RAW decoders report them. Wrappers forward it. + fn raw_as_shot(&self) -> Option { + None + } + /// What a RAW clip records beyond its as-shot values (recorded output choices, LUT, CDL, + /// curves, HDRx, RMD sidecar): only camera RAW decoders report it. Wrappers forward it. + fn raw_clip_info(&self) -> Option { + None + } + /// Write `develop` into the clip's RAW sidecar (an RMD file beside an R3D), creating it or + /// updating it, and decode with what it now says as the clip's recorded settings. Returns the + /// sidecar's path. Only on a person's explicit action (Source Settings ▸ Update RMD): it + /// changes a file beside the media. Other sources answer [`MediaError::Unsupported`]. + fn write_raw_sidecar(&self, develop: &RawDevelop) -> Result { + let _ = develop; + Err(MediaError::Unsupported("this media has no RAW sidecar".into())) + } /// `frames` audio frames starting at sample index `start` (at `sample_rate`, media time). fn audio(&self, start: i64, frames: usize, sample_rate: u32) -> Result; /// Like [`audio`](Self::audio) for audio stream `stream` (an index into @@ -228,7 +314,8 @@ pub type SharedSource = Arc; pub const STILL_EXTENSIONS: &[&str] = &["png", "jpg", "jpeg", "gif", "webp", "tif", "tiff", "bmp", "tga", "dpx", "exr", "hdr", "dng"]; pub const AUDIO_EXTENSIONS: &[&str] = &["wav", "wave", "bwf", "aif", "aiff", "mp3", "mp2", "m4a", "aac", "flac", "ogg", "opus"]; pub const VIDEO_EXTENSIONS: &[&str] = &[ - "mp4", "m4v", "mov", "mkv", "webm", "avi", "mxf", "mts", "m2ts", "m2t", "ts", "mpg", "mpeg", "vob", "mod", "tod", "m2v", "m1v", "mpv", "3gp", "y4m", "apv", + "r3d", "mp4", "m4v", "mov", "mkv", "webm", "avi", "mxf", "mts", "m2ts", "m2t", "ts", "mpg", "mpeg", "vob", "mod", "tod", "m2v", "m1v", "mpv", "3gp", "y4m", + "apv", ]; pub fn is_importable(path: &Path) -> bool { @@ -305,7 +392,7 @@ mod importable_tests { #[test] fn camera_extensions_match_regardless_of_case() { - for name in ["clip.mov", "clip.MOV", "clip.Mov", "shot.MP4", "IMG_0001.JPG", "sound.WAV", "take.M2TS"] { + for name in ["clip.mov", "clip.MOV", "clip.Mov", "shot.MP4", "IMG_0001.JPG", "sound.WAV", "take.M2TS", "DSC_0001.R3D", "DSC_0002.r3d"] { assert!(is_importable(Path::new(name)), "{name}"); } assert!(!is_importable(Path::new("notes.txt"))); diff --git a/crates/media/src/raw.rs b/crates/media/src/raw.rs new file mode 100644 index 0000000000..83bded9eb1 --- /dev/null +++ b/crates/media/src/raw.rs @@ -0,0 +1,816 @@ +//! Camera RAW development settings (Clip ▸ Source Settings… and Project Settings ▸ Camera RAW). +//! +//! The picture of a camera RAW clip is developed from the sensor data every time it is decoded, +//! so these are decode settings, not colour corrections. For R3D they are RED's IPP2 pipeline as +//! the RED SDK exposes it (`ImageProcessingSettings`): the RAW adjustments applied before +//! development (ISO, white balance, exposure), the output colour space and gamma curve, and for a +//! display-referred gamma the output tone map, highlight rolloff and HDR peak; then the +//! processing options (chroma noise reduction, flashing pixel adjustment, the clip's metadata +//! curves, creative LUT and CDL) and the decode quality. +//! +//! A clip either decodes with the project's Camera RAW settings (Decode Using ▸ Project, the +//! default of a clip that has none of its own) or with its own (Decode Using ▸ Clip). The +//! project's settings of a project made before they existed are [`RawDevelop::default`] (scene +//! linear Rec. 709, as shot), which is what every clip got then; a new project starts from +//! [`RawDevelop::STANDARD`] (Rec. 709, BT.1886). + +use serde::{Deserialize, Serialize}; + +use filmcraft_color::{ColorSpace, Curve, LogCurve}; + +/// The decoder's colour space for the developed picture (the RGB primaries; RED IPP2's +/// `ImageColorSpace`). Only the spaces the compositor can interpret are offered. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum RawColorSpace { + #[default] + Rec709, + Rec2020, + #[serde(rename = "p3d65")] + P3D65, + /// REDWideGamutRGB, the camera's wide gamut for grading. + #[serde(rename = "redWideGamutRgb")] + RedWideGamutRgb, +} + +/// The decoder's gamma curve (RED IPP2's `ImageGammaCurve`). Only curves the compositor can +/// interpret are offered. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum RawGamma { + /// Scene-linear light, no output transform (the SDK's half-float output). + #[default] + Linear, + /// The BT.1886 display curve, with the output tone map and highlight rolloff. + #[serde(rename = "bt1886")] + Bt1886, + /// The sRGB display curve, with the output tone map and highlight rolloff. + Srgb, + /// RED Log3G10 for grading (REDWideGamutRGB only). + #[serde(rename = "log3g10")] + Log3G10, + /// SMPTE ST 2084 (PQ) for HDR, with the HDR peak (Rec. 2020 only). + #[serde(rename = "st2084")] + St2084, + /// Hybrid Log-Gamma for HDR (Rec. 2020 only). + Hlg, +} + +/// Output tone mapping of a display-referred RAW output. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum RawToneMap { + /// The setting recorded in the clip (or its RMD sidecar). + #[default] + Clip, + None, + Low, + Medium, + High, +} + +/// Highlight rolloff of a display-referred RAW output (no effect on a log or linear output). +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum RawRollOff { + /// The setting recorded in the clip (or its RMD sidecar). + #[default] + Clip, + None, + Hard, + Medium, + Soft, + VerySoft, +} + +/// Flashing pixel adjustment (RED SDK `FlashingPixelAdjust`; full and half premium decodes). +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum RawFlashing { + #[default] + Off, + Mild, + Medium, + Strong, +} + +/// Decode quality (RED SDK `VideoDecodeMode`): the largest decode a preview may use. A full-size +/// request (an export, a monitor at 100 %) always decodes at full resolution, premium; a preview +/// decodes at the smallest size that covers it, but never larger than this. The SDK has one decode +/// mode per request and does not tell playback from export, so neither does this setting. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum RawQuality { + /// Full resolution, premium (a half-size preview decodes at half, premium). + #[default] + FullPremium, + HalfPremium, + /// Half resolution, good: faster than premium. + HalfGood, + QuarterGood, + EighthGood, + SixteenthGood, +} + +/// Where a clip's development settings come from. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum DecodeUsing { + /// The clip's own settings. What a clip with settings of its own from before this choice + /// existed reads as. + #[default] + Clip, + /// The project's Camera RAW settings (Project Settings ▸ Camera RAW). + Project, +} + +/// What a camera RAW decoder outputs: see the module documentation. Copy and comparable, so a +/// setting is a cache key and an undo comparison. Decoders ignore what they have no equivalent +/// for. [`RawDevelop::default`] is what every project made before these settings existed got. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct RawDevelop { + /// Clip settings only: decode with these (Clip) or with the project's (Project). + pub decode_using: DecodeUsing, + /// None = the project's decode quality (in the project's own settings: full, premium). + #[serde(skip_serializing_if = "Option::is_none")] + pub quality: Option, + pub color_space: RawColorSpace, + pub gamma: RawGamma, + pub tone_map: RawToneMap, + pub rolloff: RawRollOff, + /// Peak of an ST 2084 output in cd/m² ([`RAW_HDR_PEAK_RANGE`]); None = the clip's. + #[serde(skip_serializing_if = "Option::is_none")] + pub hdr_peak_nits: Option, + /// Chroma noise reduction (full and half premium decodes); None = as recorded. + #[serde(skip_serializing_if = "Option::is_none")] + pub chroma_noise_reduction: Option, + /// Flashing pixel adjustment; None = as recorded. + #[serde(skip_serializing_if = "Option::is_none")] + pub flashing_pixel: Option, + /// Apply the luma and RGB curves recorded in the clip (or its RMD). + pub metadata_curves: bool, + /// Apply the creative 3D LUT the clip (or its RMD) carries. Display and log gammas only. + pub creative_lut: bool, + /// Apply the clip's (or its RMD's) ASC CDL. Display and log gammas only. + pub cdl: bool, + /// ISO the sensor data is developed at, one of [`RAW_ISO_VALUES`]; None = as shot. + #[serde(skip_serializing_if = "Option::is_none")] + pub iso: Option, + /// White balance colour temperature in Kelvin ([`RAW_KELVIN_RANGE`]); None = as shot. + #[serde(skip_serializing_if = "Option::is_none")] + pub kelvin: Option, + /// White balance tint ([`RAW_TINT_RANGE`], hundredths); None = as shot. + #[serde(skip_serializing_if = "Option::is_none")] + pub tint: Option, + /// Exposure adjustment in stops ([`RAW_EXPOSURE_RANGE`], hundredths); None = as shot. + #[serde(skip_serializing_if = "Option::is_none")] + pub exposure: Option, +} + +impl Default for RawDevelop { + /// Scene-linear light in Rec. 709 primaries, as shot, no look: what every clip developed + /// before these settings existed, and the Camera RAW settings of a project from then. + fn default() -> Self { + RawDevelop { + decode_using: DecodeUsing::Clip, + quality: None, + color_space: RawColorSpace::Rec709, + gamma: RawGamma::Linear, + tone_map: RawToneMap::Clip, + rolloff: RawRollOff::Clip, + hdr_peak_nits: None, + chroma_noise_reduction: None, + flashing_pixel: None, + metadata_curves: true, + creative_lut: false, + cdl: false, + iso: None, + kelvin: None, + tint: None, + exposure: None, + } + } +} + +/// The output choice of Source Settings before colour space and gamma were separate lists +/// (project files and `clip.sourceSettings {output}` of early builds of this feature). Still read. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum RawOutput { + LinearRec709, + Rec709, + #[serde(rename = "log3g10")] + Log3G10, +} + +impl RawOutput { + pub const ALL: [RawOutput; 3] = [RawOutput::LinearRec709, RawOutput::Rec709, RawOutput::Log3G10]; + pub fn id(self) -> &'static str { + match self { + RawOutput::LinearRec709 => "linearRec709", + RawOutput::Rec709 => "rec709", + RawOutput::Log3G10 => "log3g10", + } + } + /// The colour space and gamma curve the output stood for. + pub fn space_gamma(self) -> (RawColorSpace, RawGamma) { + match self { + RawOutput::LinearRec709 => (RawColorSpace::Rec709, RawGamma::Linear), + RawOutput::Rec709 => (RawColorSpace::Rec709, RawGamma::Bt1886), + RawOutput::Log3G10 => (RawColorSpace::RedWideGamutRgb, RawGamma::Log3G10), + } + } + pub fn parse(s: &str) -> Option { + Self::ALL.into_iter().find(|o| o.id().eq_ignore_ascii_case(s)) + } +} + +/// How a [`RawDevelop`] reads: every field optional, and the `output` of the first version +/// mapped onto colour space and gamma when those are absent. +#[derive(Deserialize)] +#[serde(default, rename_all = "camelCase")] +struct RawDevelopIn { + decode_using: DecodeUsing, + quality: Option, + output: Option, + color_space: Option, + gamma: Option, + tone_map: RawToneMap, + rolloff: RawRollOff, + hdr_peak_nits: Option, + chroma_noise_reduction: Option, + flashing_pixel: Option, + metadata_curves: Option, + creative_lut: bool, + cdl: bool, + iso: Option, + kelvin: Option, + tint: Option, + exposure: Option, +} + +impl Default for RawDevelopIn { + fn default() -> Self { + let d = RawDevelop::default(); + RawDevelopIn { + decode_using: d.decode_using, + quality: None, + output: None, + color_space: None, + gamma: None, + tone_map: d.tone_map, + rolloff: d.rolloff, + hdr_peak_nits: None, + chroma_noise_reduction: None, + flashing_pixel: None, + metadata_curves: None, + creative_lut: false, + cdl: false, + iso: None, + kelvin: None, + tint: None, + exposure: None, + } + } +} + +impl<'de> Deserialize<'de> for RawDevelop { + fn deserialize>(deserializer: D) -> std::result::Result { + let w = RawDevelopIn::deserialize(deserializer)?; + let (space, gamma) = w.output.map(RawOutput::space_gamma).unwrap_or_default(); + Ok(RawDevelop { + decode_using: w.decode_using, + quality: w.quality, + color_space: w.color_space.unwrap_or(space), + gamma: w.gamma.unwrap_or(gamma), + tone_map: w.tone_map, + rolloff: w.rolloff, + hdr_peak_nits: w.hdr_peak_nits, + chroma_noise_reduction: w.chroma_noise_reduction, + flashing_pixel: w.flashing_pixel, + metadata_curves: w.metadata_curves.unwrap_or(true), + creative_lut: w.creative_lut, + cdl: w.cdl, + iso: w.iso, + kelvin: w.kelvin, + tint: w.tint, + exposure: w.exposure, + }) + } +} + +/// The ISO values a RAW decoder develops at (RED SDK 9.2.1 `ImageProcessingLimits::ISOList`; the +/// helper checks a request against the SDK's own list, so a future SDK's additions are refused +/// here first). The Nikon ZR records 64 to 25600. +pub const RAW_ISO_VALUES: [u32; 28] = [ + 50, 64, 80, 100, 125, 160, 200, 250, 320, 400, 500, 640, 800, 1000, 1280, 1600, 2000, 2500, 3200, 4000, 5000, 6400, 8000, 10000, 12800, 16000, 20000, 25600, +]; +/// White balance colour temperatures a RAW decoder accepts, in Kelvin (RED SDK `KelvinMin..Max`). +pub const RAW_KELVIN_RANGE: std::ops::RangeInclusive = 1700..=100_000; +/// White balance tints a RAW decoder accepts (RED SDK `TintMin..Max`). +pub const RAW_TINT_RANGE: std::ops::RangeInclusive = -100.0..=100.0; +/// Exposure adjustments a RAW decoder accepts, in stops (RED SDK `ExposureAdjustMin..Max`). +pub const RAW_EXPOSURE_RANGE: std::ops::RangeInclusive = -8.0..=8.0; +/// Peaks of an ST 2084 output, in cd/m² (the PQ curve's own range; the helper checks the SDK's +/// `HdrPeakNitsMin..Max` as well). +pub const RAW_HDR_PEAK_RANGE: std::ops::RangeInclusive = 100..=10_000; + +/// A decimal with two places kept as an integer count of hundredths, so a setting that holds one +/// stays exactly comparable and hashable (a cache key, an undo comparison). It reads and writes as +/// a plain number (`-0.33`); anything that is not a finite number within ±1 000 000 is refused. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct Hundredths(pub i32); + +impl Hundredths { + /// The nearest hundredth of `v`, None unless `v` is finite and within ±1 000 000. + pub fn from_f64(v: f64) -> Option { + if !v.is_finite() || v.abs() > 1_000_000.0 { + return None; + } + Some(Hundredths((v * 100.0).round() as i32)) + } + pub fn as_f64(self) -> f64 { + f64::from(self.0) / 100.0 + } +} + +impl Serialize for Hundredths { + fn serialize(&self, serializer: S) -> std::result::Result { + serializer.serialize_f64(self.as_f64()) + } +} + +impl<'de> Deserialize<'de> for Hundredths { + fn deserialize>(deserializer: D) -> std::result::Result { + let v = f64::deserialize(deserializer)?; + Hundredths::from_f64(v).ok_or_else(|| serde::de::Error::custom(format!("{v} is not a finite decimal within range"))) + } +} + +/// The development values recorded in a RAW clip (or in its RMD sidecar, which overrides them), +/// which its decoder uses until Source Settings override them: ISO, white balance colour +/// temperature (Kelvin) and tint, and the exposure adjustment in stops (a camera ISO without an +/// SDK equivalent is recorded as the nearest one plus a small adjustment). +#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct RawAsShot { + pub iso: u32, + pub kelvin: f32, + pub tint: f32, + pub exposure: f32, +} + +/// What a RAW clip records beyond [`RawAsShot`], as its decoder reports it: the recorded output +/// choices and processing options (what "as recorded" means), whether it carries a creative LUT, +/// a CDL, curves or an HDRx track, its RMD sidecar, and the decoder's own ranges and defaults. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(default, rename_all = "camelCase")] +pub struct RawClipInfo { + /// The recorded colour space and gamma when they are ones offered here. + pub color_space: Option, + pub gamma: Option, + /// The recorded tone map and rolloff (never `Clip`). + pub tone_map: Option, + pub rolloff: Option, + pub hdr_peak_nits: Option, + pub chroma_noise_reduction: bool, + pub flashing_pixel: RawFlashing, + /// The clip (or its RMD) carries a creative 3D LUT, and whether it asks for it applied. + pub lut: bool, + pub lut_enabled: bool, + /// Whether the clip asks for its CDL applied, and the CDL: saturation, then slope, offset and + /// power of red, green and blue. + pub cdl_enabled: bool, + pub cdl: [f32; 10], + /// The clip records curves that are not the identity. + pub curves: bool, + /// The clip has an HDRx (second exposure) track to blend. + pub hdrx: bool, + /// The RMD sidecar the decoder read the clip's settings from, if there was one. + pub rmd: Option, + /// Where the clip's RMD sidecar is (or would be written). + pub rmd_path: String, + /// The decoder's range and default of the HDR peak, and its default tone map and rolloff. + pub hdr_peak_range: (u32, u32), + pub hdr_peak_default: u32, + pub tone_map_default: Option, + pub rolloff_default: Option, +} + +macro_rules! choice { + ($ty:ident, $all:expr, $( $v:ident => $id:literal, $label:literal ;)*) => { + impl $ty { + pub const ALL: [$ty; $all] = [$($ty::$v),*]; + /// Stable id used in commands and project files. + pub fn id(self) -> &'static str { + match self { $($ty::$v => $id),* } + } + pub fn label(self) -> &'static str { + match self { $($ty::$v => $label),* } + } + pub fn parse(s: &str) -> Option<$ty> { + Self::ALL.into_iter().find(|o| o.id().eq_ignore_ascii_case(s)) + } + } + }; +} + +choice!(RawColorSpace, 4, + Rec709 => "rec709", "Rec. 709"; + Rec2020 => "rec2020", "Rec. 2020"; + P3D65 => "p3d65", "P3 D65"; + RedWideGamutRgb => "redWideGamutRgb", "REDWideGamutRGB"; +); +choice!(RawGamma, 6, + Linear => "linear", "Linear"; + Bt1886 => "bt1886", "BT.1886"; + Srgb => "srgb", "sRGB"; + Log3G10 => "log3g10", "Log3G10"; + St2084 => "st2084", "ST 2084 (PQ)"; + Hlg => "hlg", "Hybrid Log-Gamma"; +); +choice!(RawToneMap, 5, + Clip => "clip", "As recorded"; + None => "none", "None"; + Low => "low", "Low"; + Medium => "medium", "Medium"; + High => "high", "High"; +); +choice!(RawRollOff, 6, + Clip => "clip", "As recorded"; + None => "none", "None"; + Hard => "hard", "Hard"; + Medium => "medium", "Medium"; + Soft => "soft", "Soft"; + VerySoft => "verySoft", "Very Soft"; +); +choice!(RawFlashing, 4, + Off => "off", "None"; + Mild => "mild", "Mild"; + Medium => "medium", "Medium"; + Strong => "strong", "Strong"; +); +choice!(RawQuality, 6, + FullPremium => "fullPremium", "Full Res. Premium"; + HalfPremium => "halfPremium", "Half Res. Premium"; + HalfGood => "halfGood", "Half Res. Good"; + QuarterGood => "quarterGood", "Quarter Res. Good"; + EighthGood => "eighthGood", "Eighth Res. Good"; + SixteenthGood => "sixteenthGood", "Sixteenth Res. Good"; +); +choice!(DecodeUsing, 2, + Clip => "clip", "Clip"; + Project => "project", "Project"; +); + +impl RawQuality { + /// The smallest divisor a preview at this quality may decode at (1 full, 2 half, …). + pub fn divisor(self) -> u32 { + match self { + RawQuality::FullPremium => 1, + RawQuality::HalfPremium | RawQuality::HalfGood => 2, + RawQuality::QuarterGood => 4, + RawQuality::EighthGood => 8, + RawQuality::SixteenthGood => 16, + } + } + /// A half-resolution decode at this quality is the faster "good" one. + pub fn half_good(self) -> bool { + self == RawQuality::HalfGood + } +} + +impl RawGamma { + /// A display gamma, which the output tone map and highlight rolloff shape. + pub fn is_display(self) -> bool { + matches!(self, RawGamma::Bt1886 | RawGamma::Srgb | RawGamma::St2084 | RawGamma::Hlg) + } + /// The curve the developed code values are linearised with (the compositor's own, so its + /// interpretation and the decoder's output agree). + pub fn curve(self) -> Curve { + match self { + RawGamma::Linear => Curve::Linear, + RawGamma::Bt1886 => Curve::Sdr, + RawGamma::Srgb => Curve::Srgb, + RawGamma::Log3G10 => Curve::Log(LogCurve::Log3G10), + RawGamma::St2084 => Curve::Pq, + RawGamma::Hlg => Curve::Hlg, + } + } +} + +impl RawDevelop { + /// The project Camera RAW settings of a new project: IPP2 into Rec. 709 with the BT.1886 + /// display curve, the SDK's default output tone map and highlight rolloff (Medium and Soft + /// in RED SDK 9.2.1, which the helper reports and a real-clip test checks), as shot. + pub const STANDARD: RawDevelop = RawDevelop { + decode_using: DecodeUsing::Clip, + quality: None, + color_space: RawColorSpace::Rec709, + gamma: RawGamma::Bt1886, + tone_map: RawToneMap::Medium, + rolloff: RawRollOff::Soft, + hdr_peak_nits: None, + chroma_noise_reduction: None, + flashing_pixel: None, + metadata_curves: true, + creative_lut: false, + cdl: false, + iso: None, + kelvin: None, + tint: None, + exposure: None, + }; + + /// The settings a clip decodes with: its own (Decode Using ▸ Clip), or the project's + /// (Decode Using ▸ Project, or no settings of its own), with the clip's decode quality when + /// it has one, else the project's. + pub fn effective(project: &RawDevelop, clip: Option<&RawDevelop>) -> RawDevelop { + match clip { + Some(c) if c.decode_using == DecodeUsing::Clip => RawDevelop { quality: c.quality.or(project.quality), ..*c }, + Some(c) => RawDevelop { decode_using: DecodeUsing::Project, quality: c.quality.or(project.quality), ..*project }, + None => RawDevelop { decode_using: DecodeUsing::Project, ..*project }, + } + } + + /// The decode quality in effect (full, premium unless set). + pub fn quality(&self) -> RawQuality { + self.quality.unwrap_or_default() + } + + /// A short stable text of the settings, for cache keys (`"rec709/bt1886/medium/soft"`, then + /// whatever else is not the default: `"/iso800/k5600/t0/e-33"`, `"/q:halfGood"`…). + pub fn key(&self) -> String { + let mut key = format!("{}/{}/{}/{}", self.color_space.id(), self.gamma.id(), self.tone_map.id(), self.rolloff.id()); + if let Some(q) = self.quality { + key.push_str(&format!("/q:{}", q.id())); + } + if let Some(p) = self.hdr_peak_nits { + key.push_str(&format!("/peak{p}")); + } + if let Some(c) = self.chroma_noise_reduction { + key.push_str(if c { "/cnr1" } else { "/cnr0" }); + } + if let Some(f) = self.flashing_pixel { + key.push_str(&format!("/fpa:{}", f.id())); + } + if !self.metadata_curves { + key.push_str("/nocurves"); + } + if self.creative_lut { + key.push_str("/lut"); + } + if self.cdl { + key.push_str("/cdl"); + } + if let Some(iso) = self.iso { + key.push_str(&format!("/iso{iso}")); + } + if let Some(k) = self.kelvin { + key.push_str(&format!("/k{k}")); + } + if let Some(t) = self.tint { + key.push_str(&format!("/t{}", t.0)); + } + if let Some(e) = self.exposure { + key.push_str(&format!("/e{}", e.0)); + } + key + } + /// Whether any RAW adjustment (ISO, Kelvin, tint, exposure) overrides the clip's own value. + pub fn adjusts_raw(&self) -> bool { + self.iso.is_some() || self.kelvin.is_some() || self.tint.is_some() || self.exposure.is_some() + } + /// The RAW adjustments back to as shot; the other settings stay. + pub fn with_as_shot(self) -> Self { + RawDevelop { iso: None, kelvin: None, tint: None, exposure: None, ..self } + } + /// Whether the decoder can develop into `space` with `gamma` and the compositor interpret the + /// result: display curves need display primaries, Log3G10 REDWideGamutRGB, the HDR curves + /// Rec. 2020; linear light goes with any of them. + pub fn supports(space: RawColorSpace, gamma: RawGamma) -> bool { + match gamma { + RawGamma::Linear => true, + RawGamma::Bt1886 | RawGamma::Srgb => space != RawColorSpace::RedWideGamutRgb, + RawGamma::Log3G10 => space == RawColorSpace::RedWideGamutRgb, + RawGamma::St2084 | RawGamma::Hlg => space == RawColorSpace::Rec2020, + } + } + /// A gamma curve that goes with `space`, for a colour space chosen on its own: Log3G10 for + /// REDWideGamutRGB, BT.1886 for the display spaces. + pub fn gamma_for(space: RawColorSpace) -> RawGamma { + if space == RawColorSpace::RedWideGamutRgb { RawGamma::Log3G10 } else { RawGamma::Bt1886 } + } + /// A colour space that goes with `gamma`, for a gamma curve chosen on its own: + /// REDWideGamutRGB for Log3G10, Rec. 2020 for the HDR curves, Rec. 709 otherwise. + pub fn space_for(gamma: RawGamma) -> RawColorSpace { + match gamma { + RawGamma::Log3G10 => RawColorSpace::RedWideGamutRgb, + RawGamma::St2084 | RawGamma::Hlg => RawColorSpace::Rec2020, + _ => RawColorSpace::Rec709, + } + } + /// The colour space interpretation the compositor needs for frames developed with these + /// settings, which the decoder hands over as linear light in the developed primaries; None + /// when the frames' own linear Rec. 709 tag says it (no interpretation, the fast path). + pub fn interpretation(&self) -> Option { + match (self.color_space, self.gamma) { + (RawColorSpace::Rec709, _) => None, + (RawColorSpace::Rec2020, RawGamma::St2084) => Some(ColorSpace::Rec2100Pq), + (RawColorSpace::Rec2020, RawGamma::Hlg) => Some(ColorSpace::Rec2100Hlg), + (RawColorSpace::Rec2020, _) => Some(ColorSpace::Rec2020), + (RawColorSpace::P3D65, _) => Some(ColorSpace::P3D65), + // scene-linear camera gamut: the compositor tone maps it as log media (the frames are + // linear already, so only the gamut and tone stages apply) + (RawColorSpace::RedWideGamutRgb, _) => Some(ColorSpace::Log3G10Rwg), + } + } + /// Every interpretation [`interpretation`](Self::interpretation) can give (so an engine can + /// tell one it set from a person's own Interpret Footage choice). + pub const INTERPRETATIONS: [ColorSpace; 5] = + [ColorSpace::Rec2100Pq, ColorSpace::Rec2100Hlg, ColorSpace::Rec2020, ColorSpace::P3D65, ColorSpace::Log3G10Rwg]; + /// The creative LUT and the CDL apply (display and log gammas: the SDK's linear half-float + /// output ignores them). + pub fn grades_apply(&self) -> bool { + self.gamma != RawGamma::Linear + } + /// Every set value is one the decoder accepts ([`RAW_ISO_VALUES`] and the ranges above) and + /// the colour space and gamma go together ([`supports`](Self::supports)); the first offending + /// one otherwise. Project files and command parameters are checked with this before a decoder + /// sees them. + pub fn validate(&self) -> std::result::Result<(), String> { + if !Self::supports(self.color_space, self.gamma) { + return Err(format!("the {} gamma curve does not go with the {} colour space", self.gamma.label(), self.color_space.label())); + } + if let Some(iso) = self.iso + && !RAW_ISO_VALUES.contains(&iso) + { + return Err(format!("ISO {iso} is not one of the decoder's ISO values ({})", RAW_ISO_VALUES.map(|v| v.to_string()).join(", "))); + } + if let Some(k) = self.kelvin + && !RAW_KELVIN_RANGE.contains(&k) + { + return Err(format!("colour temperature {k} K is outside {}..{} K", RAW_KELVIN_RANGE.start(), RAW_KELVIN_RANGE.end())); + } + if let Some(t) = self.tint + && !RAW_TINT_RANGE.contains(&t.as_f64()) + { + return Err(format!("tint {} is outside {}..{}", t.as_f64(), RAW_TINT_RANGE.start(), RAW_TINT_RANGE.end())); + } + if let Some(e) = self.exposure + && !RAW_EXPOSURE_RANGE.contains(&e.as_f64()) + { + return Err(format!("exposure adjustment {} is outside {}..{} stops", e.as_f64(), RAW_EXPOSURE_RANGE.start(), RAW_EXPOSURE_RANGE.end())); + } + if let Some(p) = self.hdr_peak_nits + && !RAW_HDR_PEAK_RANGE.contains(&p) + { + return Err(format!("HDR peak {p} cd/m² is outside {}..{}", RAW_HDR_PEAK_RANGE.start(), RAW_HDR_PEAK_RANGE.end())); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + /// The JSON names are the command's ids, so a project file and `clip.sourceSettings` agree. + #[test] + fn json_names_are_the_command_ids() { + fn check(all: &[T], id: fn(T) -> &'static str, parse: fn(&str) -> Option) { + for v in all { + assert_eq!(serde_json::to_value(v).unwrap(), serde_json::Value::String(id(*v).into()), "{v:?}"); + assert_eq!(parse(id(*v)), Some(*v)); + } + } + check(&RawColorSpace::ALL, RawColorSpace::id, RawColorSpace::parse); + check(&RawGamma::ALL, RawGamma::id, RawGamma::parse); + check(&RawToneMap::ALL, RawToneMap::id, RawToneMap::parse); + check(&RawRollOff::ALL, RawRollOff::id, RawRollOff::parse); + check(&RawFlashing::ALL, RawFlashing::id, RawFlashing::parse); + check(&RawQuality::ALL, RawQuality::id, RawQuality::parse); + check(&DecodeUsing::ALL, DecodeUsing::id, DecodeUsing::parse); + let d = RawDevelop { + color_space: RawColorSpace::RedWideGamutRgb, + gamma: RawGamma::Log3G10, + tone_map: RawToneMap::High, + rolloff: RawRollOff::VerySoft, + ..Default::default() + }; + assert_eq!( + serde_json::to_value(d).unwrap(), + json!({"decodeUsing": "clip", "colorSpace": "redWideGamutRgb", "gamma": "log3g10", "toneMap": "high", "rolloff": "verySoft", "metadataCurves": true, "creativeLut": false, "cdl": false}) + ); + assert_eq!(serde_json::from_value::(serde_json::to_value(d).unwrap()).unwrap(), d); + assert!(serde_json::from_str::(r#"{"gamma": "rec2020"}"#).is_err()); + assert_eq!(d.key(), "redWideGamutRgb/log3g10/high/verySoft"); + } + + /// Projects from before colour space and gamma were separate keep their output: `output` + /// maps onto them, and a develop without either is scene-linear Rec. 709 as before. + #[test] + fn the_first_versions_output_still_reads() { + for (text, space, gamma) in [ + (r#"{"output": "linearRec709", "toneMap": "clip", "rolloff": "clip"}"#, RawColorSpace::Rec709, RawGamma::Linear), + (r#"{"output": "rec709", "toneMap": "medium"}"#, RawColorSpace::Rec709, RawGamma::Bt1886), + (r#"{"output": "log3g10"}"#, RawColorSpace::RedWideGamutRgb, RawGamma::Log3G10), + (r#"{"iso": 800}"#, RawColorSpace::Rec709, RawGamma::Linear), + ] { + let d: RawDevelop = serde_json::from_str(text).unwrap(); + assert_eq!((d.color_space, d.gamma, d.decode_using), (space, gamma, DecodeUsing::Clip), "{text}"); + assert!(d.metadata_curves && !d.creative_lut && !d.cdl, "{text}"); + } + let explicit: RawDevelop = serde_json::from_str(r#"{"output": "rec709", "gamma": "srgb"}"#).unwrap(); + assert_eq!((explicit.color_space, explicit.gamma), (RawColorSpace::Rec709, RawGamma::Srgb), "the new fields win"); + assert!(serde_json::from_str::(r#"{"output": "rec2020"}"#).is_err()); + assert_eq!(serde_json::from_str::("{}").unwrap(), RawDevelop::default()); + } + + /// The RAW adjustments read and write as plain numbers (two decimals kept exactly), are left + /// out when as shot, extend the cache key, and are range-checked; hostile numbers are refused. + #[test] + fn raw_adjustments_round_trip_as_numbers_and_are_range_checked() { + let d = RawDevelop { iso: Some(800), kelvin: Some(5600), tint: Some(Hundredths(-1250)), exposure: Hundredths::from_f64(-0.333), ..Default::default() }; + let v = serde_json::to_value(d).unwrap(); + assert_eq!(v["iso"], json!(800)); + assert_eq!(v["kelvin"], json!(5600)); + assert_eq!(v["tint"], json!(-12.5)); + assert_eq!(v["exposure"], json!(-0.33)); + assert_eq!(serde_json::from_value::(v).unwrap(), d); + assert_eq!(d.key(), "rec709/linear/clip/clip/iso800/k5600/t-1250/e-33"); + assert!(d.adjusts_raw() && !RawDevelop::default().adjusts_raw()); + assert_eq!(d.with_as_shot(), RawDevelop::default()); + assert!(!serde_json::to_string(&RawDevelop::default()).unwrap().contains("iso"), "as shot is written as nothing"); + assert!(d.validate().is_ok() && RawDevelop::default().validate().is_ok() && RawDevelop::STANDARD.validate().is_ok()); + for bad in [ + RawDevelop { iso: Some(1250), ..Default::default() }, + RawDevelop { iso: Some(0), ..Default::default() }, + RawDevelop { kelvin: Some(1699), ..Default::default() }, + RawDevelop { kelvin: Some(100_001), ..Default::default() }, + RawDevelop { tint: Some(Hundredths(10_001)), ..Default::default() }, + RawDevelop { exposure: Some(Hundredths(-801)), ..Default::default() }, + RawDevelop { hdr_peak_nits: Some(99), ..Default::default() }, + RawDevelop { hdr_peak_nits: Some(10_001), ..Default::default() }, + RawDevelop { gamma: RawGamma::Log3G10, ..Default::default() }, + RawDevelop { color_space: RawColorSpace::RedWideGamutRgb, gamma: RawGamma::Bt1886, ..Default::default() }, + RawDevelop { color_space: RawColorSpace::P3D65, gamma: RawGamma::St2084, ..Default::default() }, + ] { + assert!(bad.validate().is_err(), "{bad:?}"); + } + assert_eq!(Hundredths::from_f64(12.345), Some(Hundredths(1235))); + assert_eq!(Hundredths::from_f64(-0.005), Some(Hundredths(-1))); + for hostile in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY, 1e9, -1e300] { + assert_eq!(Hundredths::from_f64(hostile), None); + } + for text in [r#"{"tint": "warm"}"#, r#"{"tint": 1e300}"#, r#"{"exposure": null, "iso": -1}"#, r#"{"kelvin": 5600.5}"#, r#"{"hdrPeakNits": -1}"#] { + assert!(serde_json::from_str::(text).is_err(), "{text}"); + } + let as_shot: RawDevelop = serde_json::from_str(r#"{"iso": null, "tint": 0}"#).unwrap(); + assert_eq!(as_shot, RawDevelop { tint: Some(Hundredths(0)), ..Default::default() }, "an explicit 0 tint is an override, null is as shot"); + assert!(RAW_ISO_VALUES.windows(2).all(|w| w[0] < w[1])); + } + + /// Decode Using picks the clip's settings or the project's; the clip's decode quality applies + /// either way, else the project's. + #[test] + fn decode_using_picks_the_clip_or_the_project() { + let project = RawDevelop { quality: Some(RawQuality::HalfGood), ..RawDevelop::STANDARD }; + let own = RawDevelop { gamma: RawGamma::Srgb, iso: Some(400), ..Default::default() }; + let e = RawDevelop::effective(&project, Some(&own)); + assert_eq!((e.gamma, e.iso, e.quality(), e.decode_using), (RawGamma::Srgb, Some(400), RawQuality::HalfGood, DecodeUsing::Clip)); + let follows = RawDevelop { decode_using: DecodeUsing::Project, quality: Some(RawQuality::QuarterGood), ..own }; + let e = RawDevelop::effective(&project, Some(&follows)); + assert_eq!((e.gamma, e.iso, e.quality(), e.decode_using), (RawGamma::Bt1886, None, RawQuality::QuarterGood, DecodeUsing::Project)); + let e = RawDevelop::effective(&project, None); + assert_eq!(e, RawDevelop { decode_using: DecodeUsing::Project, ..project }); + assert_eq!(RawDevelop::effective(&RawDevelop::default(), None).gamma, RawGamma::Linear, "an old project keeps linear light"); + assert_eq!(RawQuality::HalfGood.divisor(), 2); + assert!(RawQuality::HalfGood.half_good() && !RawQuality::HalfPremium.half_good()); + } + + /// Each supported combination has an interpretation the compositor knows, and every one of + /// those is listed in INTERPRETATIONS; Rec. 709 needs none. + #[test] + fn every_supported_output_has_an_interpretation() { + for space in RawColorSpace::ALL { + for gamma in RawGamma::ALL { + let d = RawDevelop { color_space: space, gamma, ..Default::default() }; + assert_eq!(d.validate().is_ok(), RawDevelop::supports(space, gamma)); + if !RawDevelop::supports(space, gamma) { + continue; + } + match d.interpretation() { + None => assert_eq!(space, RawColorSpace::Rec709), + Some(cs) => assert!(RawDevelop::INTERPRETATIONS.contains(&cs), "{space:?} {gamma:?} {cs:?}"), + } + } + } + assert_eq!( + RawDevelop { color_space: RawColorSpace::Rec2020, gamma: RawGamma::St2084, ..Default::default() }.interpretation(), + Some(ColorSpace::Rec2100Pq) + ); + } +} diff --git a/crates/media/src/reader.rs b/crates/media/src/reader.rs index c3852edbf1..2deab0746c 100644 --- a/crates/media/src/reader.rs +++ b/crates/media/src/reader.rs @@ -13,6 +13,11 @@ use crate::{MediaError, Opener, Result, SharedSource}; /// Random-access, read-only bytes. pub trait ByteReader: Send + Sync { + /// Native file backing this reader, when available. Optional external decoders may use it; + /// memory, remote and browser readers must not manufacture a filesystem path. + fn local_path(&self) -> Option<&std::path::Path> { + None + } /// Total length in bytes. fn len(&self) -> u64; /// Fill `buf` entirely from `offset`; `UnexpectedEof` past the end. Asynchronous readers fail @@ -174,6 +179,9 @@ impl Drop for FileReader { #[cfg(any(unix, windows))] impl ByteReader for FileReader { + fn local_path(&self) -> Option<&std::path::Path> { + Some(&self.path) + } fn len(&self) -> u64 { self.len } diff --git a/crates/platform/Cargo.toml b/crates/platform/Cargo.toml index 9e07580669..64233573cc 100644 --- a/crates/platform/Cargo.toml +++ b/crates/platform/Cargo.toml @@ -21,6 +21,7 @@ type_complexity = "allow" undocumented_unsafe_blocks = "deny" [dependencies] +filmcraft-media = { workspace = true } filmcraft-bitstream = { workspace = true } filmcraft-codecs = { workspace = true } filmcraft-color = { workspace = true } @@ -32,6 +33,13 @@ filmcraft-isobmff = { workspace = true } filmcraft-time = { workspace = true } log = { workspace = true } +[features] +default = [] +red-sdk = ["dep:cc", "dep:libc"] + +[build-dependencies] +cc = { version = "1", optional = true } + [target.'cfg(target_os = "macos")'.dependencies] objc2-core-foundation = "0.3.2" objc2-core-media = "0.3.2" @@ -50,6 +58,8 @@ windows = { version = "0.62", features = [ "Win32_Media_MediaFoundation", "Win32_System_Com", "Win32_System_LibraryLoader", + "Win32_Security", + "Win32_System_Memory", "Win32_UI_WindowsAndMessaging", ] } @@ -57,6 +67,9 @@ windows = { version = "0.62", features = [ # wgpu), so building needs no libva headers and a system without libva simply decodes in software. [target.'cfg(target_os = "linux")'.dependencies] libloading = "0.8" +# The red-sdk helper's shared frame memory (`red::shm`): shm_open / mmap through libc (MIT OR +# Apache-2.0; already in the lockfile). +libc = { version = "0.2", optional = true } [dev-dependencies] filmcraft-testkit = { workspace = true } diff --git a/crates/platform/README.md b/crates/platform/README.md index 00371ba970..f86bcca894 100644 --- a/crates/platform/README.md +++ b/crates/platform/README.md @@ -3,6 +3,17 @@ OS media integration for FilmCraft (layer L5): hardware video decoding through the operating system's codecs, behind `filmcraft_codecs::VideoDecoder`, and hardware H.264 and H.265 (HEVC) encoding, behind `filmcraft_export::VideoEncoder`. It holds OS media FFI and nothing else. + +The optional `red-sdk` feature (Windows x64 and Linux x64, off by default; proposed in +[ADR 0002](../../docs/adr/0002-optional-vendor-sdk-helpers.md)) adds Nikon ZR R3D NE sources +through RED's R3D SDK, which the user supplies (`RED_R3D_SDK_DIR`). `build.rs` compiles the +original C++ helper `src/red/helper.cpp` against it; the SDK runs only in that helper process, two +of which serve every clip. `red` speaks the helper's FCRD9 protocol over pipes and reads decoded +frames in place from shared memory: `red::shm` is the backend's one `#[allow(unsafe_code)]` +module (a read-only view of a named section on Windows, of a POSIX shared memory object on +Linux), under the rules of ADR 0001. Without the feature none of it is compiled and nothing of +RED's is needed. Setup, behaviour, the protocol contract, tests and measurements: +[docs/r3d-ne.md](../../docs/r3d-ne.md). It is the one crate of the workspace allowed to contain `unsafe`, under the rules of [ADR 0001](../../docs/adr/0001-platform-ffi.md) and [AGENTS.md](../../AGENTS.md) §0.3. diff --git a/crates/platform/build.rs b/crates/platform/build.rs new file mode 100644 index 0000000000..b9f16d24b4 --- /dev/null +++ b/crates/platform/build.rs @@ -0,0 +1,84 @@ +fn main() -> Result<(), Box> { + println!("cargo:rerun-if-env-changed=RED_R3D_SDK_DIR"); + println!("cargo:rerun-if-changed=src/red/helper.cpp"); + #[cfg(feature = "red-sdk")] + build_red()?; + Ok(()) +} + +/// The helper is built for the target the SDK ships libraries for: Windows x64 (MSVC) and Linux +/// x64 (the system C++ compiler). Every other target fails with a clear message. +#[cfg(feature = "red-sdk")] +fn build_red() -> Result<(), Box> { + use std::{env, path::PathBuf}; + let target = env::var("TARGET")?; + let linux = target == "x86_64-unknown-linux-gnu"; + if target != "x86_64-pc-windows-msvc" && !linux { + return Err("red-sdk currently supports Windows x64 (MSVC) and Linux x64 (GNU) only".into()); + } + // MSVC does not accept canonicalize()'s Windows verbatim prefix in /I arguments. + let sdk = std::path::absolute(PathBuf::from(env::var("RED_R3D_SDK_DIR").map_err(|_| "Set RED_R3D_SDK_DIR to an extracted RED R3D SDK 9.2.1 folder")?))?; + let out = PathBuf::from(env::var_os("OUT_DIR").ok_or("OUT_DIR missing")?); + let header = sdk.join("Include/R3DSDK.h"); + println!("cargo:rerun-if-changed={}", header.display()); + let exe = if linux { build_red_linux(&sdk, &out)? } else { build_red_windows(&sdk, &out)? }; + println!("cargo:rustc-env=FILMCRAFT_BUILT_RED_HELPER={}", exe.display()); + Ok(()) +} + +#[cfg(feature = "red-sdk")] +fn build_red_windows(sdk: &std::path::Path, out: &std::path::Path) -> Result> { + let exe = out.join("filmcraft-red-helper.exe"); + let library = sdk.join("Lib/win64/R3DSDK-2017MD.lib"); + println!("cargo:rerun-if-changed={}", library.display()); + let compiler = cc::Build::new().cpp(true).get_compiler(); + let result = compiler + .to_command() + .args(["/nologo", "/std:c++17", "/EHsc", "/MD", "/O2", "/utf-8"]) + .arg(format!("/I{}", sdk.join("Include").display())) + .arg("src/red/helper.cpp") + .arg(format!("/Fo{}", out.join("red-helper.obj").display())) + .arg(format!("/Fe{}", exe.display())) + .arg(library) + .status()?; + if !result.success() { + return Err("RED helper compilation failed".into()); + } + for name in ["REDR3D-x64.dll", "REDDecoder-x64.dll", "REDCuda-x64.dll", "REDOpenCL-x64.dll"] { + let dll = sdk.join("Redistributable/win").join(name); + println!("cargo:rerun-if-changed={}", dll.display()); + std::fs::copy(dll, out.join(name))?; + } + Ok(exe) +} + +/// Linux: the position-independent C++11-ABI static library (the system g++ defaults to PIE +/// executables and the new libstdc++ ABI), plus what the SDK's notes ask for: pthreads, dl and +/// libuuid. The runtime `.so` files land beside the helper; the Rust side points +/// `LD_LIBRARY_PATH` at that directory so `REDR3D-x64.so` finds `REDCuda-x64.so` and the others. +#[cfg(feature = "red-sdk")] +fn build_red_linux(sdk: &std::path::Path, out: &std::path::Path) -> Result> { + let exe = out.join("filmcraft-red-helper"); + let library = sdk.join("Lib/linux64/libR3DSDKPIC-cpp11.a"); + println!("cargo:rerun-if-changed={}", library.display()); + let compiler = cc::Build::new().cpp(true).get_compiler(); + let result = compiler + .to_command() + .args(["-std=c++17", "-O2", "-fPIC", "-pthread", "-Wall"]) + .arg(format!("-I{}", sdk.join("Include").display())) + .arg("src/red/helper.cpp") + .arg("-o") + .arg(&exe) + .arg(library) + .args(["-ldl", "-luuid", "-pthread"]) + .status()?; + if !result.success() { + return Err("RED helper compilation failed".into()); + } + for name in ["REDR3D-x64.so", "REDDecoder-x64.so", "REDCuda-x64.so", "REDOpenCL-x64.so"] { + let so = sdk.join("Redistributable/linux").join(name); + println!("cargo:rerun-if-changed={}", so.display()); + std::fs::copy(so, out.join(name))?; + } + Ok(exe) +} diff --git a/crates/platform/src/lib.rs b/crates/platform/src/lib.rs index 8744da0eb3..8b88abd5e7 100644 --- a/crates/platform/src/lib.rs +++ b/crates/platform/src/lib.rs @@ -44,6 +44,8 @@ pub mod media_foundation; pub mod nvdec; #[cfg(any(target_os = "windows", all(target_os = "linux", target_pointer_width = "64")))] pub mod nvenc; +#[cfg(all(feature = "red-sdk", any(target_os = "windows", target_os = "linux"), target_arch = "x86_64"))] +pub mod red; #[cfg_attr(not(target_os = "linux"), allow(dead_code))] pub mod vaapi; #[cfg(target_os = "macos")] @@ -68,6 +70,8 @@ pub enum Availability { /// harmless). Streams they do not take, and every stream while hardware decoding is Off, keep /// using FilmCraft's own decoders. pub fn register() -> Availability { + #[cfg(all(feature = "red-sdk", any(target_os = "windows", target_os = "linux"), target_arch = "x86_64"))] + filmcraft_codecs::register_reader_opener(red::reader_opener); #[cfg(all(target_os = "linux", target_pointer_width = "64"))] filmcraft_export::register_encoder(nvenc::export::factory); #[cfg(target_os = "macos")] diff --git a/crates/platform/src/red/helper.cpp b/crates/platform/src/red/helper.cpp new file mode 100644 index 0000000000..1397fec76f --- /dev/null +++ b/crates/platform/src/red/helper.cpp @@ -0,0 +1,1196 @@ +// Original FilmCraft SDK adapter. Requires locally supplied RED headers/libraries. +// No SDK source or headers are redistributed here. See docs/r3d-ne.md. +// A separate process contains native faults in the vendor decoder. +#include +#include +#ifdef _WIN32 +#include +#include +#include +#include +#else +#include +#include +#include +#include +#include +#include +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +constexpr size_t MAX_PIXELS = 33554432; +constexpr size_t MAX_AUDIO = 1048576; +// Video requests the helper holds while every slot is busy (the app keeps slots + 1 in flight). +constexpr size_t MAX_WAITING = 64; +// A delivered frame the app has not handed back after this long is a fatal helper error, as a +// request that takes this long is a fatal error in the app. +constexpr auto SLOT_TIMEOUT = std::chrono::seconds(120); +void require(bool ok, const char* message) { if (!ok) throw std::runtime_error(message); } +// A protocol violation on stdin (a bad `F`): the state is no longer shared, so the helper ends. +struct fatal : std::runtime_error { using std::runtime_error::runtime_error; }; +// A bounded integer from the environment: unset means the default, anything else is an error. +size_t env_number(const char* name, size_t lo, size_t hi, size_t fallback) { + const char* value = std::getenv(name); + if (!value) return fallback; + char* end = nullptr; + const auto n = std::strtoull(value, &end, 10); + if (end == value || *end != 0 || n < lo || n > hi) + throw std::runtime_error(std::string(name) + " must be " + std::to_string(lo) + ".." + std::to_string(hi)); + return static_cast(n); +} +std::mutex output_mutex; +// Replies go down a named pipe the helper creates (FCRD5): a 31 MB half-resolution 6K frame +// through the 4 KB stdout pipe cost ~30 ms to write and ~20 ms to read, one page-sized hop +// at a time, and a decode slot stays busy until its reply is written. Since FCRD6 frames do not +// go down the pipe at all (see `Shared`); the pipe carries headers, metadata, errors and audio. +// Until the pipe is connected (start-up failures), stdout carries the same framing, so the app +// can still read the error. +// +// On Linux stdout is the reply pipe: the app's pipe grows to 1 MiB (`F_SETPIPE_SZ`, the +// unprivileged limit), nothing frame-sized goes down it since FCRD6, and the name argument +// only names the shared frame memory. +#ifdef _WIN32 +HANDLE output = INVALID_HANDLE_VALUE; +constexpr DWORD OUTPUT_BUFFER = 8u << 20; +void write_all(const void* bytes, size_t length) { + if (output == INVALID_HANDLE_VALUE) { + require(length == 0 || std::fwrite(bytes, 1, length, stdout) == length, "output closed"); + return; + } + const auto* p = static_cast(bytes); + while (length > 0) { + const DWORD chunk = static_cast(length < (1u << 30) ? length : (1u << 30)); + DWORD written = 0; + require(WriteFile(output, p, chunk, &written, nullptr) && written > 0 && written <= chunk, "output closed"); + p += written; + length -= written; + } +} +void flush_output() { + if (output == INVALID_HANDLE_VALUE) require(std::fflush(stdout) == 0, "output closed"); +} +#else +void write_all(const void* bytes, size_t length) { + require(length == 0 || std::fwrite(bytes, 1, length, stdout) == length, "output closed"); +} +void flush_output() { require(std::fflush(stdout) == 0, "output closed"); } +#endif +// FCRD6 framing: little-endian status:u32, request id:u64, length:u64, slot:u32, then the +// payload. Status 0: `length` payload bytes follow. Status 1: an error string follows. Status 2: +// no bytes follow; the frame is the first `length` bytes of shared slot `slot`. ID 0 is startup. +void reply(uint64_t id, uint32_t status, const void* bytes, uint64_t length, uint32_t slot = 0) { + std::lock_guard lock(output_mutex); + unsigned char header[24]; + std::memcpy(header, &status, 4); + std::memcpy(header + 4, &id, 8); + std::memcpy(header + 12, &length, 8); + std::memcpy(header + 20, &slot, 4); + write_all(header, sizeof header); + if (status != 2) write_all(bytes, static_cast(length)); + flush_output(); +} +// The name the app chose for this helper's reply pipe and shared frame memory: letters, digits +// and dashes, at most 64 (a UTF-16 argument on Windows, bytes elsewhere). +#ifdef _WIN32 +using Name = std::wstring; +#else +using Name = std::string; +#endif +Name checked_name(const Name::value_type* name) { + Name value; + size_t n = 0; + for (const auto* c = name; *c; ++c, ++n) { + require(n < 64 && ((*c >= 'a' && *c <= 'z') || (*c >= 'A' && *c <= 'Z') || (*c >= '0' && *c <= '9') || *c == '-'), "invalid reply pipe name"); + value += *c; + } + require(n > 0, "invalid reply pipe name"); + return value; +} +#ifdef _WIN32 +// The reply pipe `\\.\pipe\`: one instance, no remote clients, created before anything +// else can fail, so the app connects as soon as the process is up. +void open_output(const Name& name) { + std::wstring path = L"\\\\.\\pipe\\" + name; + HANDLE pipe = CreateNamedPipeW(path.c_str(), PIPE_ACCESS_OUTBOUND | FILE_FLAG_FIRST_PIPE_INSTANCE, + PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT | PIPE_REJECT_REMOTE_CLIENTS, 1, OUTPUT_BUFFER, 4096, 0, nullptr); + require(pipe != INVALID_HANDLE_VALUE, "cannot create the reply pipe"); + if (!ConnectNamedPipe(pipe, nullptr) && GetLastError() != ERROR_PIPE_CONNECTED) { + CloseHandle(pipe); + throw std::runtime_error("the app did not connect to the reply pipe"); + } + output = pipe; +} +#else +// stdout is the reply pipe; widen it (best effort: a kernel that refuses keeps 64 KiB, which +// only costs an audio reply a few more hops). +void open_output(const Name&) { fcntl(STDOUT_FILENO, F_SETPIPE_SZ, 1 << 20); } +#endif +void error(uint64_t id, const char* message) { reply(id, 1, message, std::strlen(message)); } +#ifdef _WIN32 +std::string utf8(const wchar_t* input) { + int n = WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS, input, -1, nullptr, 0, nullptr, nullptr); + require(n > 0, "invalid Unicode path"); + std::string value(static_cast(n), '\0'); + require(WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS, input, -1, value.data(), n, nullptr, nullptr) == n, "invalid Unicode path"); + value.pop_back(); + return value; +} +#endif +struct Sdk { + explicit Sdk(const char* path, bool gpu) { + auto status = R3DSDK::InitializeSdk(path, gpu ? OPTION_RED_DECODER : OPTION_RED_NONE); + if (status != R3DSDK::ISInitializeOK) { + R3DSDK::FinalizeSdk(); + throw std::runtime_error("RED SDK initialization failed; check the 9.2.1 runtime libraries beside the helper or RED_R3D_LIBRARY_PATH"); + } + } + // SDK 9.2.1 on Linux faults inside FinalizeSdk() once the managed CUDA decoder has processed + // a frame (deterministic; releasing the jobs, the decoder and the clip first, or waiting, + // does not help). The helper process ends right after, so there it is left unfinalized. + bool finalize = true; + ~Sdk() { if (finalize) R3DSDK::FinalizeSdk(); } +}; +// 512-byte aligned decode buffers (the SDK wants at least 16; sector alignment suits the +// output pipe), through each C runtime's own aligned allocator. +void* aligned_allocate(size_t bytes) { +#ifdef _WIN32 + return _aligned_malloc(bytes, 512); +#else + void* data = nullptr; + return posix_memalign(&data, 512, bytes) == 0 ? data : nullptr; +#endif +} +void aligned_release(void* data) { +#ifdef _WIN32 + _aligned_free(data); +#else + std::free(data); +#endif +} +// A half-float NaN in the last pixel before each decode: a decode that writes fewer rows than +// the size rule says (a reused slot would otherwise still hold the previous frame there) is +// rejected by the app's non-finite check instead of being shown. The 16-bit integer outputs +// read it as a code value, so there it only replaces a stale pixel with white. +void sentinel(void* buffer, size_t bytes) { + if (bytes >= 6) std::memset(static_cast(buffer) + bytes - 6, 0xFF, 6); +} +struct Aligned { + std::unique_ptr data; + explicit Aligned(size_t bytes) : data(aligned_allocate(bytes), &aligned_release) { + require(data != nullptr, "decode buffer allocation failed"); + } +}; +// The frame slots the app reads in place (FCRD6): a pagefile-backed section `Local\-frames` +// (Windows) or a POSIX shared memory object `/-frames` (Linux, a tmpfs file under +// /dev/shm) of `slots` x `slot_bytes` (the clip's full-resolution frame, rounded to pages), +// created before the metadata reply names it. The SDK decodes straight into a slot, the reply +// carries the slot index, and the app hands the slot back with `F `. Pages cost physical +// memory only once touched, so a clip played at half resolution never backs the full-resolution +// tail of a slot. The memory outlives this process for as long as the app keeps its view, so a +// helper that dies mid-frame leaves the app reading valid memory. +struct Shared { +#ifdef _WIN32 + HANDLE mapping = nullptr; +#else + std::string path; + size_t total = 0; +#endif + unsigned char* base = nullptr; + size_t slots = 0, slot_bytes = 0; + Shared(const Name& name, size_t count, size_t bytes) : slots(count), slot_bytes((bytes + 4095) / 4096 * 4096) { + require(count > 0 && slot_bytes > 0 && slot_bytes <= SIZE_MAX / count, "frame slots too large"); + const uint64_t total = static_cast(slot_bytes) * count; +#ifdef _WIN32 + const std::wstring path = L"Local\\" + name + L"-frames"; + mapping = CreateFileMappingW(INVALID_HANDLE_VALUE, nullptr, PAGE_READWRITE, static_cast(total >> 32), static_cast(total), path.c_str()); + require(mapping != nullptr, "cannot create the shared frame memory"); + if (GetLastError() == ERROR_ALREADY_EXISTS) { + CloseHandle(mapping); + mapping = nullptr; + throw std::runtime_error("the shared frame memory name is taken"); + } + base = static_cast(MapViewOfFile(mapping, FILE_MAP_ALL_ACCESS, 0, 0, static_cast(total))); + if (!base) { + CloseHandle(mapping); + mapping = nullptr; + throw std::runtime_error("cannot map the shared frame memory"); + } +#else + // Created exclusively (a taken name is an error, never reused), sized once, mapped + // shared, and never resized afterwards: the app checks the size before it maps. + require(total <= static_cast(INT64_MAX), "frame slots too large"); + path = "/" + name + "-frames"; + const int fd = shm_open(path.c_str(), O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0600); + require(fd >= 0, errno == EEXIST ? "the shared frame memory name is taken" : "cannot create the shared frame memory"); + if (ftruncate(fd, static_cast(total)) != 0) { + close(fd); + shm_unlink(path.c_str()); + throw std::runtime_error("cannot size the shared frame memory"); + } + void* view = mmap(nullptr, static_cast(total), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + close(fd); + if (view == MAP_FAILED) { + shm_unlink(path.c_str()); + throw std::runtime_error("cannot map the shared frame memory"); + } + base = static_cast(view); + this->total = static_cast(total); +#endif + } + Shared(const Shared&) = delete; + Shared& operator=(const Shared&) = delete; + ~Shared() { +#ifdef _WIN32 + if (base) UnmapViewOfFile(base); + if (mapping) CloseHandle(mapping); +#else + if (base) munmap(base, total); + // The app removes the name once it has mapped the object; a helper that ends before + // that (a start-up failure) removes it here, and a name already gone is fine. + shm_unlink(path.c_str()); +#endif + } + unsigned char* slot(size_t index) { return base + index * slot_bytes; } +}; + +void gpu_ok(R3DSDK::R3DStatus status, const char* operation) { + if (status != R3DSDK::R3DStatus_Ok) + throw std::runtime_error(std::string(operation) + " (RED GPU status " + std::to_string(status) + ")"); +} +struct Gpu { + std::unique_ptr decoder{nullptr, &R3DSDK::R3DDecoder::ReleaseDecoder}; + std::string description; + Gpu() { + R3DSDK::R3DDecoderOptions* raw = nullptr; + gpu_ok(R3DSDK::R3DDecoderOptions::CreateOptions(&raw), "Create RED GPU options"); + std::unique_ptr options(raw, &R3DSDK::R3DDecoderOptions::ReleaseOptions); + require(raw != nullptr, "RED GPU options missing"); + std::vector cuda; + std::vector opencl; + if (R3DSDK::R3DDecoderOptions::GetCudaDeviceList(cuda) == R3DSDK::R3DStatus_Ok && !cuda.empty()) { + gpu_ok(options->useDevice(cuda.front()), "Select RED CUDA device"); + description = "CUDA " + std::string(cuda.front().name, strnlen(cuda.front().name, sizeof(cuda.front().name))); + } else { + gpu_ok(R3DSDK::R3DDecoderOptions::GetOpenCLDeviceList(opencl), "Enumerate RED OpenCL devices"); + require(!opencl.empty(), "R3D NE requires a compatible CUDA or OpenCL GPU; none found"); + gpu_ok(options->useDevice(opencl.front()), "Select RED OpenCL device"); + description = "OpenCL " + std::string(opencl.front().name, strnlen(opencl.front().name, sizeof(opencl.front().name))); + } + // Keep the SDK default image-thread count; one image thread serializes all work. + // Three GPU streams measured best (3–4 frames in flight: 25–27 fps at half resolution on + // a current desktop GPU; more buys about one frame). The knobs are for bake-offs and smaller cards. + gpu_ok(options->setGPUConcurrentFrameCount(env_number("FILMCRAFT_R3D_GPU_FRAMES", 1, 8, 3)), "Set RED GPU concurrency"); + gpu_ok(options->setMemoryPoolSize(env_number("FILMCRAFT_R3D_HOST_POOL_MB", 1024, 65536, 8192)), "Set RED host memory limit"); + gpu_ok(options->setGPUMemoryPoolSize(env_number("FILMCRAFT_R3D_GPU_POOL_MB", 1024, 65536, 6144)), "Set RED GPU memory limit"); + R3DSDK::R3DDecoder* instance = nullptr; + gpu_ok(R3DSDK::R3DDecoder::CreateDecoder(options.get(), &instance), "Create RED GPU decoder"); + decoder.reset(instance); + require(instance != nullptr, "RED GPU decoder missing"); + } +}; + +// One clip the app has opened in this helper (FCRD8: one helper serves every clip). The SDK's +// `Clip` is cheap to keep (an index, ~6 ms to open inside a running SDK against a helper process +// start of 0.2-0.8 s), so a timeline of a hundred clips costs one process and its pools, not a +// hundred. Requests hold their clip's `shared_ptr`, so closing it (`C`) while a frame is in +// flight frees it once that frame is written. +struct ClipState { + std::unique_ptr clip; + size_t width = 0, height = 0, channels = 0; + uint64_t frames = 0, samples = 0; + int64_t num = 0, den = 0, rate = 0, floating = 0, bits = 0; + // The lens squeeze the camera recorded (its desqueeze factor, 1.33 for a 1.33x anamorphic + // lens), or 0 when the clip records none. + double squeeze = 0; + // CPU decodes of one clip run one at a time (the SDK's clip object is not documented as + // safe for concurrent decodes), and so does an RMD write (`W`); different clips decode in + // parallel. + std::mutex decode; + // The clip's recorded IPP2 settings, or its RMD sidecar's when it has one (the SDK's + // GetDefaultImageProcessingSettings); each request develops a copy (`develop`). Replaced + // after an RMD write, so read and written under `settings_mutex` (`settings()`). + std::mutex settings_mutex; + R3DSDK::ImageProcessingSettings base; + bool rmd = false; + std::string rmd_path; + size_t tracks = 1; + bool lut = false; + R3DSDK::ImageProcessingSettings settings() { + std::lock_guard lock(settings_mutex); + return base; + } +}; +// The desqueeze factor a clip records: `pixel_aspect_ratio` (1.0 normally), else the older +// `anamorphic_mode` code; 0 when neither is there. +double recorded_squeeze(const R3DSDK::Clip& clip) { + if (clip.MetadataExists(R3DSDK::RMD_PIXEL_ASPECT_RATIO)) { + return clip.MetadataItemType(R3DSDK::RMD_PIXEL_ASPECT_RATIO) == R3DSDK::MetadataTypeFloat + ? static_cast(clip.MetadataItemAsFloat(R3DSDK::RMD_PIXEL_ASPECT_RATIO)) + : static_cast(clip.MetadataItemAsInt(R3DSDK::RMD_PIXEL_ASPECT_RATIO)); + } + if (clip.MetadataExists(R3DSDK::RMD_ANAMORPHIC_MODE) && clip.MetadataItemType(R3DSDK::RMD_ANAMORPHIC_MODE) == R3DSDK::MetadataTypeInt) { + // 0 = none, 1 = 2x, 2 = 1.3x, 3 = 1.25x, 4 = 1.6x, 5 = 1.8x, 6 = 1.5x + static const double modes[] = {1.0, 2.0, 1.3, 1.25, 1.6, 1.8, 1.5}; + const unsigned int mode = clip.MetadataItemAsInt(R3DSDK::RMD_ANAMORPHIC_MODE); + if (mode < sizeof(modes) / sizeof(modes[0])) return modes[mode]; + } + return 0; +} +// (Re)read what the clip records: its settings (the RMD sidecar's when one is present), whether +// it has a sidecar, an HDRx track or a creative LUT. `written`: an RMD was just written (the SDK's +// RmdSidecarPresent says what it found when the clip was loaded, not since; measured). +void describe(ClipState& c, bool written = false) { + R3DSDK::ImageProcessingSettings settings; + c.clip->GetDefaultImageProcessingSettings(settings); + const char* path = c.clip->GetRmdPath(); + std::lock_guard lock(c.settings_mutex); + c.base = settings; + c.rmd = written || c.clip->RmdSidecarPresent(); + c.rmd_path = path ? path : ""; + c.tracks = c.clip->VideoTrackCount(); + c.lut = c.clip->Camera3DLutPresent() || settings.Lut3D != nullptr; +} +// Open `path` as a clip of this helper, or throw why not. `slot_bytes` (GPU helpers) bounds the +// full-resolution frame the shared slots can hold. +std::shared_ptr open_clip(const std::string& path, size_t slot_bytes) { + require(R3DSDK::IdentifyFile(path.c_str()) == R3DSDK::FileId_R3D_NE, + "This backend supports Nikon ZR R3D NE only; file is unsupported, damaged, or unreadable"); + auto state = std::make_shared(); + state->clip = std::make_unique(path.c_str()); + auto& clip = *state->clip; + require(clip.Status() == R3DSDK::LSClipLoaded, "RED SDK could not open the R3D NE clip"); + const size_t w = clip.Width(), h = clip.Height(), channels = clip.AudioChannelCount(); + require(w > 0 && h > 0 && w <= 8192 && h <= 8192 && w <= MAX_PIXELS / h, "unsupported R3D dimensions"); + require(channels <= 16, "unsupported R3D channel count"); + require(slot_bytes == 0 || w * h * 6 <= slot_bytes, "clip larger than the helper's frame slots"); + state->width = w; + state->height = h; + state->channels = channels; + state->frames = clip.VideoFrameCount(); + state->samples = clip.AudioSampleCount(); + state->num = clip.MetadataItemAsInt(R3DSDK::RMD_FRAMERATE_NUMERATOR); + state->den = clip.MetadataItemAsInt(R3DSDK::RMD_FRAMERATE_DENOMINATOR); + state->rate = clip.MetadataItemAsInt(R3DSDK::RMD_SAMPLERATE); + state->floating = clip.MetadataItemAsInt(R3DSDK::RMD_AUDIO_FORMAT); + state->bits = clip.MetadataItemAsInt(R3DSDK::RMD_SAMPLE_SIZE); + require(state->frames > 0 && state->num > 0 && state->den > 0, "invalid R3D timing metadata"); + require(channels == 0 || (state->rate > 0 && state->rate <= 384000 && state->floating <= 1 && (state->bits == 24 || state->bits == 32)), + "invalid R3D audio metadata"); + describe(*state); + state->squeeze = recorded_squeeze(clip); + return state; +} +// The open clips by the app's handle. +std::mutex clips_mutex; +std::map> clips; +constexpr size_t MAX_CLIPS = 4096; +std::shared_ptr find_clip(uint64_t handle) { + std::lock_guard lock(clips_mutex); + const auto it = clips.find(handle); + require(it != clips.end(), "unknown clip handle (reopen the clip)"); + return it->second; +} + +// Slots own every object the SDK accesses until the callback and the reply finish, and a slot +// stays the app's until it is freed. Only this fixed ring is touched by callbacks: no pipe I/O +// or allocation there. The stdin thread never waits for a slot (the command that frees one +// arrives on stdin): video requests queue here and the pipeline thread submits them as slots +// free, between writing the replies. +struct Pipeline { + enum class State { Free, Decoding, Delivered }; + struct Slot { + Pipeline* owner = nullptr; + size_t index = 0; + State state = State::Free; + uint64_t id = 0; + // The clip this slot decodes: held until the reply is written, so a clip the app closes + // (`C`) mid-frame stays alive for the SDK's callback. + std::shared_ptr clip; + size_t frame = 0; + uint64_t divisor = 0; + size_t bytes = 0; + // This request's development settings: the job points here until its callback, so + // requests in flight with different settings never share one structure. + R3DSDK::ImageProcessingSettings settings; + R3DSDK::R3DStatus status = R3DSDK::R3DStatus_ErrorProcessing; + std::chrono::steady_clock::time_point submitted, completed_at, written_at; + std::unique_ptr job{nullptr, &R3DSDK::R3DDecoder::ReleaseDecodeJob}; + }; + struct Request { + uint64_t id = 0; + std::shared_ptr clip; + size_t frame = 0; + uint64_t divisor = 0; + size_t bytes = 0, width = 0; + R3DSDK::VideoDecodeMode mode = R3DSDK::DECODE_HALF_RES_PREMIUM; + // Developed per request (`develop`): copied into the slot the request decodes in. + R3DSDK::ImageProcessingSettings settings; + R3DSDK::VideoPixelType pixel_type = R3DSDK::PixelType_HalfFloat_RGB_Interleaved; + }; + Gpu& gpu; + Shared& shared; + std::array slots; + std::array ready{}; + size_t count = 4, head = 0, queued = 0; + std::deque waiting; + bool stopping = false; + std::chrono::steady_clock::time_point stop_requested{}; + const bool timing = std::getenv("FILMCRAFT_R3D_TIMING") != nullptr; + std::mutex mutex; + std::condition_variable changed; + std::thread worker; + + Pipeline(Gpu& device, Shared& memory) : gpu(device), shared(memory) { + count = memory.slots; + require(count >= 1 && count <= slots.size(), "unsupported slot count"); + for (size_t i = 0; i < count; ++i) { + auto& slot = slots[i]; + slot.owner = this; + slot.index = i; + R3DSDK::R3DDecodeJob* raw = nullptr; + gpu_ok(R3DSDK::R3DDecoder::CreateDecodeJob(&raw), "Create RED GPU job"); + slot.job.reset(raw); + require(raw != nullptr, "RED GPU job missing"); + } + worker = std::thread([this] { + try { run(); } + catch (const std::exception& e) { + // A failed pipe cannot report another reply, and a slot the app never freed + // leaves the pipeline inconsistent. Report what can be reported and exit the + // isolated helper without unwinding buffers SDK callbacks still reference; the + // app fails the worker and reopens the clip. + try { error(0, e.what()); } catch (...) {} + std::_Exit(1); + } catch (...) { std::_Exit(1); } + }); + } + ~Pipeline() { + { + std::lock_guard lock(mutex); + stopping = true; + changed.notify_all(); + } + if (worker.joinable()) worker.join(); + } + // Five slots: the SDK keeps three or four frames in flight while the app repacks one. + static size_t slot_count() { return env_number("FILMCRAFT_R3D_SLOTS", 1, 8, 5); } + static void completed(R3DSDK::R3DDecodeJob* job, R3DSDK::R3DStatus status) noexcept { + try { + auto& slot = *static_cast(job->privateData); + auto& self = *slot.owner; + std::lock_guard lock(self.mutex); + if (self.queued >= self.count) std::_Exit(1); + slot.status = status; + slot.completed_at = std::chrono::steady_clock::now(); + self.ready[(self.head + self.queued) % self.count] = slot.index; + ++self.queued; + self.changed.notify_all(); + } catch (...) { std::_Exit(1); } + } + Slot* free_slot() { + for (size_t i = 0; i < count; ++i) { + if (slots[i].state == State::Free) return &slots[i]; + } + return nullptr; + } + bool decoding() const { + for (size_t i = 0; i < count; ++i) { + if (slots[i].state == State::Decoding) return true; + } + return false; + } + // From the stdin thread: queue a video request (the pipeline thread submits it). + void enqueue(const Request& request) { + std::lock_guard lock(mutex); + require(waiting.size() < MAX_WAITING, "too many video requests outstanding"); + waiting.push_back(request); + changed.notify_all(); + } + // From the stdin thread: the app has finished with a delivered slot. + void release(uint64_t index) { + std::lock_guard lock(mutex); + if (index >= count || slots[index].state != State::Delivered) throw fatal("freed a frame slot that was not the app's"); + auto& slot = slots[index]; + slot.state = State::Free; + if (timing) + std::fprintf(stderr, "R3D slot %zu held %.3f ms\n", slot.index, + std::chrono::duration(std::chrono::steady_clock::now() - slot.written_at).count()); + changed.notify_all(); + } + void run() { + std::unique_lock lock(mutex); + for (;;) { + changed.wait_for(lock, std::chrono::seconds(1), [&] { return queued != 0 || stopping || (!waiting.empty() && free_slot() != nullptr); }); + const auto now = std::chrono::steady_clock::now(); + for (size_t i = 0; i < count; ++i) { + if (slots[i].state == State::Delivered && now - slots[i].written_at > SLOT_TIMEOUT) + throw std::runtime_error("the app held a decoded RED frame for 120 seconds"); + } + while (queued != 0) { + auto& slot = slots[ready[head]]; + head = (head + 1) % count; + --queued; + // The slot is the app's (or free again) before the reply goes out: the app's + // `F` can arrive the moment the header lands, ahead of this thread retaking the + // lock, and must find the slot delivered. + const bool ok = slot.status == R3DSDK::R3DStatus_Ok; + slot.written_at = std::chrono::steady_clock::now(); + slot.state = ok ? State::Delivered : State::Free; + // the SDK is done with the clip; a clip closed meanwhile goes now + auto finished = std::move(slot.clip); + lock.unlock(); + finished.reset(); + if (ok) + reply(slot.id, 2, nullptr, slot.bytes, static_cast(slot.index)); + else + error(slot.id, ("RED GPU frame processing failed (RED GPU status " + std::to_string(slot.status) + ")").c_str()); + if (timing) { + // Queue wait inside the SDK and its decode are not separable from here; the + // header write is; the app's hold of the slot is logged when it is freed. + const auto written = std::chrono::steady_clock::now(); + std::fprintf(stderr, "R3D frame %zu divisor %llu submit-to-callback %.3f ms, callback-to-written %.3f ms\n", slot.frame, + static_cast(slot.divisor), + std::chrono::duration(slot.completed_at - slot.submitted).count(), + std::chrono::duration(written - slot.completed_at).count()); + } + lock.lock(); + } + while (!stopping && !waiting.empty()) { + Slot* slot = free_slot(); + if (!slot) break; + const Request request = waiting.front(); + waiting.pop_front(); + slot->state = State::Decoding; + slot->id = request.id; + slot->frame = request.frame; + slot->divisor = request.divisor; + slot->bytes = request.bytes; + slot->settings = request.settings; + slot->clip = request.clip; + sentinel(shared.slot(slot->index), request.bytes); + auto& job = *slot->job; + job.clip = slot->clip->clip.get(); + job.videoTrackNo = 0; + job.videoFrameNo = request.frame; + job.callback = completed; + job.privateData = slot; + job.mode = request.mode; + job.pixelType = request.pixel_type; + job.outputBuffer = shared.slot(slot->index); + job.outputBufferSize = request.bytes; + job.bytesPerRow = request.width * 6; // informational: the SDK packs rows at floor(w / divisor) regardless + job.imageProcessingSettings = &slot->settings; + job.outputFrameMetadata = nullptr; + slot->submitted = std::chrono::steady_clock::now(); + lock.unlock(); + // Submitted without the lock: a rejected submission has no callback and is + // routed through `completed`, which takes it. + const auto status = gpu.decoder->decode(&job); + if (status != R3DSDK::R3DStatus_Ok) completed(&job, status); + lock.lock(); + } + // EOF: finish what the SDK still holds (the callbacks reference the slots and the + // section), then let the buffers go. Frames the app still holds need nothing. A + // decode the SDK never completes (a starved GPU: helpers of an app killed with + // a hundred of them alive outlived it by hours) must not keep the process, its + // memory and its CUDA context alive: after ten seconds exit without unwinding. + if (stopping) { + if (queued == 0 && !decoding()) return; + if (stop_requested == std::chrono::steady_clock::time_point{}) stop_requested = now; + if (now - stop_requested > std::chrono::seconds(10)) { + std::fprintf(stderr, "filmcraft-red-helper: a decode did not complete within 10 s of shutdown; exiting without it\n"); + std::_Exit(0); + } + } + } + } +}; + +// The development fields of a request (FCRD9), as small integers the app sends and this helper +// range-checks (anything else is a malformed request, answered with an error, never applied): +// half 0 = premium, 1 = good (a half-resolution decode); space 0 Rec.709, 1 Rec.2020, 2 P3 D65, +// 3 REDWideGamutRGB; gamma 0 linear, 1 BT.1886, 2 sRGB, 3 Log3G10, 4 ST 2084, 5 Hybrid Log-Gamma; +// tone map 0 = as recorded, 1 None, 2 Low, 3 Medium, 4 High; rolloff 0 = as recorded, 1 None, +// 2 Hard, 3 Medium, 4 Soft, 5 Very Soft; peak 0 = as recorded, else cd/m² within the SDK's +// HdrPeakNits limits; options a flag word (1 chroma noise reduction set, 2 its value, 4 apply the +// metadata curves, 8 apply the creative LUT, 16 apply the CDL); flashing 0 = as recorded, 1 Off, +// 2 Mild, 3 Medium, 4 Strong. Then the RAW adjustments (`adjust`). +struct Develop { + uint64_t half = 0, space = 0, gamma = 0, tonemap = 0, rolloff = 0, peak = 0, options = 0, flashing = 0; + uint64_t flags = 0, iso = 0, kelvin = 0; + int64_t tint = 0, exposure = 0; +}; +bool read_develop(std::istringstream& in, Develop& d, bool with_half) { + if (with_half) in >> d.half; + in >> d.space >> d.gamma >> d.tonemap >> d.rolloff >> d.peak >> d.options >> d.flashing >> d.flags >> d.iso >> d.kelvin >> d.tint >> d.exposure; + return !in.fail(); +} +// The colour space and gamma curve go together the way the app's `RawDevelop::supports` says: +// display curves with display primaries, Log3G10 with REDWideGamutRGB, the HDR curves with +// Rec.2020, linear light with any of them. +bool supported(uint64_t space, uint64_t gamma) { + switch (gamma) { + case 0: return space <= 3; + case 1: case 2: return space <= 2; + case 3: return space == 3; + case 4: case 5: return space == 1; + default: return false; + } +} +const float IDENTITY_CURVE[10] = {0.0f, 0.0f, 0.25f, 0.25f, 0.5f, 0.5f, 0.75f, 0.75f, 1.0f, 1.0f}; +// Development of one request, from the clip's recorded IPP2 settings (or its RMD's; as-shot ISO, +// white balance and exposure are kept unless `adjust` replaces them): the colour space and gamma +// curve, and for a display curve the tone map, highlight rolloff and HDR peak (0 keeps the +// recorded choice); the processing options. Anything out of range is a malformed request. +// +// The pixel type follows the gamma: the SDK's half-float output is always linear light (its header +// says the gamma curve is ignored, and with it the IPP2 output transform, the curves and the CDL), +// so linear decodes as half floats in the chosen primaries and every other curve as 16-bit +// integer RGB, 0..65535 code values of the curve, which the app linearises. The graded pipeline +// with nothing graded is used for a log output: the GPU decoder rejects Primary_Development_Only +// (InvalidJobParameter). Both pixel types are 6 bytes a pixel. +R3DSDK::VideoPixelType develop(R3DSDK::ImageProcessingSettings& s, const Develop& d) { + using Limits = R3DSDK::ImageProcessingLimits; + static const R3DSDK::ImageColorSpace spaces[] = {R3DSDK::ImageColorRec709, R3DSDK::ImageColorRec2020, R3DSDK::ImageColorDCIP3D65, R3DSDK::ImageColorREDWideGamutRGB}; + static const R3DSDK::ImageGammaCurve gammas[] = {R3DSDK::ImageGammaLinear, R3DSDK::ImageGammaBT1886, R3DSDK::ImageGammaSRGB, + R3DSDK::ImageGammaLog3G10, R3DSDK::ImageGammaHDR2084, R3DSDK::ImageGammaHybridLogGamma}; + static const R3DSDK::ToneMap tone_maps[] = {R3DSDK::ToneMap_None, R3DSDK::ToneMap_Low, R3DSDK::ToneMap_Medium, R3DSDK::ToneMap_High}; + static const R3DSDK::RollOff roll_offs[] = {R3DSDK::RollOff_None, R3DSDK::RollOff_Hard, R3DSDK::RollOff_Medium, R3DSDK::RollOff_Soft, R3DSDK::RollOff_Very_Soft}; + static const R3DSDK::FlashingPixelAdjust flashing[] = {R3DSDK::FlashingPixelAdjustOff, R3DSDK::FlashingPixelAdjustMild, R3DSDK::FlashingPixelAdjustMedium, + R3DSDK::FlashingPixelAdjustStrong}; + require(d.half <= 1 && d.space <= 3 && d.gamma <= 5 && d.tonemap <= 4 && d.rolloff <= 5 && d.options <= 31 && d.flashing <= 4, + "unsupported development setting"); + require(supported(d.space, d.gamma), "unsupported colour space and gamma curve"); + require((d.options & 1) || !(d.options & 2), "malformed chroma noise reduction"); + if (d.peak != 0) { + if (!(d.peak >= Limits::HdrPeakNitsMin && d.peak <= Limits::HdrPeakNitsMax)) throw std::runtime_error("HDR peak out of range: " + std::to_string(d.peak)); + s.HdrPeakNits = static_cast(d.peak); + } + s.Version = R3DSDK::ColorVersion3; + s.ImagePipelineMode = R3DSDK::Full_Graded; + s.ColorSpace = spaces[d.space]; + s.GammaCurve = gammas[d.gamma]; + if (d.options & 1) s.ChromaNoiseReduction = (d.options & 2) != 0; + if (d.flashing != 0) s.FlashingPixelAdjustment = flashing[d.flashing - 1]; + if (!(d.options & 4)) { + for (int i = 0; i < 10; ++i) { + s.UserCurve[i] = IDENTITY_CURVE[i]; + s.UserCurveRed[i] = IDENTITY_CURVE[i]; + s.UserCurveGreen[i] = IDENTITY_CURVE[i]; + s.UserCurveBlue[i] = IDENTITY_CURVE[i]; + } + } + if (d.gamma == 0) { + s.OutputToneMap = R3DSDK::ToneMap_None; + s.HighlightRollOff = R3DSDK::RollOff_None; + s.Lut3DEnabled = false; + s.CdlEnabled = false; + return R3DSDK::PixelType_HalfFloat_RGB_Interleaved; + } + if (d.gamma == 3) { + s.OutputToneMap = R3DSDK::ToneMap_None; + s.HighlightRollOff = R3DSDK::RollOff_None; + } else { + if (d.tonemap != 0) s.OutputToneMap = tone_maps[d.tonemap - 1]; + if (d.rolloff != 0) s.HighlightRollOff = roll_offs[d.rolloff - 1]; + } + s.Lut3DEnabled = (d.options & 8) != 0 && s.Lut3D != nullptr; + s.CdlEnabled = (d.options & 16) != 0; + return R3DSDK::PixelType_16Bit_RGB_Interleaved; +} + +// The RAW adjustments of one request (FCRD7), applied before development: `flags` says which of +// ISO (bit 0), Kelvin (bit 1), tint (bit 2) and exposure (bit 3) replace the clip's recorded +// value; a clear bit keeps it and its field must be 0. Tint and exposure arrive in hundredths. +// Every value is checked against the SDK's own limits (ImageProcessingLimits, so a future SDK's +// wider range is honoured by this check and refused by the app's first): anything else is a +// malformed request, answered with an error, never applied. +void adjust(R3DSDK::ImageProcessingSettings& s, uint64_t flags, uint64_t iso, uint64_t kelvin, int64_t tint, int64_t exposure) { + using Limits = R3DSDK::ImageProcessingLimits; + require(flags <= 15, "unsupported RAW adjustment flags"); + if (flags & 1) { + bool listed = false; + for (size_t i = 0; i < Limits::ISOCount; ++i) listed = listed || Limits::ISOList[i] == iso; + if (!listed) throw std::runtime_error("unsupported ISO " + std::to_string(iso)); + s.ISO = static_cast(iso); + } else { + require(iso == 0, "malformed RAW adjustment"); + } + if (flags & 2) { + const double k = static_cast(kelvin); + if (!(k >= Limits::KelvinMin && k <= Limits::KelvinMax)) throw std::runtime_error("colour temperature out of range: " + std::to_string(kelvin) + " K"); + s.Kelvin = static_cast(k); + } else { + require(kelvin == 0, "malformed RAW adjustment"); + } + if (flags & 4) { + const double t = static_cast(tint) / 100.0; + if (!(t >= Limits::TintMin && t <= Limits::TintMax)) throw std::runtime_error("tint out of range: " + std::to_string(t)); + s.Tint = static_cast(t); + } else { + require(tint == 0, "malformed RAW adjustment"); + } + if (flags & 8) { + const double e = static_cast(exposure) / 100.0; + if (!(e >= Limits::ExposureAdjustMin && e <= Limits::ExposureAdjustMax)) throw std::runtime_error("exposure adjustment out of range: " + std::to_string(e) + " stops"); + s.ExposureAdjust = static_cast(e); + } else { + require(exposure == 0, "malformed RAW adjustment"); + } +} + +std::string clip_metadata(ClipState& c, size_t slots, size_t slot_bytes, const std::string& backend); +// CPU work of the helper: audio of any clip, and video when there is no GPU pipeline (the CPU +// helper serves thumbnails and the frames of a GPU that is unavailable). A few threads take it +// from two queues, audio first, so a thumbnail never makes the sound wait; the stdin thread only +// queues, so a slot release (`F`) is never stuck behind a decode. Different clips decode in +// parallel; one clip's decodes take turns (`ClipState::decode`). +struct CpuWork { + struct Job { + uint64_t id = 0; + std::shared_ptr clip; + bool audio = false; + // a clip open (`O`): opened on a pool thread, so a card's worth of opens at import run + // several at a time instead of one after another on the stdin thread (~5 ms each) + bool open = false; + // an RMD sidecar write (`W`): on a pool thread, holding the clip's decode lock + bool rmd = false; + Develop develop; + uint64_t handle = 0; + std::string path; + size_t slots = 0, slot_bytes = 0; + std::string backend; + uint64_t start = 0, count = 0; + // video + R3DSDK::VideoDecodeMode mode = R3DSDK::DECODE_HALF_RES_PREMIUM; + R3DSDK::VideoPixelType pixel_type = R3DSDK::PixelType_HalfFloat_RGB_Interleaved; + R3DSDK::ImageProcessingSettings settings; + size_t bytes = 0; + }; + std::mutex mutex; + std::condition_variable changed; + std::deque audio, video; + bool stopping = false; + std::vector threads; + const bool timing = std::getenv("FILMCRAFT_R3D_TIMING") != nullptr; + + explicit CpuWork(size_t count) { + for (size_t i = 0; i < count; ++i) { + threads.emplace_back([this] { + try { serve(); } + catch (const std::exception& e) { + try { error(0, e.what()); } catch (...) {} + std::_Exit(1); + } catch (...) { std::_Exit(1); } + }); + } + } + ~CpuWork() { + { + std::lock_guard lock(mutex); + stopping = true; + changed.notify_all(); + } + for (auto& t : threads) { + if (t.joinable()) t.join(); + } + } + // Threads for CPU work: four by default (the SDK's own decoder threads do the heavy part). + static size_t thread_count() { return env_number("FILMCRAFT_R3D_CPU_THREADS", 1, 16, 4); } + void enqueue(Job job) { + std::lock_guard lock(mutex); + // opens and audio are short and someone waits on them: they go before frames + auto& queue = (job.audio || job.open || job.rmd) ? audio : video; + require(queue.size() < MAX_WAITING, "too many requests outstanding"); + queue.push_back(std::move(job)); + changed.notify_one(); + } + void serve() { + for (;;) { + Job job; + { + std::unique_lock lock(mutex); + changed.wait(lock, [&] { return stopping || !audio.empty() || !video.empty(); }); + if (!audio.empty()) { + job = std::move(audio.front()); + audio.pop_front(); + } else if (!video.empty()) { + job = std::move(video.front()); + video.pop_front(); + } else { + return; // stopping, nothing left + } + } + try { + run(job); + } catch (const std::exception& e) { + error(job.id, e.what()); + } + } + } + void run(Job& job) { + if (job.rmd) { + auto& c = *job.clip; + std::lock_guard lock(c.decode); + auto settings = c.settings(); + develop(settings, job.develop); + adjust(settings, job.develop.flags, job.develop.iso, job.develop.kelvin, job.develop.tint, job.develop.exposure); + require(c.clip->CreateOrUpdateRmd(settings), "RED SDK could not write the RMD sidecar"); + describe(c, true); + const auto metadata = clip_metadata(c, job.slots, job.slot_bytes, job.backend); + reply(job.id, 0, metadata.data(), metadata.size()); + return; + } + if (job.open) { + auto state = open_clip(job.path, job.slot_bytes); + { + std::lock_guard lock(clips_mutex); + require(clips.size() < MAX_CLIPS || clips.count(job.handle), "too many open clips"); + clips[job.handle] = state; + } + const auto metadata = clip_metadata(*state, job.slots, job.slot_bytes, job.backend); + reply(job.id, 0, metadata.data(), metadata.size()); + return; + } + auto& c = *job.clip; + std::lock_guard lock(c.decode); + if (job.audio) { + const size_t bytes = static_cast(job.count) * c.channels * 4; + Aligned buffer(bytes); + size_t samples = static_cast(job.count); + auto result = c.floating ? c.clip->DecodeFloatAudio(job.start, &samples, buffer.data.get(), bytes) + : c.clip->DecodeAudio(job.start, &samples, buffer.data.get(), bytes); + require(result == R3DSDK::DSDecodeOK && samples <= job.count, "RED audio decode failed"); + reply(job.id, 0, buffer.data.get(), samples * c.channels * 4); + return; + } + Aligned buffer(job.bytes); + sentinel(buffer.data.get(), job.bytes); + R3DSDK::VideoDecodeJob decode; + decode.Mode = job.mode; + decode.PixelType = job.pixel_type; + decode.OutputBuffer = buffer.data.get(); + decode.OutputBufferSize = job.bytes; + decode.ImageProcessing = &job.settings; + decode.HdrProcessing = nullptr; + decode.OutputFrameMetadata = nullptr; + const auto begin = std::chrono::steady_clock::now(); + require(c.clip->DecodeVideoFrame(static_cast(job.start), decode) == R3DSDK::DSDecodeOK, "RED video decode failed"); + if (timing) + std::fprintf(stderr, "R3D frame %llu SDK %.3f ms (CPU)\n", static_cast(job.start), + std::chrono::duration(std::chrono::steady_clock::now() - begin).count()); + reply(job.id, 0, buffer.data.get(), job.bytes); + } +}; + +// A recorded SDK setting as the request code that names it (0 when it is none of them). +template uint64_t code_of(T value, const T (&values)[N]) { + for (size_t i = 0; i < N; ++i) { + if (values[i] == value) return i + 1; + } + return 0; +} +bool curves_recorded(const R3DSDK::ImageProcessingSettings& s) { + for (int i = 0; i < 10; ++i) { + if (s.UserCurve[i] != IDENTITY_CURVE[i] || s.UserCurveRed[i] != IDENTITY_CURVE[i] || s.UserCurveGreen[i] != IDENTITY_CURVE[i] || + s.UserCurveBlue[i] != IDENTITY_CURVE[i]) + return true; + } + return false; +} +// The clip's metadata for the app (FCRD9, the reply to `O` and `W`): dimensions, timing, audio, +// this helper's slots and slot size; then its timecode, the processing backend, the as-shot ISO, +// white balance and exposure adjustment; then what else it records (the request codes of its +// colour space, gamma, tone map, rolloff and flashing pixel adjustment plus one, 0 for one +// without a code; its HDR peak, chroma noise reduction, whether it carries a creative LUT and +// asks for it and for its CDL, whether its curves are not the identity, its video track count, +// whether an RMD sidecar is present, the SDK's HDR peak range and default, its default tone map +// and rolloff codes, and the CDL: saturation, then slope, offset and power of red, green and +// blue); then the RMD sidecar's path; then the recorded lens squeeze (0: none recorded). +std::string clip_metadata(ClipState& c, size_t slots, size_t slot_bytes, const std::string& backend) { + using Limits = R3DSDK::ImageProcessingLimits; + static const R3DSDK::ImageColorSpace spaces[] = {R3DSDK::ImageColorRec709, R3DSDK::ImageColorRec2020, R3DSDK::ImageColorDCIP3D65, R3DSDK::ImageColorREDWideGamutRGB}; + static const R3DSDK::ImageGammaCurve gammas[] = {R3DSDK::ImageGammaLinear, R3DSDK::ImageGammaBT1886, R3DSDK::ImageGammaSRGB, + R3DSDK::ImageGammaLog3G10, R3DSDK::ImageGammaHDR2084, R3DSDK::ImageGammaHybridLogGamma}; + static const R3DSDK::ToneMap tone_maps[] = {R3DSDK::ToneMap_None, R3DSDK::ToneMap_Low, R3DSDK::ToneMap_Medium, R3DSDK::ToneMap_High}; + static const R3DSDK::RollOff roll_offs[] = {R3DSDK::RollOff_None, R3DSDK::RollOff_Hard, R3DSDK::RollOff_Medium, R3DSDK::RollOff_Soft, R3DSDK::RollOff_Very_Soft}; + static const R3DSDK::FlashingPixelAdjust flashing[] = {R3DSDK::FlashingPixelAdjustOff, R3DSDK::FlashingPixelAdjustMild, R3DSDK::FlashingPixelAdjustMedium, + R3DSDK::FlashingPixelAdjustStrong}; + std::ostringstream info; + info << "FCRD9 " << c.width << ' ' << c.height << ' ' << c.frames << ' ' << c.num << ' ' << c.den << ' ' + << c.channels << ' ' << c.rate << ' ' << c.samples << ' ' << c.floating << ' ' << c.bits << ' ' + << slots << ' ' << slot_bytes << '\n'; + const char* tc = c.clip->Timecode(0); + if (tc) info << tc; + info << '\n' << backend; + std::lock_guard lock(c.settings_mutex); + const auto& s = c.base; + info << '\n' << s.ISO << ' ' << std::setprecision(9) << s.Kelvin << ' ' << s.Tint << ' ' << s.ExposureAdjust; + info << '\n' << code_of(s.ColorSpace, spaces) << ' ' << code_of(s.GammaCurve, gammas) << ' ' << code_of(s.OutputToneMap, tone_maps) << ' ' + << code_of(s.HighlightRollOff, roll_offs) << ' ' << s.HdrPeakNits << ' ' << (s.ChromaNoiseReduction ? 1 : 0) << ' ' + << code_of(s.FlashingPixelAdjustment, flashing) << ' ' << (c.lut ? 1 : 0) << ' ' << (s.Lut3DEnabled ? 1 : 0) << ' ' << (s.CdlEnabled ? 1 : 0) << ' ' + << (curves_recorded(s) ? 1 : 0) << ' ' << c.tracks << ' ' << (c.rmd ? 1 : 0) << ' ' << Limits::HdrPeakNitsMin << ' ' << Limits::HdrPeakNitsMax << ' ' + << Limits::HdrPeakNitsDefault << ' ' << code_of(Limits::OutputToneMapDefault, tone_maps) << ' ' << code_of(Limits::HighlightRollOffDefault, roll_offs) + << ' ' << s.CdlSaturation; + for (const auto* sop : {&s.CdlRed, &s.CdlGreen, &s.CdlBlue}) info << ' ' << sop->Slope << ' ' << sop->Offset << ' ' << sop->Power; + info << '\n' << c.rmd_path; + info << '\n' << c.squeeze; + return info.str(); +} + +// Longest command line: an `O` carries a path. +constexpr size_t MAX_LINE = 16384; + +void run(const char* libraries, const Name& name, size_t requested_slot_bytes) { + const char* requested = std::getenv("FILMCRAFT_R3D_PROCESSING"); + require(!requested || std::strcmp(requested, "gpu") == 0 || std::strcmp(requested, "cpu") == 0, + "FILMCRAFT_R3D_PROCESSING must be gpu or cpu"); + const bool use_gpu = !requested || std::strcmp(requested, "cpu") != 0; + Sdk sdk(libraries, use_gpu); + // Declared after the SDK, so destroyed before it: the open clips go before FinalizeSdk (a clip + // released after it faults), and after the pipeline and the CPU work, which may hold them. + struct CloseClips { + ~CloseClips() { + std::lock_guard lock(clips_mutex); + clips.clear(); + } + } close_clips; + std::unique_ptr gpu; + if (use_gpu) gpu = std::make_unique(); +#ifndef _WIN32 + if (gpu) sdk.finalize = false; +#endif + // Built before the start-up reply so a failed job allocation or mapping is the start-up + // error, and the slot count and size reach the app (it keeps that many requests outstanding, + // plus one, and maps the slots). The slots hold the largest frame the app expects; a clip + // larger than that is refused at `O` and the app starts a helper with larger slots. + std::unique_ptr shared; + std::unique_ptr pipeline; + if (gpu) { + require(requested_slot_bytes > 0 && requested_slot_bytes <= MAX_PIXELS * 6, "invalid frame slot size"); + shared = std::make_unique(name, Pipeline::slot_count(), requested_slot_bytes); + pipeline = std::make_unique(*gpu, *shared); + } + const size_t slots = pipeline ? pipeline->count : 0, slot_bytes = shared ? shared->slot_bytes : 0; + const std::string backend = gpu ? gpu->description : "CPU (explicit override)"; + CpuWork cpu(CpuWork::thread_count()); + { + std::ostringstream info; + info << "FCRD9 " << slots << ' ' << slot_bytes << ' ' << cpu.threads.size() << '\n' << backend; + const auto start = info.str(); + reply(0, 0, start.data(), start.size()); + } + + std::string line; + while (std::getline(std::cin, line)) { + uint64_t id = 0; + try { + require(line.size() <= MAX_LINE, "decode request exceeds line limit"); + std::istringstream command(line); + char op = 0; + uint64_t start = 0, count = 0, handle = 0; + std::string extra; + command >> op; + if (op == 'F') { + command >> start; + if (command.fail() || (command >> extra)) throw fatal("malformed frame release"); + if (!pipeline) throw fatal("frame release without shared frames"); + pipeline->release(start); + continue; + } + if (op == 'C') { + command >> handle; + if (command.fail() || (command >> extra)) throw fatal("malformed clip close"); + std::shared_ptr closing; + { + std::lock_guard lock(clips_mutex); + const auto it = clips.find(handle); + if (it != clips.end()) { + closing = std::move(it->second); + clips.erase(it); + } + } + continue; // freed here, or by the last request that still holds it + } + command >> id >> handle; + require(!command.fail() && id != 0 && handle != 0, "malformed decode request"); + if (op == 'O') { + // the rest of the line after one space is the path (UTF-8, spaces allowed) + std::string path; + std::getline(command, path); + require(path.size() > 1 && path[0] == ' ', "malformed clip open"); + path.erase(0, 1); + CpuWork::Job job; + job.id = id; + job.open = true; + job.handle = handle; + job.path = std::move(path); + job.slots = slots; + job.slot_bytes = slot_bytes; + job.backend = backend; + cpu.enqueue(std::move(job)); + continue; + } + auto clip = find_clip(handle); + if (op == 'W') { + // `W `: write them into the clip's RMD sidecar + CpuWork::Job job; + require(read_develop(command, job.develop, false) && !(command >> extra), "malformed RMD write"); + // checked here as well, so a malformed write is answered before it is queued + R3DSDK::ImageProcessingSettings check = clip->settings(); + develop(check, job.develop); + adjust(check, job.develop.flags, job.develop.iso, job.develop.kelvin, job.develop.tint, job.develop.exposure); + job.id = id; + job.clip = clip; + job.rmd = true; + job.slots = slots; + job.slot_bytes = slot_bytes; + job.backend = backend; + cpu.enqueue(std::move(job)); + continue; + } + command >> start; + require(!command.fail(), "malformed decode request"); + if (op == 'V') { + Develop d; + command >> count; + require(!command.fail() && read_develop(command, d, true) && !(command >> extra) && start < clip->frames, "video frame out of range"); + require(count == 1 || count == 2 || count == 4 || count == 8 || count == 16, "unsupported preview divisor"); + // SDK 9.2.1 writes floor(w/count) x floor(h/count) pixels, packed at that width whatever + // bytesPerRow says, and nothing past OutputBufferSize (measured; docs/r3d-ne.md). The app + // computes the same size and rejects any other reply length. + const size_t rw = clip->width / count, rh = clip->height / count; + require(rw > 0 && rh > 0, "unsupported preview divisor"); + R3DSDK::ImageProcessingSettings settings = clip->settings(); + const auto pixel_type = develop(settings, d); + adjust(settings, d.flags, d.iso, d.kelvin, d.tint, d.exposure); + // half-float and 16-bit integer RGB are both 6 bytes per pixel + const size_t bytes = rw * rh * 6; + const auto mode = count == 1 ? R3DSDK::DECODE_FULL_RES_PREMIUM : count == 2 ? (d.half ? R3DSDK::DECODE_HALF_RES_GOOD : R3DSDK::DECODE_HALF_RES_PREMIUM) + : count == 4 ? R3DSDK::DECODE_QUARTER_RES_GOOD : count == 8 ? R3DSDK::DECODE_EIGHT_RES_GOOD + : R3DSDK::DECODE_SIXTEENTH_RES_GOOD; + if (pipeline) { + Pipeline::Request request; + request.id = id; + request.clip = clip; + request.frame = static_cast(start); + request.divisor = count; + request.bytes = bytes; + request.width = rw; + request.mode = mode; + request.settings = settings; + request.pixel_type = pixel_type; + pipeline->enqueue(request); + continue; + } + CpuWork::Job job; + job.id = id; + job.clip = clip; + job.start = start; + job.mode = mode; + job.pixel_type = pixel_type; + job.settings = settings; + job.bytes = bytes; + cpu.enqueue(std::move(job)); + } else if (op == 'A') { + command >> count; + require(!command.fail() && !(command >> extra), "malformed audio request"); + require(clip->channels > 0 && count > 0 && count <= MAX_AUDIO && start < clip->samples, "audio request out of range"); + CpuWork::Job job; + job.id = id; + job.clip = clip; + job.audio = true; + job.start = start; + job.count = count; + cpu.enqueue(std::move(job)); + } else { + throw std::runtime_error("unknown decode request"); + } + } catch (const fatal& e) { + // Reported before anything is torn down (the SDK's teardown can take a while, and + // callbacks may still reference the slots), then out, as the pipeline thread does. + std::fprintf(stderr, "filmcraft-red-helper: %s\n", e.what()); + try { error(0, e.what()); } catch (...) {} + std::_Exit(1); + } catch (const std::exception& e) { + if (id != 0) { + error(id, e.what()); + } else { + // a line too malformed to name its request cannot be answered: the state is no + // longer shared, so the helper ends + std::fprintf(stderr, "filmcraft-red-helper: %s\n", e.what()); + try { error(0, e.what()); } catch (...) {} + std::_Exit(1); + } + } + } + require(std::cin.eof(), "decode request exceeds line limit"); +} +} + +constexpr const char* USAGE = "usage: filmcraft-red-helper "; +// The frame slot size argument: decimal bytes (0 for a CPU-only helper). +size_t slot_bytes_argument(const char* text) { + char* end = nullptr; + const auto n = std::strtoull(text, &end, 10); + require(end != text && *end == 0 && n <= MAX_PIXELS * 6, "invalid frame slot size"); + return static_cast(n); +} +int serve(const char* libraries, const Name::value_type* name_argument, const char* slot_bytes) { + try { + const auto name = checked_name(name_argument); + open_output(name); + run(libraries, name, slot_bytes_argument(slot_bytes)); + return 0; + } catch (const std::exception& e) { + try { error(0, e.what()); } catch (...) {} + } catch (...) { + try { error(0, "unexpected RED SDK failure"); } catch (...) {} + } + return 1; +} + +#ifdef _WIN32 +int wmain(int argc, wchar_t** argv) { + _setmode(_fileno(stdout), _O_BINARY); + // Do not display Windows fault dialogs for a failed worker. + SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX); + try { + require(argc == 4, USAGE); + auto libraries = utf8(argv[1]), slots = utf8(argv[3]); + return serve(libraries.c_str(), argv[2], slots.c_str()); + } catch (const std::exception& e) { + try { error(0, e.what()); } catch (...) {} + } + return 1; +} +#else +int main(int argc, char** argv) { + // A closed reply pipe must surface as a write error (the writer thread then exits the + // helper), not as a silent SIGPIPE death. Paths arrive as UTF-8 bytes already. + std::signal(SIGPIPE, SIG_IGN); + if (argc != 4) { + try { error(0, USAGE); } catch (...) {} + return 1; + } + return serve(argv[1], argv[2], argv[3]); +} +#endif diff --git a/crates/platform/src/red/mod.rs b/crates/platform/src/red/mod.rs new file mode 100644 index 0000000000..5a7518c402 --- /dev/null +++ b/crates/platform/src/red/mod.rs @@ -0,0 +1,3154 @@ +//! Optional SDK source (Windows x64 and Linux x64). The proprietary decoder runs outside the +//! editor process. +//! +//! Frames come back through shared memory (since FCRD6, [`shm`], the backend's one FFI module); the +//! rest of this module is safe code over pipes. +#[allow(unsafe_code)] +mod shm; + +use std::collections::{HashMap, VecDeque}; +use std::io::{BufRead, Read, Write}; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Stdio}; +use std::sync::{Arc, Condvar, Mutex, OnceLock, mpsc}; +use std::time::{Duration, Instant}; + +use filmcraft_color::{ColorInfo, Transfer}; +use filmcraft_frame::{AudioBuffer, VideoFrame}; +use filmcraft_media::{ + AudioStreamInfo, FrameRequest, MediaError, MediaInfo, MediaKind, MediaSource, RawAsShot, RawClipInfo, RawColorSpace, RawDevelop, RawFlashing, RawGamma, + RawRollOff, RawToneMap, Result, SharedReader, SharedSource, VideoStreamInfo, +}; +use filmcraft_time::{FrameRate, TICKS_PER_SECOND, Tick}; + +const MAX_PIXELS: usize = 33_554_432; +const MAX_SAMPLES: usize = 1_048_576; +const MAX_REPLY: usize = MAX_PIXELS * 6; +const TIMEOUT: Duration = Duration::from_secs(120); + +fn failure(message: impl Into) -> MediaError { + MediaError::Decode(message.into()) +} +fn io(error: std::io::Error) -> MediaError { + failure(format!("RED decoder: {error}")) +} + +/// `FILMCRAFT_R3D_TIMING` is set: the helper's per-frame timings reach the app's stderr, and this +/// side logs each request's wait for a slot, round trip, reply read and conversion. +fn timing() -> bool { + static TIMING: OnceLock = OnceLock::new(); + *TIMING.get_or_init(|| std::env::var_os("FILMCRAFT_R3D_TIMING").is_some()) +} + +/// The helper's stdin, shared by the request writers and the slot leases (a lease hands its +/// slot back from wherever it is dropped). +type Commands = Arc>>; + +/// The helper's frame slots: its section mapped read-only, with the geometry its metadata +/// declared, and the stdin to hand slots back on. +struct SharedFrames { + mapping: shm::Mapping, + slots: u32, + slot_bytes: usize, + commands: Commands, +} + +/// A decoded frame the helper handed over in one of its shared slots. The bytes are read in +/// place; dropping the lease tells the helper the slot is free (`F `), wherever that +/// happens: after the repack, inside a cancelled requester's abandoned channel, or in the +/// reader discarding an orphan. A helper that has exited makes the write fail, which is fine: +/// its slots died with it, while the pages stay readable for as long as the mapping lives. +struct SlotLease { + shared: Arc, + index: u32, + len: usize, + delivered: Instant, +} +impl SlotLease { + fn bytes(&self) -> Result<&[u8]> { + let offset = (self.index as usize).checked_mul(self.shared.slot_bytes).ok_or_else(|| failure("RED frame slot out of range"))?; + self.shared.mapping.slice(offset, self.len) + } +} +impl Drop for SlotLease { + fn drop(&mut self) { + let mut commands = self.shared.commands.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let _ = commands.write_all(format!("F {}\n", self.index).as_bytes()).and_then(|_| commands.flush()); + } +} + +/// One answer from the helper: bytes that came down the pipe (metadata, audio, and video from a +/// CPU-only helper), a video frame waiting in a shared slot, or the helper's error for this +/// request. An error answers one request (an unreadable clip, a malformed setting) and leaves the +/// helper serving every other clip (FCRD8: one helper serves them all). +enum Reply { + Inline(Vec), + Slot(SlotLease), + Error(String), +} +impl Reply { + fn into_inline(self) -> Result> { + match self { + Self::Inline(bytes) => Ok(bytes), + Self::Slot(_) => Err(failure("RED decoder answered with a frame slot where bytes were expected")), + Self::Error(message) => Err(failure(format!("RED decoder: {message}"))), + } + } +} + +/// Reply header: status `u32`, request id `u64`, payload length `u64`, slot `u32`. +const HEADER: usize = 24; +/// Status of a reply whose frame is in a shared slot (no payload follows). +const STATUS_SHARED: u32 = 2; + +fn read_reply(reader: &mut impl Read, shared: &OnceLock>) -> Result<(u64, Result)> { + let mut header = [0u8; HEADER]; + reader.read_exact(&mut header).map_err(io)?; + let t0 = Instant::now(); + let status = u32::from_le_bytes(header[..4].try_into().map_err(|_| failure("invalid RED response"))?); + let id = u64::from_le_bytes(header[4..12].try_into().map_err(|_| failure("invalid RED request id"))?); + let size = u64::from_le_bytes(header[12..20].try_into().map_err(|_| failure("invalid RED response"))?); + let slot = u32::from_le_bytes(header[20..].try_into().map_err(|_| failure("invalid RED response"))?); + if size > MAX_REPLY as u64 || (status == 1 && size > 16_384) { + return Err(failure("RED decoder response exceeds memory limit")); + } + if status == STATUS_SHARED { + // The frame is in the helper's section; nothing follows the header. The slot and the + // length are the helper's word, checked against the geometry it declared at start-up + // (and the mapping checks every read against the bytes it mapped). + let shared = shared.get().ok_or_else(|| failure("RED decoder answered with a frame slot before sharing frames"))?; + if slot >= shared.slots || size as usize > shared.slot_bytes { + return Err(failure("RED frame slot out of range")); + } + return Ok((id, Ok(Reply::Slot(SlotLease { shared: shared.clone(), index: slot, len: size as usize, delivered: t0 })))); + } + if status > 1 || slot != 0 { + return Err(failure("invalid RED response")); + } + // Frame-sized replies come from the plane pool: a fresh 31 MB allocation per frame at 24 fps + // is page faults and a zero fill every time (`video_frame` hands the buffer back). + let mut bytes = filmcraft_frame::pool::take_u8(size as usize); + bytes.try_reserve_exact(size as usize).map_err(|_| failure("RED response allocation failed"))?; + bytes.resize(size as usize, 0); + let t1 = Instant::now(); + reader.read_exact(&mut bytes).map_err(io)?; + if timing() && size >= 1 << 20 { + let ms = |d: Duration| d.as_secs_f64() * 1e3; + log::info!("R3D reply {id}: {:.1} MB, buffer {:.1} ms, pipe read {:.1} ms", size as f64 / 1e6, ms(t1.saturating_duration_since(t0)), ms(t1.elapsed())); + } + if status != 0 { + return Ok((id, Ok(Reply::Error(String::from_utf8_lossy(&bytes).into_owned())))); + } + Ok((id, Ok(Reply::Inline(bytes)))) +} + +struct PendingRequest { + sender: mpsc::SyncSender>, + started: Instant, + /// Requests outstanding when this one was registered, this one included. + in_flight: usize, +} +impl PendingRequest { + fn new(sender: mpsc::SyncSender>) -> Self { + Self { sender, started: Instant::now(), in_flight: 1 } + } +} +/// One answered request: when its reply arrived, how many were in flight with it, how long it took. +#[derive(Clone, Copy)] +struct Answered { + at: Instant, + in_flight: usize, + /// Requests outstanding when this one was answered, this one included. + outstanding: usize, + took: Duration, +} +/// Replies remembered for the throughput estimate ([`Pending::rate`]). +const RATE_WINDOW: usize = 32; +/// Replies older than this before the newest say nothing about the current throughput. +const RATE_SPAN: Duration = Duration::from_secs(2); +#[derive(Default)] +struct Pending { + next: u64, + failed: Option, + requests: HashMap, + answered: VecDeque, + /// The helper's decode slots (0 until known): the pipeline's throughput shows only in replies + /// to requests that had that many in flight with them. + slots: usize, + /// The last throughput read off saturated replies (see [`Self::rate`]), and when. + rate: Option, + rate_at: Option, +} +impl Pending { + fn fail(&mut self, message: &str) { + self.failed = Some(message.into()); + for (_, sender) in self.requests.drain() { + let _ = sender.sender.send(Err(failure(message))); + } + } + fn expired(&self, timeout: Duration) -> bool { + self.requests.values().any(|request| request.started.elapsed() >= timeout) + } + fn dispatch(&mut self, id: u64, reply: Result) -> Result<()> { + // After start-up the helper sends ID 0 only for its own fatal error, then exits. + if id == 0 && !self.requests.contains_key(&0) { + return Err(match reply { + Ok(Reply::Error(message)) => failure(format!("RED decoder: {message}")), + Err(e) => e, + Ok(_) => failure("unexpected RED start-up reply"), + }); + } + let request = self.requests.remove(&id).ok_or_else(|| failure("unexpected RED response id"))?; + // the helper's own fatal error after start-up arrives as ID 0 (handled above); at + // start-up its error reply is the start-up failure + let reply = match reply { + Ok(Reply::Error(message)) if id == 0 => Err(failure(format!("RED decoder: {message}"))), + other => other, + }; + if id != 0 && matches!(reply, Ok(Reply::Inline(_) | Reply::Slot(_))) { + if self.answered.len() >= RATE_WINDOW { + self.answered.pop_front(); + } + self.answered.push_back(Answered { + at: Instant::now(), + in_flight: request.in_flight, + outstanding: self.requests.len().saturating_add(1), + took: request.started.elapsed(), + }); + } + // A cancelled requester may have dropped its receiver. Still consume its reply. + let _ = request.sender.send(reply); + Ok(()) + } + /// Frames per second the helper delivers when kept busy: `in flight / latency` (Little's + /// law, with everything in flight: a request waiting in the helper's queue is part of the + /// latency of the ones it waits behind) averaged over the recent replies to requests that + /// had at least the slot count in flight when they were registered and still when they were + /// answered. Both ends, because Little's law holds for a system kept full: a burst into an + /// idle helper registers its fifth request with five in flight, yet that one took a free + /// slot and never queued, and its short latency read as 32 fps from a pipeline that + /// completes 26 when full (measured with five slots: the plan then stepped the + /// 60p clip up to every second frame and cascaded back to every fifth). Replies from a lull + /// say nothing about capacity (with one request in flight the SDK idles at ~4 fps, and + /// playback that asks for less would read that as the pipeline falling behind and ask for + /// less again, down to every tenth frame), so through a lull the last saturated reading + /// stands. A request that was cancelled after submission still counts, since the helper + /// decoded it. `None` until three saturated replies. + fn rate(&mut self) -> Option { + let newest = self.answered.back()?.at; + let full = self.slots.max(1); + let saturated = + |a: &&Answered| newest.saturating_duration_since(a.at) <= RATE_SPAN && !a.took.is_zero() && a.in_flight >= full && a.outstanding >= full; + let rates: Vec = self.answered.iter().filter(saturated).map(|a| a.in_flight as f64 / a.took.as_secs_f64()).collect(); + if rates.len() >= 3 + && let mean = rates.iter().sum::() / rates.len() as f64 + && mean.is_finite() + && mean > 0.0 + { + self.rate = Some(mean); + self.rate_at = Some(newest); + } + self.rate + } + /// [`Self::rate`] and whether it was read within [`RATE_SPAN`] of now. + fn delivery(&mut self) -> (Option, bool) { + let rate = self.rate(); + (rate, self.rate_at.is_some_and(|at| at.elapsed() <= RATE_SPAN)) + } +} + +/// The requests outstanding on one helper, shared by the callers, the response reader and the +/// watchdog. `changed` wakes callers waiting for a free slot. +#[derive(Default)] +struct Registry { + pending: Mutex, + changed: Condvar, + /// The helper's frame slots, mapped once its metadata has said how many and how large + /// (a GPU helper; a CPU-only one sends its frames inline and never sets this). + shared: OnceLock>, +} +impl Registry { + fn lock(&self) -> std::sync::MutexGuard<'_, Pending> { + self.pending.lock().unwrap_or_else(std::sync::PoisonError::into_inner) + } + /// Register a request once fewer than `limit` are outstanding. The helper has that many + /// decode slots (plus one line waiting in its pipe): a request sent beyond them would only + /// queue in the pipe, where it cannot be cancelled and will occupy a slot for a frame playback + /// has moved past. Waiting here is cancellable, and a cancelled wait sends nothing. + fn register(&self, limit: usize, sender: mpsc::SyncSender>) -> Result { + let started = Instant::now(); + let mut pending = self.lock(); + loop { + if let Some(error) = &pending.failed { + return Err(failure(error)); + } + if pending.requests.len() < limit { + break; + } + if filmcraft_media::cancel::cancelled() { + return Err(MediaError::Cancelled); + } + if started.elapsed() >= TIMEOUT { + return Err(failure("RED decoder slots stayed busy for 120 seconds")); + } + pending = self.changed.wait_timeout(pending, Duration::from_millis(10)).unwrap_or_else(std::sync::PoisonError::into_inner).0; + } + pending.next = pending.next.checked_add(1).ok_or_else(|| failure("RED request ids exhausted"))?; + let id = pending.next; + let mut request = PendingRequest::new(sender); + request.in_flight = pending.requests.len().saturating_add(1); + pending.requests.insert(id, request); + Ok(id) + } + fn unregister(&self, id: u64) { + self.lock().requests.remove(&id); + self.changed.notify_all(); + } + fn dispatch(&self, id: u64, reply: Result) -> Result<()> { + let result = self.lock().dispatch(id, reply); + self.changed.notify_all(); + result + } + fn fail(&self, message: &str) { + self.lock().fail(message); + self.changed.notify_all(); + } +} + +static PIPE_SERIAL: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); +/// Helper processes alive right now (logged with every start and close: a count that climbs +/// is the first sign of a source opened once per frame worker or per settings change). +static LIVE_PROCESSES: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); + +/// Longest helper stderr line kept whole; longer ones arrive in pieces. +const STDERR_LINE_MAX: u64 = 4096; + +/// Forward the helper's stderr to the app log, one record a line, until the helper exits. Its +/// per-frame timings (`FILMCRAFT_R3D_TIMING`, which keeps stderr inherited instead) would be +/// debug records; everything else it says there is a problem report. +fn forward_stderr(pid: u32, stderr: std::process::ChildStderr) { + let spawned = std::thread::Builder::new().name("red-stderr".into()).spawn(move || { + let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(move || { + let mut reader = std::io::BufReader::new(stderr); + let mut line = String::new(); + loop { + line.clear(); + match (&mut reader).take(STDERR_LINE_MAX).read_line(&mut line) { + Ok(0) | Err(_) => break, + Ok(_) => {} + } + let text = line.trim_end(); + if text.is_empty() { + continue; + } + if text.starts_with("R3D frame") || text.starts_with("R3D slot") { + log::debug!("R3D helper {pid}: {text}"); + } else { + log::warn!("R3D helper {pid}: {text}"); + } + } + })); + }); + if let Err(e) = spawned { + log::warn!("R3D helper {pid}: stderr not forwarded: {e}"); + } +} + +/// How long a helper may take to create its reply pipe (it does so before loading the SDK). +#[cfg(windows)] +const CONNECT_TIMEOUT: Duration = Duration::from_secs(10); + +/// Where the helper's replies come from: on Windows the named pipe it creates, on Linux its +/// stdout (see `connect_replies`). +type Replies = Box; + +/// Open the helper's reply pipe `\\.\pipe\` as a file (the helper creates it first +/// thing and waits for this connection). A helper that exits first reports its start-up error +/// on stdout in the reply framing; that message is returned. +#[cfg(windows)] +fn connect_replies(child: &mut Child, mut stdout: std::process::ChildStdout, name: &str) -> Result { + let path = format!(r"\\.\pipe\{name}"); + let started = Instant::now(); + loop { + let error = match std::fs::File::open(&path) { + Ok(f) => return Ok(Box::new(f)), + Err(e) => e, + }; + if let Ok(Some(status)) = child.try_wait() { + let message = match read_reply(&mut stdout, &OnceLock::new()) { + Ok((_, Err(e))) => e.to_string(), + _ => format!("RED helper exited ({status}) before creating its reply pipe"), + }; + return Err(failure(message)); + } + if started.elapsed() >= CONNECT_TIMEOUT { + return Err(failure(format!("RED helper reply pipe unavailable: {error}"))); + } + std::thread::sleep(Duration::from_millis(5)); + } +} + +/// Linux: the helper's stdout is its reply pipe (it widens the pipe to 1 MiB itself; since +/// FCRD6 nothing frame-sized goes down it), and the name only names the shared frame memory. +#[cfg(target_os = "linux")] +fn connect_replies(_: &mut Child, stdout: std::process::ChildStdout, _: &str) -> Result { + Ok(Box::new(stdout)) +} + +/// Linux: `REDR3D-x64.so` loads `REDCuda-x64.so`, `REDOpenCL-x64.so` and `REDDecoder-x64.so` by +/// name, so the runtime directory goes first on the helper's `LD_LIBRARY_PATH` (ahead of whatever +/// the app inherited). A directory `join_paths` rejects (one containing `:`) is used on its own. +#[cfg(target_os = "linux")] +fn library_search_path(libraries: &Path) -> std::ffi::OsString { + let inherited = std::env::var_os("LD_LIBRARY_PATH").unwrap_or_default(); + let paths = std::iter::once(libraries.to_path_buf()).chain(std::env::split_paths(&inherited)); + std::env::join_paths(paths).unwrap_or_else(|_| libraries.as_os_str().to_os_string()) +} + +struct Worker { + child: Arc>, + commands: Commands, + registry: Arc, + shutdown: mpsc::Sender<()>, + /// Requests outstanding at once: the helper's decode slots plus one waiting in its pipe. + limit: usize, + /// The helper process, for the log: its id, when it started, and whether it is the GPU or + /// the CPU helper. + pid: u32, + started: Instant, + label: String, +} + +impl Worker { + /// A helper that serves any number of clips (FCRD8; clips are opened in it with `O`): the GPU + /// helper (`cpu_only` false), whose shared frame slots hold `slot_bytes` each, or the CPU + /// helper (audio, thumbnails, metadata probes, and frames while the GPU is unavailable). + fn open_hub(cpu_only: bool, slot_bytes: usize) -> Result<(Self, HubInfo)> { + #[cfg(test)] + if !cpu_only && tests::FAIL_GPU_STARTS.load(std::sync::atomic::Ordering::Relaxed) { + return Err(failure("RED decoder: Create RED GPU decoder (RED GPU status 13; simulated)")); + } + let helper_name = format!("filmcraft-red-helper{}", std::env::consts::EXE_SUFFIX); + let beside_app = std::env::current_exe().ok().and_then(|p| p.parent().map(|p| p.join(&helper_name))); + let helper = std::env::var_os("FILMCRAFT_R3D_HELPER") + .map(PathBuf::from) + .or_else(|| beside_app.filter(|p| p.is_file())) + .unwrap_or_else(|| PathBuf::from(env!("FILMCRAFT_BUILT_RED_HELPER"))); + let libraries = std::env::var_os("RED_R3D_LIBRARY_PATH") + .map(PathBuf::from) + .or_else(|| helper.parent().map(Path::to_path_buf)) + .ok_or_else(|| failure("RED helper directory missing"))?; + let mut command = Command::new(&helper); + // Audio, thumbnails and the import probe need no GPU. + if cpu_only { + command.env("FILMCRAFT_R3D_PROCESSING", "cpu"); + } + // Replies come down a named pipe the helper creates (see `connect_replies`); its name is + // unguessable and the helper allows one instance, so nothing else can serve or take it. + // The same name names the helper's shared frame memory. + let pipe_name = format!("filmcraft-red-{}-{}-{:016x}", std::process::id(), PIPE_SERIAL.fetch_add(1, std::sync::atomic::Ordering::Relaxed), { + use std::hash::{BuildHasher, Hasher}; + let mut h = std::collections::hash_map::RandomState::new().build_hasher(); + h.write_u128(std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_nanos())); + h.finish() + }); + let slots_argument = if cpu_only { 0 } else { slot_bytes }; + command.arg(&libraries).arg(&pipe_name).arg(slots_argument.to_string()).stdin(Stdio::piped()).stdout(Stdio::piped()).stderr(if timing() { + Stdio::inherit() + } else { + Stdio::piped() // forwarded to the app log (`forward_stderr`) + }); + #[cfg(windows)] + { + use std::os::windows::process::CommandExt; + command.creation_flags(0x08000000); // CREATE_NO_WINDOW: the helper has no interactive UI. + } + #[cfg(target_os = "linux")] + command.env("LD_LIBRARY_PATH", library_search_path(&libraries)); + let mut child = command + .spawn() + .map_err(|e| failure(format!("Cannot start RED helper {}: {e}. Rebuild with red-sdk or set FILMCRAFT_R3D_HELPER.", helper.display())))?; + // Until the app holds the helper's frame memory, leaving here removes its name (Linux; + // a Windows section dies with its handles). + let _frame_memory_name = shm::Name::new(&pipe_name); + let pid = child.id(); + let started = Instant::now(); + let label = format!("{} helper", if cpu_only { "CPU" } else { "GPU" }); + let live = LIVE_PROCESSES.fetch_add(1, std::sync::atomic::Ordering::Relaxed).saturating_add(1); + log::info!("R3D helper {pid} ({label}) started from {}; {live} live", helper.display()); + if let Some(stderr) = child.stderr.take() { + forward_stderr(pid, stderr); + } + let pipes = (child.stdin.take(), child.stdout.take()); + let (Some(stdin), Some(stdout)) = pipes else { + let _ = child.kill(); + let _ = child.wait(); + LIVE_PROCESSES.fetch_sub(1, std::sync::atomic::Ordering::Relaxed); + return Err(failure("RED helper pipes unavailable")); + }; + let mut stdout = match connect_replies(&mut child, stdout, &pipe_name) { + Ok(f) => f, + Err(e) => { + let _ = child.kill(); + let _ = child.wait(); + LIVE_PROCESSES.fetch_sub(1, std::sync::atomic::Ordering::Relaxed); + log::warn!("R3D helper {pid} ({label}) did not connect: {e}"); + return Err(e); + } + }; + let (tx, replies) = mpsc::sync_channel(1); + let registry = Arc::new(Registry::default()); + registry.lock().requests.insert(0, PendingRequest::new(tx)); + let child = Arc::new(Mutex::new(child)); + let reader_registry = registry.clone(); + let reader_child = child.clone(); + let thread = std::thread::Builder::new().name("red-responses".into()).spawn(move || { + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| -> Result<()> { + loop { + let (id, reply) = read_reply(&mut stdout, &reader_registry.shared)?; + reader_registry.dispatch(id, reply)?; + } + })); + let message = match result { + Ok(Err(e)) => e.to_string(), + _ => "RED response worker panicked or stopped".into(), + }; + // An orderly close fails the registry first ("RED worker closed"); only a reader + // that stopped on its own has something to report. + let closing = match &reader_registry.lock().failed { + None => { + log::warn!("R3D helper {pid} replies ended: {message}"); + false + } + Some(why) => why == CLOSED, + }; + reader_registry.fail(&message); + // an orderly close is letting the helper exit by itself (Drop kills it if it does not) + if !closing { + let _ = reader_child.lock().unwrap_or_else(std::sync::PoisonError::into_inner).kill(); + } + }); + let (shutdown, stop) = mpsc::channel(); + let commands: Commands = Arc::new(Mutex::new(Box::new(stdin))); + let mut worker = Self { child, commands, registry, shutdown, limit: 1, pid, started, label }; + if let Err(e) = thread { + worker.fail(&format!("RED response thread: {e}")); + return Err(io(e)); + } + // Cancelled callers drop their receiver immediately, but their submitted jobs still + // retain a deadline. A stalled orphan must also terminate the isolated helper. + let watchdog_registry = worker.registry.clone(); + let watchdog_child = worker.child.clone(); + let watchdog = std::thread::Builder::new().name("red-timeouts".into()).spawn(move || { + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + loop { + match stop.recv_timeout(Duration::from_millis(100)) { + Ok(()) | Err(mpsc::RecvTimeoutError::Disconnected) => return, + Err(mpsc::RecvTimeoutError::Timeout) => {} + } + let pending = watchdog_registry.lock(); + if pending.failed.is_some() { + return; + } + if pending.expired(TIMEOUT) { + drop(pending); + watchdog_registry.fail("RED request exceeded 120-second deadline"); + let _ = watchdog_child.lock().unwrap_or_else(std::sync::PoisonError::into_inner).kill(); + return; + } + } + })); + if result.is_err() { + watchdog_registry.fail("RED timeout worker panicked"); + let _ = watchdog_child.lock().unwrap_or_else(std::sync::PoisonError::into_inner).kill(); + } + }); + if let Err(e) = watchdog { + return Err(io(e)); + } + let metadata = worker.receive(replies, false)?.into_inline()?; + let parsed = HubInfo::parse(&metadata)?; + // A GPU helper's frames wait in its shared slots: map them now, before any request, + // sized exactly as its metadata says (fewer bytes in the section fail the mapping). + if parsed.slots > 0 { + let len = parsed.slot_bytes.checked_mul(parsed.slots).ok_or_else(|| failure("RED frame slots too large"))?; + let mapping = match shm::Mapping::open(&pipe_name, len) { + Ok(mapping) => mapping, + Err(e) => { + worker.fail(&e.to_string()); + return Err(e); + } + }; + let shared = SharedFrames { mapping, slots: parsed.slots as u32, slot_bytes: parsed.slot_bytes, commands: worker.commands.clone() }; + if worker.registry.shared.set(Arc::new(shared)).is_err() { + return Err(failure("RED frame slots mapped twice")); + } + } + // The GPU helper keeps its slots busy plus one request waiting; the CPU helper's threads + // each take one, with as many waiting (audio jumps the queue inside the helper). + worker.limit = if parsed.slots > 0 { parsed.slots.saturating_add(1) } else { parsed.threads.max(1).saturating_mul(2) }; + worker.registry.lock().slots = if parsed.slots > 0 { parsed.slots } else { parsed.threads.max(1) }; + log::info!( + "R3D helper {} ({}) ready in {:.1} s: {}, {} slots of {:.1} MB, {} CPU threads", + worker.pid, + worker.label, + worker.started.elapsed().as_secs_f64(), + parsed.processing, + parsed.slots, + parsed.slot_bytes as f64 / 1e6, + parsed.threads + ); + Ok((worker, parsed)) + } + fn fail(&self, message: &str) { + // The orderly close (`Drop`) fails the registry too; that one is not a problem. + if message != CLOSED && self.registry.lock().failed.is_none() { + log::warn!("R3D helper {} ({}) failed: {message}", self.pid, self.label); + } + self.registry.fail(message); + let _ = self.child.lock().unwrap_or_else(std::sync::PoisonError::into_inner).kill(); + } + fn receive(&self, replies: mpsc::Receiver>, cancellable: bool) -> Result { + let started = Instant::now(); + let result = loop { + if cancellable && filmcraft_media::cancel::cancelled() { + return Err(MediaError::Cancelled); // The reader consumes and discards the eventual reply. + } + match replies.recv_timeout(Duration::from_millis(10)) { + Ok(reply) => break reply, + Err(mpsc::RecvTimeoutError::Timeout) if started.elapsed() < TIMEOUT => continue, + Err(e) => break Err(failure(format!("RED worker stopped responding: {e}"))), + } + }; + if let Err(e) = &result { + self.fail(&e.to_string()); + } + result + } + fn request(&self, command: &str) -> Result { + let (op, arguments) = command.split_once(' ').ok_or_else(|| failure("invalid RED command"))?; + let (sender, receiver) = mpsc::sync_channel(1); + let t0 = Instant::now(); + let id = self.registry.register(self.limit, sender)?; + let t1 = Instant::now(); + // Cancelled on the way to the pipe: nothing is sent, so nothing is decoded for it. + if filmcraft_media::cancel::cancelled() { + self.registry.unregister(id); + return Err(MediaError::Cancelled); + } + // Only the short write is serialized. Waiting and conversion never hold this lock. + let mut commands = self.commands.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if let Err(e) = commands.write_all(format!("{op} {id} {arguments}").as_bytes()).and_then(|_| commands.flush()) { + drop(commands); + self.fail(&format!("RED request write: {e}")); + return Err(io(e)); + } + drop(commands); + // the helper's error for this request: an error for its caller, the helper serves on + let result = self.receive(receiver, true).and_then(|reply| match reply { + Reply::Error(message) => Err(failure(format!("RED decoder: {message}"))), + other => Ok(other), + }); + if timing() && op == "V" { + let in_flight = self.registry.lock().requests.len(); + let ms = |d: Duration| d.as_secs_f64() * 1e3; + log::info!( + "R3D request {id} ({}): slot wait {:.1} ms, round trip {:.1} ms, {in_flight} still in flight, {}", + arguments.trim_end(), + ms(t1.saturating_duration_since(t0)), + ms(t1.elapsed()), + match &result { + Ok(_) => "ok", + Err(MediaError::Cancelled) => "cancelled", + Err(_) => "failed", + } + ); + } + result + } +} +/// The failure message of an orderly close: callers still waiting get it as their error. +const CLOSED: &str = "RED worker closed"; + +/// Whether the last helper closed exited with success (tests). +#[cfg(test)] +static LAST_CLOSE_SUCCESS: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); + +/// How long an orderly close waits for the helper to exit by itself (it finishes the decodes in +/// flight and releases the SDK) before killing it. +const ORDERLY_EXIT: Duration = Duration::from_secs(3); + +impl Drop for Worker { + fn drop(&mut self) { + let _ = self.shutdown.send(()); + let earlier = self.registry.lock().failed.clone(); + // An orderly close (idle, or replaced by a helper with larger slots) ends the helper's + // input: it exits 0 by itself, so the log tells a helper that really failed (a non-zero exit + // code) from an orderly close. A helper that + // failed, or does not exit in time, is killed as before. + // closed first: the reply reader then takes the helper's end of output as this close + self.registry.fail(CLOSED); + let exited = earlier.is_none() && { + *self.commands.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = Box::new(std::io::sink()); + let started = Instant::now(); + loop { + match self.child.lock().unwrap_or_else(std::sync::PoisonError::into_inner).try_wait() { + Ok(Some(_)) => break true, + Ok(None) if started.elapsed() < ORDERLY_EXIT => {} + Ok(None) | Err(_) => break false, + } + std::thread::sleep(Duration::from_millis(10)); + } + }; + if !exited { + self.fail(CLOSED); + } + let status = self.child.lock().unwrap_or_else(std::sync::PoisonError::into_inner).wait(); + #[cfg(test)] + LAST_CLOSE_SUCCESS.store(status.as_ref().is_ok_and(|s| s.success()), std::sync::atomic::Ordering::Relaxed); + let live = LIVE_PROCESSES.fetch_sub(1, std::sync::atomic::Ordering::Relaxed).saturating_sub(1); + let status = match status { + Ok(s) => s.to_string(), + Err(e) => format!("not reaped: {e}"), + }; + let why = earlier.map(|m| format!(" after failing: {m}")).unwrap_or_default(); + log::info!("R3D helper {} ({}) closed after {:.1} s ({status}){why}; {live} live", self.pid, self.label, self.started.elapsed().as_secs_f64()); + } +} + +/// The protocol tag: the first field of a helper's start-up reply and of every clip's metadata. +/// A helper built for another version is refused (its requests would be misread). +const PROTOCOL: &str = "FCRD9"; + +/// A helper's start-up reply (FCRD9): `FCRD9 `, then its +/// processing backend. The clips it serves are opened afterwards (`O`), each answered with its +/// [`Metadata`]. +#[derive(Debug)] +struct HubInfo { + /// Decode slots the GPU pipeline runs at once (0 for the CPU helper). + slots: usize, + /// Bytes per shared frame slot (0 for the CPU helper): whole pages, at least the largest + /// full-resolution frame the app asked for. + slot_bytes: usize, + /// Threads for audio and CPU frames in the helper. + threads: usize, + processing: String, +} +impl HubInfo { + fn parse(bytes: &[u8]) -> Result { + if bytes.len() > 1024 { + return Err(failure("oversized RED start-up reply")); + } + let text = std::str::from_utf8(bytes).map_err(|_| failure("invalid RED start-up encoding"))?; + let mut lines = text.lines(); + let fields: Vec<_> = lines.next().unwrap_or("").split_whitespace().collect(); + if fields.len() != 4 || fields.first() != Some(&PROTOCOL) { + return Err(failure("incompatible RED helper protocol; rebuild the app and its helper together")); + } + let n = |i: usize| -> Result { fields.get(i).and_then(|s| s.parse().ok()).ok_or_else(|| failure("invalid RED start-up number")) }; + let (slots, slot_bytes, threads) = (n(1)?, n(2)?, n(3)?); + if slots > 64 + || threads == 0 + || threads > 64 + || (slots == 0) != (slot_bytes == 0) + || (slots > 0 && (!slot_bytes.is_multiple_of(4096) || slot_bytes > MAX_REPLY as u64 + 4096 || slot_bytes.saturating_mul(slots) > MAX_SHARED)) + { + return Err(failure("RED helper reports unsupported slots")); + } + let processing = lines.next().ok_or_else(|| failure("RED helper did not report its processing backend; rebuild the helper"))?; + if processing.len() > 300 + || processing.chars().any(char::is_control) + || !(processing.starts_with("CUDA ") || processing.starts_with("OpenCL ") || processing == "CPU (explicit override)") + { + return Err(failure("invalid RED processing backend")); + } + Ok(Self { slots: slots as usize, slot_bytes: slot_bytes as usize, threads: threads as usize, processing: processing.into() }) + } +} + +#[derive(Debug)] +struct Metadata { + width: u32, + height: u32, + frames: i64, + rate: FrameRate, + channels: usize, + sample_rate: u32, + samples: u64, + float_audio: bool, + bits: u32, + timecode: Option, + duration: Tick, + /// The clip's recorded ISO, white balance and exposure adjustment (FCRD7): what a request + /// without RAW adjustments develops with, shown by Source Settings as the defaults. + as_shot: RawAsShot, + /// What else the clip (or its RMD sidecar) records (FCRD9): Source Settings' "as recorded". + clip: RawClipInfo, + /// The pixel aspect ratio of the lens squeeze the clip records + /// ([`filmcraft_media::lens_squeeze_par`]); square when it records none. + par: (u32, u32), +} +/// The most shared frame memory a helper may ask the app to map. +const MAX_SHARED: u64 = 16 << 30; +impl Metadata { + fn parse(bytes: &[u8]) -> Result { + if bytes.len() > 16384 { + return Err(failure("oversized RED metadata")); + } + let text = std::str::from_utf8(bytes).map_err(|_| failure("invalid RED metadata encoding"))?; + let mut lines = text.lines(); + let fields: Vec<_> = lines.next().unwrap_or("").split_whitespace().collect(); + if fields.len() != 13 || fields.first() != Some(&PROTOCOL) { + return Err(failure("incompatible RED helper protocol")); + } + let n = |i: usize| -> Result { fields.get(i).and_then(|s| s.parse().ok()).ok_or_else(|| failure("invalid RED metadata number")) }; + let (w, h, frames, num, den, channels, rate, samples, floating, bits, slots, slot_bytes) = + (n(1)?, n(2)?, n(3)?, n(4)?, n(5)?, n(6)?, n(7)?, n(8)?, n(9)?, n(10)?, n(11)?, n(12)?); + if slots > 64 { + return Err(failure("RED helper reports an unsupported slot count")); + } + if w == 0 + || h == 0 + || w > 8192 + || h > 8192 + || w * h > MAX_PIXELS as u64 + || frames == 0 + || frames > i64::MAX as u64 + || num == 0 + || den == 0 + || num > 1_000_000 + || den > 1_000_000 + || num > den * 1000 + || channels > 16 + || samples > i64::MAX as u64 + || floating > 1 + || rate > 384_000 + || bits > 32 + || (channels > 0 && (rate == 0 || ![24, 32].contains(&bits))) + { + return Err(failure("RED clip metadata outside supported limits")); + } + // A GPU helper shares slots of at least one full-resolution frame, in whole pages; a + // CPU-only helper shares nothing. + if (slots == 0) != (slot_bytes == 0) + || (slots > 0 + && (slot_bytes < w * h * 6 || !slot_bytes.is_multiple_of(4096) || slot_bytes > MAX_REPLY as u64 + 4096 || slot_bytes * slots > MAX_SHARED)) + { + return Err(failure("RED helper reports unsupported shared frame memory")); + } + let ticks = frames as i128 * den as i128 * TICKS_PER_SECOND as i128 / num as i128; + if ticks == 0 || ticks > i64::MAX as i128 { + return Err(failure("R3D duration outside supported range")); + } + let frame_rate = FrameRate { num: num as i64, den: den as i64 }; + let tc = lines.next().unwrap_or(""); + let timecode = if tc.is_empty() { + None + } else { + Some(filmcraft_time::parse_timecode(tc, frame_rate, tc.contains(';'), 0).map_err(|_| failure("invalid R3D timecode"))?) + }; + let processing = lines.next().ok_or_else(|| failure("RED helper did not report its processing backend; rebuild the helper"))?; + if processing.len() > 300 + || processing.chars().any(char::is_control) + || !(processing.starts_with("CUDA ") || processing.starts_with("OpenCL ") || processing == "CPU (explicit override)") + { + return Err(failure("invalid RED processing backend")); + } + // FCRD7: `ISO Kelvin Tint ExposureAdjust` as the SDK reports them for the clip. The values + // are hostile numbers from a helper: finite and within the SDK's own limits (with room + // for an SDK that widens them), or the clip is refused. + let raw: Vec<_> = + lines.next().ok_or_else(|| failure("RED helper did not report the clip's RAW settings; rebuild the helper"))?.split_whitespace().collect(); + let f = |i: usize| -> Result { + raw.get(i).and_then(|s| s.parse::().ok()).filter(|v| v.is_finite()).ok_or_else(|| failure("invalid RED as-shot value")) + }; + let iso = raw.first().and_then(|s| s.parse::().ok()).ok_or_else(|| failure("invalid RED as-shot ISO"))?; + let (kelvin, tint, exposure) = (f(1)?, f(2)?, f(3)?); + if raw.len() != 4 + || iso == 0 + || iso > 1_000_000 + || !(1000.0..=1_000_000.0).contains(&kelvin) + || !(-1000.0..=1000.0).contains(&tint) + || !(-100.0..=100.0).contains(&exposure) + { + return Err(failure("RED as-shot settings outside supported limits")); + } + let clip = parse_clip_info(lines.next(), lines.next())?; + // The recorded lens squeeze (the desqueeze factor; 0 when none is recorded). A helper + // from before it sends no line; a value that is no usable ratio is ignored, not refused: + // the clip still plays, with square pixels. + let par = match lines.next().map(str::trim) { + None | Some("") => (1, 1), + Some(v) => v.parse::().ok().and_then(filmcraft_media::lens_squeeze_par).unwrap_or((1, 1)), + }; + Ok(Self { + width: w as u32, + height: h as u32, + frames: frames as i64, + rate: frame_rate, + channels: channels as usize, + sample_rate: rate as u32, + samples, + float_audio: floating == 1, + bits: bits as u32, + timecode, + duration: Tick(ticks as i64), + as_shot: RawAsShot { iso, kelvin, tint, exposure }, + clip, + par, + }) + } +} + +struct State { + /// The last frame: its index, preview divisor, development settings and image. + last: Option<(i64, u32, RawDevelop, Arc)>, +} + +/// The two helpers every R3D source shares (FCRD8). The GPU helper decodes foreground frames +/// into its shared slots; the CPU helper serves audio, thumbnails, metadata probes and the frames +/// of a GPU that is unavailable. A clip is opened in a helper (`O`, ~6 ms inside a running SDK) +/// the first time a source needs it there, so a timeline of a hundred clips is two processes and +/// a cut is a clip already open, not a helper start (helpers per source would cost a GPU helper +/// start of 0.2-0.8 s at each cut and a process and its pools per clip: a hundred-clip timeline +/// ran over a hundred of them). +/// +/// A helper that fails or exits is started again by the next request; its `generation` tells the +/// sources their clips must be opened again. One idle for [`Hub::idle`] is closed (the GPU one +/// gives its memory back) and started again when it is next needed. +struct Hub { + cpu_only: bool, + idle: Duration, + state: Mutex, + /// A [`Hub::warm`] start is under way. + warming: std::sync::atomic::AtomicBool, +} +#[derive(Default)] +struct HubState { + worker: Option>, + /// Counts helper starts: a source's clip handle is valid for the generation it was opened in. + generation: u64, + /// The current GPU helper's frame slot size. + slot_bytes: usize, + last_used: Option, + /// The GPU helper's failed starts, and until when frames decode on the CPU helper instead. + backoff: GpuBackoff, +} +/// The frame slot size the GPU helper starts with: a Nikon ZR 6K frame (6048 × 3402, 6 bytes a +/// pixel); a larger clip starts a helper with larger slots. +const DEFAULT_SLOT_BYTES: usize = 6048 * 3402 * 6; +/// Idle time after which the GPU helper is closed (its CUDA context and pools hold 1-2 GB). +const GPU_IDLE: Duration = Duration::from_secs(120); +/// Idle time after which the CPU helper is closed. +const CPU_IDLE: Duration = Duration::from_secs(60); +/// Clip handles, unique across both helpers and every source. +static HANDLES: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1); +fn next_handle() -> u64 { + HANDLES.fetch_add(1, std::sync::atomic::Ordering::Relaxed) +} +fn gpu_hub() -> &'static Hub { + static HUB: OnceLock = OnceLock::new(); + HUB.get_or_init(|| Hub::new(false, GPU_IDLE)) +} +fn cpu_hub() -> &'static Hub { + static HUB: OnceLock = OnceLock::new(); + HUB.get_or_init(|| Hub::new(true, CPU_IDLE)) +} +/// Lock `slot`, giving up when the caller's work is cancelled instead of waiting behind a start +/// that takes a moment (another frame starting the helper). +fn lock_cancellably(slot: &Mutex) -> Result> { + let started = Instant::now(); + loop { + match slot.try_lock() { + Ok(guard) => return Ok(guard), + Err(std::sync::TryLockError::Poisoned(poisoned)) => return Ok(poisoned.into_inner()), + Err(std::sync::TryLockError::WouldBlock) => {} + } + if filmcraft_media::cancel::cancelled() { + return Err(MediaError::Cancelled); + } + if started.elapsed() >= TIMEOUT { + return Err(failure("RED helper start stayed locked for 120 seconds")); + } + std::thread::sleep(Duration::from_millis(2)); + } +} +impl Hub { + fn new(cpu_only: bool, idle: Duration) -> Self { + start_hub_reaper(); + Self { cpu_only, idle, state: Mutex::new(HubState::default()), warming: Default::default() } + } + fn lock(&self) -> std::sync::MutexGuard<'_, HubState> { + self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner) + } + /// The helper and its generation: the running one, or one started now (again after a + /// failure, or with larger slots when a clip needs `slot_bytes`). While the GPU helper is + /// backing off after a failed start, an error without trying. + fn worker(&self, slot_bytes: usize) -> Result<(Arc, u64)> { + let mut st = lock_cancellably(&self.state)?; + let running = st.worker.as_ref().filter(|w| w.registry.lock().failed.is_none()).cloned(); + if let Some(worker) = running + && (self.cpu_only || st.slot_bytes >= slot_bytes) + { + st.last_used = Some(Instant::now()); + return Ok((worker, st.generation)); + } + if !self.cpu_only && st.backoff.until.is_some_and(|until| Instant::now() < until) { + return Err(failure("RED GPU helper unavailable (backing off after a failed start)")); + } + // The old helper (failed, or with slots too small) goes; requests still running on it + // hold it until they end. + st.worker = None; + let want = if self.cpu_only { 0 } else { slot_bytes.max(DEFAULT_SLOT_BYTES).max(st.slot_bytes) }; + match Worker::open_hub(self.cpu_only, want) { + Ok((worker, info)) => { + st.generation = st.generation.wrapping_add(1); + st.slot_bytes = info.slot_bytes; + st.backoff = GpuBackoff::default(); + st.last_used = Some(Instant::now()); + let worker = Arc::new(worker); + st.worker = Some(worker.clone()); + if !self.cpu_only { + log::info!("R3D NE processing: {}", info.processing); + } + Ok((worker, st.generation)) + } + Err(MediaError::Cancelled) => Err(MediaError::Cancelled), + Err(e) => { + if !self.cpu_only { + st.backoff.failures = st.backoff.failures.saturating_add(1); + let delay = backoff_delay(st.backoff.failures); + st.backoff.until = Some(Instant::now() + delay); + log::warn!( + "R3D NE GPU unavailable: {e}; frames decode on the CPU helper for the next {} s (failed start {})", + delay.as_secs(), + st.backoff.failures + ); + } + Err(e) + } + } + } + /// Have the helper running for frames of `slot_bytes` soon, without waiting for it: a running + /// one counts as used (it is not closed as idle), a missing one is started on a thread of its + /// own. Still frames decode on the CPU helper, and playback that starts after a scrub must + /// find the GPU helper ready: started by its first frame, the first half second of playback + /// was dropped frames (17 of 236 in a measured run). + fn warm(&'static self, slot_bytes: usize) { + { + let Ok(mut st) = self.state.try_lock() else { return }; // starting or busy: nothing to do + let running = st.worker.as_ref().is_some_and(|w| w.registry.lock().failed.is_none()); + if running && (self.cpu_only || st.slot_bytes >= slot_bytes) { + st.last_used = Some(Instant::now()); + return; + } + if st.backoff.until.is_some_and(|until| Instant::now() < until) { + return; + } + } + if self.warming.swap(true, std::sync::atomic::Ordering::AcqRel) { + return; + } + let spawned = std::thread::Builder::new().name("red-helper-warm".into()).spawn(move || { + let started = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| self.worker(slot_bytes).map(|_| ()))); + if let Ok(Err(e)) = &started { + log::debug!("R3D helper warm start: {e}"); + } + self.warming.store(false, std::sync::atomic::Ordering::Release); + }); + if spawned.is_err() { + self.warming.store(false, std::sync::atomic::Ordering::Release); + } + } + /// The GPU helper is backing off after a failed start: frames go to the CPU helper. + fn unavailable(&self) -> bool { + self.lock().backoff.until.is_some_and(|until| Instant::now() < until) + } + /// What the running helper delivers (see [`Pending::delivery`]); nothing before it runs. + fn delivery(&self) -> (Option, bool) { + // a hint, asked for every frame: never wait for a helper being started under the lock + let worker = match self.state.try_lock() { + Ok(st) => st.worker.clone(), + Err(std::sync::TryLockError::Poisoned(p)) => p.into_inner().worker.clone(), + Err(std::sync::TryLockError::WouldBlock) => return (None, false), + }; + worker.map_or((None, false), |w| w.registry.lock().delivery()) + } + /// The clip `handle` opened in helper `generation` is no longer needed: closed there (`C`), if + /// that helper still runs. Best effort: a helper gone took its clips with it. + fn close_clip(&self, generation: u64, handle: u64) { + let worker = { + let st = self.lock(); + if st.generation != generation { + return; + } + st.worker.clone() + }; + if let Some(worker) = worker { + let mut commands = worker.commands.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let _ = commands.write_all(format!("C {handle}\n").as_bytes()).and_then(|_| commands.flush()); + } + } + /// Close the helper if nothing has asked it for anything for [`Self::idle`] and nothing is in + /// flight; the next request starts it again (the sources reopen their clips). + fn sweep(&self) { + let closing = { + let mut st = self.lock(); + let idle = st.last_used.is_none_or(|t| t.elapsed() >= self.idle); + let quiet = st.worker.as_ref().is_some_and(|w| w.registry.lock().requests.is_empty()); + if !(idle && quiet) { + return; + } + st.worker.take() + }; + if let Some(worker) = closing { + log::info!("R3D helper {} ({}) closed: idle for {} s", worker.pid, worker.label, self.idle.as_secs()); + drop(worker); + } + } + /// The running helper, if any (tests). + #[cfg(test)] + fn running(&self) -> Option> { + self.lock().worker.clone() + } +} +/// The thread that closes idle helpers (started with the first hub; its sweep runs under +/// `catch_unwind`, and a thread that could not be started only means idle helpers stay up). +fn start_hub_reaper() { + static STARTED: OnceLock<()> = OnceLock::new(); + STARTED.get_or_init(|| { + let spawned = std::thread::Builder::new().name("red-helper-reaper".into()).spawn(|| { + loop { + std::thread::sleep(Duration::from_secs(1)); + if std::panic::catch_unwind(|| { + gpu_hub().sweep(); + cpu_hub().sweep(); + }) + .is_err() + { + log::warn!("R3D helper reaper: a sweep panicked; the next runs anyway"); + } + } + }); + if let Err(e) = spawned { + log::warn!("R3D helper reaper not started: {e}; idle helpers stay up"); + } + }); +} +/// Open `path` as clip `handle` in `worker` (`O`): the clip's metadata, or the helper's reason. +fn open_in(worker: &Worker, handle: u64, path: &Path) -> Result> { + let path = path.to_str().ok_or_else(|| failure("R3D path is not valid Unicode"))?; + if path.is_empty() || path.contains(['\n', '\r']) || path.len() > 8192 { + return Err(failure("unsupported R3D path")); + } + worker.request(&format!("O {handle} {path}\n"))?.into_inline() +} + +/// The GPU helper failed to become ready (an exit before its start-up reply: CUDA +/// initialisation or the SDK's pool allocation failing on a full GPU, a missing runtime). Frames +/// decode on the CPU helper until `until`, and each failure doubles the wait, from +/// [`BACKOFF_FIRST`] to [`BACKOFF_MAX`]; a start that succeeds clears it. Without it a failed +/// start was retried by the next frame request at once, each try a process on a GPU that had +/// just refused one. +#[derive(Default)] +struct GpuBackoff { + failures: u32, + until: Option, +} +const BACKOFF_FIRST: Duration = Duration::from_secs(2); +const BACKOFF_MAX: Duration = Duration::from_secs(60); +/// The wait after the `failures`th consecutive failed start. +fn backoff_delay(failures: u32) -> Duration { + let doublings = failures.saturating_sub(1).min(16); + BACKOFF_FIRST.saturating_mul(1u32 << doublings).min(BACKOFF_MAX) +} + +pub struct RedSource { + info: MediaInfo, + metadata: Metadata, + state: Mutex, + path: PathBuf, + /// This clip in the GPU helper and in the CPU helper ([`Hub`]): the helper generation it was + /// opened in and its handle there. Opened on first use; a helper started again since means + /// the clip is opened again. + gpu_clip: Mutex>, + cpu_clip: Mutex>, + /// Clip ▸ Source Settings…: how the helper develops the picture ([`MediaSource::set_develop`]). + develop: Mutex, + /// What the clip records (as shot, and the rest): from its metadata, and again after an RMD + /// sidecar write changed it ([`MediaSource::write_raw_sidecar`]). + recorded: Mutex<(RawAsShot, RawClipInfo)>, +} + +/// The position of `v` in `all` (the request codes of the development fields follow the order of +/// the settings' `ALL` lists, which the helper's tables mirror). +fn code(all: &[T], v: T) -> usize { + all.iter().position(|x| *x == v).unwrap_or(0) +} + +/// The development fields of a request (FCRD9), after `V ` (with the +/// half-resolution quality first: `half`, 0 premium or 1 good) or after `W `: colour +/// space and gamma (their place in [`RawColorSpace::ALL`] and [`RawGamma::ALL`]), tone map and +/// rolloff (0 = as recorded), the HDR peak (0 = as recorded), an options word (1 chroma noise +/// reduction set, 2 its value, 4 apply the metadata curves, 8 the creative LUT, 16 the CDL), the +/// flashing pixel adjustment (0 = as recorded), then the RAW adjustments as a flag word (bit 0 ISO, +/// 1 Kelvin, 2 tint, 3 exposure; a clear bit means as shot) followed by the ISO, the Kelvin, the +/// tint in hundredths and the exposure adjustment in hundredths of a stop, each 0 unless its bit +/// is set. The helper checks every value against the SDK's limits and answers anything else with +/// an error; the same ranges are checked here first ([`RawDevelop::validate`]) so a bad project +/// value is an error message, not a helper round trip. +fn wire(d: &RawDevelop, half: Option) -> Result { + d.validate().map_err(|e| failure(format!("RAW development setting refused: {e}")))?; + let options = d.chroma_noise_reduction.map_or(0, |on| 1 | if on { 2 } else { 0 }) + | if d.metadata_curves { 4 } else { 0 } + | if d.creative_lut { 8 } else { 0 } + | if d.cdl { 16 } else { 0 }; + let flashing = d.flashing_pixel.map_or(0, |f| code(&RawFlashing::ALL, f) + 1); + let flags = u8::from(d.iso.is_some()) | u8::from(d.kelvin.is_some()) << 1 | u8::from(d.tint.is_some()) << 2 | u8::from(d.exposure.is_some()) << 3; + let fields = format!( + "{} {} {} {} {} {options} {flashing} {flags} {} {} {} {}", + code(&RawColorSpace::ALL, d.color_space), + code(&RawGamma::ALL, d.gamma), + code(&RawToneMap::ALL, d.tone_map), + code(&RawRollOff::ALL, d.rolloff), + d.hdr_peak_nits.unwrap_or(0), + d.iso.unwrap_or(0), + d.kelvin.unwrap_or(0), + d.tint.map_or(0, |t| t.0), + d.exposure.map_or(0, |e| e.0), + ); + Ok(match half { + Some(good) => format!("{} {fields}", u8::from(good)), + None => fields, + }) +} + +/// Parse the FCRD9 lines after the as-shot line of a clip's metadata: what else it records +/// (codes as in [`wire`], 0 for a recorded value without one; the HDR peak, flags, track count, +/// the SDK's HDR peak range and default tone map and rolloff, the CDL) and its RMD sidecar path. +/// Hostile numbers from a helper: every one bounded, or the clip is refused. +fn parse_clip_info(line: Option<&str>, path: Option<&str>) -> Result { + let line = line.ok_or_else(|| failure("RED helper did not report the clip's recorded settings; rebuild the helper"))?; + let f: Vec<&str> = line.split_whitespace().collect(); + if f.len() != 28 { + return Err(failure("invalid RED recorded settings")); + } + let n = |i: usize, max: u64| -> Result { + f.get(i).and_then(|s| s.parse::().ok()).filter(|v| *v <= max).ok_or_else(|| failure("invalid RED recorded setting")) + }; + let pick = |i: usize, max: u64| -> Result> { Ok(n(i, max)?.checked_sub(1).map(|v| v as usize)) }; + let flag = |i: usize| -> Result { Ok(n(i, 1)? == 1) }; + let mut cdl = [0f32; 10]; + for (k, out) in cdl.iter_mut().enumerate() { + *out = f + .get(18 + k) + .and_then(|s| s.parse::().ok()) + .filter(|v| v.is_finite() && v.abs() <= 1000.0) + .ok_or_else(|| failure("invalid RED recorded CDL"))?; + } + let path = path.unwrap_or(""); + if path.len() > 8192 || path.chars().any(char::is_control) { + return Err(failure("invalid RED RMD path")); + } + let (lo, hi, default) = (n(13, 100_000)? as u32, n(14, 100_000)? as u32, n(15, 100_000)? as u32); + let rmd = flag(12)?; + Ok(RawClipInfo { + color_space: pick(0, 4)?.and_then(|i| RawColorSpace::ALL.get(i).copied()), + gamma: pick(1, 6)?.and_then(|i| RawGamma::ALL.get(i).copied()), + // tone map and rolloff codes count from None (1): the ALL lists start with "as recorded" + tone_map: pick(2, 4)?.and_then(|i| RawToneMap::ALL.get(i + 1).copied()), + rolloff: pick(3, 5)?.and_then(|i| RawRollOff::ALL.get(i + 1).copied()), + hdr_peak_nits: Some(n(4, 100_000)? as u32).filter(|p| *p > 0), + chroma_noise_reduction: flag(5)?, + flashing_pixel: pick(6, 4)?.and_then(|i| RawFlashing::ALL.get(i).copied()).unwrap_or_default(), + lut: flag(7)?, + lut_enabled: flag(8)?, + cdl_enabled: flag(9)?, + cdl, + curves: flag(10)?, + hdrx: n(11, 16)? > 1, + rmd: rmd.then(|| path.to_string()), + rmd_path: path.to_string(), + hdr_peak_range: (lo, hi), + hdr_peak_default: default, + tone_map_default: pick(16, 4)?.and_then(|i| RawToneMap::ALL.get(i + 1).copied()), + rolloff_default: pick(17, 5)?.and_then(|i| RawRollOff::ALL.get(i + 1).copied()), + }) +} + +/// What a developed frame holds: linear light (the helper's half floats, or the code values of +/// any other gamma linearised here) in the developed primaries. The compositor's interpretation +/// of a frame whose primaries are not Rec. 709 comes from the clip's development +/// (`Project::color_space_override`), since container metadata cannot name them all. +fn frame_color(d: &RawDevelop) -> ColorInfo { + let primaries = match d.color_space { + RawColorSpace::Rec2020 => filmcraft_color::Primaries::Bt2020, + RawColorSpace::P3D65 => filmcraft_color::Primaries::P3D65, + RawColorSpace::Rec709 | RawColorSpace::RedWideGamutRgb => filmcraft_color::Primaries::Bt709, + }; + ColorInfo { primaries, ..linear_color() } +} + +/// Code value → linear light for each of the helper's 65 536 16-bit code values of `gamma`, as +/// IEEE halves: the compositor's own curve ([`filmcraft_color::transform::decode_channel`], full +/// range), so its interpretation of the clip and these frames agree exactly (mid grey encoded by +/// the curve comes back as mid grey). Built once per curve. +fn linearise_table(gamma: RawGamma) -> &'static [u16] { + static TABLES: [OnceLock>; 6] = [const { OnceLock::new() }; 6]; + let slot = code(&RawGamma::ALL, gamma); + let table = TABLES.get(slot).unwrap_or(&TABLES[0]); + table.get_or_init(|| { + (0..=u16::MAX) + .map(|c| { + let v = f64::from(c) / 65535.0; + let l = filmcraft_color::transform::decode_channel(v, gamma.curve(), filmcraft_color::Range::Full); + filmcraft_frame::f32_to_half(if l.is_finite() { l as f32 } else { 0.0 }) + }) + .collect() + }) +} + +impl RedSource { + /// Open `path`: its metadata through the CPU helper (the clip is opened there, ~6 ms, and + /// stays open for the source's audio and thumbnails); no GPU work until a frame is asked for. + fn open(name: &str, path: &Path, len: u64) -> Result { + let (worker, generation) = cpu_hub().worker(0)?; + let handle = next_handle(); + let bytes = open_in(&worker, handle, path)?; + let m = match Metadata::parse(&bytes) { + Ok(m) => m, + Err(e) => { + cpu_hub().close_clip(generation, handle); + return Err(e); + } + }; + let audio_streams = if m.channels == 0 { + vec![] + } else { + vec![AudioStreamInfo { + sample_rate: m.sample_rate, + channels: m.channels as u32, + codec: if m.float_audio { "pcm_f32be" } else { "pcm_s24be" }.into(), + bits_per_sample: Some(m.bits), + }] + }; + let info = MediaInfo { + name: name.into(), + kind: MediaKind::Movie, + duration: m.duration, + video: Some(VideoStreamInfo { + width: m.width, + height: m.height, + frame_rate: m.rate, + par: m.par, + codec: "R3D NE".into(), + pixel_format: "RAW 12-bit → linear RGB float".into(), + color: linear_color(), + has_alpha: false, + bitrate: None, + hdr: None, + }), + audio_streams, + container: "R3D NE".into(), + start_timecode: m.timecode, + file_size: Some(len), + }; + let recorded = Mutex::new((m.as_shot, m.clip.clone())); + Ok(RedSource { + info, + metadata: m, + state: Mutex::new(State { last: None }), + path: path.into(), + gpu_clip: Mutex::new(None), + recorded, + cpu_clip: Mutex::new(Some((generation, handle))), + develop: Mutex::new(RawDevelop::default()), + }) + } + /// This clip in `hub` (its slot in this source is `mine`): the helper and the clip's handle + /// there, opened now when it is not yet (or the helper was started again since). The + /// source's slot stays locked while the clip opens, so two frames of one source open it once. + fn clip_on(&self, hub: &Hub, mine: &Mutex>) -> Result<(Arc, u64)> { + let mut mine = lock_cancellably(mine)?; + let m = &self.metadata; + let frame_bytes = (m.width as usize).saturating_mul(m.height as usize).saturating_mul(6); + let (worker, generation) = hub.worker(frame_bytes)?; + if let Some((opened_in, handle)) = *mine + && opened_in == generation + { + return Ok((worker, handle)); + } + let handle = next_handle(); + let bytes = open_in(&worker, handle, &self.path)?; + let opened = Metadata::parse(&bytes); + let same = opened.as_ref().is_ok_and(|o| (o.width, o.height, o.frames) == (m.width, m.height, m.frames)); + if !same { + hub.close_clip(generation, handle); + return Err(opened.err().unwrap_or_else(|| failure("R3D clip changed on disk; reopen it"))); + } + *mine = Some((generation, handle)); + Ok((worker, handle)) + } +} +impl Drop for RedSource { + fn drop(&mut self) { + for (hub, mine) in [(gpu_hub(), &self.gpu_clip), (cpu_hub(), &self.cpu_clip)] { + if let Some((generation, handle)) = *mine.lock().unwrap_or_else(std::sync::PoisonError::into_inner) { + hub.close_clip(generation, handle); + } + } + } +} + +pub fn reader_opener(name: &str, _: &[u8], reader: &SharedReader) -> Option> { + if !filmcraft_codecs::r3d::is_r3d(name) { + return None; + } + Some((|| { + let path = reader + .local_path() + .ok_or_else(|| MediaError::Unsupported("RED SDK requires a local R3D file; browser and memory sources are not supported".into()))?; + Ok(Arc::new(RedSource::open(name, path, reader.len())?) as SharedSource) + })()) +} + +fn linear_color() -> ColorInfo { + ColorInfo { transfer: Transfer::Linear, ..ColorInfo::SRGB_FULL } +} + +/// The frame the SDK writes for a reduced decode: both dimensions rounded down, rows packed at +/// that width (SDK 9.2.1, measured on 6048×3402 for half, quarter, eighth and sixteenth; see +/// `docs/r3d-ne.md`). `None` for a divisor that is not one of the SDK's or that leaves no pixel. +/// The helper sizes its buffer by the same rule; `video_texels` and `video_linear` reject any +/// other byte count. +fn reduced_size(width: u32, height: u32, divisor: u32) -> Option<(u32, u32)> { + if ![1, 2, 4, 8, 16].contains(&divisor) { + return None; + } + let (w, h) = (width.checked_div(divisor)?, height.checked_div(divisor)?); + (w > 0 && h > 0).then_some((w, h)) +} + +/// The smallest divisor a still frame is decoded at by the CPU helper rather than the GPU one. +/// One frame through the GPU decoder takes ~160 ms at any size on a current desktop GPU (its pipeline is +/// built for throughput); the CPU decoder answers a quarter in 50-95 ms, an eighth in 30-35 ms +/// and a sixteenth in 15-22 ms, but a half in 0.6-1.3 s (6K and 4K Nikon ZR clips on a desktop CPU, +/// `real_random_frame_latency_on_each_helper`). `FILMCRAFT_R3D_STILL_CPU_DIVISOR` overrides it +/// (0: never). +fn still_cpu_divisor() -> u32 { + static DIVISOR: std::sync::OnceLock = std::sync::OnceLock::new(); + *DIVISOR.get_or_init(|| { + std::env::var("FILMCRAFT_R3D_STILL_CPU_DIVISOR") + .ok() + .and_then(|v| v.trim().parse::().ok()) + .filter(|d| [0, 1, 2, 4, 8, 16].contains(d)) + .unwrap_or(4) + }) +} + +/// Whether a frame at `divisor` goes to the CPU helper for its latency: a still frame at the CPU +/// divisor or smaller. +fn still_on_cpu(still: bool, divisor: u32) -> bool { + let from = still_cpu_divisor(); + still && from != 0 && divisor >= from +} + +/// The divisor a preview at `scale` decodes at under a decode quality whose smallest divisor is +/// `cap` ([`filmcraft_media::RawQuality::divisor`]): the covering one ([`preview_divisor`]) or +/// the quality's, whichever is smaller, falling back to the next larger decode for a clip too +/// small for it. +fn capped_divisor(width: u32, height: u32, scale: f32, cap: u32) -> u32 { + let mut d = preview_divisor(width, height, scale).max(cap).min(16); + while d > 1 && reduced_size(width, height, d).is_none() { + d /= 2; + } + d +} + +fn preview_divisor(width: u32, height: u32, scale: f32) -> u32 { + // The smallest decode that still covers the requested scale (an odd dimension loses up to + // one pixel row or column to the SDK's rounding, which a preview can spare; full requests + // never reduce). + [16, 8, 4, 2].into_iter().find(|d| scale <= 1.0 / *d as f32 && reduced_size(width, height, *d).is_some()).unwrap_or(1) +} + +/// The helper's RGB halves (native little endian on x64, 6 bytes a pixel) as RGBA halves with +/// alpha 1.0: the frame's [`filmcraft_frame::PixelData::RgbaF16`] layout, which the GPU uploads +/// as it is. A copy with two bytes inserted per pixel (a 6K half-resolution frame in a few +/// milliseconds), not a 32-bit image (82 MB written, then converted back to halves for the +/// upload, ~120 ms of a frame worker). Infinities and NaNs (an exponent field of all ones) are +/// rejected, as the f32 conversion did. +fn video_texels(bytes: &[u8], pixels: usize) -> Result> { + if pixels > MAX_PIXELS || bytes.len() != pixels * 6 { + return Err(failure("RED video buffer size mismatch")); + } + let len = pixels * 8; + let mut rgba = filmcraft_frame::pool::take_u8(len); + rgba.try_reserve_exact(len).map_err(|_| failure("RED frame allocation failed"))?; + rgba.resize(len, 0); + let mut non_finite = false; + for (rgb, texel) in bytes.as_chunks::<6>().0.iter().zip(rgba.as_chunks_mut::<8>().0.iter_mut()) { + texel[..6].copy_from_slice(rgb); + texel[6] = 0x00; + texel[7] = 0x3c; // 1.0 + non_finite |= (rgb[1] & 0x7c == 0x7c) | (rgb[3] & 0x7c == 0x7c) | (rgb[5] & 0x7c == 0x7c); + } + if non_finite { + filmcraft_frame::pool::recycle_u8(rgba); + return Err(failure("non-finite RED video sample")); + } + Ok(rgba) +} +/// The helper's 16-bit integer RGB code values (every gamma but linear, 6 bytes a pixel, read in +/// place from the slot) linearised through `table` ([`linearise_table`]) into RGBA halves with +/// alpha 1.0: the frame's [`filmcraft_frame::PixelData::RgbaF16`] layout, linear light as every +/// float frame is, which the GPU uploads as it is. A NaN half is a plain code value here: nothing +/// is rejected but a wrong size (or a table that is not one per code value). +fn video_linear(bytes: &[u8], pixels: usize, table: &[u16]) -> Result> { + if pixels > MAX_PIXELS || bytes.len() != pixels * 6 || table.len() != 65536 { + return Err(failure("RED video buffer size mismatch")); + } + let len = pixels * 8; + let mut rgba = filmcraft_frame::pool::take_u8(len); + rgba.try_reserve_exact(len).map_err(|_| failure("RED frame allocation failed"))?; + rgba.resize(len, 0); + for (rgb, texel) in bytes.as_chunks::<6>().0.iter().zip(rgba.as_chunks_mut::<8>().0.iter_mut()) { + for (value, output) in rgb.as_chunks::<2>().0.iter().zip(texel.as_chunks_mut::<2>().0.iter_mut()) { + let h = table.get(usize::from(u16::from_le_bytes(*value))).copied().unwrap_or(0); + *output = h.to_le_bytes(); + } + texel[6] = 0x00; + texel[7] = 0x3c; // 1.0 + } + Ok(rgba) +} +fn audio_samples(bytes: &[u8], channels: usize, requested: usize, floating: bool) -> Result>> { + if channels == 0 || channels > 16 || requested > MAX_SAMPLES || !bytes.len().is_multiple_of(channels * 4) || bytes.len() / (channels * 4) > requested { + return Err(failure("RED audio buffer size mismatch")); + } + let mut out = vec![Vec::with_capacity(bytes.len() / (channels * 4)); channels]; + for frame in bytes.chunks_exact(channels * 4) { + for (c, word) in frame.as_chunks::<4>().0.iter().enumerate() { + let bits = [word[0], word[1], word[2], word[3]]; + let v = if floating { f32::from_be_bytes(bits) } else { i32::from_be_bytes(bits) as f32 / 2147483648.0 }; + if !v.is_finite() { + return Err(failure("non-finite RED audio sample")); + } + out[c].push(v); + } + } + Ok(out) +} + +impl MediaSource for RedSource { + fn info(&self) -> &MediaInfo { + &self.info + } + /// Every R3D frame decodes on its own, and the helper's reply timings say how many per + /// second it manages: playback asks for the frames it can finish, not for all of them. + fn frame_delivery(&self) -> filmcraft_media::FrameDelivery { + // The GPU helper serves every clip: its reading is what any of them gets. Not started yet + // (or closed while idle), it has none; the first frames start it. + let (rate, fresh) = gpu_hub().delivery(); + filmcraft_media::FrameDelivery { independent: true, rate, fresh } + } + fn video_frame(&self, req: FrameRequest) -> Result> { + if filmcraft_media::cancel::cancelled() { + return Err(MediaError::Cancelled); + } + if !req.scale.is_finite() || req.scale <= 0.0 { + return Err(failure("invalid R3D frame scale")); + } + let m = &self.metadata; + let index = m.rate.frame_at(req.time).clamp(0, m.frames - 1); + let develop = *self.develop.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let quality = develop.quality(); + // a full-size request (an export, a monitor at 100 %) always decodes at full resolution; + // a preview at the smallest size that covers it, never larger than the decode quality + let divisor = if req.scale >= 1.0 { 1 } else { capped_divisor(m.width, m.height, req.scale, quality.divisor()) }; + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some((previous, previous_divisor, previous_develop, frame)) = &state.last + && *previous == index + && *previous_divisor == divisor + && *previous_develop == develop + { + return Ok(frame.clone()); + } + state.last = None; // Release the previous large frame before decoding another. + drop(state); + let develop_fields = wire(&develop, Some(divisor == 2 && quality.half_good()))?; + // Background work (a thumbnail) never touches the GPU: the CPU helper decodes it at the + // request's divisor. So does every frame while the GPU helper is unavailable, the frame + // whose GPU start just failed (the back-off starts with it), and a still frame (a scrub, + // a paused monitor) small enough for the CPU decoder to answer first (`still_on_cpu`). + let on_cpu = filmcraft_media::cancel::background() || still_on_cpu(filmcraft_media::cancel::still(), divisor) || gpu_hub().unavailable(); + if on_cpu && !filmcraft_media::cancel::background() { + let frame_bytes = (m.width as usize).saturating_mul(m.height as usize).saturating_mul(6); + gpu_hub().warm(frame_bytes); + } + // The reply, or the error and whether the helper itself went (killed, crashed) under it. + let decode = || -> std::result::Result { + let (worker, handle) = if on_cpu { + self.clip_on(cpu_hub(), &self.cpu_clip).map_err(|e| (e, false))? + } else { + match self.clip_on(gpu_hub(), &self.gpu_clip) { + Ok(clip) => clip, + Err(MediaError::Cancelled) => return Err((MediaError::Cancelled, false)), + Err(_) => self.clip_on(cpu_hub(), &self.cpu_clip).map_err(|e| (e, false))?, + } + }; + worker.request(&format!("V {handle} {index} {divisor} {develop_fields}\n")).map_err(|e| { + let gone = !matches!(e, MediaError::Cancelled) && worker.registry.lock().failed.is_some(); + (e, gone) + }) + }; + // A frame in flight when its helper died is asked again of the helper started in its place + // (once). Before, the frames in flight failed and the monitor showed the damaged-media slate + // for a moment after a helper crash (a GPU helper killed mid-play). + let reply = match decode() { + Err((_, true)) if !filmcraft_media::cancel::cancelled() => { + log::info!("R3D frame {index}: its helper went; asking the helper started in its place"); + decode().map_err(|(e, _)| e)? + } + r => r.map_err(|(e, _)| e)?, + }; + if filmcraft_media::cancel::cancelled() { + return Err(MediaError::Cancelled); // dropping a slot reply hands the slot back + } + let (width, height) = reduced_size(m.width, m.height, divisor).ok_or_else(|| failure("invalid R3D preview divisor"))?; + let pixels = width as usize * height as usize; + let t0 = Instant::now(); + // Repacked straight out of the helper's slot: the one copy of the frame in this process. + // The linear output is half float, repacked as it is; every other gamma comes as 16-bit + // code values, linearised into halves (linear light keeps 11 significant bits of each). + let bytes = match &reply { + Reply::Inline(bytes) => bytes.as_slice(), + Reply::Slot(lease) => lease.bytes()?, + Reply::Error(message) => return Err(failure(format!("RED decoder: {message}"))), + }; + let image = if develop.gamma == RawGamma::Linear { + VideoFrame::rgba_f16(width, height, video_texels(bytes, pixels)?) + } else { + VideoFrame::rgba_f16(width, height, video_linear(bytes, pixels, linearise_table(develop.gamma))?) + }; + if timing() { + let held = match &reply { + Reply::Slot(lease) => format!(", slot {} held {:.1} ms", lease.index, lease.delivered.elapsed().as_secs_f64() * 1e3), + Reply::Inline(_) | Reply::Error(_) => String::new(), + }; + log::info!("R3D frame {index}/{divisor}: convert {:.1} ms{held}", t0.elapsed().as_secs_f64() * 1e3); + } + match reply { + Reply::Inline(bytes) => filmcraft_frame::pool::recycle_u8(bytes), + Reply::Slot(lease) => drop(lease), // `F `: the helper may decode into it again + Reply::Error(_) => {} + } + let mut frame = image.with_pts(m.rate.tick_of(index)); + frame.color = frame_color(&develop); + let frame = Arc::new(frame); + self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner).last = Some((index, divisor, develop, frame.clone())); + Ok(frame) + } + fn raw_as_shot(&self) -> Option { + Some(self.recorded.lock().unwrap_or_else(std::sync::PoisonError::into_inner).0) + } + fn raw_clip_info(&self) -> Option { + Some(self.recorded.lock().unwrap_or_else(std::sync::PoisonError::into_inner).1.clone()) + } + /// Update RMD: `W` to the CPU helper, which writes the sidecar beside the clip + /// (`CreateOrUpdateRmd`) and answers the clip's metadata as it now reads, RMD included. The + /// GPU helper's copy of the clip still has the old recorded settings: it is closed there, so + /// the next frame opens it again with the sidecar. + fn write_raw_sidecar(&self, develop: &RawDevelop) -> Result { + let fields = wire(develop, None)?; + let (worker, handle) = self.clip_on(cpu_hub(), &self.cpu_clip)?; + let bytes = worker + .request(&format!( + "W {handle} {fields} +" + ))? + .into_inline()?; + let m = Metadata::parse(&bytes)?; + let path = m.clip.rmd.clone().ok_or_else(|| failure("the RED SDK wrote no RMD sidecar"))?; + *self.recorded.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = (m.as_shot, m.clip); + if let Some((generation, handle)) = self.gpu_clip.lock().unwrap_or_else(std::sync::PoisonError::into_inner).take() { + gpu_hub().close_clip(generation, handle); + } + self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner).last = None; + Ok(path) + } + /// The next frames are developed with `develop` (the helper takes the settings per request, + /// so nothing is restarted); the cached frame is dropped. + fn set_develop(&self, develop: &RawDevelop) -> Result<()> { + let mut current = self.develop.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if *current != *develop { + *current = *develop; + self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner).last = None; + } + Ok(()) + } + fn audio(&self, start: i64, frames: usize, sample_rate: u32) -> Result { + let m = &self.metadata; + if m.channels == 0 { + return Err(MediaError::NoStream("audio")); + } + if frames > MAX_SAMPLES || sample_rate == 0 || sample_rate > 384_000 { + return Err(failure("R3D audio request outside supported limits")); + } + if filmcraft_media::cancel::cancelled() { + return Err(MediaError::Cancelled); + } + let end = start.checked_add(frames as i64).ok_or_else(|| failure("R3D audio position overflow"))?; + let pos = |sample: i64| sample as i128 * m.sample_rate as i128; + let first = pos(start).div_euclid(sample_rate as i128).max(0).min(m.samples as i128) as u64; + let last = (pos(end).div_euclid(sample_rate as i128) + 2).max(0).min(m.samples as i128) as u64; + let count = last.saturating_sub(first); + if count > MAX_SAMPLES as u64 { + return Err(failure("R3D resampling request too large")); + } + let mut output = AudioBuffer::silence(sample_rate, m.channels, frames); + if frames == 0 || count == 0 { + return Ok(output); + } + // The CPU helper decodes audio before any queued frame, so sound never waits on video. + let (worker, handle) = self.clip_on(cpu_hub(), &self.cpu_clip)?; + let bytes = worker.request(&format!("A {handle} {first} {count}\n"))?.into_inline()?; + let samples = audio_samples(&bytes, m.channels, count as usize, m.float_audio)?; + for i in 0..frames { + let p = pos(start + i as i64); + let absolute = p.div_euclid(sample_rate as i128); + if absolute < first as i128 || absolute >= m.samples as i128 { + continue; + } + let local = (absolute - first as i128) as usize; + let fraction = p.rem_euclid(sample_rate as i128) as f32 / sample_rate as f32; + for (channel, src) in samples.iter().enumerate() { + let a = src.get(local).copied().ok_or_else(|| failure("RED decoder returned too few audio samples"))?; + let b = src.get(local + 1).copied().unwrap_or(a); + // Use f64 for interpolation so finite large float PCM cannot overflow b - a. + let value = (a as f64 * (1.0 - fraction as f64) + b as f64 * fraction as f64) as f32; + if !value.is_finite() { + return Err(failure("non-finite resampled RED audio")); + } + output.channels[channel][i] = value; + } + } + Ok(output) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + + /// Set: every GPU helper start fails as a helper whose CUDA decoder could not be created + /// (the process is never started). For the back-off test. + pub(super) static FAIL_GPU_STARTS: AtomicBool = AtomicBool::new(false); + + /// Helper processes alive on this machine right now, from the process list (not this + /// process's own count): what the GPU actually sees. + fn helper_processes() -> usize { + #[cfg(windows)] + { + let out = std::process::Command::new("tasklist").args(["/FI", "IMAGENAME eq filmcraft-red-helper.exe", "/FO", "CSV", "/NH"]).output().unwrap(); + String::from_utf8_lossy(&out.stdout).lines().filter(|l| l.contains("filmcraft-red-helper.exe")).count() + } + #[cfg(not(windows))] + { + // `comm` is the executable name cut to fifteen characters. + std::fs::read_dir("/proc") + .unwrap() + .flatten() + .filter(|entry| std::fs::read_to_string(entry.path().join("comm")).is_ok_and(|comm| comm.trim_end() == "filmcraft-red-h")) + .count() + } + } + /// Wait for the helper processes to be gone (a close reaps synchronously, but the process + /// list lags a little on Windows). + fn wait_for_no_helpers() { + let started = Instant::now(); + while helper_processes() > 0 && started.elapsed() < Duration::from_secs(10) { + std::thread::sleep(Duration::from_millis(50)); + } + assert_eq!(helper_processes(), 0, "helper processes left behind"); + } + /// Sample the process list every few milliseconds until `stop`, keeping the peak. + struct Sampler { + stop: AtomicBool, + processes: AtomicUsize, + samples: AtomicUsize, + } + impl Sampler { + fn new() -> Self { + Self { stop: AtomicBool::new(false), processes: AtomicUsize::new(0), samples: AtomicUsize::new(0) } + } + fn run(&self) { + while !self.stop.load(Ordering::Relaxed) { + self.processes.fetch_max(helper_processes(), Ordering::Relaxed); + self.samples.fetch_add(1, Ordering::Relaxed); + std::thread::sleep(Duration::from_millis(10)); + } + } + } + /// A GPU helper of its own (not the shared one) with the test clip open in it: the worker, + /// the clip's handle, its metadata and the helper's start-up reply. + fn gpu_helper_with_clip(path: &Path) -> (Worker, u64, Metadata, HubInfo) { + let (worker, hub) = Worker::open_hub(false, DEFAULT_SLOT_BYTES).unwrap(); + let handle = next_handle(); + let m = Metadata::parse(&open_in(&worker, handle, path).unwrap()).unwrap(); + (worker, handle, m, hub) + } + /// Close both helpers and wait for their processes to be gone: a clean slate (a source's + /// clips are opened again by its next request). + fn close_hubs() { + for hub in [gpu_hub(), cpu_hub()] { + let worker = hub.lock().worker.take(); + drop(worker); + } + wait_for_no_helpers(); + } + /// `count` sources opened on the test clip: open in the CPU helper (their metadata came from + /// it), not yet in the GPU one. + fn open_sources(path: &Path, count: usize) -> Vec { + let len = std::fs::metadata(path).unwrap().len(); + let sources: Vec<_> = (0..count).map(|_| RedSource::open("test.R3D", path, len).unwrap()).collect(); + for source in &sources { + assert!(source.gpu_clip.lock().unwrap().is_none() && source.cpu_clip.lock().unwrap().is_some()); + } + sources + } + /// Every source asked for `request` at once from `threads` threads (each takes the next + /// source off a shared queue); the results in source order. + fn burst(sources: &[RedSource], threads: usize, request: impl Fn(&RedSource) -> T + Sync) -> Vec { + let queue = Mutex::new((0..sources.len()).collect::>()); + let results = Mutex::new(Vec::new()); + std::thread::scope(|scope| { + for _ in 0..threads { + let (queue, results, request) = (&queue, &results, &request); + scope.spawn(move || { + loop { + // taken under a lock released before the request (a `while let` guard + // would live through the body and serialise the threads) + let next = queue.lock().unwrap().pop_front(); + let Some(i) = next else { break }; + let result = request(&sources[i]); + results.lock().unwrap().push((i, result)); + } + }); + } + }); + let mut results = results.into_inner().unwrap(); + results.sort_by_key(|(i, _)| *i); + results.into_iter().map(|(_, result)| result).collect() + } + + #[test] + #[ignore = "requires RED_R3D_TEST_CLIP pointing to user-owned Nikon ZR footage; run alone (--test-threads=1)"] + fn real_two_helpers_serve_every_source() { + // Regression: a helper per source and per role ran over a hundred processes for a + // hundred-clip timeline. Now one GPU and one CPU helper serve every clip. + let path = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + close_hubs(); + let mut sources = open_sources(&path, 24); + assert_eq!(helper_processes(), 1, "opening 24 sources probed them through the one CPU helper"); + assert!(gpu_hub().running().is_none(), "no GPU work before a frame is asked for"); + let (w, h) = (sources[0].metadata.width, sources[0].metadata.height); + let quarter = preview_divisor(w, h, 0.25); + // A first frame from every source at once, and their audio at once: two processes. + let sampler = Sampler::new(); + let (frames, sounds) = std::thread::scope(|scope| { + let sampler = &sampler; + scope.spawn(move || sampler.run()); + let frames = burst(&sources, 8, |source| source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).map(|f| (f.width, f.height))); + let sounds = burst(&sources, 8, |source| source.audio(0, 4800, 48000).map(|a| a.channels.len())); + sampler.stop.store(true, Ordering::Relaxed); + (frames, sounds) + }); + for (i, frame) in frames.iter().enumerate() { + assert_eq!(frame.as_ref().map_err(|e| e.to_string()), Ok(&(w / quarter, h / quarter)), "source {i}"); + } + for (i, sound) in sounds.iter().enumerate() { + assert_eq!(sound.as_ref().map_err(|e| e.to_string()), Ok(&sources[0].metadata.channels), "source {i}"); + } + let peak = sampler.processes.load(Ordering::Relaxed); + println!( + "24 sources, a frame and audio from each at once on 8 threads: peak {peak} helper processes over {} samples", + sampler.samples.load(Ordering::Relaxed) + ); + assert!(peak <= 2, "{peak} helper processes"); + let generation = gpu_hub().lock().generation; + assert!(sources.iter().all(|s| s.gpu_clip.lock().unwrap().is_some_and(|(g, _)| g == generation)), "every clip is open in the one GPU helper"); + // A source that goes closes its clips; the helpers stay for the others. + drop(sources.pop()); + assert_eq!(helper_processes(), 2); + sources[0].video_frame(FrameRequest { time: sources[0].metadata.rate.frame_duration(), scale: 0.25 }).unwrap(); + // A helper that dies is started again by the next request, which opens its clip again. + let old = gpu_hub().running().unwrap(); + { + let mut child = old.child.lock().unwrap(); + child.kill().unwrap(); + child.wait().unwrap(); + } + let started = Instant::now(); + while old.registry.lock().failed.is_none() && started.elapsed() < Duration::from_secs(5) { + std::thread::sleep(Duration::from_millis(10)); + } + drop(old); + let again = sources[1].video_frame(FrameRequest { time: sources[1].metadata.rate.tick_of(2), scale: 0.25 }).unwrap(); + assert_eq!((again.width, again.height), (w / quarter, h / quarter)); + assert_eq!(gpu_hub().lock().generation, generation.wrapping_add(1), "a new GPU helper"); + assert!(sources[1].gpu_clip.lock().unwrap().is_some_and(|(g, _)| g == generation.wrapping_add(1))); + // A helper idle for its idle time is closed; the next request starts it again. + cpu_hub().lock().last_used = Some(Instant::now() - CPU_IDLE); + cpu_hub().sweep(); + assert!(cpu_hub().running().is_none(), "the idle CPU helper was not closed"); + assert!(sources[2].audio(0, 4800, 48000).is_ok()); + assert!(cpu_hub().running().is_some()); + drop(sources); + close_hubs(); + } + #[test] + #[ignore = "benchmark: requires RED_R3D_TEST_CLIP; run alone (--test-threads=1 --nocapture)"] + fn real_random_frame_latency_on_each_helper() { + // A paused monitor asks for one frame at a time, anywhere in the clip (a scrub, a click on + // the ruler): its latency, not the helpers' throughput, is what the user waits for. + let path = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + close_hubs(); + let source = open_sources(&path, 1).remove(0); + let frames = source.metadata.frames.max(1); + source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.125 }).unwrap(); // start the GPU helper + filmcraft_media::cancel::with_background(true, || source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.0625 }).unwrap()); + for scale in [1.0f32, 0.5, 0.25, 0.125, 0.0625] { + for (name, background) in [("gpu", false), ("cpu", true)] { + let mut ms: Vec = (0..10i64) + .map(|k| { + let index = (k * 7919 + i64::from(scale.to_bits() % 97)) % frames; + let t0 = Instant::now(); + filmcraft_media::cancel::with_background(background, || { + source.video_frame(FrameRequest { time: source.metadata.rate.tick_of(index), scale }).unwrap() + }); + t0.elapsed().as_secs_f64() * 1e3 + }) + .collect(); + ms.sort_by(f64::total_cmp); + println!("scale {scale:>6}: {name} p50 {:6.1} ms max {:6.1} ms", ms[ms.len() / 2], ms[ms.len() - 1]); + } + } + // the two decoders develop the same picture (a still frame from one, playback from the other) + for scale in [0.25f32, 0.125] { + let at = source.metadata.rate.tick_of(frames / 2); + let gpu = source.video_frame(FrameRequest { time: at, scale }).unwrap(); + source.state.lock().unwrap().last = None; + let cpu = filmcraft_media::cancel::with_still(true, || source.video_frame(FrameRequest { time: at, scale }).unwrap()); + let (a, b) = (gpu.to_rgba8(), cpu.to_rgba8()); + assert_eq!((gpu.width, gpu.height), (cpu.width, cpu.height)); + let diff = a.iter().zip(&b).map(|(x, y)| f64::from(x.abs_diff(*y))).sum::() / a.len().max(1) as f64; + println!("scale {scale}: mean |gpu - cpu| {diff:.2} of 255"); + assert!(diff < 2.0, "the CPU and GPU decoders differ by {diff} of 255 on average at {scale}"); + } + drop(source); + close_hubs(); + } + + #[test] + #[ignore = "requires RED_R3D_TEST_CLIP pointing to user-owned Nikon ZR footage; run alone (--test-threads=1)"] + fn real_a_still_frame_decodes_on_the_cpu_and_warms_the_gpu_for_playback() { + let path = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + close_hubs(); + let source = open_sources(&path, 1).remove(0); + let t0 = Instant::now(); + let still = filmcraft_media::cancel::with_still(true, || source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.125 }).unwrap()); + let took = t0.elapsed(); + assert_eq!(still.width, source.metadata.width / 8); + assert!(source.gpu_clip.lock().unwrap().is_none(), "the still frame came from the CPU helper"); + println!("still frame at an eighth in {:.1} ms", took.as_secs_f64() * 1e3); + // the GPU helper starts meanwhile, so playback finds it running + let started = Instant::now(); + while gpu_hub().running().is_none() && started.elapsed() < Duration::from_secs(20) { + std::thread::sleep(Duration::from_millis(20)); + } + assert!(gpu_hub().running().is_some(), "a still frame warms the GPU helper"); + let generation = gpu_hub().lock().generation; + source.video_frame(FrameRequest { time: source.metadata.rate.tick_of(1), scale: 0.125 }).unwrap(); + assert_eq!(gpu_hub().lock().generation, generation, "playback used the warmed helper"); + drop(source); + close_hubs(); + } + + #[test] + #[ignore = "requires RED_R3D_TEST_CLIP pointing to user-owned Nikon ZR footage; run alone (--test-threads=1)"] + fn real_frames_in_flight_when_the_gpu_helper_dies_still_arrive() { + // Regression: a GPU helper killed mid-play failed the frames in flight, three of them in a + // row, and the monitor showed the damaged-media slate until the new helper delivered. + let path = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + close_hubs(); + let source = open_sources(&path, 1).remove(0); + let frames = source.metadata.frames.max(1); + source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.5 }).unwrap(); + for round in 0..3i64 { + let old = gpu_hub().running().expect("the GPU helper runs"); + let results = std::thread::scope(|scope| { + let asks: Vec<_> = (0..6i64) + .map(|k| { + let source = &source; + scope.spawn(move || { + let index = (round * 31 + k * 7 + 1) % frames; + source.video_frame(FrameRequest { time: source.metadata.rate.tick_of(index), scale: 0.5 }).map(|f| f.width) + }) + }) + .collect(); + std::thread::sleep(Duration::from_millis(30)); // the requests are on their way + { + let mut child = old.child.lock().unwrap(); + child.kill().unwrap(); + child.wait().unwrap(); + } + asks.into_iter().map(|a| a.join().unwrap()).collect::>() + }); + for (k, r) in results.iter().enumerate() { + assert_eq!(r.as_ref().map_err(|e| e.to_string()), Ok(&(source.metadata.width / 2)), "round {round}, frame {k}"); + } + drop(old); + } + drop(source); + close_hubs(); + } + + #[test] + #[ignore = "requires RED_R3D_TEST_CLIP pointing to user-owned Nikon ZR footage; run alone (--test-threads=1)"] + fn real_an_idle_helper_closes_by_itself_with_exit_code_0() { + // Regression: every close, idle reaps included, killed the helper ("exit code: 1" in every + // log), so the closes of helpers that really failed did not stand out + let path = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + close_hubs(); + let source = open_sources(&path, 1).remove(0); + assert!(source.audio(0, 4800, 48000).is_ok()); + drop(source); + LAST_CLOSE_SUCCESS.store(false, Ordering::Relaxed); + cpu_hub().lock().last_used = Some(Instant::now() - CPU_IDLE); + cpu_hub().sweep(); + assert!(cpu_hub().running().is_none(), "the idle CPU helper was closed"); + assert!(LAST_CLOSE_SUCCESS.load(Ordering::Relaxed), "it exited by itself, with success"); + close_hubs(); + } + + #[test] + fn still_frames_go_to_the_cpu_helper_from_a_quarter_down() { + assert!(!still_on_cpu(false, 16), "playback stays on the GPU"); + assert!(!still_on_cpu(true, 1) && !still_on_cpu(true, 2), "a half or full still frame is quicker on the GPU"); + assert!(still_on_cpu(true, 4) && still_on_cpu(true, 8) && still_on_cpu(true, 16)); + } + + #[test] + #[ignore = "requires RED_R3D_TEST_CLIP pointing to user-owned Nikon ZR footage; run alone (--test-threads=1)"] + fn real_thumbnails_use_the_cpu_helper_and_a_failed_gpu_start_backs_off() { + let path = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + close_hubs(); + let sources = open_sources(&path, 24); + let (w, h) = (sources[0].metadata.width, sources[0].metadata.height); + // the smallest decode the SDK offers (a sixteenth: 6048x3402 rounds down to 378x212) + let divisor = preview_divisor(w, h, 0.05); + println!("thumbnails decode at 1/{divisor}"); + assert_eq!(divisor, 16, "thumbnails of a clip that does not divide exactly still decode at a sixteenth"); + // Thumbnails of every source at once: the CPU helper only, never the GPU. + let sampler = Sampler::new(); + let results = std::thread::scope(|scope| { + let sampler = &sampler; + scope.spawn(move || sampler.run()); + let results = burst(&sources, 8, |source| { + filmcraft_media::cancel::with_background(true, || { + source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.05 }).map(|f| (f.width, f.height)) + }) + }); + sampler.stop.store(true, Ordering::Relaxed); + results + }); + for (i, result) in results.iter().enumerate() { + assert_eq!(result.as_ref().map_err(|e| e.to_string()), Ok(&(w / divisor, h / divisor)), "source {i}"); + } + let processes = sampler.processes.load(Ordering::Relaxed); + println!("{} thumbnails on 8 threads: peak {processes} helper processes", sources.len()); + assert_eq!(processes, 1, "thumbnails ran on more than the CPU helper"); + assert!(gpu_hub().running().is_none(), "a thumbnail started the GPU helper"); + // A GPU helper that fails to become ready: the frame falls back to the CPU helper, the GPU + // backs off (a WARN line names the reason), and the next frames stay on the CPU until the + // back-off ends; a start that succeeds clears it. + let source = &sources[0]; + FAIL_GPU_STARTS.store(true, Ordering::Relaxed); + let fallback = source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap(); + FAIL_GPU_STARTS.store(false, Ordering::Relaxed); + let quarter = preview_divisor(w, h, 0.25); + assert_eq!((fallback.width, fallback.height), (w / quarter, h / quarter)); + assert!(gpu_hub().running().is_none()); + assert!(gpu_hub().unavailable()); + assert_eq!(gpu_hub().lock().backoff.failures, 1); + source.video_frame(FrameRequest { time: source.metadata.rate.frame_duration(), scale: 0.25 }).unwrap(); + assert!(gpu_hub().running().is_none(), "a frame during the back-off started the GPU helper"); + gpu_hub().lock().backoff.until = None; + source.video_frame(FrameRequest { time: source.metadata.rate.tick_of(2), scale: 0.25 }).unwrap(); + assert!(gpu_hub().running().is_some(), "the GPU helper was not started once the back-off ended"); + assert_eq!(gpu_hub().lock().backoff.failures, 0); + drop(sources); + close_hubs(); + } + #[test] + fn gpu_back_off_doubles_from_two_seconds_to_a_minute() { + assert_eq!(backoff_delay(0), Duration::from_secs(2)); + assert_eq!(backoff_delay(1), Duration::from_secs(2)); + assert_eq!(backoff_delay(2), Duration::from_secs(4)); + assert_eq!(backoff_delay(3), Duration::from_secs(8)); + assert_eq!(backoff_delay(5), Duration::from_secs(32)); + assert_eq!(backoff_delay(6), Duration::from_secs(60)); + assert_eq!(backoff_delay(u32::MAX), Duration::from_secs(60)); + let mut backoff = GpuBackoff::default(); + assert!(backoff.until.is_none_or(|t| Instant::now() >= t)); + backoff.failures = 1; + backoff.until = Some(Instant::now() + backoff_delay(1)); + assert!(backoff.until.is_some_and(|t| Instant::now() < t)); + } + #[test] + fn malformed_native_files_fail_without_crashing_the_editor() { + let path = std::env::temp_dir().join(format!("filmcraft-red-invalid-{}.R3D", std::process::id())); + for length in [0, 1, 64, 4096] { + std::fs::write(&path, vec![0xa5; length]).unwrap(); + assert!(RedSource::open("invalid.R3D", &path, length as u64).is_err()); + } + std::fs::remove_file(path).unwrap(); + // the helper answered each with an error and serves on (one helper serves every clip) + assert!(cpu_hub().running().is_some_and(|w| w.registry.lock().failed.is_none()), "an unreadable file took the CPU helper down"); + } + #[test] + fn reduced_sizes_follow_the_measured_sdk_rounding() { + // SDK 9.2.1 on 6048×3402 (docs/r3d-ne.md): both dimensions round down. + assert_eq!(reduced_size(6048, 3402, 1), Some((6048, 3402))); + assert_eq!(reduced_size(6048, 3402, 2), Some((3024, 1701))); + assert_eq!(reduced_size(6048, 3402, 4), Some((1512, 850))); + assert_eq!(reduced_size(6048, 3402, 8), Some((756, 425))); + assert_eq!(reduced_size(6048, 3402, 16), Some((378, 212))); + assert_eq!(reduced_size(4095, 2159, 16), Some((255, 134))); + // Only the SDK's divisors, and never an empty frame. + for divisor in [0, 3, 5, 6, 7, 9, 32, u32::MAX] { + assert_eq!(reduced_size(6048, 3402, divisor), None, "divisor {divisor}"); + } + assert_eq!(reduced_size(15, 3402, 16), None); + assert_eq!(reduced_size(6048, 15, 16), None); + assert_eq!(reduced_size(1, 1, 1), Some((1, 1))); + assert_eq!(reduced_size(u32::MAX, u32::MAX, 16), Some((u32::MAX / 16, u32::MAX / 16))); + } + #[test] + fn previews_pick_the_smallest_covering_decode_and_never_lower_export_resolution() { + assert_eq!(preview_divisor(6048, 3402, 1.0), 1); + assert_eq!(preview_divisor(6048, 3402, 0.5), 2); + assert_eq!(preview_divisor(6048, 3402, 0.3), 2); + assert_eq!(preview_divisor(6048, 3402, 0.25), 4); + assert_eq!(preview_divisor(6048, 3402, 0.125), 8); + assert_eq!(preview_divisor(6048, 3402, 0.0625), 16); + assert_eq!(preview_divisor(6048, 3402, 0.01), 16); + assert_eq!(preview_divisor(4096, 2160, 0.125), 8); + assert_eq!(preview_divisor(4095, 2159, 0.1), 8); + // A clip too small for a divisor falls back to the next larger decode, down to full. + assert_eq!(preview_divisor(15, 15, 0.01), 8); + assert_eq!(preview_divisor(1, 1, 0.01), 1); + for (divisor, (w, h)) in [16, 8, 4].into_iter().map(|d| (d, reduced_size(6048, 3402, d).unwrap())) { + // The size the app expects for the divisor it picked is what the helper sizes its + // buffer to, and a reply of any other length is refused. + assert_eq!((w, h), (6048 / divisor, 3402 / divisor)); + assert!(video_linear(&vec![0; w as usize * h as usize * 6 + 6], w as usize * h as usize, linearise_table(RawGamma::Bt1886)).is_err()); + assert!(video_linear(&vec![0; w as usize * h as usize * 6], w as usize * h as usize, linearise_table(RawGamma::Bt1886)).is_ok()); + } + } + + /// Update RMD on a copy of the test clip in a temporary folder (never the user's original): + /// the sidecar is written beside the copy only when asked, holds the settings (a fresh open + /// reads them as the clip's recorded ones and says an RMD is in effect), and the source's own + /// GPU copy of the clip is reopened with it. + #[test] + #[ignore = "requires RED_R3D_TEST_CLIP pointing to user-owned Nikon ZR footage (copied to a temp folder); run alone (--test-threads=1)"] + fn real_rmd_sidecar_is_written_beside_a_copy_and_read_back() { + let original = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + let dir = std::env::temp_dir().join(format!( + "filmcraft-rmd-{}-{}", + std::process::id(), + std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_nanos() + )); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join(original.file_name().unwrap()); + std::fs::copy(&original, &path).unwrap(); + let open = |p: &Path| RedSource::open("copy.R3D", p, std::fs::metadata(p).unwrap().len()).unwrap(); + let source = open(&path); + let before = source.raw_clip_info().unwrap(); + println!("before: {before:?}"); + assert!(before.rmd.is_none(), "a fresh copy has no sidecar"); + assert!(Path::new(&before.rmd_path).parent().is_some_and(|p| p == dir), "the sidecar goes beside the clip: {}", before.rmd_path); + assert!(!Path::new(&before.rmd_path).exists(), "nothing is written by opening"); + let as_shot = source.raw_as_shot().unwrap(); + let iso = if as_shot.iso == 1600 { 3200 } else { 1600 }; + let frame = || source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap().to_linear_f32(); + let as_recorded = frame(); + let d = RawDevelop { iso: Some(iso), kelvin: Some(3200), gamma: RawGamma::Bt1886, tone_map: RawToneMap::Low, ..Default::default() }; + let written = source.write_raw_sidecar(&d).unwrap(); + println!("wrote {written}"); + assert!(Path::new(&written).exists() && Path::new(&written).parent().is_some_and(|p| p == dir), "{written}"); + let after = source.raw_clip_info().unwrap(); + assert_eq!(after.rmd.as_deref(), Some(written.as_str()), "the panel says an RMD is in effect"); + let recorded = source.raw_as_shot().unwrap(); + assert_eq!((recorded.iso, recorded.kelvin.round() as u32), (iso, 3200), "the RMD's settings are now the clip's recorded ones"); + assert_eq!(after.tone_map, Some(RawToneMap::Low)); + assert!(source.gpu_clip.lock().unwrap().is_none(), "the GPU copy of the clip is opened again with the sidecar"); + assert_ne!(frame(), as_recorded, "as recorded is now the RMD's development"); + // a fresh open (an import) reads the sidecar + drop(source); + let again = open(&path); + assert_eq!(again.raw_clip_info().unwrap().rmd.as_deref(), Some(written.as_str())); + assert_eq!(again.raw_as_shot().unwrap().iso, iso); + // a hostile development is refused before the helper is asked + assert!(again.write_raw_sidecar(&RawDevelop { iso: Some(1250), ..Default::default() }).is_err()); + drop(again); + close_hubs(); + std::fs::remove_dir_all(&dir).unwrap(); + } + + #[test] + #[ignore = "requires RED_R3D_TEST_CLIP pointing to user-owned Nikon ZR footage"] + fn real_clip_video_audio_and_seeking() { + let path = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + let reader: SharedReader = Arc::new(filmcraft_media::reader::FileReader::open(&path).unwrap()); + filmcraft_codecs::register_reader_opener(reader_opener); + let source = filmcraft_media::reader::open_reader("test.R3D", reader, &filmcraft_codecs::reader_openers(), &filmcraft_codecs::openers()).unwrap(); + let info = source.info(); + let v = info.video.as_ref().unwrap(); + assert_eq!(v.codec, "R3D NE"); + assert!(info.duration > Tick::ZERO); + let first = source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap(); + assert!(first.to_linear_f32().as_chunks::<4>().0.iter().any(|p| p[..3].iter().any(|v| *v != 0.0))); + // A quarter request decodes at quarter resolution, rounded down (6048×3402 → 1512×850), + // and the picture is the half-resolution frame at half its size: a 2×2 box average of the + // half frame (dropping its odd last row/column) is close to the quarter decode, and far + // closer than a misaligned (flipped or one-row-shifted) comparison. + assert_eq!((first.width, first.height), (v.width / 4, v.height / 4)); + let half = source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.5 }).unwrap(); + assert_eq!((half.width, half.height), (v.width / 2, v.height / 2)); + let (qw, qh) = (first.width as usize, first.height as usize); + let (half_px, quarter_px) = (half.to_linear_f32(), first.to_linear_f32()); + let half_at = |x: usize, y: usize, c: usize| half_px[(y * half.width as usize + x) * 4 + c]; + let quarter_at = |x: usize, y: usize, c: usize| quarter_px[(y * qw + x) * 4 + c]; + let error = |map: &dyn Fn(usize, usize) -> (usize, usize)| -> f64 { + let mut sum = 0.0; + for y in 0..qh { + for x in 0..qw { + let (sx, sy) = map(x, y); + for c in 0..3 { + let box_mean = + (half_at(2 * sx, 2 * sy, c) + half_at(2 * sx + 1, 2 * sy, c) + half_at(2 * sx, 2 * sy + 1, c) + half_at(2 * sx + 1, 2 * sy + 1, c)) + / 4.0; + sum += (box_mean - quarter_at(x, y, c)).abs() as f64; + } + } + } + sum / (qw * qh * 3) as f64 + }; + let mean = quarter_px.as_chunks::<4>().0.iter().map(|p| (p[0] + p[1] + p[2]) as f64 / 3.0).sum::() / (qw * qh) as f64; + let aligned = error(&|x, y| (x, y)); + let flipped = error(&|x, y| (x, qh - 1 - y)); + let shifted = error(&|x, y| (x, (y + 1).min(qh - 1))); + println!("quarter vs box-filtered half: mean level {mean:.4}, mean abs error aligned {aligned:.5}, flipped {flipped:.5}, shifted one row {shifted:.5}"); + // Measured on a 6K 59.94p clip: aligned 0.051 at level 0.607 (Quarter Good against box-filtered + // Half Premium), flipped 0.96, shifted 0.070. + assert!(mean > 0.0 && aligned < 0.15 * mean, "quarter decode does not match the half decode: {aligned} vs level {mean}"); + assert!(aligned < flipped && aligned < shifted, "quarter decode is misaligned: {aligned} aligned, {flipped} flipped, {shifted} shifted"); + let last_time = info.duration - v.frame_rate.frame_duration(); + let last = source.video_frame(FrameRequest { time: last_time, scale: 0.25 }).unwrap(); + assert_eq!(last.pts, last_time); + let again = source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap(); + assert_eq!(first.to_linear_f32(), again.to_linear_f32()); + let full = source.video_frame(FrameRequest::full(Tick::ZERO)).unwrap(); + assert_eq!((full.width, full.height), (v.width, v.height)); + // Source Settings: every colour space and gamma decodes, as linear light; the display + // curves differ from the linear default (tone mapped); back at the default the picture is + // the same again + let linear = first.to_linear_f32(); + let frame_with = |d: RawDevelop| { + source.set_develop(&d).unwrap(); + source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap() + }; + let display = frame_with(RawDevelop::STANDARD); + assert_eq!((display.color.transfer, display.width), (Transfer::Linear, first.width)); + assert_ne!(display.to_linear_f32(), linear, "the display output is tone mapped, not linear"); + for space in RawColorSpace::ALL { + for gamma in RawGamma::ALL.into_iter().filter(|g| RawDevelop::supports(space, *g)) { + let f = frame_with(RawDevelop { color_space: space, gamma, ..RawDevelop::STANDARD }).to_linear_f32(); + let mean = f.as_chunks::<4>().0.iter().map(|p| f64::from(p[0] + p[1] + p[2]) / 3.0).sum::() / (f.len() / 4) as f64; + println!("{space:?} {gamma:?}: mean linear level {mean:.4}"); + assert!(mean.is_finite() && mean > 0.0, "{space:?} {gamma:?}"); + } + } + // the SDK's Log3G10 encoding is the published curve the compositor decodes with: the same + // frame developed into scene-linear REDWideGamutRGB and into Log3G10 (then linearised + // here) agrees to within the 16-bit quantisation of the log code values + let rwg_linear = frame_with(RawDevelop { color_space: RawColorSpace::RedWideGamutRgb, gamma: RawGamma::Linear, ..Default::default() }).to_linear_f32(); + let rwg_log = frame_with(RawDevelop { color_space: RawColorSpace::RedWideGamutRgb, gamma: RawGamma::Log3G10, ..Default::default() }).to_linear_f32(); + let (mut diff, mut level) = (0f64, 0f64); + for (a, b) in rwg_linear.as_chunks::<4>().0.iter().zip(rwg_log.as_chunks::<4>().0) { + for c in 0..3 { + diff += f64::from((a[c] - b[c]).abs()); + level += f64::from(a[c].abs()); + } + } + println!( + "REDWideGamutRGB linear vs Log3G10 linearised: mean abs difference {:.5} of level {:.5}", + diff / level.max(1e-9), + level / (rwg_linear.len() / 4 * 3) as f64 + ); + assert!(diff < 0.01 * level, "Log3G10 linearised does not match the linear output: {diff} vs {level}"); + // the decode quality caps a preview: a half-size preview at quarter quality is a quarter + source.set_develop(&RawDevelop { quality: Some(filmcraft_media::RawQuality::QuarterGood), ..Default::default() }).unwrap(); + let capped = source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.5 }).unwrap(); + assert_eq!(capped.width, v.width / 4); + let half_good = frame_with(RawDevelop { quality: Some(filmcraft_media::RawQuality::HalfGood), ..Default::default() }); + assert_eq!(half_good.width, v.width / 4, "a quarter request decodes at a quarter at any quality"); + source.set_develop(&RawDevelop { quality: Some(filmcraft_media::RawQuality::HalfGood), ..Default::default() }).unwrap(); + assert_eq!(source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.5 }).unwrap().width, v.width / 2, "half, good"); + assert_eq!(source.video_frame(FrameRequest::full(Tick::ZERO)).unwrap().width, v.width, "a full-size request is full at any quality"); + // the processing options decode (their effect on a clean frame may be nil) + for d in [ + RawDevelop { chroma_noise_reduction: Some(false), flashing_pixel: Some(RawFlashing::Strong), ..RawDevelop::STANDARD }, + RawDevelop { metadata_curves: false, cdl: true, creative_lut: true, ..RawDevelop::STANDARD }, + RawDevelop { color_space: RawColorSpace::Rec2020, gamma: RawGamma::St2084, hdr_peak_nits: Some(1000), ..RawDevelop::STANDARD }, + ] { + assert!(frame_with(d).to_linear_f32().iter().all(|v| v.is_finite()), "{d:?}"); + } + // what the clip records: Nikon R3D NE has no HDRx track; the SDK's default tone map and + // rolloff are the ones a new project starts with + let clip = source.raw_clip_info().expect("an R3D source reports what the clip records"); + println!("recorded: {clip:?}"); + assert!(!clip.hdrx, "Nikon R3D NE records no HDRx track"); + assert_eq!((clip.tone_map_default, clip.rolloff_default), (Some(RawDevelop::STANDARD.tone_map), Some(RawDevelop::STANDARD.rolloff))); + assert!(clip.hdr_peak_range.0 <= clip.hdr_peak_default && clip.hdr_peak_default <= clip.hdr_peak_range.1, "{clip:?}"); + source.set_develop(&RawDevelop::default()).unwrap(); + assert_eq!(source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap().to_linear_f32(), linear); + // RAW adjustments: the helper reports the clip's as-shot values; developing the same frame + // at another ISO, colour temperature, tint or exposure changes the picture, each on its + // own, and back at as shot (no adjustment set) the picture is the linear one again + let as_shot = source.raw_as_shot().expect("an R3D source reports its as-shot settings"); + assert!(filmcraft_media::RAW_ISO_VALUES.contains(&as_shot.iso), "{as_shot:?}"); + assert!(filmcraft_media::RAW_KELVIN_RANGE.contains(&(as_shot.kelvin.round() as u32)), "{as_shot:?}"); + println!("as shot: {as_shot:?}"); + let other_iso = if as_shot.iso == 3200 { 800 } else { 3200 }; + let mut pictures = vec![linear.clone()]; + for (name, develop) in [ + ("ISO", RawDevelop { iso: Some(other_iso), ..Default::default() }), + ("Kelvin", RawDevelop { kelvin: Some(if as_shot.kelvin > 4000.0 { 3200 } else { 6500 }), ..Default::default() }), + ("tint", RawDevelop { tint: Some(filmcraft_media::Hundredths(4000)), ..Default::default() }), + ("exposure", RawDevelop { exposure: Some(filmcraft_media::Hundredths(100)), ..Default::default() }), + ] { + source.set_develop(&develop).unwrap(); + let picture = source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap().to_linear_f32(); + assert!(!pictures.contains(&picture), "{name} changed nothing"); + pictures.push(picture); + } + // setting the as-shot ISO explicitly is the as-shot picture (the SDK takes the same value) + source.set_develop(&RawDevelop { iso: Some(as_shot.iso), ..Default::default() }).unwrap(); + assert_eq!( + source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap().to_linear_f32(), + linear, + "the as-shot ISO is the as-shot picture" + ); + source.set_develop(&RawDevelop::default()).unwrap(); + assert_eq!( + source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).unwrap().to_linear_f32(), + linear, + "as shot reproduces the previous output" + ); + // an ISO the SDK does not list is refused before any request is sent + assert!(source.set_develop(&RawDevelop { iso: Some(1250), ..Default::default() }).is_ok(), "stored; refused when a frame is asked for"); + assert!(source.video_frame(FrameRequest { time: Tick::ZERO, scale: 0.25 }).is_err()); + source.set_develop(&RawDevelop::default()).unwrap(); + if let Some(a) = info.audio() { + let audio = source.audio(0, 2048, a.sample_rate).unwrap(); + assert_eq!(audio.channels.len(), a.channels as usize); + assert!(audio.channels.iter().flatten().all(|x| x.is_finite())); + let lead = source.audio(-1024, 2048, a.sample_rate).unwrap(); + assert!(lead.channels.iter().all(|c| c[..1024].iter().all(|x| *x == 0.0))); + for (ch, expected) in lead.channels.iter().zip(&audio.channels) { + assert_eq!(&ch[1024..], &expected[..1024]); + } + let resampled = source.audio(0, 1024, 44100).unwrap(); + let left = source.audio(0, 512, 44100).unwrap(); + let right = source.audio(512, 512, 44100).unwrap(); + for c in 0..a.channels as usize { + assert_eq!(resampled.channels[c], [left.channels[c].as_slice(), right.channels[c].as_slice()].concat()); + } + assert!(source.audio(i64::MAX, 2, a.sample_rate).is_err()); + assert!(source.audio(0, 2, 0).is_err()); + assert!( + source + .audio(info.duration.to_units_floor(a.sample_rate as i64) + a.sample_rate as i64, 128, a.sample_rate) + .unwrap() + .channels + .iter() + .flatten() + .all(|x| *x == 0.0) + ); + } + println!( + "verified {}x{} {}/{} fps, duration {} ticks, {} audio streams", + v.width, + v.height, + v.frame_rate.num, + v.frame_rate.den, + info.duration.0, + info.audio_streams.len() + ); + if info.has_audio() { + // A video decode may hold this lock for seconds; audio must not wait for it. + let independent = RedSource::open("test.R3D", &path, std::fs::metadata(&path).unwrap().len()).unwrap(); + independent.audio(0, 64, 48000).unwrap(); + std::thread::scope(|scope| { + let guard = independent.state.lock().unwrap(); + let (send, receive) = mpsc::channel(); + let independent = &independent; + scope.spawn(move || send.send(independent.audio(64, 64, 48000)).unwrap()); + let result = receive.recv_timeout(Duration::from_secs(2)); + drop(guard); // Release even on regression, so the test cannot deadlock on join. + assert!(result.expect("audio waited for video lock").is_ok()); + }); + } + } + #[test] + #[ignore = "requires RED_R3D_TEST_CLIP pointing to user-owned Nikon ZR footage"] + fn real_pipeline_reuses_slots_routes_frames_and_contains_failures() { + use std::sync::atomic::{AtomicBool, Ordering}; + let path = PathBuf::from(std::env::var_os("RED_R3D_TEST_CLIP").expect("set RED_R3D_TEST_CLIP")); + let (worker, clip, m, hub) = gpu_helper_with_clip(&path); + assert!(m.frames >= 4); + assert!(hub.slots >= 1 && hub.slot_bytes >= m.width as usize * m.height as usize * 6, "a GPU helper shares its frame slots"); + let divisor = preview_divisor(m.width, m.height, 0.5); + let frame_bytes = (m.width / divisor) as usize * (m.height / divisor) as usize * 6; + // A frame's bytes, copied out of its slot so the slot goes back to the helper at once; + // holding every slot's lease while asking for more would wait for a slot forever. + let copy = |reply: Reply| -> Vec { + match reply { + Reply::Slot(lease) => { + assert!(lease.index < hub.slots as u32); + lease.bytes().unwrap().to_vec() + } + Reply::Inline(_) | Reply::Error(_) => panic!("a GPU helper delivers frames in shared slots"), + } + }; + // Every reduced decode returns exactly the bytes of the rounded-down size (docs/r3d-ne.md). + for d in [16, 8, 4, 2] { + let (w, h) = reduced_size(m.width, m.height, d).unwrap(); + assert_eq!(copy(worker.request(&format!("V {clip} 0 {d} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap()).len(), w as usize * h as usize * 6, "divisor {d}"); + } + // The clip's as-shot values are reported, and a frame decodes with every RAW adjustment + // set within the SDK's limits. + assert!(filmcraft_media::RAW_ISO_VALUES.contains(&m.as_shot.iso), "{:?}", m.as_shot); + assert_eq!(copy(worker.request(&format!("V {clip} 0 {divisor} 0 0 0 0 0 0 4 0 15 800 5600 -1250 33\n")).unwrap()).len(), frame_bytes); + for round in 0..2 { + let barrier = std::sync::Barrier::new(4); + let results = std::thread::scope(|scope| { + let handles: Vec<_> = (0..4) + .map(|frame| { + let worker = &worker; + let barrier = &barrier; + scope.spawn(move || { + barrier.wait(); + copy(worker.request(&format!("V {clip} {frame} {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap()) + }) + }) + .collect(); + handles.into_iter().map(|h| h.join().unwrap()).collect::>() + }); + for (frame, bytes) in results.iter().enumerate() { + assert_eq!(bytes.len(), frame_bytes); + assert_eq!( + bytes, + ©(worker.request(&format!("V {clip} {frame} {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap()), + "round {round}, frame {frame}" + ); + } + } + // Every slot's frame cancelled after submission: the reader discards the replies and + // hands every slot back, or the next request would wait for a slot until the deadline. + for _ in 0..2 { + let cancel = Arc::new(AtomicBool::new(false)); + std::thread::scope(|scope| { + let (worker, cancel) = (&worker, &cancel); + let requests: Vec<_> = (0..hub.slots) + .map(|frame| { + scope.spawn(move || { + filmcraft_media::cancel::with_cancel(cancel, || worker.request(&format!("V {clip} {frame} {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n"))) + }) + }) + .collect(); + // Wait for registration, not a guessed decode duration. + let started = Instant::now(); + while worker.registry.lock().requests.len() < hub.slots && started.elapsed() < Duration::from_secs(2) { + std::thread::sleep(Duration::from_millis(1)); + } + cancel.store(true, Ordering::Relaxed); + for request in requests { + assert!(matches!(request.join().unwrap(), Err(MediaError::Cancelled))); + } + }); + // An abandoned reply must not be delivered to the next request or poison the worker. + assert_eq!(copy(worker.request(&format!("V {clip} 1 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap()).len(), frame_bytes); + } + // Leases dropped before their frames arrive (the requester gave up) also come back. + { + let (send, _dropped) = mpsc::sync_channel::>(1); + let id = worker.registry.register(worker.limit, send).unwrap(); + worker.commands.lock().unwrap().write_all(format!("V {id} {clip} 0 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n").as_bytes()).unwrap(); + } + for frame in 0..hub.slots { + assert_eq!(copy(worker.request(&format!("V {clip} {frame} {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap()).len(), frame_bytes); + } + // More callers than the helper has slots: the extra ones wait in Rust (never more than + // slots + 1 outstanding), and every frame still comes back right. + assert_eq!(worker.limit, hub.slots + 1); + let peak = std::sync::atomic::AtomicUsize::new(0); + std::thread::scope(|scope| { + for frame in 0..(hub.slots + 3) { + let (worker, peak) = (&worker, &peak); + scope.spawn(move || { + let bytes = copy(worker.request(&format!("V {clip} {} {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n", frame % 4)).unwrap()); + assert_eq!(bytes.len(), frame_bytes); + peak.fetch_max(worker.registry.lock().requests.len(), Ordering::Relaxed); + }); + } + }); + assert!(peak.load(Ordering::Relaxed) <= hub.slots + 1, "outstanding requests reached {}", peak.load(Ordering::Relaxed)); + // A delivered slot is the app's until it is freed: other frames decode around it into + // the other slots and its bytes do not change (a torn picture would be the symptom). + { + let reply = worker.request(&format!("V {clip} 3 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap(); + let Reply::Slot(lease) = &reply else { panic!("shared frame expected") }; + let before = lease.bytes().unwrap().to_vec(); + for frame in 0..(hub.slots + 2) { + assert_eq!(copy(worker.request(&format!("V {clip} {} {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n", frame % 3)).unwrap()).len(), frame_bytes); + } + std::thread::sleep(Duration::from_millis(200)); + assert_eq!(lease.bytes().unwrap(), &before[..], "a held slot was written while the app owned it"); + } + // A frame read while its helper is gone: the section outlives the process. + { + let reply = worker.request(&format!("V {clip} 0 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap(); + let Reply::Slot(lease) = &reply else { panic!("shared frame expected") }; + let before = lease.bytes().unwrap().to_vec(); + let mut child = worker.child.lock().unwrap(); + child.kill().unwrap(); + child.wait().unwrap(); + drop(child); + assert_eq!(lease.bytes().unwrap(), &before[..]); + assert!(worker.request(&format!("V {clip} 1 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).is_err()); + } + let (worker, clip, _, _) = gpu_helper_with_clip(&path); + assert_eq!(copy(worker.request(&format!("V {clip} 1 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap()).len(), frame_bytes); + // Force a stalled request deadline without a two-minute test delay. Watchdog must + // kill the process and wake all pending receivers, including abandoned requests. + let (send, receiver) = mpsc::sync_channel(1); + worker.registry.lock().requests.insert(999, PendingRequest { sender: send, started: Instant::now() - TIMEOUT, in_flight: 1 }); + assert!(receiver.recv_timeout(Duration::from_secs(2)).unwrap().is_err()); + assert!(worker.request(&format!("V {clip} 2 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).is_err()); + let status = worker.child.lock().unwrap().wait().unwrap(); + assert!(!status.success()); + // RAW adjustments the SDK does not accept (an unlisted ISO, a field set without its + // flag, a value outside the SDK's limits) are malformed requests: the helper answers an + // error naming the value instead of applying or crashing, and serves on (one helper + // serves every clip; the app never sends these anyway, `wire` refuses them first). + let (worker, clip, _, _) = gpu_helper_with_clip(&path); + for (bad, expected) in [ + ("0 0 0 0 0 0 4 0 1 1250 0 0 0", "unsupported ISO 1250"), + ("0 0 0 0 0 0 4 0 0 800 0 0 0", "malformed RAW adjustment"), + ("0 0 0 0 0 0 4 0 2 0 100001 0 0", "colour temperature out of range"), + // FCRD9 development fields: each beyond its table, a colour space and gamma that do + // not go together, a chroma value without its flag, a peak beyond the SDK's limits, + // too few or too many fields, a negative one + ("2 0 0 0 0 0 4 0 0 0 0 0 0", "unsupported development setting"), + ("0 4 0 0 0 0 4 0 0 0 0 0 0", "unsupported development setting"), + ("0 0 6 0 0 0 4 0 0 0 0 0 0", "unsupported development setting"), + ("0 0 1 5 0 0 4 0 0 0 0 0 0", "unsupported development setting"), + ("0 0 1 0 6 0 4 0 0 0 0 0 0", "unsupported development setting"), + ("0 0 0 0 0 0 32 0 0 0 0 0 0", "unsupported development setting"), + ("0 0 0 0 0 0 4 5 0 0 0 0 0", "unsupported development setting"), + ("0 0 3 0 0 0 4 0 0 0 0 0 0", "unsupported colour space and gamma curve"), + ("0 3 1 0 0 0 4 0 0 0 0 0 0", "unsupported colour space and gamma curve"), + ("0 2 4 0 0 0 4 0 0 0 0 0 0", "unsupported colour space and gamma curve"), + ("0 0 0 0 0 0 6 0 0 0 0 0 0", "malformed chroma noise reduction"), + ("0 1 4 0 0 1 4 0 0 0 0 0 0", "HDR peak out of range"), + ("0 1 4 0 0 4000000000 4 0 0 0 0 0 0", "HDR peak out of range"), + ("0 0 0 0 0 0 4 0 0 0 0 0", "video frame out of range"), + ("0 0 0 0 0 0 4 0 0 0 0 0 0 0", "video frame out of range"), + ("0 0 1 -1 0 0 4 0 0 0 0 0 0", ""), // refused however the stream reads a negative unsigned + ] { + let Err(e) = worker.request(&format!("V {clip} 0 {divisor} {bad}\n")) else { panic!("{bad} was accepted") }; + assert!(e.to_string().contains(expected), "{bad}: {e}"); + } + // a request for a clip the helper does not have is an error too, and the helper serves on + let Err(e) = worker.request(&format!("V {} 0 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n", clip + 1_000_000)) else { + panic!("an unknown clip handle was accepted") + }; + assert!(e.to_string().contains("unknown clip handle"), "{e}"); + assert_eq!(copy(worker.request(&format!("V {clip} 0 {divisor} 0 0 0 0 0 0 4 0 0 0 0 0 0\n")).unwrap()).len(), frame_bytes); + } + #[test] + fn metadata_preserves_fractional_rate_and_rejects_hostile_values() { + // FCRD9: the as-shot line, what else the clip records, and its RMD path + const REC: &str = "1 2 3 4 1000 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1"; + let text = |head: &str, raw: &str, rec: &str, path: &str| format!("{head}\n01:00:00;00\nCUDA test device\n{raw}\n{rec}\n{path}"); + const HEAD: &str = "FCRD9 6048 3402 300 30000 1001 2 48000 480480 1 32 4 123453440"; + let valid = text(HEAD, "1280 5600 -2.5 -0.0339", REC, "/footage/DSC_0001.RMD").into_bytes(); + let m = Metadata::parse(&valid).unwrap(); + assert_eq!(m.duration, Tick::from_rational(300, 1001, 30000)); + assert_eq!(m.timecode, Some(107892)); + assert_eq!(m.as_shot, RawAsShot { iso: 1280, kelvin: 5600.0, tint: -2.5, exposure: -0.0339 }); + let c = &m.clip; + assert_eq!( + (c.color_space, c.gamma, c.tone_map, c.rolloff), + (Some(RawColorSpace::Rec709), Some(RawGamma::Bt1886), Some(RawToneMap::Medium), Some(RawRollOff::Soft)) + ); + assert_eq!( + (c.hdr_peak_nits, c.chroma_noise_reduction, c.flashing_pixel, c.lut, c.hdrx, c.rmd.as_deref()), + (Some(1000), true, RawFlashing::Off, false, false, None) + ); + assert_eq!( + (c.hdr_peak_range, c.hdr_peak_default, c.tone_map_default, c.rolloff_default), + ((1000, 10000), 1000, Some(RawToneMap::Medium), Some(RawRollOff::Medium)) + ); + assert_eq!((c.cdl[0], c.cdl[1], c.cdl[2], c.cdl[3]), (1.0, 1.0, 0.0, 1.0)); + assert_eq!(c.rmd_path, "/footage/DSC_0001.RMD"); + // an RMD in effect, an HDRx clip, recorded values without a code here + let other = text(HEAD, "800 5600 0 0", "0 0 0 0 0 0 0 1 1 1 1 2 1 1000 10000 1000 0 0 1 1 0 1 1 0 1 1 0 1", "/clips/A001.RMD"); + let o = Metadata::parse(other.as_bytes()).unwrap().clip; + assert_eq!((o.color_space, o.gamma, o.tone_map, o.hdr_peak_nits), (None, None, None, None)); + assert_eq!((o.lut, o.lut_enabled, o.cdl_enabled, o.curves, o.hdrx, o.rmd.as_deref()), (true, true, true, true, true, Some("/clips/A001.RMD"))); + let cpu = Metadata::parse(text("FCRD9 6048 3402 300 30000 1001 2 48000 480480 1 32 0 0", "800 5600 0 0", REC, "").as_bytes()).unwrap(); + assert_eq!(cpu.as_shot.iso, 800); + // the recorded lens squeeze: none from a helper that sends no line, the exact ratio from + // one that does, square pixels for a value that is no usable ratio + assert_eq!((m.par, cpu.par), ((1, 1), (1, 1))); + // the squeeze line follows the RMD path + let head = text(HEAD, "800 5600 0 0", REC, "/footage/DSC_0001.RMD"); + for (line, par) in [ + ("1.33", (133, 100)), + ("1.33000004", (133, 100)), + ("1.33333337", (4, 3)), + ("2", (2, 1)), + ("1", (1, 1)), + ("0", (1, 1)), + ("", (1, 1)), + ("-1.33", (1, 1)), + ("nan", (1, 1)), + ("inf", (1, 1)), + ("1e30", (1, 1)), + ("garbage", (1, 1)), + ] { + assert_eq!(Metadata::parse(format!("{head}\n{line}").as_bytes()).unwrap().par, par, "{line}"); + } + // older helpers are refused: their requests would be misread + for older in [ + "FCRD8 6048 3402 300 30000 1001 2 48000 480480 1 32 4 123453440", + "FCRD7 6048 3402 300 30000 1001 2 48000 480480 1 32 4 123453440", + "FCRD6 6048 3402 300 30000 1001 2 48000 480480 1 32 4 123453440", + "FCRD5 6048 3402 300 30000 1001 2 48000 480480 1 32 4", + ] { + assert!(Metadata::parse(text(older, "800 5600 0 0", REC, "").as_bytes()).is_err(), "{older}"); + } + assert!(Metadata::parse(b"FCRD8 6048 3402 300 30000 1001 2 48000 480480 1 32 4 123453440\n01:00:00;00\nCUDA test device\n800 5600 0 0").is_err()); + assert!(Metadata::parse(format!("{HEAD}\n01:00:00;00\nCUDA test device\n800 5600 0 0").as_bytes()).is_err(), "no recorded line"); + assert!(Metadata::parse(format!("{HEAD}\n01:00:00;00\nCUDA test device").as_bytes()).is_err()); + for raw in [ + "", + "800", + "800 5600 0", + "800 5600 0 0 0", + "0 5600 0 0", + "-800 5600 0 0", + "800 nan 0 0", + "800 inf 0 0", + "800 5600 1e9 0", + "800 5600 0 -200", + "8e2 5600 0 0", + ] { + assert!(Metadata::parse(text(HEAD, raw, REC, "").as_bytes()).is_err(), "{raw:?}"); + } + // hostile recorded settings: wrong counts, codes out of range, non-finite CDL, a path with + // a control character + for rec in [ + "", + "1 2 3 4", + "5 2 3 4 1000 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1", + "1 7 3 4 1000 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1", + "1 2 5 4 1000 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1", + "1 2 3 6 1000 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1", + "1 2 3 4 1000 2 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1", + "1 2 3 4 1000 1 5 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1", + "1 2 3 4 1000 1 1 0 0 0 0 17 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1", + "1 2 3 4 -1 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1", + "1 2 3 4 1000 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 nan 1 0 1 1 0 1 1 0 1", + "1 2 3 4 1000 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1e9 0 1 1 0 1 1 0 1", + "1 2 3 4 1000 1 1 0 0 0 0 1 0 1000 10000 1000 3 3 1 1 0 1 1 0 1 1 0 1 7", + ] { + assert!(Metadata::parse(text(HEAD, "800 5600 0 0", rec, "").as_bytes()).is_err(), "{rec:?}"); + } + assert!(Metadata::parse(text(HEAD, "800 5600 0 0", REC, "a\u{7}b").as_bytes()).is_err()); + for s in [ + "FCRD9 0 2 1 24 1 0 0 0 0 0 4 4096", + "FCRD9 2 2 1 0 1 0 0 0 0 0 4 4096", + "FCRD9 2 2 1 24 0 0 0 0 0 0 4 4096", + "FCRD9 2 2 1 24 1 17 48000 10 1 32 4 4096", + "FCRD9 2 2 1 24 1 2 0 10 1 32 4 4096", + "FCRD9 2 2 1 24 1 2 48000 10 1 32 65 4096", + // shared slots must hold a full frame, in whole pages, within reason, and only a + // GPU helper has them + "FCRD9 6048 3402 300 30000 1001 2 48000 480480 1 32 4 123449344", + "FCRD9 6048 3402 300 30000 1001 2 48000 480480 1 32 4 123453442", + "FCRD9 6048 3402 300 30000 1001 2 48000 480480 1 32 4 0", + "FCRD9 6048 3402 300 30000 1001 2 48000 480480 1 32 0 123453440", + "FCRD9 6048 3402 300 30000 1001 2 48000 480480 1 32 64 4294967296", + "FCRD9 2 2 1 24 1 0 0 0 0 0 1 201334784", + ] { + assert!(Metadata::parse(text(s, "800 5600 0 0", REC, "").as_bytes()).is_err(), "{s}"); + } + for end in 0..valid.len() { + assert!(std::panic::catch_unwind(|| Metadata::parse(&valid[..end])).is_ok()); + } + for i in 0..valid.len() { + let mut mutated = valid.to_vec(); + mutated[i] ^= 0xff; + assert!(std::panic::catch_unwind(|| Metadata::parse(&mutated)).is_ok()); + } + } + #[test] + fn start_up_replies_name_the_slots_and_reject_hostile_values() { + let gpu = HubInfo::parse(b"FCRD9 5 123453440 3\nCUDA Example GPU").unwrap(); + assert_eq!((gpu.slots, gpu.slot_bytes, gpu.threads), (5, 123_453_440, 3)); + assert_eq!(gpu.processing, "CUDA Example GPU"); + let cpu = HubInfo::parse(b"FCRD9 0 0 3\nCPU (explicit override)").unwrap(); + assert_eq!((cpu.slots, cpu.slot_bytes), (0, 0)); + for bad in [ + "FCRD7 5 123453440 3\nCUDA x", + "FCRD8 5 123453440 3\nCUDA x", + "FCRD9 5 123453440\nCUDA x", + "FCRD9 5 123453440 3 1\nCUDA x", + "FCRD9 65 123453440 3\nCUDA x", + "FCRD9 5 123453441 3\nCUDA x", + "FCRD9 5 0 3\nCUDA x", + "FCRD9 0 4096 3\nCPU (explicit override)", + "FCRD9 5 123453440 0\nCUDA x", + "FCRD9 5 123453440 65\nCUDA x", + "FCRD9 64 4294967296 3\nCUDA x", + "FCRD9 5 123453440 3", + "FCRD9 5 123453440 3\nsomething else", + "FCRD9 -5 123453440 3\nCUDA x", + ] { + assert!(HubInfo::parse(bad.as_bytes()).is_err(), "{bad:?}"); + } + let valid = b"FCRD9 5 123453440 3\nCUDA Example GPU"; + for end in 0..valid.len() { + assert!(std::panic::catch_unwind(|| HubInfo::parse(&valid[..end])).is_ok()); + } + } + #[test] + fn float_audio_preserves_headroom_and_integer_audio_sign() { + let bytes: Vec<_> = [2.5f32, -4.0, 0.25, 0.5].iter().flat_map(|v| v.to_be_bytes()).collect(); + assert_eq!(audio_samples(&bytes, 2, 2, true).unwrap(), vec![vec![2.5, 0.25], vec![-4.0, 0.5]]); + assert_eq!(audio_samples(&i32::MIN.to_be_bytes(), 1, 1, false).unwrap()[0], vec![-1.0]); + assert!(audio_samples(&f32::NAN.to_be_bytes(), 1, 1, true).is_err()); + assert!(audio_samples(&bytes[..15], 2, 2, true).is_err()); + assert!(audio_samples(&[], 0, 1, false).is_err()); + } + #[test] + #[ignore = "diagnostic conversion benchmark; run without other workloads"] + fn benchmark_half_conversion() { + let pixels = 3024 * 1701; + let bytes: Vec<_> = (0..pixels * 3).flat_map(|sample| ((sample % 0x4400) as u16).to_le_bytes()).collect(); + let start = Instant::now(); + let texels = video_texels(&bytes, pixels).unwrap(); + let repack = start.elapsed(); + let start = Instant::now(); + let floats = VideoFrame::rgba_f16(3024, 1701, texels).to_linear_f32(); + let widen = start.elapsed(); + assert_eq!(floats.len(), pixels * 4); + println!("half conversion: repack {:.3} ms, widen to f32 {:.3} ms", repack.as_secs_f64() * 1000.0, widen.as_secs_f64() * 1000.0); + } + #[test] + fn half_rgb_preserves_linear_highlights() { + let bytes: Vec<_> = [0x3c00u16, 0x4000, 0xbc00].iter().flat_map(|v| v.to_le_bytes()).collect(); + let texels = video_texels(&bytes, 1).unwrap(); + assert_eq!(texels, vec![0x00, 0x3c, 0x00, 0x40, 0x00, 0xbc, 0x00, 0x3c]); + assert_eq!(VideoFrame::rgba_f16(1, 1, texels).to_linear_f32(), vec![1.0, 2.0, -1.0, 1.0]); + assert_eq!(filmcraft_frame::half_to_f32(1), 2.0f32.powi(-24)); + assert!(video_texels(&bytes[..5], 1).is_err()); + // infinity and NaN in any channel are rejected; the largest finite half is not + assert!(video_texels(&[0, 124, 0, 0, 0, 0], 1).is_err()); + assert!(video_texels(&[0, 0, 0, 0, 1, 0xfc], 1).is_err()); + assert!(video_texels(&[0xff, 0x7b, 0xff, 0xfb, 0, 0], 1).is_ok()); + // 16-bit code values (every other gamma) are linearised through the curve's table and reject + // nothing but a wrong size + let codes: Vec = [0u16, 32768, 65535].iter().flat_map(|v| v.to_le_bytes()).collect(); + let lin = |bytes: &[u8], g: RawGamma| VideoFrame::rgba_f16(1, 1, video_linear(bytes, 1, linearise_table(g)).unwrap()).to_linear_f32(); + let got = lin(&codes, RawGamma::Bt1886); + let want = filmcraft_color::srgb_to_linear(32768.0 / 65535.0); + assert!(got[0] == 0.0 && (got[1] - want).abs() < want * 1e-3 && (got[2] - 1.0).abs() < 1e-3 && got[3] == 1.0, "{got:?}"); + assert_eq!(video_linear(&[0, 124, 0, 0, 0, 0], 1, linearise_table(RawGamma::Srgb)).map(|v| v.len()).ok(), Some(8), "a NaN half is a plain code value"); + assert!(video_linear(&codes[..5], 1, linearise_table(RawGamma::Srgb)).is_err()); + assert!(video_linear(&codes, 2, linearise_table(RawGamma::Srgb)).is_err()); + assert!(video_linear(&codes, 1, &[0; 16]).is_err(), "a table that is not one per code value"); + } + /// Every gamma's table is the compositor's own curve: mid grey (18 % scene light, or the + /// display's mid grey for the display curves) encoded by the curve comes back as itself, and + /// every code value gives a finite half (PQ peaks at 10 000 cd/m² over reference white). + #[test] + fn linearised_code_values_round_trip_mid_grey() { + use filmcraft_color::transform::encode_channel; + for gamma in RawGamma::ALL.into_iter().filter(|g| *g != RawGamma::Linear) { + let table = linearise_table(gamma); + assert_eq!(table.len(), 65536); + assert!(table.iter().all(|h| filmcraft_frame::half_to_f32(*h).is_finite()), "{gamma:?}"); + let code = (encode_channel(0.18, gamma.curve(), filmcraft_color::Range::Full) * 65535.0).round() as u16; + let back = filmcraft_frame::half_to_f32(table[usize::from(code)]); + assert!((back - 0.18).abs() < 0.18 * 2e-3, "{gamma:?}: code {code} → {back}"); + // monotonic: a brighter code is never darker light + assert!(table.windows(2).all(|w| filmcraft_frame::half_to_f32(w[1]) >= filmcraft_frame::half_to_f32(w[0])), "{gamma:?}"); + } + let top = filmcraft_frame::half_to_f32(linearise_table(RawGamma::St2084)[65535]); + assert!((top - 10_000.0 / 203.0).abs() < 0.1, "{top}"); + } + #[test] + fn response_ids_route_out_of_order_and_discard_cancelled_receivers() { + let mut pending = Pending::default(); + let mut receivers = Vec::new(); + for id in 1..=4 { + let (send, receive) = mpsc::sync_channel(1); + pending.requests.insert(id, PendingRequest::new(send)); + receivers.push(receive); + } + drop(receivers.pop()); // ID 4 was cancelled, but its response must still be drained. + for id in [4, 2, 3, 1] { + pending.dispatch(id, Ok(Reply::Inline(vec![id as u8]))).unwrap(); + } + for (index, receiver) in receivers.iter().enumerate() { + assert_eq!(receiver.recv().unwrap().unwrap().into_inline().unwrap(), vec![(index + 1) as u8]); + } + assert!(pending.requests.is_empty()); + assert!(pending.dispatch(1, Ok(Reply::Inline(vec![]))).is_err()); // duplicate + assert!(pending.dispatch(u64::MAX, Ok(Reply::Inline(vec![]))).is_err()); // unknown + } + #[test] + fn requests_wait_for_a_free_slot_instead_of_failing() { + use std::sync::atomic::{AtomicBool, Ordering}; + let registry = Registry::default(); + let (a, _keep_a) = mpsc::sync_channel(1); + let (b, _keep_b) = mpsc::sync_channel(1); + assert_eq!(registry.register(2, a).unwrap(), 1); + assert_eq!(registry.register(2, b).unwrap(), 2); + // A third caller waits; cancelled while waiting, it registers nothing. + let cancel = Arc::new(AtomicBool::new(false)); + let waited = std::thread::scope(|scope| { + let (registry, cancel) = (®istry, &cancel); + let waiter = scope.spawn(move || { + let (c, _receiver) = mpsc::sync_channel(1); + filmcraft_media::cancel::with_cancel(cancel, || registry.register(2, c)) + }); + std::thread::sleep(Duration::from_millis(60)); + assert!(!waiter.is_finished()); + cancel.store(true, Ordering::Relaxed); + waiter.join().unwrap() + }); + assert!(matches!(waited, Err(MediaError::Cancelled))); + assert_eq!(registry.lock().requests.len(), 2); + // A reply frees a slot and wakes the next waiter. + let woken = std::thread::scope(|scope| { + let registry = ®istry; + let waiter = scope.spawn(move || { + let (c, _receiver) = mpsc::sync_channel(1); + registry.register(2, c) + }); + std::thread::sleep(Duration::from_millis(60)); + assert!(!waiter.is_finished()); + registry.dispatch(1, Ok(Reply::Inline(vec![]))).unwrap(); + waiter.join().unwrap() + }); + assert_eq!(woken.unwrap(), 3); + // A failed worker wakes waiters with its error instead of leaving them for the deadline. + let failed = std::thread::scope(|scope| { + let registry = ®istry; + let waiter = scope.spawn(move || { + let (c, _receiver) = mpsc::sync_channel(1); + registry.register(2, c) + }); + std::thread::sleep(Duration::from_millis(60)); + registry.fail("helper exited"); + waiter.join().unwrap() + }); + assert!(failed.unwrap_err().to_string().contains("helper exited")); + } + #[test] + fn throughput_is_the_best_recent_in_flight_over_latency() { + let mut pending = Pending::default(); + assert!(pending.rate().is_none()); + // `outstanding` others are still in flight when the reply arrives (never answered here) + let mut fillers = Vec::new(); + let answer = |pending: &mut Pending, in_flight: usize, ms: u64, ok: bool| { + let (send, _receiver) = mpsc::sync_channel(1); + let id = pending.next + 1; + pending.next = id; + let request = PendingRequest { sender: send, started: Instant::now() - Duration::from_millis(ms), in_flight }; + pending.requests.insert(id, request); + pending.dispatch(id, if ok { Ok(Reply::Inline(vec![])) } else { Err(failure("gpu")) }).unwrap(); + }; + // a helper with four slots: one request at a time (the SDK idles, 220 ms a frame) says + // nothing about its capacity + pending.slots = 4; + for id in 1_000..1_003 { + let (send, receiver) = mpsc::sync_channel(1); + fillers.push(receiver); + pending.requests.insert(id, PendingRequest::new(send)); + } + for _ in 0..3 { + answer(&mut pending, 1, 220, true); + } + assert!(pending.rate().is_none()); + // four in flight at 160 ms each: 25 fps, once three such replies are in + answer(&mut pending, 4, 160, true); + answer(&mut pending, 4, 160, true); + assert!(pending.rate().is_none()); + for _ in 0..2 { + answer(&mut pending, 4, 160, true); + } + let busy = pending.rate().unwrap(); + assert!((24.0..26.0).contains(&busy), "{busy}"); + // a lull does not lower it; a fast saturated reply moves the mean, not the reading + answer(&mut pending, 1, 220, true); + assert_eq!(pending.rate(), Some(busy)); + answer(&mut pending, 4, 120, true); + let bumped = pending.rate().unwrap(); + assert!((26.0..28.0).contains(&bumped), "{bumped}"); + // failed replies and the start-up reply are not throughput + answer(&mut pending, 8, 1, false); + assert_eq!(pending.rate(), Some(bumped)); + // five in flight (one waiting in the helper's pipe) at 200 ms each is the same 25 fps: + // the wait for a slot is part of the latency, so all five count + for _ in 0..4 { + answer(&mut pending, 5, 200, true); + } + let piped = pending.rate().unwrap(); + assert!((24.0..27.0).contains(&piped), "{piped}"); + // the window is bounded, and through a long lull the last saturated reading stands + for _ in 0..(2 * RATE_WINDOW) { + answer(&mut pending, 2, 100, true); + } + assert_eq!(pending.answered.len(), RATE_WINDOW); + assert_eq!(pending.rate(), Some(piped)); + // a burst into an idle helper: registered with six in flight, but answered with the + // pipeline drained behind it (a free slot, no queueing), so its 60 fps is not a reading + for id in 1_000..1_003 { + pending.requests.remove(&id); + } + for _ in 0..3 { + answer(&mut pending, 6, 100, true); + } + assert_eq!(pending.rate(), Some(piped)); + drop(fillers); + } + #[test] + fn development_settings_are_sent_as_small_range_checked_integers() { + use filmcraft_media::{Hundredths, RawQuality}; + // half space gamma toneMap rolloff peak options flashing | flags iso kelvin tint exposure + assert_eq!(wire(&RawDevelop::default(), Some(false)).unwrap(), "0 0 0 0 0 0 4 0 0 0 0 0 0", "linear, as shot, the curves as recorded"); + let bt1886 = RawDevelop { gamma: RawGamma::Bt1886, tone_map: RawToneMap::High, rolloff: RawRollOff::VerySoft, ..Default::default() }; + assert_eq!(wire(&bt1886, Some(true)).unwrap(), "1 0 1 4 5 0 4 0 0 0 0 0 0"); + assert_eq!(wire(&bt1886, None).unwrap(), "0 1 4 5 0 4 0 0 0 0 0 0", "an RMD write has no decode quality"); + let pq = RawDevelop { + color_space: RawColorSpace::Rec2020, + gamma: RawGamma::St2084, + hdr_peak_nits: Some(1000), + chroma_noise_reduction: Some(true), + flashing_pixel: Some(RawFlashing::Strong), + metadata_curves: false, + creative_lut: true, + cdl: true, + quality: Some(RawQuality::HalfGood), + ..Default::default() + }; + assert_eq!(wire(&pq, Some(false)).unwrap(), "0 1 4 0 0 1000 27 4 0 0 0 0 0"); + let off = RawDevelop { chroma_noise_reduction: Some(false), flashing_pixel: Some(RawFlashing::Off), ..Default::default() }; + assert_eq!(wire(&off, None).unwrap(), "0 0 0 0 0 5 1 0 0 0 0 0", "chroma off is set (1) without its value (2)"); + // every code is within the helper's tables + for space in RawColorSpace::ALL { + for gamma in RawGamma::ALL.into_iter().filter(|g| RawDevelop::supports(space, *g)) { + let fields: Vec = + wire(&RawDevelop { color_space: space, gamma, ..Default::default() }, None).unwrap().split(' ').map(|f| f.parse().unwrap()).collect(); + assert!(fields[0] <= 3 && fields[1] <= 5, "{space:?} {gamma:?}"); + } + } + // the RAW adjustments: a flag word, then ISO, Kelvin, tint and exposure in hundredths + let all = RawDevelop { iso: Some(800), kelvin: Some(4300), tint: Some(Hundredths(-1250)), exposure: Some(Hundredths(67)), ..bt1886 }; + assert_eq!(wire(&all, None).unwrap(), "0 1 4 5 0 4 0 15 800 4300 -1250 67"); + let some = RawDevelop { kelvin: Some(100_000), exposure: Some(Hundredths(-800)), ..Default::default() }; + assert_eq!(wire(&some, None).unwrap(), "0 0 0 0 0 4 0 10 0 100000 0 -800"); + // out-of-range values and unsupported combinations never reach the helper + for bad in [ + RawDevelop { iso: Some(1250), ..Default::default() }, + RawDevelop { kelvin: Some(1699), ..Default::default() }, + RawDevelop { tint: Some(Hundredths(10_001)), ..Default::default() }, + RawDevelop { exposure: Some(Hundredths(801)), ..Default::default() }, + RawDevelop { hdr_peak_nits: Some(50), ..Default::default() }, + RawDevelop { gamma: RawGamma::Log3G10, ..Default::default() }, + RawDevelop { color_space: RawColorSpace::P3D65, gamma: RawGamma::Hlg, ..Default::default() }, + ] { + assert!(wire(&bad, None).is_err(), "{bad:?}"); + } + // frames are linear light in the developed primaries + assert_eq!(frame_color(&RawDevelop::default()).transfer, Transfer::Linear); + assert_eq!(frame_color(&bt1886).transfer, Transfer::Linear); + assert_eq!(frame_color(&pq).primaries, filmcraft_color::Primaries::Bt2020); + // the decode quality caps a preview's decode, a full-size request is always full + assert_eq!(capped_divisor(6048, 3402, 0.5, 1), 2); + assert_eq!(capped_divisor(6048, 3402, 0.5, 4), 4); + assert_eq!(capped_divisor(6048, 3402, 0.06, 2), 16, "a smaller preview still decodes smaller"); + assert_eq!(capped_divisor(6048, 3402, 0.99, 16), 16); + assert_eq!(capped_divisor(15, 15, 0.9, 16), 8, "a clip too small for the quality falls back"); + assert_eq!(capped_divisor(1, 1, 0.5, 16), 1); + } + #[test] + fn fatal_helper_error_after_start_up_carries_its_message() { + let mut pending = Pending::default(); + let error = pending.dispatch(0, Err(failure("decode request exceeds line limit"))).unwrap_err(); + assert!(error.to_string().contains("line limit"), "{error}"); + assert!(pending.dispatch(0, Ok(Reply::Inline(b"FCRD8".to_vec()))).is_err()); + } + #[test] + fn deadlines_include_cancelled_requests_and_failure_wakes_every_waiter() { + let mut pending = Pending::default(); + let (send, cancelled) = mpsc::sync_channel(1); + pending.requests.insert(1, PendingRequest::new(send)); + drop(cancelled); + assert!(pending.expired(Duration::ZERO)); + let (send, waiting) = mpsc::sync_channel(1); + pending.requests.insert(2, PendingRequest::new(send)); + pending.fail("helper exited"); + assert!(waiting.recv_timeout(Duration::from_millis(100)).unwrap().is_err()); + assert!(pending.requests.is_empty()); + assert!(pending.failed.is_some()); + } + /// A reply header as the helper writes it (FCRD6). + fn header(status: u32, id: u64, length: u64, slot: u32) -> Vec { + let mut packet = status.to_le_bytes().to_vec(); + packet.extend_from_slice(&id.to_le_bytes()); + packet.extend_from_slice(&length.to_le_bytes()); + packet.extend_from_slice(&slot.to_le_bytes()); + packet + } + /// Stands in for the helper's stdin: what the app wrote, readable by the test. + struct Sink(Arc>>); + impl Write for Sink { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.0.lock().unwrap().extend_from_slice(bytes); + Ok(bytes.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + /// The helper's side of the shared frames (a section of `slots` pages) and the app's view + /// of it, with a sink for the app's `F` commands. + fn shared_frames(tag: &str, slots: u32) -> (shm::TestSection, OnceLock>, Arc>>) { + let name = format!("filmcraft-red-test-{tag}-{}", std::process::id()); + let section = shm::TestSection::create(&name, slots as usize * 4096).unwrap(); + let written = Arc::new(Mutex::new(Vec::new())); + let mapping = shm::Mapping::open(&name, slots as usize * 4096).unwrap(); + let shared = + OnceLock::from(Arc::new(SharedFrames { mapping, slots, slot_bytes: 4096, commands: Arc::new(Mutex::new(Box::new(Sink(written.clone())))) })); + (section, shared, written) + } + #[test] + fn protocol_rejects_truncation_and_oversized_replies() { + let mut packet = header(0, 42, 3, 0); + packet.extend_from_slice(b"abc"); + let (id, bytes) = read_reply(&mut &packet[..], &OnceLock::new()).unwrap(); + assert_eq!(id, 42); + assert_eq!(bytes.unwrap().into_inline().unwrap(), b"abc"); + for end in 0..packet.len() { + assert!(read_reply(&mut &packet[..end], &OnceLock::new()).is_err()); + } + for i in 0..packet.len() { + let mut changed = packet.clone(); + changed[i] ^= 0xff; + assert!(std::panic::catch_unwind(|| read_reply(&mut &changed[..], &OnceLock::new())).is_ok()); + } + let mut error_packet = packet.clone(); + error_packet[..4].copy_from_slice(&1u32.to_le_bytes()); + let (id, result) = read_reply(&mut &error_packet[..], &OnceLock::new()).unwrap(); + assert_eq!(id, 42); + // an error answers its request only: the helper serves on + assert!(matches!(result, Ok(Reply::Error(ref message)) if message == "abc")); + // an inline reply names no slot, a status beyond the protocol's is rejected, a shared + // frame needs shared memory + let mut slotted = packet.clone(); + slotted[20] = 1; + assert!(read_reply(&mut &slotted[..], &OnceLock::new()).is_err()); + let mut unknown = packet.clone(); + unknown[0] = 3; + assert!(read_reply(&mut &unknown[..], &OnceLock::new()).is_err()); + assert!(read_reply(&mut &header(STATUS_SHARED, 42, 3, 0)[..], &OnceLock::new()).is_err()); + packet[12..20].copy_from_slice(&u64::MAX.to_le_bytes()); + assert!(read_reply(&mut &packet[..], &OnceLock::new()).is_err()); + } + #[test] + fn shared_frames_are_read_in_place_and_handed_back() { + let (section, shared, written) = shared_frames("slots", 2); + section.write(4096, b"frame one").unwrap(); + section.write(0, b"frame zero").unwrap(); + // the reply carries only the header; the bytes are the slot's, exactly `length` of them + let (id, reply) = read_reply(&mut &header(STATUS_SHARED, 7, 9, 1)[..], &shared).unwrap(); + assert_eq!(id, 7); + let Reply::Slot(lease) = reply.unwrap() else { panic!("shared frame expected") }; + assert_eq!(lease.bytes().unwrap(), b"frame one"); + assert!(written.lock().unwrap().is_empty()); + // the helper may change a slot it has not handed over; this one is read only when freed + section.write(0, b"FRAME ZERO").unwrap(); + drop(lease); + assert_eq!(std::str::from_utf8(&written.lock().unwrap()).unwrap(), "F 1\n"); + let (_, reply) = read_reply(&mut &header(STATUS_SHARED, 8, 10, 0)[..], &shared).unwrap(); + let Reply::Slot(lease) = reply.unwrap() else { panic!("shared frame expected") }; + assert_eq!(lease.bytes().unwrap(), b"FRAME ZERO"); + // a frame nobody waits for still hands its slot back: dispatched into a dropped + // channel, or discarded with an unknown id + let mut pending = Pending::default(); + let (send, receiver) = mpsc::sync_channel(1); + pending.requests.insert(8, PendingRequest::new(send)); + drop(receiver); + pending.dispatch(8, Ok(Reply::Slot(lease))).unwrap(); + assert_eq!(std::str::from_utf8(&written.lock().unwrap()).unwrap(), "F 1\nF 0\n"); + // bounds: a slot beyond the count or a length beyond the slot is the helper's fault, and + // the mapping checks every read against the bytes it mapped + assert!(read_reply(&mut &header(STATUS_SHARED, 9, 1, 2)[..], &shared).is_err()); + assert!(read_reply(&mut &header(STATUS_SHARED, 9, 4097, 1)[..], &shared).is_err()); + assert!(shared.get().unwrap().mapping.slice(4096, 4097).is_err()); + assert!(shared.get().unwrap().mapping.slice(usize::MAX, 1).is_err()); + assert_eq!(shared.get().unwrap().mapping.slice(8192, 0).unwrap(), b""); + assert_eq!(shared.get().unwrap().mapping.len(), 8192); + // audio and metadata never come in a slot + let (_, reply) = read_reply(&mut &header(STATUS_SHARED, 10, 4, 0)[..], &shared).unwrap(); + assert!(reply.unwrap().into_inline().is_err()); + assert_eq!(std::str::from_utf8(&written.lock().unwrap()).unwrap(), "F 1\nF 0\nF 0\n"); + } + #[test] + fn shared_frame_mapping_needs_the_section_at_the_declared_size() { + let name = format!("filmcraft-red-test-size-{}", std::process::id()); + assert!(shm::Mapping::open(&name, 4096).is_err()); + let section = shm::TestSection::create(&name, 8192).unwrap(); + assert!(shm::Mapping::open(&name, 8192 + 4096).is_err()); + assert!(shm::Mapping::open(&name, 0).is_err()); + assert!(shm::Mapping::open("not a valid name", 4096).is_err()); + let mapping = shm::Mapping::open(&name, 8192).unwrap(); + section.write(8000, b"tail").unwrap(); + assert_eq!(mapping.slice(8000, 4).unwrap(), b"tail"); + // the view outlives the section's creator's handle, as it must outlive a dead helper + drop(section); + assert_eq!(mapping.slice(8000, 4).unwrap(), b"tail"); + } +} diff --git a/crates/platform/src/red/shm.rs b/crates/platform/src/red/shm.rs new file mode 100644 index 0000000000..dbea4e9b66 --- /dev/null +++ b/crates/platform/src/red/shm.rs @@ -0,0 +1,349 @@ +//! The helper's frame slots, read in place (FCRD6). The one FFI module of this backend +//! (docs/adr/0001-platform-ffi.md): a read-only view of the memory the helper creates, through +//! the official `windows` bindings (a named, pagefile-backed section) or `libc` (a POSIX shared +//! memory object under `/dev/shm`). Everything else in `red` is safe code; the view is handed +//! out as byte slices with their bounds checked here, and both arms present the same API. + +use filmcraft_media::{MediaError, Result}; + +fn failure(message: impl Into) -> MediaError { + MediaError::Decode(message.into()) +} + +/// The name the app chose for the helper (letters, digits and dashes) and a non-empty length. +fn valid(name: &str, len: usize) -> Result<()> { + if len == 0 || name.is_empty() || name.len() > 128 || !name.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'-') { + return Err(failure("invalid RED shared frame memory")); + } + Ok(()) +} + +/// The helper's frame memory name while the app does not yet hold the memory: held across the +/// worker's start-up and dropped on every path out of it, it removes the name on Linux, where +/// a helper killed before the app mapped its object would otherwise leave an empty object under +/// `/dev/shm` (a name already gone, the usual case, is fine). A Windows section dies with its +/// handles, so there is nothing to remove. +pub(super) struct Name(String); +impl Name { + pub(super) fn new(name: &str) -> Self { + Self(name.to_owned()) + } +} +impl Drop for Name { + fn drop(&mut self) { + remove(&self.0); + } +} + +#[cfg(target_os = "linux")] +pub(super) use posix_object::*; +#[cfg(windows)] +pub(super) use windows_section::*; + +#[cfg(windows)] +mod windows_section { + use super::{failure, valid}; + use filmcraft_media::Result; + use windows::Win32::Foundation::{CloseHandle, HANDLE}; + use windows::Win32::System::Memory::{FILE_MAP_READ, MEMORY_MAPPED_VIEW_ADDRESS, MapViewOfFile, OpenFileMappingW, UnmapViewOfFile}; + use windows::core::PCWSTR; + + fn wide(name: &str) -> Vec { + format!("Local\\{name}-frames").encode_utf16().chain(std::iter::once(0)).collect() + } + + /// Nothing to do: a section lives exactly as long as a handle or view of it does. + pub(in super::super) fn remove(_: &str) {} + + /// A read-only view of `len` bytes of a named section another process created. + pub(in super::super) struct Mapping { + handle: HANDLE, + view: MEMORY_MAPPED_VIEW_ADDRESS, + len: usize, + } + + // SAFETY: the view is plain memory owned by this process for the life of the value; the handle + // and base address are used only through `&self` methods that read, and the section is never + // written through this view (it is mapped read-only), so sharing it between threads is sound. + unsafe impl Send for Mapping {} + // SAFETY: as above; `slice` only reads, and the drop runs once. + unsafe impl Sync for Mapping {} + + impl Mapping { + /// Open the section `Local\-frames` for reading and map its first `len` bytes. + /// Fails if the section does not exist or is smaller than `len` (the view then covers + /// memory that the section guarantees to be readable, whatever the other process does + /// later). + pub(in super::super) fn open(name: &str, len: usize) -> Result { + valid(name, len)?; + let wide = wide(name); + // SAFETY: `wide` is a NUL-terminated UTF-16 string that outlives the call; the handle + // returned is owned by this value and closed in `drop`. + let handle = unsafe { OpenFileMappingW(FILE_MAP_READ.0, false, PCWSTR(wide.as_ptr())) } + .map_err(|e| failure(format!("RED shared frame memory unavailable: {e}")))?; + // SAFETY: `handle` is an open file-mapping handle with read access; a view of `len` + // bytes from offset 0 is requested and the call fails (null) rather than mapping less + // when the section is smaller than `len`, so a non-null view is `len` readable bytes + // until `UnmapViewOfFile`. + let view = unsafe { MapViewOfFile(handle, FILE_MAP_READ, 0, 0, len) }; + if view.Value.is_null() { + let error = windows::core::Error::from_thread(); + // SAFETY: `handle` is open and owned here; nothing else refers to it. + let _ = unsafe { CloseHandle(handle) }; + return Err(failure(format!("RED shared frame memory could not be mapped: {error}"))); + } + Ok(Self { handle, view, len }) + } + /// Bytes `offset..offset + len` of the view. + /// + /// The helper writes a slot only while it decodes into it and hands the slot over before + /// its reply, and the app hands it back (`F `) only once it has finished reading; a + /// helper that exits leaves the pages as they are (a pagefile-backed section lives while + /// this view does). So within the protocol the bytes do not change under the slice. A + /// helper that broke the protocol could change them, as any process can change a file + /// another has mapped; the readers tolerate any byte values (they are validated, never + /// trusted), and no Rust invariant beyond the slice's own contents rests on them. + pub(in super::super) fn slice(&self, offset: usize, len: usize) -> Result<&[u8]> { + let end = offset.checked_add(len).ok_or_else(|| failure("RED frame slot out of range"))?; + if end > self.len { + return Err(failure("RED frame slot out of range")); + } + if len == 0 { + return Ok(&[]); + } + // SAFETY: `offset + len <= self.len`, and the view is `self.len` readable bytes (see + // `open`) for as long as `self` lives, which the returned lifetime does not exceed; the + // memory is initialized (a section's pages read as zero until written) and `u8` has no + // invalid values. Concurrent modification by the other process is discussed above. + Ok(unsafe { std::slice::from_raw_parts((self.view.Value as *const u8).add(offset), len) }) + } + #[cfg(test)] + pub(in super::super) fn len(&self) -> usize { + self.len + } + } + + impl Drop for Mapping { + fn drop(&mut self) { + // SAFETY: `view` came from a successful `MapViewOfFile` and `handle` from + // `OpenFileMappingW`, both owned by this value and released exactly once here; no + // slice borrowed from `self` can outlive it. + unsafe { + let _ = UnmapViewOfFile(self.view); + let _ = CloseHandle(self.handle); + } + } + } + + /// A section created in this process, for the tests: the helper's side of the protocol. + #[cfg(test)] + pub(in super::super) struct TestSection { + handle: HANDLE, + view: MEMORY_MAPPED_VIEW_ADDRESS, + len: usize, + } + #[cfg(test)] + impl TestSection { + pub(in super::super) fn create(name: &str, len: usize) -> Result { + use windows::Win32::Foundation::INVALID_HANDLE_VALUE; + use windows::Win32::System::Memory::{CreateFileMappingW, FILE_MAP_ALL_ACCESS, PAGE_READWRITE}; + let wide = wide(name); + // SAFETY: a pagefile-backed section (no file handle) of `len` bytes with a NUL-terminated + // name that outlives the call; the handle is owned by the returned value. + let handle = + unsafe { CreateFileMappingW(INVALID_HANDLE_VALUE, None, PAGE_READWRITE, (len as u64 >> 32) as u32, len as u32, PCWSTR(wide.as_ptr())) } + .map_err(|e| failure(format!("test section: {e}")))?; + // SAFETY: `handle` is an open read-write mapping of at least `len` bytes. + let view = unsafe { MapViewOfFile(handle, FILE_MAP_ALL_ACCESS, 0, 0, len) }; + if view.Value.is_null() { + // SAFETY: owned, open handle. + let _ = unsafe { CloseHandle(handle) }; + return Err(failure("test section could not be mapped")); + } + Ok(Self { handle, view, len }) + } + pub(in super::super) fn write(&self, offset: usize, bytes: &[u8]) -> Result<()> { + let end = offset.checked_add(bytes.len()).filter(|end| *end <= self.len).ok_or_else(|| failure("test section write out of range"))?; + let _ = end; + // SAFETY: `offset + bytes.len() <= self.len` within a writable view this value owns, and + // `bytes` cannot overlap the view (it is a Rust slice of this process's own memory). + unsafe { std::ptr::copy_nonoverlapping(bytes.as_ptr(), (self.view.Value as *mut u8).add(offset), bytes.len()) }; + Ok(()) + } + } + #[cfg(test)] + impl Drop for TestSection { + fn drop(&mut self) { + // SAFETY: as for `Mapping`. + unsafe { + let _ = UnmapViewOfFile(self.view); + let _ = CloseHandle(self.handle); + } + } + } +} + +/// Linux: the helper creates the POSIX shared memory object `/-frames` (a file on the +/// tmpfs at `/dev/shm`) exclusively, sizes it once with `ftruncate` and never resizes it; the +/// app opens it read-only, checks its size against the geometry the helper declared, maps it +/// shared and then removes the name (the mapping keeps the object alive, as a view keeps a +/// Windows section alive, so a helper that dies mid-frame leaves the app reading valid memory). +/// A process of the same user could still shrink the object under the mapping (reading past +/// the new end would then fault), as it could ptrace the app; nothing short of that can. +#[cfg(target_os = "linux")] +mod posix_object { + use super::{failure, valid}; + use filmcraft_media::Result; + use std::ffi::CString; + use std::os::fd::{AsRawFd, FromRawFd, OwnedFd}; + + /// The object's POSIX name, `/-frames`. + fn object(name: &str) -> Result { + CString::new(format!("/{name}-frames")).map_err(|_| failure("invalid RED shared frame memory")) + } + + /// Remove the object's name, if it has one (its memory lives on in every mapping of it). + pub(in super::super) fn remove(name: &str) { + if valid(name, 1).is_err() { + return; + } + if let Ok(object) = object(name) { + // SAFETY: `object` is a NUL-terminated string that outlives the call; the call + // touches no memory of this process and its result (a name already gone) is ignored. + let _ = unsafe { libc::shm_unlink(object.as_ptr()) }; + } + } + + /// A read-only mapping of `len` bytes of a shared memory object another process created. + pub(in super::super) struct Mapping { + base: *const u8, + len: usize, + } + + // SAFETY: the mapping is plain memory owned by this process for the life of the value; the + // base address is used only through `&self` methods that read, and the object is never + // written through this mapping (it is mapped read-only), so sharing it between threads is sound. + unsafe impl Send for Mapping {} + // SAFETY: as above; `slice` only reads, and the drop runs once. + unsafe impl Sync for Mapping {} + + impl Mapping { + /// Open `/-frames` for reading and map its first `len` bytes. Fails if the object + /// does not exist or is smaller than `len` (a mapping past the end of the object would + /// fault when read), and removes the object's name once the mapping holds it. + pub(in super::super) fn open(name: &str, len: usize) -> Result { + valid(name, len)?; + let object = object(name)?; + // SAFETY: `object` is a NUL-terminated string that outlives the call; the descriptor + // returned is owned by `file` below and closed with it. + let fd = unsafe { libc::shm_open(object.as_ptr(), libc::O_RDONLY | libc::O_CLOEXEC, 0) }; + if fd < 0 { + return Err(failure(format!("RED shared frame memory unavailable: {}", std::io::Error::last_os_error()))); + } + // SAFETY: `fd` is an open descriptor that nothing else owns. + let file = std::fs::File::from(unsafe { OwnedFd::from_raw_fd(fd) }); + let size = file.metadata().map_err(|e| failure(format!("RED shared frame memory unavailable: {e}")))?.len(); + if size < len as u64 { + return Err(failure("RED shared frame memory is smaller than the helper declared")); + } + // SAFETY: a read-only, shared mapping of `len` bytes from offset 0 of an open + // descriptor whose object is at least `len` bytes (checked above); the address hint + // is null, `MAP_FAILED` is checked, and the mapping outlives the descriptor. + let base = unsafe { libc::mmap(std::ptr::null_mut(), len, libc::PROT_READ, libc::MAP_SHARED, file.as_raw_fd(), 0) }; + if base == libc::MAP_FAILED { + return Err(failure(format!("RED shared frame memory could not be mapped: {}", std::io::Error::last_os_error()))); + } + // The name is the app's to remove now that the mapping holds the object. + // SAFETY: as in `remove`. + let _ = unsafe { libc::shm_unlink(object.as_ptr()) }; + Ok(Self { base: base as *const u8, len }) + } + /// Bytes `offset..offset + len` of the mapping. + /// + /// The helper writes a slot only while it decodes into it and hands the slot over before + /// its reply, and the app hands it back (`F `) only once it has finished reading; a + /// helper that exits leaves the pages as they are (the object lives while this mapping + /// does). So within the protocol the bytes do not change under the slice. A helper that + /// broke the protocol could change them, as any process can change a file another has + /// mapped; the readers tolerate any byte values (they are validated, never trusted), and + /// no Rust invariant beyond the slice's own contents rests on them. + pub(in super::super) fn slice(&self, offset: usize, len: usize) -> Result<&[u8]> { + let end = offset.checked_add(len).ok_or_else(|| failure("RED frame slot out of range"))?; + if end > self.len { + return Err(failure("RED frame slot out of range")); + } + if len == 0 { + return Ok(&[]); + } + // SAFETY: `offset + len <= self.len`, and the mapping is `self.len` readable bytes + // (see `open`) for as long as `self` lives, which the returned lifetime does not + // exceed; the memory is initialized (a shared memory object's pages read as zero + // until written) and `u8` has no invalid values. Concurrent modification by the + // other process is discussed above. + Ok(unsafe { std::slice::from_raw_parts(self.base.add(offset), len) }) + } + #[cfg(test)] + pub(in super::super) fn len(&self) -> usize { + self.len + } + } + + impl Drop for Mapping { + fn drop(&mut self) { + // SAFETY: `base` and `len` came from a successful `mmap` owned by this value and are + // unmapped exactly once here; no slice borrowed from `self` can outlive it. + let _ = unsafe { libc::munmap(self.base as *mut libc::c_void, self.len) }; + } + } + + /// An object created in this process, for the tests: the helper's side of the protocol. + #[cfg(test)] + pub(in super::super) struct TestSection { + object: CString, + base: *mut u8, + len: usize, + } + #[cfg(test)] + impl TestSection { + pub(in super::super) fn create(name: &str, len: usize) -> Result { + let object = object(name)?; + // SAFETY: `object` is NUL-terminated and outlives the call; the descriptor returned + // is owned by `file` below. + let fd = unsafe { libc::shm_open(object.as_ptr(), libc::O_RDWR | libc::O_CREAT | libc::O_EXCL | libc::O_CLOEXEC, 0o600) }; + if fd < 0 { + return Err(failure(format!("test section: {}", std::io::Error::last_os_error()))); + } + // SAFETY: `fd` is an open descriptor that nothing else owns. + let file = std::fs::File::from(unsafe { OwnedFd::from_raw_fd(fd) }); + if let Err(e) = file.set_len(len as u64) { + super::remove(name); + return Err(failure(format!("test section: {e}"))); + } + // SAFETY: a read-write shared mapping of `len` bytes of an object just sized to `len`. + let base = unsafe { libc::mmap(std::ptr::null_mut(), len, libc::PROT_READ | libc::PROT_WRITE, libc::MAP_SHARED, file.as_raw_fd(), 0) }; + if base == libc::MAP_FAILED { + super::remove(name); + return Err(failure("test section could not be mapped")); + } + Ok(Self { object, base: base as *mut u8, len }) + } + pub(in super::super) fn write(&self, offset: usize, bytes: &[u8]) -> Result<()> { + let end = offset.checked_add(bytes.len()).filter(|end| *end <= self.len).ok_or_else(|| failure("test section write out of range"))?; + let _ = end; + // SAFETY: `offset + bytes.len() <= self.len` within a writable mapping this value owns, + // and `bytes` cannot overlap it (it is a Rust slice of this process's own memory). + unsafe { std::ptr::copy_nonoverlapping(bytes.as_ptr(), self.base.add(offset), bytes.len()) }; + Ok(()) + } + } + #[cfg(test)] + impl Drop for TestSection { + fn drop(&mut self) { + // SAFETY: as for `Mapping`; the name may already be gone (the app removes it). + unsafe { + let _ = libc::munmap(self.base as *mut libc::c_void, self.len); + let _ = libc::shm_unlink(self.object.as_ptr()); + } + } + } +} diff --git a/crates/project/src/lib.rs b/crates/project/src/lib.rs index 05693820f0..5c70f66afa 100644 --- a/crates/project/src/lib.rs +++ b/crates/project/src/lib.rs @@ -178,6 +178,12 @@ pub struct Interpretation { /// is edited into a sequence. None = one stereo clip of the first two channels. #[serde(default, skip_serializing_if = "Option::is_none")] pub audio_channels: Option, + /// Clip ▸ Source Settings… for camera RAW: how the decoder develops the picture. None = the + /// project's Camera RAW settings (Decode Using ▸ Project); before those existed, it meant the + /// decoder's default (scene-linear Rec. 709, as shot), which is what a project from before + /// them still gets through its project settings. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub develop: Option, } /// Modify ▸ Audio Channels: the clip channel format and, per audio clip, the source channels it @@ -246,6 +252,12 @@ mod hex_u64 { } impl MediaClip { + /// Camera RAW with development settings (Clip ▸ Source Settings…): today R3D NE through the + /// RED SDK, by the codec its decoder reported or, before it was opened, the file name. + pub fn is_camera_raw(&self) -> bool { + self.info.video.as_ref().is_some_and(|v| v.codec == "R3D NE") + || matches!(&self.media, MediaRef::File { path } if path.rsplit('.').next().is_some_and(|e| e.eq_ignore_ascii_case("r3d"))) + } pub fn frame_rate(&self) -> FrameRate { self.interpret.frame_rate.unwrap_or_else(|| self.info.frame_rate()) } @@ -1158,6 +1170,13 @@ pub struct ProjectSettings { /// Project Settings ▸ Scratch Disks. Schema v11. #[serde(default)] pub scratch: ScratchDisks, + /// Project Settings ▸ Camera RAW: how camera RAW clips that decode with the project's + /// settings (Source Settings ▸ Decode Using ▸ Project) are developed. A new project starts + /// from `RawDevelop::STANDARD` (Rec. 709, BT.1886); a project from before this setting reads + /// as `RawDevelop::default()` (scene-linear Rec. 709, as shot), what its clips got then. + /// Schema v16. + #[serde(default)] + pub camera_raw: filmcraft_media::RawDevelop, } fn title_safe_default() -> (f64, f64) { @@ -1225,6 +1244,7 @@ impl Default for ProjectSettings { action_safe: action_safe_default(), capture_format: capture_format_default(), scratch: ScratchDisks::default(), + camera_raw: filmcraft_media::RawDevelop::STANDARD, } } } @@ -1750,6 +1770,20 @@ fn resolve_auto_points_sized(e: &mut EffectInstance, frame: (u32, u32), source: } impl Project { + /// The development settings a camera RAW clip decodes with: its own (Source Settings ▸ + /// Decode Using ▸ Clip) or the project's Camera RAW settings (Decode Using ▸ Project, or no + /// settings of its own). Decode Using is reported as it is. None for other media. + pub fn develop_of(&self, m: &MediaClip) -> Option { + m.is_camera_raw().then(|| filmcraft_media::RawDevelop::effective(&self.settings.camera_raw, m.interpret.develop.as_ref())) + } + + /// The colour space a clip's frames are interpreted in when not by their own metadata: + /// Interpret Footage's override, else for camera RAW the interpretation its development + /// needs (the decoder hands over linear light in the developed primaries). + pub fn color_space_override(&self, m: &MediaClip) -> Option { + m.interpret.color_space.or_else(|| self.develop_of(m).and_then(|d| d.interpretation())) + } + /// Lazy upgrade for graphic layers written before they had stable uids: assign a unique /// [`crate::graphic_design::LayerExtra::uid`] to every uid-less or duplicated graphic layer /// in sequences and source graphics. diff --git a/crates/render/src/colorman.rs b/crates/render/src/colorman.rs index e645b9c3a3..fd6ec50e15 100644 --- a/crates/render/src/colorman.rs +++ b/crates/render/src/colorman.rs @@ -17,9 +17,10 @@ use filmcraft_project::{ItemId, Project}; use crate::Image; -/// The Interpret Footage colour-space override of an item (following subclips to their media). +/// The Interpret Footage colour-space override of an item (following subclips to their media), or +/// for camera RAW the interpretation its development needs ([`Project::color_space_override`]). pub fn override_of(project: &Project, item: ItemId) -> Option { - project.resolve_media(item).and_then(|(_, media, _)| media.interpret.color_space) + project.resolve_media(item).and_then(|(_, media, _)| project.color_space_override(media)) } /// The colour space a frame of `item` is interpreted in. @@ -75,11 +76,11 @@ pub fn decode(project: &Project, item: ItemId, frame: &VideoFrame, n: usize, pip } // RGB stills/generators carry full-range code values; YUV carries its own range flag let range = match frame.data { - filmcraft_frame::PixelData::Rgba8(_) | filmcraft_frame::PixelData::RgbaF32(_) => Range::Full, + filmcraft_frame::PixelData::Rgba8(_) | filmcraft_frame::PixelData::RgbaF32(_) | filmcraft_frame::PixelData::RgbaF16(_) => Range::Full, _ => frame.color.range, }; let t = input_transform(cs, range, pipe, source_peak_nits(project, item)); - let float_src = matches!(frame.data, filmcraft_frame::PixelData::RgbaF32(_)); + let float_src = matches!(frame.data, filmcraft_frame::PixelData::RgbaF32(_) | filmcraft_frame::PixelData::RgbaF16(_)); // float frames are linear already: only the gamut/tone stages apply to them let (w, h, px) = frame.to_linear_f32_decimated_with(n, if float_src { None } else { Some(&t.table) }); let mut img = Image { w, h, px }; @@ -168,6 +169,104 @@ mod tests { assert_eq!(HdrMetadata { max_cll: Some(80.0), ..Default::default() }.peak_nits(), Some(203.0)); } + /// Camera RAW: the decoder hands over linear light in the developed primaries (the 16-bit code + /// values of a display or log gamma linearised with the compositor's own curve), and the + /// clip's development decides how the compositor interprets it. For every colour space and + /// gamma Source Settings offers, that gives exactly what the compositor makes of the code + /// values themselves; mid grey comes back as mid grey, and a BT.1886 or sRGB grey reaches the + /// monitor as the code value the decoder wrote. + #[test] + fn camera_raw_outputs_are_interpreted_as_their_code_values_and_mid_grey_round_trips() { + use filmcraft_color::transform::{decode_channel, encode_channel}; + use filmcraft_media::{MediaSource, RawColorSpace, RawDevelop, RawGamma}; + use filmcraft_project::{ItemKind, Label, MediaClip, MediaRef}; + let clip_with = |d: RawDevelop| { + let mut p = Project::new("raw"); + let g = filmcraft_media::Generator::BlackVideo; + let mut info = filmcraft_media::generators::GeneratorSource::new(g, 2, 1, Default::default(), filmcraft_time::Tick(1)).info().clone(); + if let Some(v) = info.video.as_mut() { + v.codec = "R3D NE".into(); + v.color = ColorInfo { transfer: Transfer::Linear, ..ColorInfo::SRGB_FULL }; + } + let interpret = filmcraft_project::Interpretation { develop: Some(d), ..Default::default() }; + let clip = MediaClip { + media: MediaRef::File { path: "/footage/DSC_0001.R3D".into() }, + info, + interpret, + mark_in: None, + mark_out: None, + markers: vec![], + offline: false, + proxy: None, + identity: None, + }; + let id = p.add_item("DSC_0001.R3D", Label::Iris, ItemKind::Media(clip), None); + (p, id) + }; + let half = |v: f64| filmcraft_frame::f32_to_half(v as f32).to_le_bytes(); + for pipe in [ColorPipeline::REC709, ColorPipeline { working: WorkingSpace::Rec2100Pq, ..ColorPipeline::REC709 }] { + for space in RawColorSpace::ALL { + for gamma in RawGamma::ALL.into_iter().filter(|g| RawDevelop::supports(space, *g)) { + let d = RawDevelop { color_space: space, gamma, ..Default::default() }; + let (p, id) = clip_with(d); + let curve = gamma.curve(); + // a neutral mid grey and a colour, as the decoder's 16-bit code values + let grey = if gamma == RawGamma::Linear { 0.18 } else { (encode_channel(0.18, curve, Range::Full) * 65535.0).round() / 65535.0 }; + let colour = [0.55, 0.40, 0.25].map(|v| if gamma == RawGamma::Linear { v * 0.4 } else { v }); + // what the source delivers: linear light as halves, tagged linear in its primaries + let lin = |v: f64| if gamma == RawGamma::Linear { v } else { decode_channel(v, curve, Range::Full) }; + let mut bytes = Vec::new(); + for px in [[grey; 3], colour] { + for v in px { + bytes.extend(half(lin(v))); + } + bytes.extend(half(1.0)); + } + let primaries = match space { + RawColorSpace::Rec2020 => Primaries::Bt2020, + RawColorSpace::P3D65 => Primaries::P3D65, + _ => Primaries::Bt709, + }; + let mut f = VideoFrame::rgba_f16(2, 1, bytes); + f.color = ColorInfo { transfer: Transfer::Linear, primaries, ..ColorInfo::SRGB_FULL }; + let got = decode(&p, id, &f, 1, &pipe); + // the compositor's own reading of those code values in the interpreted space + let cs = d.interpretation().unwrap_or(match gamma { + RawGamma::Linear => ColorSpace::Linear709, + RawGamma::Srgb => ColorSpace::Srgb, + _ => ColorSpace::Rec709, + }); + // Rec. 709 frames need no interpretation: they say they are linear Rec. 709 (the fast path) + assert_eq!(source_space(&p, id, &f), d.interpretation().unwrap_or(ColorSpace::Linear709), "{space:?} {gamma:?}"); + let t = InputTransform::new(cs, Range::Full, &pipe, None); + for (i, px) in [[grey; 3], colour].into_iter().enumerate() { + let code = px.map(|v| v as f32); + let want = if gamma == RawGamma::Linear { t.apply(code) } else { t.convert(code) }; + for c in 0..3 { + let g = got.px[i * 4 + c]; + assert!( + (g - want[c]).abs() <= 2e-3 * want[c].abs().max(0.05), + "{pipe:?} {space:?} {gamma:?} pixel {i} channel {c}: {g} vs {}", + want[c] + ); + } + } + // mid grey stays mid grey (and neutral) wherever nothing tone maps it + let tone_mapped = cs.is_hdr() && !pipe.working.is_hdr(); + let g = &got.px[..3]; + assert!((g[0] - g[1]).abs() < 1e-3 && (g[1] - g[2]).abs() < 1e-3, "{space:?} {gamma:?}: grey is not neutral {g:?}"); + if !tone_mapped && gamma != RawGamma::Hlg { + assert!((g[1] - 0.18).abs() < 0.18 * 3e-3, "{pipe:?} {space:?} {gamma:?}: mid grey {}", g[1]); + } + if pipe.is_plain() && matches!(gamma, RawGamma::Bt1886 | RawGamma::Srgb) && space == RawColorSpace::Rec709 { + let shown = filmcraft_color::linear_to_srgb(g[1]) as f64; + assert!((shown - grey).abs() < 1.0 / 1023.0, "{gamma:?}: the monitor shows {shown}, the decoder wrote {grey}"); + } + } + } + } + } + #[test] fn plain_media_keeps_the_fast_path() { let f = frame([200, 100, 50], ColorInfo::SRGB_FULL); diff --git a/crates/ui-egui/examples/thumb_burst.rs b/crates/ui-egui/examples/thumb_burst.rs index 87b18f4217..a683b595fb 100644 --- a/crates/ui-egui/examples/thumb_burst.rs +++ b/crates/ui-egui/examples/thumb_burst.rs @@ -62,7 +62,7 @@ fn main() { use std::hash::{Hash, Hasher}; let mut h = std::collections::hash_map::DefaultHasher::new(); if let Some(filmcraft_project::ItemKind::Media(m)) = s.project.item(id).map(|i| &i.kind) { - filmcraft_engine::media_pool::media_key(m).hash(&mut h); + filmcraft_engine::media_pool::media_key(m, &s.project.settings.camera_raw).hash(&mut h); false.hash(&mut h); } h.finish() diff --git a/crates/ui-egui/src/frames.rs b/crates/ui-egui/src/frames.rs index 60888522e5..e1044d3996 100644 --- a/crates/ui-egui/src/frames.rs +++ b/crates/ui-egui/src/frames.rs @@ -146,6 +146,9 @@ struct Job { scale: f32, project: Arc, prio: u32, + /// Asked for with playback stopped ([`FrameServer::request_while`]): one frame someone waits for + /// (`filmcraft_media::cancel::with_still`). + still: bool, } struct Shared { @@ -162,6 +165,8 @@ struct Shared { /// frames: compositing, CPU effects. Frames cannot be ready sooner than this after they are /// started, and the workers can finish at most `workers / cost` of them per second. render_cost: Mutex, + /// What playback jobs learned about throughput-bound sources (see [`SourceRate`]). + throughput: Mutex, workers: usize, /// Always-on counters for `perf.stats`. stats: Mutex, @@ -169,6 +174,239 @@ struct Shared { stop: AtomicBool, } +/// What playback knows about the throughput-bound sources it is playing (sources whose frames +/// decode independently, [`filmcraft_media::FrameDelivery`]): the frames per second the slowest +/// of them delivers and the time a playback job on one takes from its start to its frame. Drives +/// [`throughput_plan`]. +#[derive(Clone, Copy, Debug, Default, PartialEq)] +pub struct SourceRate { + /// Frames per second (smoothed); `None` until a source reported one or enough jobs completed. + pub rate: Option, + /// Seconds from a job's start to its frame (smoothed, rising fast), with the wait for a + /// decoder slot: what the lead is sized by. + pub latency: f64, + /// The same, smoothed slowly both ways: what the workers' throughput is judged by, so one + /// slow job (a 60 ms one among 40 ms ones at 120 fps) does not read as capacity lost. + pub latency_slow: f64, +} + +#[derive(Default)] +struct Throughput { + model: SourceRate, + /// A playback job has touched a throughput-bound source, and the ones since did too. + active: bool, + /// Playback jobs in a row that touched none: a few of them (another sequence, the clip after + /// the cut) clear the model, so an ordinary source is not played with a stride. + misses: u32, + /// When jobs on such sources finished (not cancelled), for the rate when the source itself + /// reports none. + completions: VecDeque, + /// The pattern playback last planned with (see [`choose_pattern`]). + pattern: Option<(i64, i64)>, + /// The refresh window being counted (see [`Self::settle`]): since when, refreshes in it, + /// how many found the picture late (more than a pattern step behind), how many on time. + window: Option<(Instant, u32, u32, u32)>, + /// Windows in a row with the picture on time. + on_time_windows: u32, + /// When the pattern was last made sparser because the picture ran late, and how long after + /// that a denser one may be tried (doubles while trying fails, see [`Self::settle`]). + backed_off_at: Option, + cooldown: Duration, + /// A denser pattern was stepped up to at this time, from this one: a step that runs late + /// within [`Self::HOLD`] goes back to it and doubles the cooldown. + stepped_up: Option<(Instant, (i64, i64))>, + /// The source measures its own rate, and when a job last brought a fresh reading of it + /// (see [`filmcraft_media::FrameDelivery::fresh`]). + reports_rate: bool, + fresh_at: Option, + /// When the playback clock started (the preroll ended): the picture is not judged for + /// [`Self::STARTUP`] after that, while the first planned frames are still on their way. + clock_started: Option, +} + +impl Throughput { + /// Playback jobs without a throughput-bound source after which the model is dropped. + const MISSES: u32 = 12; + const COMPLETIONS: usize = 32; + /// Refreshes are counted in windows this long; a window with two in five refreshes late + /// makes the pattern sparser, whatever the capacity reads. + const WINDOW: Duration = Duration::from_millis(500); + /// Windows in a row with the picture on time that allow a denser pattern, once the + /// cooldown has passed. + const SLACK_WINDOWS: u32 = 2; + /// The cooldown after backing off: this, doubled for every step up that failed within + /// [`Self::HOLD`], up to a minute; back to this once a step up held that long. + const COOLDOWN: Duration = Duration::from_secs(4); + const MAX_COOLDOWN: Duration = Duration::from_secs(60); + const HOLD: Duration = Duration::from_secs(3); + /// Lateness is not judged this long after backing off: the frames the denser pattern left + /// queued are dropped as the playhead passes them, the new pattern's frames are started a + /// lead ahead and take the latency to arrive, and the picture catches up meanwhile; a + /// back-off every window would otherwise cascade to every n-th frame. + const RECOVER: Duration = Duration::from_secs(2); + /// A reading older than this is stale: playback asks for a few more frames at once to + /// measure the source again (see [`FrameServer::schedule_playback`]). + const FRESH_FOR: Duration = Duration::from_secs(2); + /// The picture is not judged this long after the clock starts: the preroll frames are + /// ready, the planned run beyond them is still decoding, and a window judged on that + /// backed a 6K 59.94p clip off from every third frame to every fifth at every start. + const STARTUP: Duration = Duration::from_secs(2); + + /// The playback clock is running (`false` while prerolling); `Some` once the picture may be + /// judged, with the time now. + fn judging(&mut self, running: bool, now: Instant) -> bool { + if !running { + self.clock_started = None; + return false; + } + let started = *self.clock_started.get_or_insert(now); + now.saturating_duration_since(started) >= Self::STARTUP + } + + /// The pattern to use, given what the capacity reading chose (`chosen`, with the rate's + /// hysteresis) and how the picture is doing: `lag` is how many frames the newest ready + /// picture of the pattern in use is behind the frame due (`None` while the clock has not + /// started). + /// + /// The reading is a throughput measured at saturation, within a few per cent of the + /// sequence rate for a 6K 24p clip on a 22–27 fps pipeline, so the picture itself decides + /// at the margin, counted over [`Self::WINDOW`]s of refreshes (a pipeline a fifth short of + /// a pattern still lands most of its frames, so the lag flickers between zero and several + /// frames rather than staying high): a window with two in five refreshes late → sparser + /// now, back to the pattern before a step up that just failed; the reading's denser choice + /// only after [`Self::SLACK_WINDOWS`] on time and the cooldown since backing off; while a + /// step up is held for [`Self::HOLD`] the reading's sparser verdict is ignored. Stepping up + /// beyond what the reading allows was tried: a step that fails costs the frames the + /// decoder decoded for nothing, and a 6K 59.94p clip at every third frame paid + /// 50–60 cancelled frames and a cascade of back-offs for each try at every second one; a + /// stale reading is refreshed by asking for a few more frames at once instead. + fn settle(&mut self, chosen: (i64, i64), lag: Option, now: Instant) -> (i64, i64) { + let density = |(keep, of): (i64, i64)| keep as f64 / of.max(1) as f64; + let Some(in_use) = self.pattern else { return chosen }; + let Some(lag) = lag else { + // nothing shown yet: no reason to ask for more than before + return if density(chosen) > density(in_use) { in_use } else { chosen }; + }; + let step = in_use.1.max(1); + // count this refresh; close the window once it is full + let (mut late_window, mut closed) = (false, false); + match &mut self.window { + Some((since, n, late, ok)) if now.saturating_duration_since(*since) < Self::WINDOW => { + *n += 1; + *late += (lag > step) as u32; + // on time: no further behind than the frames the pattern leaves out of each group + *ok += (lag <= step - in_use.0.clamp(1, step)) as u32; + } + w => { + if let Some((_, n, late, ok)) = *w { + closed = true; + late_window = late.saturating_mul(5) >= n.saturating_mul(2); + self.on_time_windows = if ok.saturating_mul(10) >= n.saturating_mul(9) { self.on_time_windows.saturating_add(1) } else { 0 }; + } + *w = Some((now, 1, (lag > step) as u32, (lag <= step - in_use.0.clamp(1, step)) as u32)); + } + } + let holding = self.stepped_up.filter(|(at, _)| now.saturating_duration_since(*at) < Self::HOLD); + if self.stepped_up.is_some() && holding.is_none() { + // the step up held: trying again later starts from the base cooldown + self.stepped_up = None; + self.cooldown = Self::COOLDOWN; + } + let recovering = self.backed_off_at.is_some_and(|a| now.saturating_duration_since(a) < Self::RECOVER); + if closed && late_window && !recovering && density(chosen) >= density(in_use) { + let back = match holding { + Some((_, before)) => { + self.cooldown = self.cooldown.max(Self::COOLDOWN).saturating_mul(2).min(Self::MAX_COOLDOWN); + before + } + None => { + self.cooldown = Self::COOLDOWN; + next_sparser(in_use) + } + }; + self.backed_off_at = Some(now); + self.stepped_up = None; + self.on_time_windows = 0; + return back; + } + if holding.is_some() && density(chosen) < density(in_use) { + return in_use; + } + let slack = closed && self.on_time_windows >= Self::SLACK_WINDOWS; + let cooled = self.backed_off_at.is_none_or(|a| now.saturating_duration_since(a) >= self.cooldown.max(Self::COOLDOWN)); + if density(chosen) > density(in_use) { + if !(slack && cooled) { + return in_use; + } + self.stepped_up = Some((now, in_use)); + self.on_time_windows = 0; + return chosen; + } + chosen + } + + /// The source measures its own rate and its last reading is older than [`Self::FRESH_FOR`]: + /// a sparse plan keeps too few requests in flight for the source to measure itself. + fn reading_stale(&self, now: Instant) -> bool { + self.active && self.reports_rate && self.fresh_at.is_none_or(|f| now.saturating_duration_since(f) > Self::FRESH_FOR) + } + + /// A playback job finished: `delivery` is what its sources said (`None`: none of them is + /// throughput-bound), `decoded` whether it asked any source for a frame at all (a frame past + /// the end of the sequence, a gap or a rendered segment asks none, and says nothing about + /// what is playing), `took` its start-to-finish time when it was not cancelled. + fn job(&mut self, delivery: Option, decoded: bool, took: Option) { + let Some(d) = delivery.filter(|d| d.independent) else { + if decoded { + self.misses = self.misses.saturating_add(1); + if self.misses >= Self::MISSES { + *self = Self::default(); + } + } + return; + }; + self.misses = 0; + self.active = true; + let now = Instant::now(); + self.reports_rate |= d.rate.is_some(); + if d.fresh { + self.fresh_at = Some(now); + } + if let Some(took) = took.filter(|t| t.is_finite() && *t >= 0.0) { + // late is worse than early: rise fast, settle slowly + let l = self.model.latency; + self.model.latency = if l == 0.0 { + took + } else if took > l { + l * 0.5 + took * 0.5 + } else { + l * 0.8 + took * 0.2 + }; + let s = self.model.latency_slow; + self.model.latency_slow = if s == 0.0 { took } else { s * 0.95 + took * 0.05 }; + if self.completions.len() >= Self::COMPLETIONS { + self.completions.pop_front(); + } + self.completions.push_back(now); + } + let measured = d.rate.filter(|r| r.is_finite() && *r > 0.0).or_else(|| { + // completions over the last two seconds, once there are enough to mean something + let recent: Vec<&Instant> = self.completions.iter().filter(|t| now.saturating_duration_since(**t) <= Duration::from_secs(2)).collect(); + let span = recent.first().map(|t| now.saturating_duration_since(**t).as_secs_f64()).unwrap_or(0.0); + (recent.len() >= 6 && span > 0.0).then(|| (recent.len() - 1) as f64 / span) + }); + if let Some(r) = measured { + self.model.rate = Some(match self.model.rate { + // the source's reading is its best recent moment: believe a rise slowly, a + // fall quickly, so the stride does not flap on one fast reply + Some(p) if r > p => p * 0.9 + r * 0.1, + Some(p) => p * 0.6 + r * 0.4, + None => r, + }); + } + } +} + /// Cumulative frame-job statistics (`perf.stats`): cheap enough to keep on all the time. #[derive(Clone, Debug, Default)] pub struct FrameStats { @@ -189,8 +427,9 @@ pub struct FrameStats { pub source_ms: f64, pub render_ms: f64, pub job_ms: f64, - /// The most recent jobs: (source ms, render ms). - pub recent: VecDeque<(f32, f32)>, + /// The most recent jobs: (source ms, render ms, texel preparation ms, plan-cache insert ms); + /// the last two are part of the render time of a GPU plan job. + pub recent: VecDeque<(f32, f32, f32, f32)>, } impl FrameStats { @@ -198,7 +437,7 @@ impl FrameStats { /// JSON for `perf.stats`. pub fn to_json(&self) -> serde_json::Value { - let pct = |f: &dyn Fn(&(f32, f32)) -> f32, p: f64| -> f64 { + let pct = |f: &dyn Fn(&(f32, f32, f32, f32)) -> f32, p: f64| -> f64 { let mut v: Vec = self.recent.iter().map(f).collect(); if v.is_empty() { return 0.0; @@ -213,6 +452,8 @@ impl FrameStats { "requestHitRate": if req == 0 { 0.0 } else { self.request_hits as f64 / req as f64 }, "decodeMs": {"mean": self.source_ms / n, "p50": pct(&|r| r.0, 0.5), "p95": pct(&|r| r.0, 0.95)}, "renderMs": {"mean": self.render_ms / n, "p50": pct(&|r| r.1, 0.5), "p95": pct(&|r| r.1, 0.95)}, + "prepareMs": {"p50": pct(&|r| r.2, 0.5), "p95": pct(&|r| r.2, 0.95)}, + "insertMs": {"p50": pct(&|r| r.3, 0.5), "p95": pct(&|r| r.3, 0.95)}, "jobMsMean": self.job_ms / n, }) } @@ -275,6 +516,11 @@ pub fn process_cpu_time() -> Option { thread_local! { /// (wall, cpu) spent in source `video_frame` calls by this worker during the current job. static SOURCE_TIME: Cell<(Duration, Duration)> = const { Cell::new((Duration::ZERO, Duration::ZERO)) }; + /// What the throughput-bound sources of the current job said about their delivery (the + /// slowest rate among them); `None` when none was asked for a frame. + static DELIVERY: Cell> = const { Cell::new(None) }; + /// The current job asked a source for a frame. + static DECODED: Cell = const { Cell::new(false) }; /// This worker is profiling the current job (thread CPU of source calls is measured). static PROFILING: Cell = const { Cell::new(false) }; } @@ -300,8 +546,23 @@ impl filmcraft_media::MediaSource for TimedSource { self.0.info() } fn video_frame(&self, req: filmcraft_media::FrameRequest) -> filmcraft_media::Result> { + let d = self.0.frame_delivery(); + if d.independent { + DELIVERY.with(|c| { + let m = c.get().unwrap_or_default(); + let rate = match (m.rate, d.rate) { + (Some(a), Some(b)) => Some(a.min(b)), + (a, b) => a.or(b), + }; + c.set(Some(filmcraft_media::FrameDelivery { independent: true, rate, fresh: m.fresh || d.fresh })); + }); + } + DECODED.with(|c| c.set(true)); timed_source(|| self.0.video_frame(req)) } + fn frame_delivery(&self) -> filmcraft_media::FrameDelivery { + self.0.frame_delivery() + } fn audio(&self, start: i64, frames: usize, sample_rate: u32) -> filmcraft_media::Result { self.0.audio(start, frames, sample_rate) } @@ -337,48 +598,122 @@ impl GpuPlan { } } -/// Finished plans, bounded by count and bytes (oldest use evicted first). +/// Finished plans, bounded by count and bytes (see [`eviction_key`] for which go first). #[derive(Default)] struct PlanCache { - map: HashMap, u64, usize)>, + /// Plan, when it was last looked up, when it was inserted, its bytes. + map: HashMap, u64, u64, usize)>, clock: u64, bytes: usize, + /// Bytes of the plan inserted last: what one frame of what is playing costs. + last_bytes: usize, +} + +/// Frames playback may hold ahead of the playhead when the cache holds `capacity` of them: +/// all of `depth` while they fit, else what fits beside the frame on screen and one behind it. +/// Decoders that run a little faster than the frames are due would otherwise fill the cache +/// with frames far ahead and evict the one due next (a 6K half-resolution float plan is 123 MB, +/// eight to the budget, while the prefetch runs 14 deep). +fn lookahead(capacity: usize, depth: i64) -> i64 { + depth.min(i64::try_from(capacity).unwrap_or(i64::MAX).saturating_sub(2)).max(2) +} + +/// Which cache entries go first when the cache is over budget, given when an entry was last +/// looked up and when it was inserted: entries looked up since they were inserted (shown, or +/// scrubbed past), least recently used first; then the ones never looked up, oldest first. +/// Playback prefetches frames ahead of the playhead, and nothing looks at one until it is due: +/// under plain LRU they were the oldest entries, and a 6K half-resolution frame (then 123 MB as +/// a plan, eight to the budget) decoded 26 frames ahead was evicted before the playhead reached +/// it, while the frames behind the playhead stayed. A never-used entry keeps that place only +/// while it is fresh (inserted within [`PlanCache::FRESH`] ticks of the cache's `clock`) and +/// playback has not stopped since ([`PlanCache::stale_prefetch`]): a frame playback skipped or +/// prefetched before it stopped, or one of an old revision, would otherwise stay until nothing +/// else is left, and a scrub would evict the frames it had just shown instead. +fn eviction_key(used: u64, inserted: u64, clock: u64) -> (bool, u64) { + (used == inserted && clock.saturating_sub(inserted) < PlanCache::FRESH, used) +} + +/// When a plan last looked up at `used` counts as inserted once playback stopped (see +/// [`PlanCache::stale_prefetch`]): one nobody looked up no longer counts as fresh. +fn stopped(used: u64, inserted: u64) -> u64 { + if used == inserted { inserted.saturating_sub(PlanCache::FRESH) } else { inserted } } impl PlanCache { const MAX: usize = 96; const BUDGET: usize = 1 << 30; + /// Lookups and inserts within which a prefetched plan is still due (playback refreshes look + /// up a few plans a frame; it prefetches at most [`prefetch_depth`] frames ahead). + const FRESH: u64 = 1024; + + /// Playback stopped: the plans it prefetched and nobody looked up count as used when they + /// were inserted, so they go before the frames a scrub shows next. + fn stale_prefetch(&mut self) { + for (_, used, inserted, _) in self.map.values_mut() { + *inserted = stopped(*used, *inserted); + } + } fn get(&mut self, k: &FrameKey) -> Option> { self.clock += 1; let clock = self.clock; - self.map.get_mut(k).map(|(p, used, _)| { + self.map.get_mut(k).map(|(p, used, _, _)| { *used = clock; p.clone() }) } + /// How many plans like the last one fit the budget. + fn capacity(&self) -> usize { + (Self::BUDGET / self.last_bytes.max(1)).min(Self::MAX) + } + fn insert(&mut self, k: FrameKey, p: GpuPlan) { self.clock += 1; let b = p.bytes(); + self.last_bytes = b; self.bytes += b; - if let Some((_, _, old)) = self.map.insert(k, (Arc::new(p), self.clock, b)) { + if let Some((_, _, _, old)) = self.map.insert(k, (Arc::new(p), self.clock, self.clock, b)) { self.bytes -= old; } if self.map.len() > Self::MAX || self.bytes > Self::BUDGET { - let mut v: Vec<(u64, FrameKey, usize)> = self.map.iter().map(|(k, v)| (v.1, *k, v.2)).collect(); + let clock = self.clock; + let mut v: Vec<((bool, u64), FrameKey, usize)> = self.map.iter().map(|(k, v)| (eviction_key(v.1, v.2, clock), *k, v.3)).collect(); v.sort_unstable_by_key(|x| x.0); for (_, k, b) in v { if self.map.len() <= Self::MAX && self.bytes <= Self::BUDGET { break; } - self.map.remove(&k); + if let Some((p, ..)) = self.map.remove(&k) { + recycle_plan(p); + } self.bytes -= b; } } } } +/// An evicted plan's decoded frames go back to the frame pool when nothing else holds them (a +/// 6K camera-RAW frame is 41 MB: allocated fresh for every frame, it page-faults for ~20 ms on +/// the worker; see `filmcraft_frame::pool`). +fn recycle_plan(plan: Arc) { + use filmcraft_render::plan::{FramePlan, PlanStep}; + let Ok(p) = Arc::try_unwrap(plan) else { return }; + let layers = |ls: Vec| ls.into_iter().for_each(|l| filmcraft_frame::pool::recycle(l.frame)); + match p.plan { + FramePlan::Layers { layers: ls, .. } => layers(ls), + FramePlan::Composite { steps, .. } => { + for s in steps { + match s { + PlanStep::Layers(ls) => layers(ls), + PlanStep::Transition { inputs, .. } => inputs.into_iter().for_each(&layers), + } + } + } + FramePlan::Image(_) => {} + } +} + struct Cache { map: HashMap, u64)>, bytes: usize, @@ -446,6 +781,139 @@ pub fn playback_plan(render_cost: f64, fps: f64, workers: usize) -> (i64, i64) { (frames_per_job.ceil() as i64, load.ceil() as i64) } +/// The patterns a throughput-bound plan may keep, `(keep, of)` frames, densest first: every +/// frame, three of four, two of three, every second, two of five, three of eight, every third, +/// every fourth, then every n-th. Three of four is what a 6K 24p clip gets from a pipeline that +/// manages 20 fps, instead of every second frame. Two of five is 24 fps of a 59.94p clip: a 6K +/// half-resolution pipeline delivers 25–27 fps, and without it the plan fell to every third +/// frame (20 fps) where 22.5–24 fps is possible. +const PATTERNS: [(i64, i64); 8] = [(1, 1), (3, 4), (2, 3), (1, 2), (2, 5), (3, 8), (1, 3), (1, 4)]; +const SPARSEST: i64 = 64; + +/// The densest pattern whose share of the frames the source can deliver (`ratio` = frames per +/// second it delivers over the sequence's), with hysteresis against `previous`: a sparser +/// pattern once the share is 10 % short of the current one (the source fell behind; a source +/// that just keeps up reads as exactly the sequence rate, since saturation is what is measured, +/// and must not be read as falling behind on noise), a denser one only with 5 % to spare. The +/// reading alone is not what decides at the margin: [`Throughput::settle`] backs off as soon +/// as the picture runs late and waits for slack before asking for more, so a pipeline a few +/// per cent short of every frame does not fall behind for seconds before the reading says so. +fn choose_pattern(ratio: f64, previous: Option<(i64, i64)>) -> (i64, i64) { + if !ratio.is_finite() || ratio <= 0.0 { + return previous.unwrap_or((1, SPARSEST)); + } + let density = |(keep, of): (i64, i64)| keep as f64 / of.max(1) as f64; + let fit = PATTERNS.into_iter().find(|p| density(*p) <= ratio).unwrap_or_else(|| (1, ((1.0 / ratio).ceil() as i64).clamp(5, SPARSEST))); + match previous { + Some(p) if density(p) > ratio => { + if ratio < density(p) * 0.9 { + fit + } else { + p + } + } + Some(p) if density(fit) > density(p) && ratio < density(fit) * 1.05 => p, + _ => fit, + } +} + +/// The next sparser pattern after `p`: the next of [`PATTERNS`], then every n-th. +fn next_sparser(p: (i64, i64)) -> (i64, i64) { + match PATTERNS.iter().position(|q| *q == p) { + Some(i) => PATTERNS.get(i + 1).copied().unwrap_or((1, p.1.saturating_add(1).min(SPARSEST))), + None => (1, p.1.max(1).saturating_add(1).min(SPARSEST)), + } +} + +/// Whether frame `f` is in the pattern: the frames where `f × keep / of` crosses an integer, +/// `keep` of every `of`, spread evenly and the same set whatever the playhead is at. +fn in_pattern(f: i64, keep: i64, of: i64) -> bool { + let of = of.max(1); + let keep = keep.clamp(1, of); + keep == of || f.saturating_mul(keep).div_euclid(of) != f.saturating_sub(1).saturating_mul(keep).div_euclid(of) +} + +/// Which frames playback asks for from a throughput-bound source (frames decode independently, +/// [`SourceRate`]): `(lead, keep, of)`. The plan keeps `keep` of every `of` frames +/// ([`choose_pattern`]) according to what the source delivers against what the sequence needs, +/// bounded as well by the frame workers (each holds a job for the source's latency: six workers +/// at 330 ms sustain 18 frames per second whatever the decoder manages), so that a 25 fps +/// decoder under 59.94 shows every fourth frame evenly instead of all of them each a little +/// late. `lead` is the latency in frames with a quarter and a frame of slack, so a frame started +/// that early is ready when it is due. `previous` is the pattern in use, for hysteresis. +pub fn throughput_plan(model: SourceRate, fps: f64, workers: usize, previous: Option<(i64, i64)>) -> (i64, i64, i64) { + let (keep, of) = match capacity_ratio(model, fps, workers) { + Some(r) => choose_pattern(r, previous), + None => (1, 1), + }; + let lead = (model.latency * fps * 1.25).ceil() as i64 + 1; + (lead.clamp(1, 600), keep, of) +} + +/// What the source delivers against what the sequence needs: the lesser of the source's rate +/// and the workers' bound (`workers / latency`, slowly smoothed), over `fps`; `None` with +/// nothing measured yet. +fn capacity_ratio(model: SourceRate, fps: f64, workers: usize) -> Option { + let rate = model.rate.filter(|r| r.is_finite() && *r > 0.0); + let slow = if model.latency_slow > 0.0 { model.latency_slow } else { model.latency }; + let by_workers = (slow.is_finite() && slow > 0.0).then(|| workers.max(1) as f64 / slow); + let capacity = match (rate, by_workers) { + (Some(a), Some(b)) => Some(a.min(b)), + (a, b) => a.or(b), + }; + capacity.filter(|_| fps > 0.0).map(|c| c / fps) +} + +/// The frames a playback refresh at `cur` asks for: `ahead` of them from `lead` frames on. +/// Plain (`anchored` false): every `of`-th. Anchored: the frames of the pattern `keep` of `of` +/// ([`in_pattern`]), so the set does not shift with the playhead at every refresh: the frames +/// between are left out for good, not asked for a refresh later when they could not be ready +/// either. +fn planned_frames(cur: i64, dir: i64, step: i64, lead: i64, keep: i64, of: i64, ahead: i64, anchored: bool) -> Vec { + let (step, of) = (step.max(1), of.max(1)); + let keep = keep.clamp(1, of); + let first = cur.saturating_add(lead.saturating_mul(step).saturating_mul(dir)); + let ahead = ahead.max(0); + if !anchored || keep == of { + let unit = if anchored { step } else { of.saturating_mul(step) }; + return (0..ahead).map(|i| first.saturating_add(i.saturating_mul(unit).saturating_mul(dir))).collect(); + } + let mut out = Vec::with_capacity(ahead as usize); + let mut f = first; + // every `of` consecutive frames hold at least `keep` of the pattern: the walk is bounded + for _ in 0..ahead.saturating_mul(of).saturating_add(of) { + if out.len() as i64 >= ahead { + break; + } + if in_pattern(f, keep, of) { + out.push(f); + } + f = f.saturating_add(step.saturating_mul(dir)); + } + out +} + +/// The pattern `FILMCRAFT_PLAYBACK_PATTERN=keep/of` pins (read once), if any. +fn forced_pattern() -> Option<(i64, i64)> { + static FORCED: std::sync::OnceLock> = std::sync::OnceLock::new(); + *FORCED.get_or_init(|| { + let s = std::env::var("FILMCRAFT_PLAYBACK_PATTERN").ok()?; + let (k, n) = s.trim().split_once('/')?; + let (k, n) = (k.trim().parse::().ok()?, n.trim().parse::().ok()?); + (n >= 1 && (1..=n).contains(&k) && n <= SPARSEST).then_some((k, n)) + }) +} + +/// Frames of the pattern asked for beyond the look-ahead while a throughput-bound source's +/// reading is stale: a pipelined decoder's slots and a couple more, so its replies are +/// saturated ones. +const REMEASURE_BURST: i64 = 6; + +/// How far behind the playhead a running job on a throughput-bound source is left to finish: +/// its decoder has the frame in hand (cancelling it saves nothing), and the monitor shows a late +/// frame when nothing newer is ready, looking this far back. +pub const LATE_GRACE: i64 = 6; + impl FrameServer { /// Settings ▸ Memory ▸ frame cache budget (bytes); evicts down to it right away. pub fn set_cache_budget(&self, bytes: usize) { @@ -475,6 +943,7 @@ impl FrameServer { profiling: AtomicBool::new(false), records: Mutex::new(Vec::new()), render_cost: Mutex::new(0.0), + throughput: Mutex::new(Throughput::default()), workers: workers.max(1), stats: Mutex::new(FrameStats::default()), stop: AtomicBool::new(false), @@ -494,8 +963,12 @@ impl FrameServer { Self { shared, pool, services, previews, repaint } } - /// The default number of frame workers for this machine. + /// The default number of frame workers for this machine. `FILMCRAFT_FRAME_WORKERS` (2..=32) + /// overrides it: a bake-off knob, not a user setting. pub fn default_workers() -> usize { + if let Some(n) = std::env::var("FILMCRAFT_FRAME_WORKERS").ok().and_then(|s| s.trim().parse::().ok()) { + return n.clamp(2, 32); + } std::thread::available_parallelism().map(|n| n.get().clamp(2, 6)).unwrap_or(3) } @@ -573,11 +1046,19 @@ impl FrameServer { /// [`SCRUB_KEEP_S`] before it as not wanted: a decoder seeking from a keyframe skips their /// non-reference pictures, while nearby frames stay decoded (and cached) for scrubbing back. pub fn request(&self, key: FrameKey, time: Tick, scale: f32, project: &Arc, prio: u32) { + self.request_while(key, time, scale, project, prio, false); + } + + /// [`Self::request`] from a view that also draws while playback runs (scopes, the Reference + /// monitor, thumbnails): with `playing` its frame is not a still someone waits for, so a + /// source does not move it to its still-frame path (`filmcraft_media::cancel::with_still`). + pub fn request_while(&self, key: FrameKey, time: Tick, scale: f32, project: &Arc, prio: u32, playing: bool) { let late = (prio == 0).then(|| Tick::from_seconds_f64(SCRUB_KEEP_S)); - self.request_job(key, time, scale, project, prio, false, late); + self.request_job(key, time, scale, project, prio, false, late, !playing); } - fn request_job(&self, key: FrameKey, time: Tick, scale: f32, project: &Arc, prio: u32, prefetch: bool, catch_up: Option) { + #[allow(clippy::too_many_arguments)] + fn request_job(&self, key: FrameKey, time: Tick, scale: f32, project: &Arc, prio: u32, prefetch: bool, catch_up: Option, still: bool) { let hit = || self.shared.stats.lock().unwrap_or_else(|e| e.into_inner()).request_hits += 1; if self.shared.done.lock().unwrap_or_else(|e| e.into_inner()).map.contains_key(&key) || self.shared.plans.lock().unwrap_or_else(|e| e.into_inner()).map.contains_key(&key) @@ -593,6 +1074,8 @@ impl FrameServer { if let Some(j) = q.iter_mut().find(|j| j.key == key) { j.prio = j.prio.min(prio); j.prefetch &= prefetch; + // playback wanting the frame too: decode it for throughput + j.still &= still; // the playhead only moves on: the newest (smallest) margin j.catch_up = match (j.catch_up, catch_up) { (Some(a), Some(b)) => Some(a.min(b)), @@ -620,7 +1103,7 @@ impl FrameServer { } } } - q.push_back(Job { key, queued: Instant::now(), cancel: Default::default(), prefetch, catch_up, time, scale, project: project.clone(), prio }); + q.push_back(Job { key, queued: Instant::now(), cancel: Default::default(), prefetch, catch_up, time, scale, project: project.clone(), prio, still }); drop(q); self.shared.cv.notify_one(); } @@ -636,25 +1119,84 @@ impl FrameServer { let dir = if speed < 0.0 { -1 } else { 1 }; let step = speed.abs().max(1.0) as i64; let fps = rate.as_f64() * speed.abs().max(1.0) / step as f64; - let (lead, stride) = if preroll { (1, 1) } else { playback_plan(self.render_cost(), fps, self.shared.workers) }; + let model = self.source_rate(); + // A throughput-bound source (camera RAW) sets the plan when it is the slower constraint. + let (lead, keep, of, anchored) = match (preroll, model) { + (true, _) => (1, 1, 1, false), + (false, None) => { + let (l, s) = playback_plan(self.render_cost(), fps, self.shared.workers); + (l, 1, s, false) + } + (false, Some(m)) => { + let (l, s) = playback_plan(self.render_cost(), fps, self.shared.workers); + // forward play: how late the picture is tells the plan what the reading cannot + let (in_use, judging) = { + let mut t = self.shared.throughput.lock().unwrap_or_else(|e| e.into_inner()); + (t.pattern, t.judging(true, Instant::now())) + }; + let lag = (dir > 0 && judging).then(|| self.lag_of(key, in_use)); + let (lt, keep, of) = self.throughput_plan(m, fps, lag); + // the sparser of the two + let (keep, of) = if keep * s < of { (keep, of) } else { (1, s) }; + (l.max(lt), keep, of, true) + } + }; if lead <= 1 { // the frame due now; once the clock runs, the frames before it are late - self.request_job(key, rate.tick_of(key.frame), scale, project, 0, false, (!preroll && dir > 0).then_some(Tick::ZERO)); + self.request_job(key, rate.tick_of(key.frame), scale, project, 0, false, (!preroll && dir > 0).then_some(Tick::ZERO), false); } // During preroll the clock has not started: only the frames that gate start, not a // prefetch run that walks the rest of a LongGOP and evicts the playhead pictures. - let ahead = if preroll { (PREROLL_FRAMES - 1).max(0) } else { prefetch_depth(speed) }; - for i in 0..ahead { - let f = key.frame + (lead + i * stride) * dir * step; + // A throughput-bound source whose reading has gone stale (a sparse plan keeps too few + // requests in flight for it to measure itself; a cold start read 68 fps for a 130 fps + // pipeline and the plan stayed at every second frame) is asked for a few more frames of + // the pattern at once, enough to fill its decoder's slots and measure again: frames the + // plan would ask for a moment later anyway, nothing beyond the pattern. + let burst = if !preroll && model.is_some() && self.shared.throughput.lock().unwrap_or_else(|e| e.into_inner()).reading_stale(Instant::now()) { + REMEASURE_BURST + } else { + 0 + }; + let depth = lookahead(self.plan_capacity(key.target), prefetch_depth(speed) + burst); + let ahead = if preroll { (PREROLL_FRAMES - 1).max(0) } else { depth }; + let mut prio = 1; + for f in planned_frames(key.frame, dir, step, lead, keep, of, ahead, anchored) { if f >= 0 { // forward play: frames before the playhead are late (reverse play skips nothing) let late = (!preroll && dir > 0).then(|| rate.tick_of(f) - rate.tick_of(key.frame)); - self.request_job(FrameKey { frame: f, ..key }, rate.tick_of(f), scale, project, i as u32 + 1, true, late); + self.request_job(FrameKey { frame: f, ..key }, rate.tick_of(f), scale, project, prio, true, late, false); } + prio += 1; + } + // Queued frames beyond the planned run are stale: an ordinary source may keep a few + // (its decoder reaches them in order anyway), a throughput-bound one only a pattern's + // worth (its decoder would spend the look-ahead on them and evict the frames due next). + let mut span = (lead + ahead * of) * step + if anchored { of * step } else { 26 }; + if preroll && let Some(m) = model { + // A throughput-bound source's planned run as well, from right after the preroll + // frames: the clock has not started, so every frame decoded now is ahead of it, + // and the clock must not start into frames nobody asked for. + self.shared.throughput.lock().unwrap_or_else(|e| e.into_inner()).judging(false, Instant::now()); + let (lt, keep, of) = self.throughput_plan(m, fps, None); + let (lt, n) = (lt.min(PREROLL_FRAMES), depth); + for f in planned_frames(key.frame, dir, step, lt, keep, of, n, true) { + if f >= 0 { + self.request_job(FrameKey { frame: f, ..key }, rate.tick_of(f), scale, project, prio, true, None, false); + } + prio += 1; + } + span = span.max((lt + n * of) * step + of * step); } let (target, rev, cur) = (key.target, key.revision, key.frame); - let span = (lead + ahead * stride) * step + 26; - self.retain_queue(|k| k.target != target || (k.revision == rev && (k.frame - cur) * dir >= 0 && (k.frame - cur).abs() < span)); + // Queued frames behind the playhead are late; a running one on a throughput-bound + // source is left to finish while it is within the grace (its frame is still shown). + let grace = if model.is_some() { LATE_GRACE } else { 0 }; + let wanted = |k: &FrameKey, behind: i64| k.target != target || (k.revision == rev && (k.frame - cur) * dir >= -behind && (k.frame - cur).abs() < span); + // An anchored plan whose pattern just got sparser leaves the frames it no longer keeps + // queued beyond the lead: dropped, or they would be decoded at the expense of the frames + // the pattern does keep (a failed step up left both patterns' frames in the queue). + let planned = |k: &FrameKey| !anchored || k.target != target || (k.frame - cur) * dir < lead * step || in_pattern(k.frame, keep, of); + self.retain_playback(|k| wanted(k, 0) && planned(k), |k| wanted(k, grace)); } /// Whether playback can start at `key`: the next [`PREROLL_FRAMES`] frames (up to `last`) @@ -669,13 +1211,89 @@ impl FrameServer { *self.shared.render_cost.lock().unwrap_or_else(|e| e.into_inner()) } + /// How many frames of `target` its cache holds (see [`lookahead`]); only GPU plans are + /// large enough to matter, the image cache's entries are 8-bit at the monitor's size. + fn plan_capacity(&self, target: Target) -> usize { + match target { + Target::SequencePlan(_) => self.shared.plans.lock().unwrap_or_else(|e| e.into_inner()).capacity(), + _ => usize::MAX, + } + } + + /// The plan cache's look-ahead for playback and the bytes of the plan inserted last, for + /// `perf.stats`. + pub fn plan_lookahead(&self) -> (i64, usize) { + let p = self.shared.plans.lock().unwrap_or_else(|e| e.into_inner()); + (lookahead(p.capacity(), prefetch_depth(1.0)), p.last_bytes) + } + + /// What playback learned about the throughput-bound sources it is playing, while it is + /// playing one (see [`throughput_plan`]); `None` for ordinary sources. + pub fn source_rate(&self) -> Option { + let t = self.shared.throughput.lock().unwrap_or_else(|e| e.into_inner()); + t.active.then_some(t.model) + } + + /// [`throughput_plan`] for the model, remembering the pattern for the next refresh. + fn throughput_plan(&self, model: SourceRate, fps: f64, lag: Option) -> (i64, i64, i64) { + let mut t = self.shared.throughput.lock().unwrap_or_else(|e| e.into_inner()); + let (lead, keep, of) = throughput_plan(model, fps, self.shared.workers, t.pattern); + let (keep, of) = t.settle((keep, of), lag, Instant::now()); + // `FILMCRAFT_PLAYBACK_PATTERN=keep/of` pins the pattern (a bake-off knob: what a pipeline + // does under a given demand, whatever it reads its capacity as) + let (keep, of) = forced_pattern().unwrap_or((keep, of)); + t.pattern = Some((keep, of)); + (lead, keep, of) + } + + /// How many frames the newest ready picture of the pattern in use is behind `key` (0: `key` + /// itself is ready), up to `LATE_GRACE × 4` when nothing that recent is, for + /// [`Throughput::settle`]. Only the pattern's own frames count: after a step up, the frames + /// decoded for the sparser pattern before it keep the picture looking on time for as long + /// as they were prefetched (a second), while the denser pattern's frames fall behind unseen. + fn lag_of(&self, key: FrameKey, pattern: Option<(i64, i64)>) -> i64 { + let max_back = LATE_GRACE * 4; + let (keep, of) = pattern.unwrap_or((1, 1)); + let counts = |d: i64| in_pattern(key.frame - d, keep, of); + match key.target { + Target::SequencePlan(_) => { + let plans = self.shared.plans.lock().unwrap_or_else(|e| e.into_inner()); + (0..=max_back).find(|d| counts(*d) && plans.map.contains_key(&FrameKey { frame: key.frame - d, ..key })).unwrap_or(max_back) + } + _ => (0..=max_back).find(|d| counts(*d) && self.get(&FrameKey { frame: key.frame - d, ..key }).is_some()).unwrap_or(max_back), + } + } + + /// The plan playback at `fps` would use now: `(lead, keep, of)` (see [`playback_plan`] and + /// [`throughput_plan`]), for `perf.stats`; it does not change the pattern in use. + pub fn plan_for(&self, fps: f64) -> (i64, i64, i64) { + let (l, s) = playback_plan(self.render_cost(), fps, self.shared.workers); + match self.source_rate() { + Some(m) => { + // the pattern in use (settled on the picture, see [`Throughput::settle`]), not + // only what the reading would choose + let previous = self.shared.throughput.lock().unwrap_or_else(|e| e.into_inner()).pattern; + let (lt, keep, of) = throughput_plan(m, fps, self.shared.workers, previous); + let (keep, of) = previous.unwrap_or((keep, of)); + let (keep, of) = if keep * s < of { (keep, of) } else { (1, s) }; + (l.max(lt), keep, of) + } + None => (l, 1, s), + } + } + /// Drop queued jobs that fail `keep` (e.g. stale revisions or frames far from the playhead). /// Jobs already running for such frames are cancelled: a worker blocked behind a decoder /// would otherwise go on to decode a frame playback has passed, possibly from its keyframe. pub fn retain_queue(&self, keep: impl Fn(&FrameKey) -> bool) { - self.shared.queue.lock().unwrap_or_else(|e| e.into_inner()).retain(|j| keep(&j.key)); + self.retain_playback(&keep, &keep); + } + + /// [`retain_queue`](Self::retain_queue) with its own rule for running jobs. + fn retain_playback(&self, keep_queued: impl Fn(&FrameKey) -> bool, keep_running: impl Fn(&FrameKey) -> bool) { + self.shared.queue.lock().unwrap_or_else(|e| e.into_inner()).retain(|j| keep_queued(&j.key)); for (k, c, _) in self.shared.in_flight.lock().unwrap_or_else(|e| e.into_inner()).iter() { - if !keep(k) { + if !keep_running(k) { c.store(true, Ordering::Relaxed); } } @@ -685,6 +1303,7 @@ impl FrameServer { /// they would keep the workers busy, e.g. rendering effects for frames nobody will see). pub fn stop_prefetch(&self) { self.shared.queue.lock().unwrap_or_else(|e| e.into_inner()).retain(|j| !j.prefetch); + self.shared.plans.lock().unwrap_or_else(|e| e.into_inner()).stale_prefetch(); for (_, c, prefetch) in self.shared.in_flight.lock().unwrap_or_else(|e| e.into_inner()).iter() { if *prefetch { c.store(true, Ordering::Relaxed); @@ -900,33 +1519,42 @@ fn run_job(sh: &Shared, job: &Job, pool: &Arc, services: &Arc, services: &Arc *c { *c = if *c == 0.0 { cost } else { *c * 0.8 + cost * 0.2 }; } + drop(c); + let took = (!cancelled).then(|| started.elapsed().as_secs_f64()); + sh.throughput.lock().unwrap_or_else(|e| e.into_inner()).job(DELIVERY.with(Cell::take), DECODED.with(Cell::get), took); } { let job_wall = started.elapsed(); @@ -958,7 +1595,7 @@ fn run_job(sh: &Shared, job: &Job, pool: &Arc, services: &Arc= FrameStats::RECENT { st.recent.pop_front(); } - st.recent.push_back((src_ms as f32, render_ms as f32)); + st.recent.push_back((src_ms as f32, render_ms as f32, stages.0.as_secs_f32() * 1e3, stages.1.as_secs_f32() * 1e3)); } if profiling { let (source_wall, source_cpu) = SOURCE_TIME.with(|s| s.get()); @@ -1059,7 +1696,11 @@ fn render_job(job: &Job, pool: &Arc, services: &Arc, pr #[cfg(test)] mod tests { - use super::{PlaybackMeter, Target, background_work, playback_plan}; + use super::{ + PlaybackMeter, SPARSEST, SourceRate, Target, Throughput, background_work, choose_pattern, eviction_key, in_pattern, lookahead, next_sparser, + planned_frames, playback_plan, stopped, throughput_plan, + }; + use filmcraft_media::FrameDelivery; use filmcraft_project::ItemId; #[test] @@ -1116,6 +1757,273 @@ mod tests { assert_eq!(playback_plan(0.5, 24.0, 6), (12, 3)); } + #[test] + fn throughput_plan_asks_for_what_the_source_delivers_early_enough() { + let plan = |rate: Option, latency, fps| throughput_plan(SourceRate { rate, latency, latency_slow: latency }, fps, 6, None); + // a 25 fps camera RAW decoder under 59.94 fps, 240 ms from a job's start to its frame: + // two of every five frames (24 fps, evenly), started 19 frames ahead (240 ms × 1.25 + 1) + assert_eq!(plan(Some(25.0), 0.24, 59.94), (19, 2, 5)); + // keeping up: every frame, still started early enough + assert_eq!(plan(Some(30.0), 0.1, 29.97), (5, 1, 1)); + // short of the rate: three of four frames (18 fps), not every second one + assert_eq!(plan(Some(23.0), 0.1, 24.0), (5, 3, 4)); + assert_eq!(plan(Some(31.0), 0.1, 60.0), (9, 1, 2)); + assert_eq!(plan(Some(20.5), 0.1, 60.0), (9, 1, 3)); + // six workers each holding a job for 330 ms sustain 18 fps whatever the decoder says + assert_eq!(plan(Some(25.0), 0.33, 59.94), (26, 1, 4)); + assert_eq!(throughput_plan(SourceRate { rate: Some(25.0), latency: 0.33, latency_slow: 0.33 }, 59.94, 8, None), (26, 2, 5)); + // the workers' bound follows the slow average: a spike in the fast one sizes the lead only + assert_eq!(throughput_plan(SourceRate { rate: Some(150.0), latency: 0.06, latency_slow: 0.04 }, 119.88, 6, Some((1, 1))), (10, 1, 1)); + // nothing measured yet: every frame from the playhead on + assert_eq!(plan(None, 0.0, 59.94), (1, 1, 1)); + // no rate from the source yet, but the workers' bound already + assert_eq!(plan(None, 0.24, 59.94).2, 5); + // hostile values stay bounded + assert_eq!(plan(Some(0.001), 1e9, 60.0), (600, 1, 64)); + assert_eq!(plan(Some(-1.0), f64::NAN, 60.0), (1, 1, 1)); + } + + #[test] + fn pattern_choice_has_hysteresis() { + // the densest pattern the share allows + assert_eq!(choose_pattern(1.2, None), (1, 1)); + assert_eq!(choose_pattern(0.8, None), (3, 4)); + assert_eq!(choose_pattern(0.7, None), (2, 3)); + assert_eq!(choose_pattern(0.45, None), (2, 5), "a 27 fps pipeline under 59.94p shows 24 fps"); + assert_eq!(choose_pattern(0.38, None), (3, 8)); + assert_eq!(choose_pattern(0.34, None), (1, 3)); + assert_eq!(choose_pattern(0.1, None), (1, 10)); + assert_eq!(choose_pattern(1e-9, None), (1, 64)); + // falling behind by more than 10 %: sparser; denser only with 5 % to spare + assert_eq!(choose_pattern(0.29, Some((1, 3))), (1, 4)); + assert_eq!(choose_pattern(0.31, Some((1, 3))), (1, 3)); + assert_eq!(choose_pattern(0.93, Some((1, 1))), (1, 1), "keeping up reads as ~1.0"); + assert_eq!(choose_pattern(0.34, Some((1, 4))), (1, 4)); + assert_eq!(choose_pattern(0.36, Some((1, 4))), (1, 3)); + assert_eq!(choose_pattern(1.0, Some((3, 4))), (3, 4)); + assert_eq!(choose_pattern(1.04, Some((3, 4))), (3, 4)); + assert_eq!(choose_pattern(1.06, Some((3, 4))), (1, 1)); + assert_eq!(choose_pattern(0.72, Some((3, 4))), (3, 4)); + assert_eq!(choose_pattern(0.67, Some((3, 4))), (2, 3)); + assert_eq!(choose_pattern(1.25, Some((3, 4))), (1, 1)); + assert_eq!(choose_pattern(f64::NAN, Some((1, 3))), (1, 3)); + // the next sparser pattern, then every n-th, bounded + assert_eq!(next_sparser((1, 1)), (3, 4)); + assert_eq!(next_sparser((1, 2)), (2, 5)); + assert_eq!(next_sparser((3, 8)), (1, 3)); + assert_eq!(next_sparser((1, 4)), (1, 5)); + assert_eq!(next_sparser((1, 9)), (1, 10)); + assert_eq!(next_sparser((1, SPARSEST)), (1, SPARSEST)); + // the pattern spreads its frames evenly and the same way for any playhead + let kept = |keep, of| (0..12).filter(|f| in_pattern(*f, keep, of)).collect::>(); + assert_eq!(kept(1, 3), vec![0, 3, 6, 9]); + assert_eq!(kept(3, 4), vec![0, 2, 3, 4, 6, 7, 8, 10, 11]); + assert_eq!(kept(2, 3), vec![0, 2, 3, 5, 6, 8, 9, 11]); + assert_eq!(kept(2, 5), vec![0, 3, 5, 8, 10], "gaps of three and two frames, alternating"); + assert_eq!(kept(1, 1).len(), 12); + assert!(in_pattern(i64::MAX, 3, 4) || !in_pattern(i64::MAX, 3, 4), "no overflow"); + } + + /// Refreshes every 50 ms from `from` to `to` (ms), each with `lag`; the pattern after the + /// last. The pattern in use follows what `settle` returns, as the frame server's does. + fn drive(t: &mut Throughput, chosen: (i64, i64), lag: i64, t0: super::Instant, from: u64, to: u64) -> (i64, i64) { + let mut out = t.pattern.unwrap_or(chosen); + let mut ms = from; + while ms <= to { + out = t.settle(chosen, Some(lag), t0 + std::time::Duration::from_millis(ms)); + t.pattern = Some(out); + ms += 50; + } + out + } + + #[test] + fn pattern_backs_off_when_the_picture_runs_late_and_climbs_back_after_slack() { + let t0 = super::Instant::now(); + let mut t = Throughput::default(); + // the first choice stands; nothing shown yet keeps what is in use unless sparser + assert_eq!(t.settle((1, 1), None, t0), (1, 1)); + t.pattern = Some((1, 1)); + assert_eq!(t.settle((1, 1), None, t0), (1, 1)); + assert_eq!(t.settle((3, 4), None, t0), (3, 4)); + t.pattern = Some((1, 1)); + // every frame, the reading still says so, but the picture is two frames late: once a + // half-second window of that closes, three of four + assert_eq!(drive(&mut t, (1, 1), 2, t0, 0, 450), (1, 1)); + assert_eq!(drive(&mut t, (1, 1), 2, t0, 500, 500), (3, 4)); + // on time again: no denser within the cooldown (4 s from 500 ms) + assert_eq!(drive(&mut t, (1, 1), 0, t0, 550, 4400), (3, 4)); + assert_eq!(drive(&mut t, (1, 1), 0, t0, 4450, 4600), (1, 1)); + // the step up fails within the hold: back to three of four, and the next try waits twice + // as long (8 s), then 16 s + assert_eq!(drive(&mut t, (1, 1), 2, t0, 4650, 5200), (3, 4)); + assert_eq!(drive(&mut t, (1, 1), 0, t0, 5250, 12950), (3, 4), "8 s cooldown from 5 s"); + assert_eq!(drive(&mut t, (1, 1), 0, t0, 13000, 13300), (1, 1)); + assert_eq!(drive(&mut t, (1, 1), 2, t0, 13350, 14000), (3, 4), "late in the window closing at 14 s"); + assert_eq!(drive(&mut t, (1, 1), 0, t0, 14050, 29950), (3, 4), "16 s cooldown from 14 s"); + assert_eq!(drive(&mut t, (1, 1), 0, t0, 30000, 30200), (1, 1)); + // held for three seconds: a later late episode is an ordinary back-off (base cooldown) + assert_eq!(drive(&mut t, (1, 1), 0, t0, 30250, 34000), (1, 1)); + assert_eq!(drive(&mut t, (1, 1), 2, t0, 34050, 34500), (3, 4)); + assert_eq!(drive(&mut t, (1, 1), 0, t0, 34550, 38450), (3, 4)); + assert_eq!(drive(&mut t, (1, 1), 0, t0, 38500, 38750), (1, 1), "4 s cooldown from 34.5 s"); + // a lag that flickers (most frames land, one in three is late) still counts as late + let mut t = Throughput { pattern: Some((1, 2)), ..Default::default() }; + let mut ms = 0; + let mut out = (1, 2); + while ms <= 600 { + out = t.settle((1, 2), Some(if ms % 150 == 0 { 4 } else { 0 }), t0 + std::time::Duration::from_millis(ms)); + t.pattern = Some(out); + ms += 50; + } + assert_eq!(out, (2, 5), "the next sparser pattern after every second frame"); + // sparser from the reading is always allowed, and a lag within the pattern's step is fine + let mut t = Throughput { pattern: Some((1, 3)), ..Default::default() }; + assert_eq!(t.settle((1, 4), Some(0), t0), (1, 4)); + t.pattern = Some((1, 3)); + assert_eq!(drive(&mut t, (1, 3), 3, t0, 0, 1200), (1, 3)); + assert_eq!(drive(&mut t, (1, 3), 5, t0, 1250, 1800), (1, 4)); + // right after backing off (at 1.5 s) the picture is still late: not judged for two seconds + assert_eq!(drive(&mut t, (1, 4), 9, t0, 1850, 3450), (1, 4)); + assert_eq!(drive(&mut t, (1, 4), 9, t0, 3500, 5450), (1, 5)); + assert_eq!(drive(&mut t, (1, 5), 9, t0, 5500, 5600), (1, 6)); + // the clock: not judged for two seconds after it starts, and a preroll resets that + let mut t = Throughput::default(); + assert!(!t.judging(false, t0)); + assert!(!t.judging(true, t0)); + assert!(!t.judging(true, t0 + std::time::Duration::from_millis(1900))); + assert!(t.judging(true, t0 + std::time::Duration::from_millis(2000))); + assert!(!t.judging(false, t0 + std::time::Duration::from_millis(2100))); + assert!(!t.judging(true, t0 + std::time::Duration::from_millis(2200))); + } + + #[test] + fn a_stale_reading_asks_for_a_burst_and_never_a_denser_pattern() { + use std::time::Duration; + let t0 = super::Instant::now(); + let mut t = Throughput::default(); + // nothing reported: nothing to refresh + assert!(!t.reading_stale(t0)); + t.job(Some(FrameDelivery { independent: true, rate: Some(25.0), fresh: true }), true, Some(0.2)); + let now = super::Instant::now(); + assert!(!t.reading_stale(now)); + assert!(t.reading_stale(now + Duration::from_millis(2100))); + // a source that reports no rate of its own is measured by playback: never stale + let mut u = Throughput::default(); + u.job(Some(FrameDelivery { independent: true, rate: None, fresh: false }), true, Some(0.05)); + assert!(!u.reading_stale(super::Instant::now() + Duration::from_secs(60))); + // the pattern in use is never stepped up beyond the reading's choice, however long on time + let mut t = Throughput { pattern: Some((1, 2)), ..Default::default() }; + assert_eq!(drive(&mut t, (1, 2), 0, t0, 0, 12000), (1, 2)); + t.pattern = Some((1, 1)); + assert_eq!(drive(&mut t, (1, 1), 0, t0, 12050, 16000), (1, 1), "nothing denser than every frame"); + } + + #[test] + fn anchored_plan_keeps_the_same_frames_as_the_playhead_moves() { + let frames = |cur, dir, anchored| planned_frames(cur, dir, 1, 19, 1, 3, 4, anchored); + // multiples of the stride: a refresh one frame later asks for the same frames + assert_eq!(frames(100, 1, true), vec![120, 123, 126, 129]); + assert_eq!(frames(101, 1, true), vec![120, 123, 126, 129]); + assert_eq!(frames(102, 1, true), vec![123, 126, 129, 132]); + // the plain plan shifts with every refresh (every frame gets asked for eventually) + assert_eq!(frames(100, 1, false), vec![119, 122, 125, 128]); + assert_eq!(frames(101, 1, false), vec![120, 123, 126, 129]); + // reverse play anchors the same way + assert_eq!(frames(100, -1, true), vec![81, 78, 75, 72]); + assert_eq!(frames(99, -1, true), vec![78, 75, 72, 69]); + // 2× speed: the stride is in played frames + assert_eq!(planned_frames(100, 1, 2, 19, 1, 3, 3, true), vec![138, 144, 150]); + assert_eq!(planned_frames(0, 1, 1, 1, 1, 1, 3, true), vec![1, 2, 3]); + assert!(planned_frames(i64::MAX, 1, 1, 1, 1, 2, 2, true).len() <= 2, "no overflow, and the walk ends"); + // three of four: the same frames whatever the playhead, one of every four left out + assert_eq!(planned_frames(100, 1, 1, 19, 3, 4, 4, true), vec![119, 120, 122, 123]); + assert_eq!(planned_frames(101, 1, 1, 19, 3, 4, 4, true), vec![120, 122, 123, 124]); + assert_eq!(planned_frames(100, -1, 1, 19, 3, 4, 4, true), vec![80, 79, 78, 76]); + } + + #[test] + fn throughput_model_follows_the_source_and_clears_for_ordinary_ones() { + let raw = Some(FrameDelivery { independent: true, rate: Some(25.0), fresh: true }); + let mut t = Throughput::default(); + assert!(!t.active); + t.job(raw, true, Some(0.2)); + assert!(t.active); + assert_eq!(t.model, SourceRate { rate: Some(25.0), latency: 0.2, latency_slow: 0.2 }); + // latency rises fast and settles slowly; a cancelled job (no time) still keeps it active + t.job(raw, true, Some(0.4)); + assert!((t.model.latency - 0.3).abs() < 1e-9); + assert!((t.model.latency_slow - 0.21).abs() < 1e-9); + t.job(raw, true, None); + assert!(t.active); + // a source without a rate of its own: measured from completions once there are enough + let mut u = Throughput::default(); + for _ in 0..5 { + u.job(Some(FrameDelivery { independent: true, rate: None, fresh: false }), true, Some(0.05)); + } + assert_eq!(u.model.rate, None); + u.job(Some(FrameDelivery { independent: true, rate: None, fresh: false }), true, Some(0.05)); + assert!(u.model.rate.is_some_and(|r| r > 100.0), "{:?}", u.model); + // frames that decode nothing (past the end of the sequence while it loops, a gap, a + // rendered segment) say nothing about what is playing: the model stays + for _ in 0..(4 * Throughput::MISSES) { + t.job(None, false, Some(0.001)); + } + assert!(t.active); + // playback moves on to a sequence of ordinary sources: a dozen jobs clear the model + for i in 0..Throughput::MISSES { + assert!(t.active, "{i}"); + t.job(None, true, Some(0.01)); + } + assert!(!t.active); + assert_eq!(t.model, SourceRate::default()); + // a GOP source's delivery is not throughput-bound either + t.job(Some(FrameDelivery::default()), true, Some(0.01)); + assert!(!t.active); + } + + #[test] + fn lookahead_fits_the_plan_cache() { + // 1080p plans by the hundred: the prefetch depth as it is + assert_eq!(lookahead(128, 14), 14); + assert_eq!(lookahead(usize::MAX, 24), 24); + // eight 6K half-resolution plans: six ahead, the frame on screen and one behind + assert_eq!(lookahead(8, 14), 6); + // never less than two, whatever the plan costs + assert_eq!(lookahead(1, 14), 2); + assert_eq!(lookahead(0, 14), 2); + } + + #[test] + fn prefetched_frames_outlive_the_frames_already_shown() { + // (name, last looked up, inserted): the playhead's frame is looked up every refresh, + // the frames behind it were, the frames ahead never + let mut entries = [("ahead-near", 5, 5), ("ahead-far", 8, 8), ("shown-now", 9, 4), ("shown-before", 7, 3), ("shown-long-ago", 2, 1)]; + entries.sort_by_key(|e| eviction_key(e.1, e.2, 10)); + let order: Vec<&str> = entries.iter().map(|e| e.0).collect(); + assert_eq!(order, ["shown-long-ago", "shown-before", "shown-now", "ahead-near", "ahead-far"]); + // long after: a frame nobody looked up (skipped, or of an old revision) is just old + let mut entries = [("never-looked-up", 5, 5), ("shown-now", 3000, 9)]; + entries.sort_by_key(|e| eviction_key(e.1, e.2, 3000)); + assert_eq!(entries.map(|e| e.0), ["never-looked-up", "shown-now"]); + } + + /// Playback prefetched frames ahead and stopped; a scrub then shows new frames. Over budget, + /// the stale prefetched plans go first, not the frames the scrub just showed. + #[test] + fn prefetched_plans_go_first_once_playback_stopped() { + // (last looked up, inserted) at clock 100: two prefetched, two shown by the scrub + let mut v = [(95, 95), (98, 98), (99, 96), (100, 97)]; + for e in &mut v { + e.1 = stopped(e.0, e.1); + } + let mut order: Vec = (0..v.len()).collect(); + order.sort_by_key(|&i| eviction_key(v[i].0, v[i].1, 100)); + assert_eq!(order, [0, 1, 2, 3], "the prefetched (0, 1) before the shown (2, 3)"); + assert_eq!(stopped(99, 96), 96, "a plan looked up keeps its place"); + assert_eq!(stopped(3, 3), 0, "no underflow"); + } + #[test] fn meter_resync_does_not_count_hidden_time() { let mut m = PlaybackMeter::default(); @@ -1203,6 +2111,7 @@ mod tests { scale: 0.1, project: Arc::new(filmcraft_project::Project::new("Frames")), prio: 0, + still: false, }; // what `Drop` sets before it cancels the jobs in flight server.shared.stop.store(true, Ordering::Relaxed); diff --git a/crates/ui-egui/src/i18n/de.tsv b/crates/ui-egui/src/i18n/de.tsv index b97843aeaf..59c8fb12b2 100644 --- a/crates/ui-egui/src/i18n/de.tsv +++ b/crates/ui-egui/src/i18n/de.tsv @@ -3614,3 +3614,4 @@ Shows the version, community links and credits in a FilmCraft window. Zeigt Version, Community-Links und Credits in einem FilmCraft-Fenster. Couldn't open your web browser Dein Webbrowser konnte nicht geöffnet werden Open this address in your browser yourself: Öffne diese Adresse selbst in deinem Browser: + Camera RAW… Kamera-RAW… diff --git a/crates/ui-egui/src/i18n/es.tsv b/crates/ui-egui/src/i18n/es.tsv index 14eddb4f2f..9104096052 100644 --- a/crates/ui-egui/src/i18n/es.tsv +++ b/crates/ui-egui/src/i18n/es.tsv @@ -3626,5 +3626,77 @@ Open a sequence to search its graphics Abre una secuencia para buscar en sus gráficos No text graphics in this sequence. Add text with the Type tool. Esta secuencia no tiene gráficos con texto. Añade texto con la herramienta Texto. No matching text. Ningún texto coincide. + As Shot Como se grabó + As Shot (ISO {v}) Como se grabó (ISO {v}) + as shot como se grabó + as shot (decoder not open) como se grabó (descodificador no abierto) + ISO ISO + Applies to a display gamma curve (BT.1886, sRGB, ST 2084, Hybrid Log-Gamma). Se aplica a una curva gamma de pantalla (BT.1886, sRGB, ST 2084, Hybrid Log-Gamma). + Applies to the ST 2084 (PQ) gamma curve only. Solo se aplica a la curva gamma ST 2084 (PQ). + Apply CDL Aplicar CDL + Apply Creative LUT Aplicar LUT creativa + Apply Metadata Curves Aplicar curvas de metadatos + Apply this clip's settings to every selected clip Aplicar los ajustes de este clip a todos los clips seleccionados + As recorded Como se grabó + BT.1886 BT.1886 + Blend Bias Sesgo de mezcla + Blend Type Tipo de mezcla + Camera RAW RAW de cámara + Camera RAW clips develop with these settings unless they have settings of their own (Clip > Source Settings > Decode Using > Clip). Los clips RAW de cámara se revelan con estos ajustes salvo que tengan ajustes propios (Clip > Ajustes de origen > Decodificar con > Clip). + Camera RAW… RAW de cámara… + Camera Raw RAW de cámara + Chroma Noise Reduction Reducción de ruido de croma + Color Science Ciencia del color + Color Temp Temp. de color + Decode Quality Calidad de decodificación + Decode Using Decodificar con + Display curves go with Rec. 709, Rec. 2020 and P3 D65, Log3G10 with REDWideGamutRGB, ST 2084 and Hybrid Log-Gamma with Rec. 2020; linear with any. Las curvas de pantalla van con Rec. 709, Rec. 2020 y P3 D65; Log3G10, con REDWideGamutRGB; ST 2084 e Hybrid Log-Gamma, con Rec. 2020; la lineal, con cualquiera. + Eighth Res. Good 1/8 de res. buena + Exposure Adjust Ajuste de exposición + Flashing Pixel Adjust Ajuste de píxeles parpadeantes + Full Res. Premium Res. completa premium + Full and half resolution premium decodes. Decodificaciones premium a resolución completa y media. + Gamma Curve Curva gamma + HDR Peak Nits Pico HDR (nits) + HDRx blending is not available in FilmCraft yet. La mezcla HDRx aún no está disponible en FilmCraft. + Half Res. Good Media res. buena + Half Res. Premium Media res. premium + Highlight Roll Off Atenuación de altas luces + Hybrid Log-Gamma Hybrid Log-Gamma + IPP2 IPP2 + Log3G10 Log3G10 + Mild Leve + Nikon R3D NE is developed with RED IPP2 only. Nikon R3D NE solo se revela con RED IPP2. + No RMD sidecar: the clip's recorded settings are the camera's. Sin archivo RMD: los ajustes grabados del clip son los de la cámara. + Output Tone Map Mapa de tonos de salida + P3 D65 P3 D65 + Put this clip back to the settings it records (or its RMD sidecar says) Devolver este clip a los ajustes que tiene grabados (o que indica su archivo RMD) + Quarter Res. Good 1/4 de res. buena + RED IPP2 development of the RAW picture. ISO, colour temperature, tint and exposure are applied by the SDK before development (as shot unless set); the output tone map and highlight roll-off shape a display gamma curve. Proxies keep the look they were made with. Revelado RED IPP2 de la imagen RAW. El SDK aplica ISO, temperatura de color, matiz y exposición antes del revelado (como se grabó, salvo que se ajusten); el mapa de tonos de salida y la atenuación de altas luces dan forma a una curva gamma de pantalla. Los proxies conservan el aspecto con el que se crearon. + REDWideGamutRGB REDWideGamutRGB + RMD sidecar in effect: {path} Archivo RMD en uso: {path} + RMD written: {paths} RMD escrito: {paths} + Rec. 2020 Rec. 2020 + ST 2084 (PQ) ST 2084 (PQ) + Sixteenth Res. Good 1/16 de res. buena + Standard (Rec. 709, BT.1886) Estándar (Rec. 709, BT.1886) + Strong Fuerte + The 3D LUT the clip (or its RMD) carries. La LUT 3D que lleva el clip (o su RMD). + The ASC CDL the clip (or its RMD) records. El ASC CDL que graba el clip (o su RMD). + The largest decode a preview uses; an export and a monitor at 100 % always decode at full resolution, premium. La decodificación más grande que usa una vista previa; una exportación y un monitor al 100 % siempre decodifican a resolución completa, premium. + The linear output carries no look: choose a display or log gamma curve to apply it. La salida lineal no lleva ningún aspecto: elija una curva gamma de pantalla o logarítmica para aplicarlo. + The luma and RGB curves the clip records. Las curvas de luma y RGB que graba el clip. + This clip carries no creative LUT. Este clip no lleva ninguna LUT creativa. + This clip decodes with the project's Camera RAW settings (File > Project Settings > Camera RAW). Choose Decode Using > Clip to give it settings of its own. Este clip se decodifica con los ajustes de RAW de cámara del proyecto (Archivo > Ajustes del proyecto > RAW de cámara). Elija Decodificar con > Clip para darle ajustes propios. + This clip has no HDRx track (Nikon R3D NE records none): there is nothing to blend. Este clip no tiene pista HDRx (Nikon R3D NE no la graba): no hay nada que mezclar. + Update RMD Actualizar RMD + Use Changes Usar cambios + Use Settings Usar ajustes + Use project setting Usar ajuste del proyecto + Very Soft Muy suave + Write this clip's settings into its RMD sidecar beside the clip Escribir los ajustes de este clip en su archivo RMD junto al clip + as recorded como se grabó + sRGB sRGB + {n} camera RAW clips decode using these settings. {n} clips RAW de cámara se decodifican con estos ajustes. H.265 (HEVC) needs a hardware encoder (NVENC on NVIDIA GPUs, VideoToolbox on macOS); none was found on this machine, and FilmCraft has no software H.265 encoder yet. Choose H.264 or ProRes instead. H.265 (HEVC) necesita un codificador por hardware (NVENC en GPU NVIDIA, VideoToolbox en macOS); no se encontró ninguno en este equipo y FilmCraft aún no tiene un codificador H.265 por software. Elija H.264 o ProRes. AV1 needs a hardware encoder; none was found on this machine, and FilmCraft has no software AV1 encoder yet. AV1 necesita un codificador por hardware; no se encontró ninguno en este equipo y FilmCraft aún no tiene un codificador AV1 por software. diff --git a/crates/ui-egui/src/i18n/fr.tsv b/crates/ui-egui/src/i18n/fr.tsv index 8d00deec36..b262c8fa28 100644 --- a/crates/ui-egui/src/i18n/fr.tsv +++ b/crates/ui-egui/src/i18n/fr.tsv @@ -3585,6 +3585,7 @@ Open a sequence to search its graphics Ouvrez une séquence pour rechercher dans ses graphiques No text graphics in this sequence. Add text with the Type tool. Aucun graphique avec du texte dans cette séquence. Ajoutez du texte avec l'outil Texte. No matching text. Aucun texte ne correspond. + Camera RAW… RAW de la caméra… Hide FilmCraft Masquer FilmCraft Hide Others Masquer les autres More transport controls Autres commandes de lecture diff --git a/crates/ui-egui/src/i18n/id.tsv b/crates/ui-egui/src/i18n/id.tsv index 3d383b4f0e..98bcd00e4c 100644 --- a/crates/ui-egui/src/i18n/id.tsv +++ b/crates/ui-egui/src/i18n/id.tsv @@ -3583,3 +3583,4 @@ Open a sequence to search its graphics Buka sekuens untuk mencari di grafisnya No text graphics in this sequence. Add text with the Type tool. Tidak ada grafis teks di sekuens ini. Tambahkan teks dengan alat Teks. No matching text. Tidak ada teks yang cocok. + Camera RAW… RAW Kamera… diff --git a/crates/ui-egui/src/i18n/it.tsv b/crates/ui-egui/src/i18n/it.tsv index 81fb5d6271..bb068b6f31 100644 --- a/crates/ui-egui/src/i18n/it.tsv +++ b/crates/ui-egui/src/i18n/it.tsv @@ -3609,3 +3609,4 @@ Open a sequence to search its graphics Apri una sequenza per cercare nelle sue grafiche No text graphics in this sequence. Add text with the Type tool. Nessuna grafica con testo in questa sequenza. Aggiungi testo con lo strumento Testo. No matching text. Nessun testo corrispondente. + Camera RAW… RAW della fotocamera… diff --git a/crates/ui-egui/src/i18n/ja.tsv b/crates/ui-egui/src/i18n/ja.tsv index e2b1c67021..2944bd0d6d 100644 --- a/crates/ui-egui/src/i18n/ja.tsv +++ b/crates/ui-egui/src/i18n/ja.tsv @@ -3625,5 +3625,77 @@ Open a sequence to search its graphics シーケンスを開くとグラフィックスを検索できます No text graphics in this sequence. Add text with the Type tool. このシーケンスにはテキストのグラフィックスがありません。文字ツールでテキストを追加してください。 No matching text. 一致するテキストはありません。 + As Shot 撮影時 + As Shot (ISO {v}) 撮影時 (ISO {v}) + as shot 撮影時 + as shot (decoder not open) 撮影時（デコーダー未起動） + ISO ISO + Applies to a display gamma curve (BT.1886, sRGB, ST 2084, Hybrid Log-Gamma). 表示用ガンマカーブ（BT.1886、sRGB、ST 2084、Hybrid Log-Gamma）に適用されます。 + Applies to the ST 2084 (PQ) gamma curve only. ST 2084（PQ）ガンマカーブにのみ適用されます。 + Apply CDL CDL を適用 + Apply Creative LUT クリエイティブ LUT を適用 + Apply Metadata Curves メタデータのカーブを適用 + Apply this clip's settings to every selected clip このクリップの設定を選択したすべてのクリップに適用 + As recorded 記録どおり + BT.1886 BT.1886 + Blend Bias ブレンドバイアス + Blend Type ブレンドの種類 + Camera RAW カメラ RAW + Camera RAW clips develop with these settings unless they have settings of their own (Clip > Source Settings > Decode Using > Clip). カメラ RAW クリップは、独自の設定がない限りこの設定で現像されます（クリップ > ソース設定 > デコードに使用 > クリップ）。 + Camera RAW… カメラ RAW… + Camera Raw カメラ RAW + Chroma Noise Reduction 色ノイズ低減 + Color Science カラーサイエンス + Color Temp 色温度 + Decode Quality デコード品質 + Decode Using デコードに使用 + Display curves go with Rec. 709, Rec. 2020 and P3 D65, Log3G10 with REDWideGamutRGB, ST 2084 and Hybrid Log-Gamma with Rec. 2020; linear with any. 表示用カーブは Rec. 709・Rec. 2020・P3 D65、Log3G10 は REDWideGamutRGB、ST 2084 と Hybrid Log-Gamma は Rec. 2020 と組み合わせます。リニアはどれとでも使えます。 + Eighth Res. Good 1/8 解像度・良好 + Exposure Adjust 露出補正 + Flashing Pixel Adjust 点滅ピクセル補正 + Full Res. Premium フル解像度・プレミアム + Full and half resolution premium decodes. フル解像度と 1/2 解像度のプレミアムデコード。 + Gamma Curve ガンマカーブ + HDR Peak Nits HDR ピーク（nit） + HDRx blending is not available in FilmCraft yet. HDRx のブレンドは FilmCraft ではまだ使用できません。 + Half Res. Good 1/2 解像度・良好 + Half Res. Premium 1/2 解像度・プレミアム + Highlight Roll Off ハイライトのロールオフ + Hybrid Log-Gamma Hybrid Log-Gamma + IPP2 IPP2 + Log3G10 Log3G10 + Mild 弱 + Nikon R3D NE is developed with RED IPP2 only. Nikon R3D NE は RED IPP2 でのみ現像されます。 + No RMD sidecar: the clip's recorded settings are the camera's. RMD サイドカーなし：クリップの記録設定はカメラのものです。 + Output Tone Map 出力トーンマップ + P3 D65 P3 D65 + Put this clip back to the settings it records (or its RMD sidecar says) このクリップを記録された設定（または RMD サイドカーの設定）に戻す + Quarter Res. Good 1/4 解像度・良好 + RED IPP2 development of the RAW picture. ISO, colour temperature, tint and exposure are applied by the SDK before development (as shot unless set); the output tone map and highlight roll-off shape a display gamma curve. Proxies keep the look they were made with. RAW 画像の RED IPP2 現像です。ISO・色温度・色合い・露出は現像前に SDK が適用します（設定しない限り撮影時の値）。出力トーンマップとハイライトのロールオフは表示用ガンマカーブを形作ります。プロキシは作成時のルックのままです。 + REDWideGamutRGB REDWideGamutRGB + RMD sidecar in effect: {path} RMD サイドカーを使用中: {path} + RMD written: {paths} RMD を書き込みました: {paths} + Rec. 2020 Rec. 2020 + ST 2084 (PQ) ST 2084（PQ） + Sixteenth Res. Good 1/16 解像度・良好 + Standard (Rec. 709, BT.1886) 標準（Rec. 709、BT.1886） + Strong 強 + The 3D LUT the clip (or its RMD) carries. クリップ（またはその RMD）が持つ 3D LUT。 + The ASC CDL the clip (or its RMD) records. クリップ（またはその RMD）に記録された ASC CDL。 + The largest decode a preview uses; an export and a monitor at 100 % always decode at full resolution, premium. プレビューで使う最大のデコードです。書き出しと 100 % 表示のモニターは常にフル解像度・プレミアムでデコードします。 + The linear output carries no look: choose a display or log gamma curve to apply it. リニア出力にはルックが含まれません。適用するには表示用またはログのガンマカーブを選んでください。 + The luma and RGB curves the clip records. クリップに記録された輝度と RGB のカーブ。 + This clip carries no creative LUT. このクリップにはクリエイティブ LUT がありません。 + This clip decodes with the project's Camera RAW settings (File > Project Settings > Camera RAW). Choose Decode Using > Clip to give it settings of its own. このクリップはプロジェクトのカメラ RAW 設定でデコードされます（ファイル > プロジェクト設定 > カメラ RAW）。独自の設定にするには、デコードに使用 > クリップを選んでください。 + This clip has no HDRx track (Nikon R3D NE records none): there is nothing to blend. このクリップには HDRx トラックがありません（Nikon R3D NE は記録しません）。ブレンドするものはありません。 + Update RMD RMD を更新 + Use Changes 変更を適用 + Use Settings 設定を使用 + Use project setting プロジェクト設定を使用 + Very Soft 非常にソフト + Write this clip's settings into its RMD sidecar beside the clip このクリップの設定をクリップの横の RMD サイドカーに書き込む + as recorded 記録どおり + sRGB sRGB + {n} camera RAW clips decode using these settings. {n} 個のカメラ RAW クリップがこの設定でデコードされます。 H.265 (HEVC) needs a hardware encoder (NVENC on NVIDIA GPUs, VideoToolbox on macOS); none was found on this machine, and FilmCraft has no software H.265 encoder yet. Choose H.264 or ProRes instead. H.265 (HEVC) にはハードウェアエンコーダー（NVIDIA GPU の NVENC、macOS の VideoToolbox）が必要です。このマシンには見つからず、FilmCraft にはまだソフトウェアの H.265 エンコーダーがありません。代わりに H.264 または ProRes を選んでください。 AV1 needs a hardware encoder; none was found on this machine, and FilmCraft has no software AV1 encoder yet. AV1 にはハードウェアエンコーダーが必要です。このマシンには見つからず、FilmCraft にはまだソフトウェアの AV1 エンコーダーがありません。 diff --git a/crates/ui-egui/src/i18n/ko.tsv b/crates/ui-egui/src/i18n/ko.tsv index e795ac9f36..079e0b0889 100644 --- a/crates/ui-egui/src/i18n/ko.tsv +++ b/crates/ui-egui/src/i18n/ko.tsv @@ -3604,3 +3604,4 @@ Open a sequence to search its graphics 그래픽을 검색하려면 시퀀스를 여세요 No text graphics in this sequence. Add text with the Type tool. 이 시퀀스에는 텍스트 그래픽이 없습니다. 문자 도구로 텍스트를 추가하세요. No matching text. 일치하는 텍스트가 없습니다. + Camera RAW… 카메라 RAW… diff --git a/crates/ui-egui/src/i18n/mod.rs b/crates/ui-egui/src/i18n/mod.rs index b84f938195..5c0974767e 100644 --- a/crates/ui-egui/src/i18n/mod.rs +++ b/crates/ui-egui/src/i18n/mod.rs @@ -738,6 +738,7 @@ mod tests { filmcraft_engine::project_panel::BUILTIN_COLUMNS.iter().for_each(|c| push(&mut out, c)); filmcraft_project::find::COLUMNS.iter().for_each(|c| push(&mut out, c)); filmcraft_project::FindOp::ALL.iter().for_each(|o| push(&mut out, o.label())); + filmcraft_engine::camera_raw::panel_texts().iter().for_each(|t| push(&mut out, t)); filmcraft_engine::media_browser::ALL_COLUMNS.iter().for_each(|c| push(&mut out, c)); // export: presets, settings choices, ranges and pixel aspects use filmcraft_engine::export as ex; diff --git a/crates/ui-egui/src/i18n/pt-br.tsv b/crates/ui-egui/src/i18n/pt-br.tsv index ea7642c52b..47e7a392d5 100644 --- a/crates/ui-egui/src/i18n/pt-br.tsv +++ b/crates/ui-egui/src/i18n/pt-br.tsv @@ -3586,3 +3586,4 @@ Open a sequence to search its graphics Abra uma sequência para pesquisar nos gráficos dela No text graphics in this sequence. Add text with the Type tool. Esta sequência não tem gráficos com texto. Adicione texto com a ferramenta Texto. No matching text. Nenhum texto corresponde. + Camera RAW… RAW da câmera… diff --git a/crates/ui-egui/src/i18n/zh-cn.tsv b/crates/ui-egui/src/i18n/zh-cn.tsv index dc3ee65ef1..cad735de4e 100644 --- a/crates/ui-egui/src/i18n/zh-cn.tsv +++ b/crates/ui-egui/src/i18n/zh-cn.tsv @@ -3601,5 +3601,77 @@ Open a sequence to search its graphics 打开序列以搜索其中的图形 No text graphics in this sequence. Add text with the Type tool. 此序列中没有文本图形。请用文字工具添加文本。 No matching text. 没有匹配的文本。 + As Shot 拍摄时 + As Shot (ISO {v}) 拍摄时 (ISO {v}) + as shot 拍摄时 + as shot (decoder not open) 拍摄时（解码器未打开） + ISO ISO + Applies to a display gamma curve (BT.1886, sRGB, ST 2084, Hybrid Log-Gamma). 适用于显示伽马曲线（BT.1886、sRGB、ST 2084、Hybrid Log-Gamma）。 + Applies to the ST 2084 (PQ) gamma curve only. 仅适用于 ST 2084（PQ）伽马曲线。 + Apply CDL 应用 CDL + Apply Creative LUT 应用创意 LUT + Apply Metadata Curves 应用元数据曲线 + Apply this clip's settings to every selected clip 将此剪辑的设置应用于所有选定的剪辑 + As recorded 按记录 + BT.1886 BT.1886 + Blend Bias 混合偏向 + Blend Type 混合类型 + Camera RAW 相机 RAW + Camera RAW clips develop with these settings unless they have settings of their own (Clip > Source Settings > Decode Using > Clip). 相机 RAW 剪辑使用这些设置显影，除非它们有自己的设置（剪辑 > 源设置 > 解码依据 > 剪辑）。 + Camera RAW… 相机 RAW… + Camera Raw 相机 RAW + Chroma Noise Reduction 色度降噪 + Color Science 色彩科学 + Color Temp 色温 + Decode Quality 解码质量 + Decode Using 解码依据 + Display curves go with Rec. 709, Rec. 2020 and P3 D65, Log3G10 with REDWideGamutRGB, ST 2084 and Hybrid Log-Gamma with Rec. 2020; linear with any. 显示曲线搭配 Rec. 709、Rec. 2020 和 P3 D65，Log3G10 搭配 REDWideGamutRGB，ST 2084 和 Hybrid Log-Gamma 搭配 Rec. 2020；线性可搭配任意色彩空间。 + Eighth Res. Good 1/8 分辨率 良好 + Exposure Adjust 曝光调整 + Flashing Pixel Adjust 闪烁像素调整 + Full Res. Premium 全分辨率 高级 + Full and half resolution premium decodes. 全分辨率和 1/2 分辨率的高级解码。 + Gamma Curve 伽马曲线 + HDR Peak Nits HDR 峰值尼特 + HDRx blending is not available in FilmCraft yet. FilmCraft 尚不支持 HDRx 混合。 + Half Res. Good 1/2 分辨率 良好 + Half Res. Premium 1/2 分辨率 高级 + Highlight Roll Off 高光滚降 + Hybrid Log-Gamma Hybrid Log-Gamma + IPP2 IPP2 + Log3G10 Log3G10 + Mild 轻度 + Nikon R3D NE is developed with RED IPP2 only. Nikon R3D NE 仅使用 RED IPP2 显影。 + No RMD sidecar: the clip's recorded settings are the camera's. 无 RMD 附属文件：剪辑的记录设置来自相机。 + Output Tone Map 输出色调映射 + P3 D65 P3 D65 + Put this clip back to the settings it records (or its RMD sidecar says) 将此剪辑恢复为其记录的设置（或其 RMD 附属文件中的设置） + Quarter Res. Good 1/4 分辨率 良好 + RED IPP2 development of the RAW picture. ISO, colour temperature, tint and exposure are applied by the SDK before development (as shot unless set); the output tone map and highlight roll-off shape a display gamma curve. Proxies keep the look they were made with. RAW 画面的 RED IPP2 显影。ISO、色温、色调和曝光在显影前由 SDK 应用（未设置时按拍摄时）；输出色调映射和高光滚降塑造显示伽马曲线。代理保留创建时的外观。 + REDWideGamutRGB REDWideGamutRGB + RMD sidecar in effect: {path} 正在使用 RMD 附属文件：{path} + RMD written: {paths} 已写入 RMD：{paths} + Rec. 2020 Rec. 2020 + ST 2084 (PQ) ST 2084（PQ） + Sixteenth Res. Good 1/16 分辨率 良好 + Standard (Rec. 709, BT.1886) 标准（Rec. 709、BT.1886） + Strong 强 + The 3D LUT the clip (or its RMD) carries. 剪辑（或其 RMD）附带的 3D LUT。 + The ASC CDL the clip (or its RMD) records. 剪辑（或其 RMD）记录的 ASC CDL。 + The largest decode a preview uses; an export and a monitor at 100 % always decode at full resolution, premium. 预览使用的最大解码；导出和 100% 显示的监视器始终以全分辨率高级解码。 + The linear output carries no look: choose a display or log gamma curve to apply it. 线性输出不带外观：请选择显示或对数伽马曲线以应用它。 + The luma and RGB curves the clip records. 剪辑记录的亮度和 RGB 曲线。 + This clip carries no creative LUT. 此剪辑不带创意 LUT。 + This clip decodes with the project's Camera RAW settings (File > Project Settings > Camera RAW). Choose Decode Using > Clip to give it settings of its own. 此剪辑使用项目的相机 RAW 设置解码（文件 > 项目设置 > 相机 RAW）。选择 解码依据 > 剪辑 可为其指定自己的设置。 + This clip has no HDRx track (Nikon R3D NE records none): there is nothing to blend. 此剪辑没有 HDRx 轨道（Nikon R3D NE 不记录）：没有可混合的内容。 + Update RMD 更新 RMD + Use Changes 使用更改 + Use Settings 使用设置 + Use project setting 使用项目设置 + Very Soft 非常柔和 + Write this clip's settings into its RMD sidecar beside the clip 将此剪辑的设置写入剪辑旁的 RMD 附属文件 + as recorded 按记录 + sRGB sRGB + {n} camera RAW clips decode using these settings. {n} 个相机 RAW 剪辑使用这些设置解码。 H.265 (HEVC) needs a hardware encoder (NVENC on NVIDIA GPUs, VideoToolbox on macOS); none was found on this machine, and FilmCraft has no software H.265 encoder yet. Choose H.264 or ProRes instead. H.265 (HEVC) 需要硬件编码器（NVIDIA GPU 上的 NVENC、macOS 上的 VideoToolbox）；本机未找到，且 FilmCraft 尚无软件 H.265 编码器。请改选 H.264 或 ProRes。 AV1 needs a hardware encoder; none was found on this machine, and FilmCraft has no software AV1 encoder yet. AV1 需要硬件编码器；本机未找到，且 FilmCraft 尚无软件 AV1 编码器。 diff --git a/crates/ui-egui/src/lib.rs b/crates/ui-egui/src/lib.rs index 053f5ddd6d..940181f38f 100644 --- a/crates/ui-egui/src/lib.rs +++ b/crates/ui-egui/src/lib.rs @@ -1332,6 +1332,11 @@ impl FilmcraftApp { Ok(seconds) } + /// Whether the Program or the Source monitor is playing. + pub fn any_playing(&self) -> bool { + self.playback.playing || self.source_playback.clock.playing + } + pub fn stop(&mut self) { self.playback.playing = false; self.playback.stop_at = None; @@ -1580,7 +1585,7 @@ impl FilmcraftApp { Some(filmcraft_project::ItemKind::Media(m)) => { use std::hash::{Hash, Hasher}; let mut h = std::collections::hash_map::DefaultHasher::new(); - filmcraft_engine::media_pool::media_key(m).hash(&mut h); + filmcraft_engine::media_pool::media_key(m, &p.settings.camera_raw).hash(&mut h); (m.proxy.is_some() && self.session.media.use_proxies()).hash(&mut h); h.finish() } @@ -1607,7 +1612,7 @@ impl FilmcraftApp { } let scale = width as f32 / src_w.max(1) as f32; let project = self.session.project.clone(); - self.frames.request(key, rate.tick_of(frame), scale, &project, 50); + self.frames.request_while(key, rate.tick_of(frame), scale, &project, 50, self.any_playing()); let shown = self.texture_existing(&name); if shown.is_some() { self.thumb_touched(ctx, &name); diff --git a/crates/ui-egui/src/panels/media_browser.rs b/crates/ui-egui/src/panels/media_browser.rs index 65b0e4c9b4..c4d165d811 100644 --- a/crates/ui-egui/src/panels/media_browser.rs +++ b/crates/ui-egui/src/panels/media_browser.rs @@ -730,7 +730,7 @@ fn draw_preview(app: &mut FilmcraftApp, ui: &egui::Ui, ctx: &egui::Context, e: & let tid = app.texture_for(ctx, &name, key, &img); Some((tid, vec2(img.w as f32, img.h as f32))) } else { - app.frames.request(key, rate.tick_of(frame), width as f32 / src_w.max(1) as f32, &proj, 40); + app.frames.request_while(key, rate.tick_of(frame), width as f32 / src_w.max(1) as f32, &proj, 40, app.any_playing()); app.texture_existing(&name) }; match tex { diff --git a/crates/ui-egui/src/panels/menu_dialogs.rs b/crates/ui-egui/src/panels/menu_dialogs.rs index 10122f438e..febcfadb52 100644 --- a/crates/ui-egui/src/panels/menu_dialogs.rs +++ b/crates/ui-egui/src/panels/menu_dialogs.rs @@ -37,11 +37,198 @@ pub struct Extras { pub dynamic_waveforms: bool, /// Paths handed to the operating system's opener (Edit Original, Reveal Log Files), newest last. pub opened: Vec, + /// Clip ▸ Source Settings… live preview: the last settings sent to the clip while the dialog is + /// open (None = none yet). See [`source_preview`]. + pub source_preview: Option, } impl Default for Extras { fn default() -> Self { - Extras { dialog: None, dynamic_waveforms: true, opened: Vec::new() } + Extras { dialog: None, dynamic_waveforms: true, opened: Vec::new(), source_preview: None } + } +} + +/// The live preview of the open Source Settings dialog: what was last sent to `clip.sourceSettings +/// {preview: true}` (applied, or refused with the error the dialog shows), for which clips, at +/// which project revision and when (egui time, seconds), so the next change waits for the +/// Program monitor to show that revision before it is sent (at most one apply outstanding). +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct SourcePreview { + pub sent: Value, + pub items: Value, + pub revision: u64, + pub sent_at: f64, +} + +/// Longest a live preview waits for the monitors before the next change is sent anyway (a frame +/// that never comes: no sequence open, the clip not on the Program monitor). +const PREVIEW_WAIT_MAX: f64 = 1.0; + +/// The Camera Raw panel's values as `clip.sourceSettings` / `file.projectSettings.cameraRaw` +/// take them: every control but the fixed ones (colour science, HDRx blend), with "as recorded" +/// as `null`. +fn panel_params(settings: &Value, p: &Value) -> Value { + let mut q = json!({}); + for st in settings.as_array().into_iter().flatten() { + let Some(id) = st["id"].as_str() else { continue }; + if st["kind"] == "fixed" { + continue; + } + q[id] = match &p[id] { + Value::String(v) if v == "asRecorded" => Value::Null, + v => v.clone(), + }; + } + q +} + +/// The parameters of the Source Settings dialog, as `clip.sourceSettings` takes them. Decode +/// Using goes only when it changes. A clip that decodes with the project's settings sends no +/// development settings of its own, unless one was changed (over the control channel: the panel +/// disables them), which gives the clip settings of its own, as the command does. +fn source_settings_params(d: &ClipDialogDraft) -> Value { + let mut q = panel_params(&d.info["settings"], &d.params); + let mode = d.info["develop"]["decodeUsing"].as_str().unwrap_or("clip"); + let wanted = d.params["decodeUsing"].as_str().unwrap_or(mode); + let Some(m) = q.as_object_mut() else { return q }; + m.remove("decodeUsing"); + if wanted != mode { + m.insert("decodeUsing".into(), json!(wanted)); + if wanted == "project" { + m.retain(|k, _| k == "decodeUsing" || k == "quality"); + } + } else if mode == "project" { + // only what differs from the project's settings the panel shows + let shown = panel_params(&d.info["settings"], &setting_values(&d.info["settings"])); + m.retain(|k, v| k == "quality" || shown.get(k) != Some(v)); + } + q +} + +/// The value of every control the command reported, as the dialog's parameters. +fn setting_values(settings: &Value) -> Value { + let mut p = json!({}); + for st in settings.as_array().into_iter().flatten() { + if let Some(id) = st["id"].as_str() { + p[id] = st["value"].clone(); + } + } + p +} + +/// The Camera Raw panel's values of develop `d` (the project's panel: no Decode Using). +fn develop_values(d: &filmcraft_media::RawDevelop) -> Value { + json!({ + "quality": d.quality().id(), + "colorScience": "ipp2", + "colorSpace": d.color_space.id(), + "gamma": d.gamma.id(), + "toneMap": d.tone_map.id(), + "rolloff": d.rolloff.id(), + "hdrPeakNits": d.hdr_peak_nits, + "chromaNoiseReduction": d.chroma_noise_reduction, + "flashingPixel": d.flashing_pixel.map_or("asRecorded", |f| f.id()), + "metadataCurves": d.metadata_curves, + "creativeLut": d.creative_lut, + "cdl": d.cdl, + "iso": d.iso, + "kelvin": d.kelvin, + "tint": d.tint, + "exposure": d.exposure, + }) +} + +/// Take a `clip.sourceSettings` reply into the open dialog: the controls (values, what is +/// enabled), the clip's mode and what it records, so the panel shows what the clip now develops +/// with (Decode Using switched, a colour space that brought its gamma along, Use Settings). +fn take_reply(d: &mut ClipDialogDraft, r: &Value) { + for k in ["settings", "develop", "own", "asShot", "clip", "rmd", "rmdNote", "message"] { + if let Some(v) = r.get(k) { + d.info[k] = v.clone(); + } + } + d.params = setting_values(&d.info["settings"]); +} + +/// Keep the clip in step with the Source Settings dialog: a changed control is +/// applied through `clip.sourceSettings {preview: true}` so the monitors develop the current +/// frame with it before OK. Every preview of one dialog is one undo step (the command merges +/// them); OK commits it ([`run`]) and Cancel takes it back ([`revert_source_preview`]). While a +/// control is dragged, at most one apply is outstanding: the next waits until the Program monitor +/// shows the revision of the last one (or [`PREVIEW_WAIT_MAX`] passes), and a drag that ends +/// meanwhile still sends its final value. A refused value (hostile, over the control channel) +/// shows its error and is not sent again until it changes. +fn source_preview(app: &mut FilmcraftApp, ctx: &egui::Context, d: &mut ClipDialogDraft) { + let wanted = source_settings_params(d); + if d.info["opened"].is_null() { + // a dialog opened without them (restored state): what it shows now counts as its opening values + d.info["opened"] = wanted; + return; + } + let last = app.ui.extras.source_preview.as_ref().map(|s| s.sent.clone()).unwrap_or_else(|| d.info["opened"].clone()); + if wanted == last { + return; + } + let now = ctx.input(|i| i.time); + if let Some(sp) = &app.ui.extras.source_preview { + let dragging = ctx.input(|i| i.pointer.any_down()); + let shown = app.program_picture().map(|k| k.revision); + let behind = shown.is_some_and(|r| r < sp.revision); + if dragging && behind && now - sp.sent_at < PREVIEW_WAIT_MAX { + // the last apply is still being developed: send this one when its frame is on screen + ctx.request_repaint_after(std::time::Duration::from_millis(16)); + return; + } + } + let items = d.info["items"].clone(); + let mut q = wanted.clone(); + q["items"] = items.clone(); + q["preview"] = json!(true); + let sent = match app.session.execute("clip.sourceSettings", q) { + Ok(r) => { + d.error.clear(); + take_reply(d, &r); + source_settings_params(d) + } + Err(e) => { + d.error = e.to_string(); + wanted + } + }; + app.ui.extras.source_preview = Some(SourcePreview { sent, items, revision: app.session.revision, sent_at: now }); +} + +/// The Camera Raw panel's buttons (Update RMD, Use Changes, Use Settings): the dialog's settings +/// with the action, applied as part of the dialog's live preview (one undo step; Cancel takes it +/// back, except the RMD file, which is written at once). +fn camera_raw_action(app: &mut FilmcraftApp, d: &mut ClipDialogDraft, flag: &str, now: f64) { + let items = d.info["items"].clone(); + let mut q = source_settings_params(d); + q["items"] = items.clone(); + q["preview"] = json!(true); + q[flag] = json!(true); + match app.session.execute("clip.sourceSettings", q) { + Ok(r) => { + d.error.clear(); + take_reply(d, &r); + if flag == "updateRmd" { + let paths: Vec<&str> = r["rmdWritten"].as_array().into_iter().flatten().filter_map(|w| w["path"].as_str()).collect(); + app.ui.status = tlf!("RMD written: {paths}", paths = paths.join(", ")); + } + let sent = source_settings_params(d); + app.ui.extras.source_preview = Some(SourcePreview { sent, items, revision: app.session.revision, sent_at: now }); + } + Err(e) => d.error = e.to_string(), + } +} + +/// Take back the live preview of a Source Settings dialog that closes without OK (Cancel, Escape, +/// or the dialog replaced): the clip returns to the values the dialog opened with, and the +/// history is as before it opened. +fn revert_source_preview(app: &mut FilmcraftApp) { + let Some(sp) = app.ui.extras.source_preview.take() else { return }; + if let Err(e) = app.session.execute("clip.sourceSettings", json!({"items": sp.items, "revert": true})) { + log::warn!("source settings preview not reverted: {e}"); } } @@ -80,7 +267,9 @@ fn meta(command: &str) -> Option<(&'static str, &'static str, &'static str)> { /// Commands whose menu item opens a dialog (the Project Settings items share one). fn dialog_command(id: &str) -> Option<&'static str> { Some(match id { - "file.projectSettings.general" | "file.projectSettings.scratchDisks" | "project.ingestSettings" => "project.settings", + "file.projectSettings.general" | "file.projectSettings.scratchDisks" | "project.ingestSettings" | "file.projectSettings.cameraRaw" => { + "project.settings" + } "edit.find" => "edit.find", "file.newSearchBin" => "file.newSearchBin", "file.mediaProperties" => "file.mediaProperties", @@ -289,11 +478,14 @@ fn defaults(app: &mut FilmcraftApp, cmd: &str, id: &str) -> Result<(Value, Value let tab = match id { "file.projectSettings.scratchDisks" => "scratchDisks", "project.ingestSettings" => "ingest", + "file.projectSettings.cameraRaw" => "cameraRaw", _ => "general", }; let st = s.project.settings.clone(); let general = s.execute("file.projectSettings.general", json!({})).map_err(|e| e.to_string())?; let scratch = filmcraft_engine::project_tools::scratch_paths(s); + let camera_raw = s.execute("file.projectSettings.cameraRaw", json!({})).map_err(|e| e.to_string())?; + let camera_values = setting_values(&camera_raw["settings"]); let ingest = serde_json::to_value(&st.ingest).unwrap_or_default(); let sc = &st.scratch; ( @@ -303,8 +495,9 @@ fn defaults(app: &mut FilmcraftApp, cmd: &str, id: &str) -> Result<(Value, Value "captureFormat": general["captureFormat"], "titleSafe": general["titleSafe"], "actionSafe": general["actionSafe"], "captured": sc.captured, "videoPreviews": sc.video_previews, "audioPreviews": sc.audio_previews, "autoSave": sc.auto_save, "ingest": ingest.clone(), + "cameraRaw": camera_values.clone(), }), - json!({"scratch": scratch, "ingest": ingest, "project": s.project.name}), + json!({"scratch": scratch, "ingest": ingest, "project": s.project.name, "cameraRaw": camera_raw, "cameraRawOpened": camera_values}), ) } "file.mediaProperties" => { @@ -323,7 +516,14 @@ fn defaults(app: &mut FilmcraftApp, cmd: &str, id: &str) -> Result<(Value, Value } (p, Value::Null) } - "clip.sourceSettings" => (json!({}), s.execute("clip.sourceSettings", json!({})).map_err(|e| e.to_string())?), + "clip.sourceSettings" => { + // the controls and their values come from the command; the dialog edits a copy + let info = s.execute("clip.sourceSettings", json!({})).map_err(|e| e.to_string())?; + let mut draft = ClipDialogDraft { command: cmd.into(), params: setting_values(&info["settings"]), info, error: String::new() }; + // the values the dialog opened with: the live preview applies every change from them + draft.info["opened"] = source_settings_params(&draft); + (draft.params, draft.info) + } "clip.updateMetadata" => { let n = s.state.project_selection.len().max(s.state.selection.len()); (json!({}), json!({"count": n})) @@ -448,12 +648,196 @@ fn color(ui: &mut egui::Ui, elems: &mut Elems, pre: &str, p: &mut Value, key: &s }); } +/// The value of one setting with a recorded value (`
.`): ISO as a list of the decoder's
+/// values, the others as numbers within the decoder's range, showing the clip's recorded value
+/// until set, with an As Shot button (`
..asShot`) that unsets it. The parameter is
+/// `null` for as shot, else the number.
+fn raw_value(ui: &mut egui::Ui, elems: &mut Elems, pre: &str, p: &mut Value, st: &Value) {
+    let id = st["id"].as_str().unwrap_or_default();
+    let label = crate::i18n::t(st["label"].as_str().unwrap_or(id)).to_string();
+    let as_shot = st["asShot"].as_f64().filter(|v| v.is_finite());
+    let fallback = as_shot.or_else(|| st["default"].as_f64()).unwrap_or(0.0);
+    let set = !p[id].is_null();
+    ui.horizontal(|ui| {
+        if st["kind"] == "iso" {
+            let as_shot_label = match as_shot {
+                Some(v) => tlf!("As Shot (ISO {v})", v = v.round().max(0.0) as u64),
+                None => tl!("As Shot").to_string(),
+            };
+            let mut options: Vec<(String, String)> = vec![("asShot".into(), as_shot_label)];
+            options.extend(st["options"].as_array().into_iter().flatten().filter_map(Value::as_u64).map(|v| (v.to_string(), v.to_string())));
+            let mut choice = json!(p[id].as_u64().map_or_else(|| "asShot".to_string(), |v| v.to_string()));
+            combo(ui, elems, &format!("{pre}.{id}"), "", &mut choice, &options);
+            p[id] = match choice.as_str().and_then(|c| c.parse::().ok()) {
+                Some(v) => json!(v),
+                None => Value::Null,
+            };
+        } else {
+            let mut v = p[id].as_f64().unwrap_or(fallback);
+            let (min, max) = (st["min"].as_f64().unwrap_or(f64::MIN), st["max"].as_f64().unwrap_or(f64::MAX));
+            let decimals = st["decimals"].as_u64().unwrap_or(0).min(6) as usize;
+            let step = st["step"].as_f64().filter(|s| *s > 0.0).unwrap_or(1.0);
+            let suffix = st["suffix"].as_str().unwrap_or_default();
+            let mut drag = crate::widgets::number_drag(&mut v).speed(step).fixed_decimals(decimals).suffix(suffix);
+            if min <= max {
+                drag = drag.range(min..=max);
+            }
+            let r = ui.add(drag);
+            push(elems, format!("{pre}.{id}"), &r, label.as_str());
+            if r.changed() {
+                // whole-number settings (Kelvin, the HDR peak) travel as integers
+                p[id] = if decimals == 0 { json!(v.round().max(0.0) as u64) } else { json!(v) };
+            }
+            if !set {
+                let what = if id == "hdrPeakNits" {
+                    tl!("as recorded")
+                } else if as_shot.is_some() {
+                    tl!("as shot")
+                } else {
+                    tl!("as shot (decoder not open)")
+                };
+                ui.label(RichText::new(what).weak());
+            }
+        }
+        if set {
+            let r = ui.small_button(tl!("As Shot"));
+            push(elems, format!("{pre}.{id}.asShot"), &r, "As Shot");
+            if r.clicked() {
+                p[id] = Value::Null;
+            }
+        }
+    });
+}
+
+/// The Camera Raw panel (Clip ▸ Source Settings…, and Project Settings ▸ Camera RAW with
+/// `clip` false): the controls the command reports, in its three columns as a RED Camera Raw
+/// panel lays them out (decode and colour pipeline; ISO and the output transform; white balance
+/// and exposure), each `
.`. What a control does not apply to is greyed out with the
+/// reason on hover: every development control while the clip decodes with the project's settings,
+/// the tone map and roll-off without a display gamma, the HDR peak without ST 2084, the curves,
+/// LUT and CDL with linear light, the LUT of a clip that has none, colour science and the HDRx
+/// blend always (with the reason under them). A colour space or gamma chosen that does not go
+/// with the other one brings a fitting other one along, as the command does.
+fn camera_raw_panel(ui: &mut egui::Ui, elems: &mut Elems, pre: &str, p: &mut Value, settings: &Value, clip: bool) {
+    use filmcraft_media::{RawColorSpace, RawDevelop, RawGamma};
+    // the colour space and gamma shown last frame, to tell which of them changed (from the panel
+    // or over the control channel)
+    let shown = p.get("shownPair").cloned().filter(Value::is_array);
+    let follows = clip && p["decodeUsing"] == "project";
+    let gamma = p["gamma"].as_str().unwrap_or("linear").to_string();
+    let display = RawGamma::parse(&gamma).is_some_and(RawGamma::is_display);
+    let enabled = |st: &Value| -> bool {
+        match st["id"].as_str().unwrap_or_default() {
+            _ if st["kind"] == "fixed" => false,
+            "quality" | "decodeUsing" => true,
+            _ if follows => false,
+            "toneMap" | "rolloff" => display,
+            "hdrPeakNits" => gamma == "st2084",
+            "metadataCurves" | "cdl" => gamma != "linear",
+            "creativeLut" => gamma != "linear" && st["available"] != false,
+            _ => true,
+        }
+    };
+    let all: Vec = settings.as_array().cloned().unwrap_or_default();
+    ui.horizontal_top(|ui| {
+        for column in 1..=3u64 {
+            ui.vertical(|ui| {
+                egui::Grid::new((pre, "camera-raw", column)).num_columns(2).spacing([8.0, 6.0]).show(ui, |ui| {
+                    for st in all.iter().filter(|st| st["column"] == column) {
+                        let id = st["id"].as_str().unwrap_or_default().to_string();
+                        let label = crate::i18n::t(st["label"].as_str().unwrap_or(&id)).to_string();
+                        let on = enabled(st);
+                        let note = crate::i18n::t(st["note"].as_str().unwrap_or_default()).to_string();
+                        if st["kind"] == "bool" {
+                            ui.label("");
+                        } else {
+                            ui.add_enabled(on, egui::Label::new(label.as_str()));
+                        }
+                        let r = ui.add_enabled_ui(on, |ui| match st["kind"].as_str().unwrap_or_default() {
+                            "bool" => {
+                                let recorded = st["recorded"].as_bool();
+                                let mut v = p[id.as_str()].as_bool().or(recorded).unwrap_or(false);
+                                ui.horizontal(|ui| {
+                                    let r = ui.checkbox(&mut v, label.as_str());
+                                    push(elems, format!("{pre}.{id}"), &r, label.as_str());
+                                    if r.changed() {
+                                        p[id.as_str()] = json!(v);
+                                    }
+                                    if p[id.as_str()].is_null() {
+                                        ui.label(RichText::new(tl!("as recorded")).weak());
+                                    }
+                                });
+                            }
+                            "fixed" => {
+                                let shown = st["options"][0]["label"]
+                                    .as_str()
+                                    .map(str::to_string)
+                                    .unwrap_or_else(|| format!("{:.2}", st["value"].as_f64().unwrap_or(0.0)));
+                                let r = ui.add(egui::Button::new(crate::i18n::t(&shown)).min_size(egui::vec2(140.0, 0.0)));
+                                push(elems, format!("{pre}.{id}"), &r, label.as_str());
+                            }
+                            "iso" | "number" => raw_value(ui, elems, pre, p, st),
+                            _ => {
+                                let options: Vec<(String, String)> = st["options"]
+                                    .as_array()
+                                    .into_iter()
+                                    .flatten()
+                                    .filter_map(|o| Some((o["id"].as_str()?.to_string(), o["label"].as_str().unwrap_or_default().to_string())))
+                                    .collect();
+                                combo(ui, elems, &format!("{pre}.{id}"), "", &mut p[id.as_str()], &options);
+                            }
+                        });
+                        if !note.is_empty() {
+                            r.response.on_hover_text(note);
+                        }
+                        ui.end_row();
+                    }
+                });
+                // why the blend is greyed out, where it shows (the panel's first column)
+                if let Some(blend) = all.iter().find(|st| st["id"] == "blendType" && st["column"] == column) {
+                    ui.scope(|ui| {
+                        ui.set_max_width(320.0);
+                        ui.add(egui::Label::new(RichText::new(crate::i18n::t(blend["note"].as_str().unwrap_or_default())).weak().small()).wrap());
+                    });
+                }
+            });
+            if column < 3 {
+                // space, not a separator: a vertical one in a row takes the whole height left
+                ui.add_space(18.0);
+            }
+        }
+    });
+    // a colour space or gamma that does not go with the other brings a fitting one along
+    let space = p["colorSpace"].as_str().and_then(RawColorSpace::parse);
+    let curve = p["gamma"].as_str().and_then(RawGamma::parse);
+    if let (Some(s), Some(g)) = (space, curve)
+        && !RawDevelop::supports(s, g)
+    {
+        // without a last frame to compare with, a gamma that fits one colour space only wins
+        let gamma_changed = match &shown {
+            Some(pair) => p["gamma"] != pair[1],
+            None => matches!(g, RawGamma::Log3G10 | RawGamma::St2084 | RawGamma::Hlg),
+        };
+        if gamma_changed {
+            p["colorSpace"] = json!(RawDevelop::space_for(g).id());
+        } else {
+            p["gamma"] = json!(RawDevelop::gamma_for(s).id());
+        }
+    }
+    p["shownPair"] = json!([p["colorSpace"].clone(), p["gamma"].clone()]);
+}
+
 fn pairs(v: &[&str]) -> Vec<(String, String)> {
     v.iter().map(|x| (x.to_string(), x.to_string())).collect()
 }
 
 /// Draw the open dialog, if any.
 pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) {
+    // a Source Settings dialog that went away without OK (closed from outside, or replaced by
+    // another dialog) leaves its live preview behind: take it back, as Cancel would
+    if app.ui.extras.source_preview.is_some() && app.ui.extras.dialog.as_ref().is_none_or(|d| d.command != "clip.sourceSettings") {
+        revert_source_preview(app);
+    }
     let Some(mut d) = app.ui.extras.dialog.clone() else { return };
     crate::widgets::revert_drag_on_escape(ctx, egui::Id::unique("menu-dialog-before-drag"), &mut d);
     let Some((title, pre, ok_text)) = meta(&d.command) else {
@@ -462,8 +846,12 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) {
     };
     let mut elems: Elems = Vec::new();
     let mut action: Option<&'static str> = None;
+    // a Camera Raw panel button (Update RMD, Use Changes, Use Settings), run after the widgets
+    let mut camera_action: Option<&'static str> = None;
     let mut browse: Option = None;
-    let info_only = matches!(d.command.as_str(), "file.mediaProperties" | "clip.sourceSettings" | "help.systemCompatibilityReport");
+    // Source Settings is read-only for media without development controls
+    let info_only = matches!(d.command.as_str(), "file.mediaProperties" | "help.systemCompatibilityReport")
+        || (d.command == "clip.sourceSettings" && d.info["settings"].as_array().is_none_or(Vec::is_empty));
     // the title is translated; the window keeps one id whatever the interface language
     let id = egui::Id::unique(("menu-dialog", pre));
     egui::Window::new(title).id(id).collapsible(false).resizable(false).default_width(420.0).centered(ctx).show(ctx, |ui| {
@@ -530,7 +918,12 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) {
             "project.settings" => {
                 ui.label(tlf!("Project: {name}", name = d.info["project"].as_str().unwrap_or_default()));
                 ui.horizontal(|ui| {
-                    for (k, l) in [("general", tl!("General")), ("scratchDisks", tl!("Scratch Disks")), ("ingest", tl!("Ingest Settings"))] {
+                    for (k, l) in [
+                        ("general", tl!("General")),
+                        ("scratchDisks", tl!("Scratch Disks")),
+                        ("ingest", tl!("Ingest Settings")),
+                        ("cameraRaw", tl!("Camera RAW")),
+                    ] {
                         let r = ui.selectable_label(p["tab"].as_str() == Some(k), l);
                         push(&mut elems, format!("projectSettings.tab.{k}"), &r, l);
                         if r.clicked() {
@@ -540,6 +933,21 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) {
                 });
                 ui.separator();
                 match p["tab"].as_str().unwrap_or("general") {
+                    "cameraRaw" => {
+                        // the project's Camera RAW settings: what clips that decode using the project develop with
+                        let settings = d.info["cameraRaw"]["settings"].clone();
+                        camera_raw_panel(ui, &mut elems, "projectSettings.cameraRaw", &mut p["cameraRaw"], &settings, false);
+                        ui.horizontal(|ui| {
+                            let r = ui.button(tl!("Standard (Rec. 709, BT.1886)"));
+                            push(&mut elems, "projectSettings.cameraRaw.standard", &r, "Standard");
+                            if r.clicked() {
+                                p["cameraRaw"] = develop_values(&filmcraft_media::RawDevelop::STANDARD);
+                            }
+                            let n = d.info["cameraRaw"]["clips"].as_u64().unwrap_or(0);
+                            ui.label(RichText::new(tlf!("{n} camera RAW clips decode using these settings.", n)).weak());
+                        });
+                        ui.label(RichText::new(crate::i18n::t(d.info["cameraRaw"]["message"].as_str().unwrap_or_default())).weak());
+                    }
                     "scratchDisks" => {
                         for (k, l) in [
                             ("captured", tl!("Captured and Generated:")),
@@ -686,8 +1094,43 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) {
                 text(ui, &mut elems, pre, p, "logNote", tl!("Log Note:"), 220.0);
             }
             "clip.sourceSettings" => {
-                ui.label(tlf!("Codec: {codec}", codec = d.info["codec"].as_str().unwrap_or_default()));
-                ui.label(RichText::new(d.info["message"].as_str().unwrap_or_default()).weak());
+                let settings = d.info["settings"].clone();
+                let developable = settings.as_array().is_some_and(|s| !s.is_empty());
+                ui.horizontal(|ui| {
+                    ui.label(RichText::new(tl!("Camera Raw")).strong());
+                    if developable {
+                        ui.label(RichText::new("RED · IPP2").strong());
+                    }
+                    ui.label(tlf!("Codec: {codec}", codec = d.info["codec"].as_str().unwrap_or_default()));
+                });
+                if developable {
+                    ui.separator();
+                    camera_raw_panel(ui, &mut elems, pre, p, &settings, true);
+                    ui.separator();
+                    // the panel's buttons, in a plain row (a right-to-left layout makes an
+                    // auto-sized window grow to the screen)
+                    ui.horizontal(|ui| {
+                        let many = d.info["items"].as_array().is_some_and(|i| i.len() > 1);
+                        for (flag, label, on, tip) in [
+                            ("updateRmd", tl!("Update RMD"), true, tl!("Write this clip's settings into its RMD sidecar beside the clip")),
+                            ("useChanges", tl!("Use Changes"), many, tl!("Apply this clip's settings to every selected clip")),
+                            ("useSettings", tl!("Use Settings"), true, tl!("Put this clip back to the settings it records (or its RMD sidecar says)")),
+                        ] {
+                            let r = ui.add_enabled(on, egui::Button::new(label)).on_hover_text(tip);
+                            push(&mut elems, format!("{pre}.{flag}"), &r, label);
+                            if r.clicked() {
+                                camera_action = Some(flag);
+                            }
+                        }
+                    });
+                    // what the RAW adjustments start from: as shot, or the clip's RMD sidecar
+                    let rmd = match d.info["rmd"].as_str() {
+                        Some(path) => tlf!("RMD sidecar in effect: {path}", path),
+                        None => tl!("No RMD sidecar: the clip's recorded settings are the camera's.").to_string(),
+                    };
+                    ui.label(RichText::new(rmd).weak().small());
+                }
+                ui.label(RichText::new(crate::i18n::t(d.info["message"].as_str().unwrap_or_default())).weak());
             }
             "clip.updateMetadata" => {
                 ui.label(tlf!("Write the metadata of the selected clip(s) ({n}) to XMP files next to their media?", n = d.info["count"]));
@@ -847,14 +1290,23 @@ pub fn show(app: &mut FilmcraftApp, ctx: &egui::Context) {
     if crate::widgets::escape_closes(ctx) {
         action = Some("cancel");
     }
+    // Source Settings previews live: a changed control reaches the clip (and the monitors) now
+    if d.command == "clip.sourceSettings" && !info_only && action.is_none() {
+        if let Some(flag) = camera_action {
+            camera_raw_action(app, &mut d, flag, ctx.input(|i| i.time));
+        }
+        source_preview(app, ctx, &mut d);
+    }
     match action {
         Some("cancel") => {
+            revert_source_preview(app);
             app.ui.extras.dialog = None;
             return;
         }
         Some("ok") => match run(app, &d) {
             Ok(Some(info)) => d.info["result"] = json!(info),
             Ok(None) => {
+                app.ui.extras.source_preview = None;
                 app.ui.extras.dialog = None;
                 return;
             }
@@ -893,6 +1345,12 @@ fn run(app: &mut FilmcraftApp, d: &ClipDialogDraft) -> Result, St
             if p["ingest"] != d.info["ingest"] {
                 s.execute("project.ingestSettings", p["ingest"].clone()).map_err(|e| e.to_string())?;
             }
+            // the panel's state also holds what it shows (`shownPair`): compare the parameters
+            let settings = &d.info["cameraRaw"]["settings"];
+            let wanted = panel_params(settings, &p["cameraRaw"]);
+            if wanted != panel_params(settings, &d.info["cameraRawOpened"]) {
+                s.execute("file.projectSettings.cameraRaw", wanted).map_err(|e| e.to_string())?;
+            }
             Ok(None)
         }
         "file.saveAsTemplate" => {
@@ -905,6 +1363,14 @@ fn run(app: &mut FilmcraftApp, d: &ClipDialogDraft) -> Result, St
             app.ui.status = tl!("Scene Edit Detection: analysing in the background (see Progress)").into();
             Ok(None)
         }
+        "clip.sourceSettings" => {
+            // OK keeps what the live preview applied, as one undo step with it
+            let mut q = source_settings_params(d);
+            q["items"] = d.info["items"].clone();
+            q["commit"] = json!(true);
+            s.execute("clip.sourceSettings", q).map_err(|e| e.to_string())?;
+            Ok(None)
+        }
         cmd => {
             let mut q = p.clone();
             if cmd == "markers.addFlashCue" && q["name"].as_str().is_some_and(str::is_empty) {
diff --git a/crates/ui-egui/src/panels/monitor.rs b/crates/ui-egui/src/panels/monitor.rs
index 018a2cd440..3353f7973c 100644
--- a/crates/ui-egui/src/panels/monitor.rs
+++ b/crates/ui-egui/src/panels/monitor.rs
@@ -229,7 +229,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
                 .frames
                 .get_plan(&key)
                 .map(|p| (key, p))
-                .or_else(|| app.frames.nearest_plan(key, 6))
+                .or_else(|| app.frames.nearest_plan(key, crate::frames::LATE_GRACE))
                 .or_else(|| asked.iter().find_map(|k| app.frames.get_plan(k).map(|p| (*k, p))));
             match found {
                 Some((k, plan)) => (app.gpu_present(k, &plan).map(|(id, _)| id), Some(k)),
@@ -266,7 +266,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect, which: Which)
                 t => t,
             };
             let rkey = FrameKey { target: rtarget, frame: rf, size: size_key, revision: rev, draft: false };
-            app.frames.request(rkey, rate.tick_of(rf), scale, &project, 1);
+            app.frames.request_while(rkey, rate.tick_of(rf), scale, &project, 1, app.any_playing());
             let rpic = fit(ra, display_w, frame_size.1 as f32);
             if mv.transparency_grid {
                 monitor_view::transparency_grid(ui.painter(), rpic, ui.clip_rect());
diff --git a/crates/ui-egui/src/panels/multicam.rs b/crates/ui-egui/src/panels/multicam.rs
index 70f0ffe0d0..e0f961c9d0 100644
--- a/crates/ui-egui/src/panels/multicam.rs
+++ b/crates/ui-egui/src/panels/multicam.rs
@@ -344,7 +344,7 @@ pub fn grid(app: &mut FilmcraftApp, ui: &mut egui::Ui, area: Rect) {
     }
     let tex = match app.frames.get(&key) {
         Some(img) => Some(app.texture_for(&ctx, "monitor-multicam", key, &img)),
-        None => match app.frames.nearest(key.target, frame, key.size, key.revision, 6) {
+        None => match app.frames.nearest(key.target, frame, key.size, key.revision, crate::frames::LATE_GRACE) {
             Some(img) => Some(app.texture_for(&ctx, "monitor-multicam", FrameKey { frame: frame - 1, ..key }, &img)),
             None => app.texture_existing("monitor-multicam").map(|(id, _)| id),
         },
@@ -451,7 +451,7 @@ pub fn show_edit_cameras(app: &mut FilmcraftApp, ctx: &egui::Context) {
         let has_video = q.angle_video_track_index(a).is_some();
         let key = FrameKey { target: Target::MulticamAngle(src, a as u32), frame, size: (scale * 1000.0) as u32, revision: rev, draft: false };
         let tex = if has_video {
-            app.frames.request(key, rate.tick_of(frame), scale, &project, 2);
+            app.frames.request_while(key, rate.tick_of(frame), scale, &project, 2, app.any_playing());
             app.frames.get(&key).map(|img| app.texture_for(ctx, &format!("edit-cameras-{}-{a}", src.0), key, &img))
         } else {
             None
diff --git a/crates/ui-egui/src/panels/reference.rs b/crates/ui-egui/src/panels/reference.rs
index 07f2ded344..fbb532030a 100644
--- a/crates/ui-egui/src/panels/reference.rs
+++ b/crates/ui-egui/src/panels/reference.rs
@@ -50,7 +50,7 @@ pub fn show(app: &mut FilmcraftApp, ui: &mut egui::Ui, rect: Rect) {
             let scale = crate::panels::monitor::quantize_scale((pic.width() * ppp / w).clamp(1.0 / 32.0, 1.0));
             let key = FrameKey { target: Target::Sequence(seq_id), frame, size: (scale * 1000.0) as u32, revision: app.session.revision, draft: false };
             let project = app.session.project.clone();
-            app.frames.request(key, rate.tick_of(frame), scale, &project, 3);
+            app.frames.request_while(key, rate.tick_of(frame), scale, &project, 3, app.any_playing());
             let tex = match app.frames.get(&key) {
                 Some(img) => Some(app.texture_for(&ctx, "reference-monitor", key, &img)),
                 None => app.texture_existing("reference-monitor").map(|x| x.0),
diff --git a/crates/ui-egui/src/panels/scopes.rs b/crates/ui-egui/src/panels/scopes.rs
index bf8747961f..34aa0b945a 100644
--- a/crates/ui-egui/src/panels/scopes.rs
+++ b/crates/ui-egui/src/panels/scopes.rs
@@ -70,7 +70,7 @@ pub fn frame_signal(app: &mut FilmcraftApp, ctx: &egui::Context, slot: &str, fra
     } else {
         let fkey = FrameKey { target: Target::Sequence(seq_id), frame, size: 250, revision: rev, draft: false };
         let project = app.session.project.clone();
-        app.frames.request(fkey, rate.tick_of(frame), 0.25, &project, prio);
+        app.frames.request_while(fkey, rate.tick_of(frame), 0.25, &project, prio, app.any_playing());
         app.frames.get(&fkey).map(|img| Signal::from_rgba8(img.w, img.h, &img.px))
     };
     match fresh {
diff --git a/crates/ui-egui/src/perf.rs b/crates/ui-egui/src/perf.rs
index ce10ce8ef2..0144811916 100644
--- a/crates/ui-egui/src/perf.rs
+++ b/crates/ui-egui/src/perf.rs
@@ -29,7 +29,14 @@ pub fn stats(app: &FilmcraftApp) -> Value {
     frames["workers"] = json!(app.frames.workers());
     frames["queued"] = json!(app.frames.queue_len());
     frames["renderCostMs"] = json!(app.frames.render_cost() * 1e3);
-    frames["cache"] = json!({"images": images, "imageMB": image_bytes as f64 / 1e6, "plans": plans, "planMB": plan_bytes as f64 / 1e6});
+    // a throughput-bound source (camera RAW) while one plays: its rate, latency and the plan
+    let fps = app.session.sequence_rate().as_f64();
+    let (lead, keep, of) = app.frames.plan_for(fps);
+    frames["plan"] = json!({"lead": lead, "keep": keep, "of": of});
+    frames["source"] =
+        app.frames.source_rate().map_or(Value::Null, |m| json!({"rate": m.rate, "latencyMs": m.latency * 1e3, "latencySlowMs": m.latency_slow * 1e3}));
+    let (lookahead, last_plan) = app.frames.plan_lookahead();
+    frames["cache"] = json!({"images": images, "imageMB": image_bytes as f64 / 1e6, "plans": plans, "planMB": plan_bytes as f64 / 1e6, "lookahead": lookahead, "lastPlanMB": last_plan as f64 / 1e6});
     v["frames"] = frames;
     v["gpu"] = app.gpu.as_ref().map_or(Value::Null, |g| {
         json!({
diff --git a/crates/ui-egui/tests/it/clip_dialogs_ui.rs b/crates/ui-egui/tests/it/clip_dialogs_ui.rs
index 2d9bbf20fa..9dd1e66206 100644
--- a/crates/ui-egui/tests/it/clip_dialogs_ui.rs
+++ b/crates/ui-egui/tests/it/clip_dialogs_ui.rs
@@ -466,3 +466,225 @@ fn clip_context_menu_adds_a_marker_at_the_playhead_with_one_undo() {
         assert_eq!(serde_json::to_value(&*d.app().session.project).unwrap(), serde_json::to_value(&*after).unwrap());
     }
 }
+
+/// A hand-made R3D NE item `name` at `path` (the SDK is optional; the dialog works on the project).
+fn add_r3d(d: &mut Driver, name: &str, path: &str) -> u64 {
+    let ocean = d.item_named("Ocean_Sunset.mp4");
+    let app = d.app();
+    let mut p = (*app.session.project).clone();
+    let mut clip = p.item(filmcraft_engine::project::ItemId(ocean)).unwrap().as_media().unwrap().clone();
+    clip.media = filmcraft_engine::project::MediaRef::File { path: path.into() };
+    clip.info.video.as_mut().unwrap().codec = "R3D NE".into();
+    let id = p.add_item(name, Label::Iris, ItemKind::Media(clip), None);
+    app.session.project = std::sync::Arc::new(p);
+    id.0
+}
+
+/// Clip ▸ Source Settings… on an R3D NE clip is a Camera Raw panel: every control in its three
+/// columns with an automation id, the three buttons; a new project's clip decodes using the
+/// project's settings until Decode Using ▸ Clip; a typed value is applied on OK as that clip's
+/// setting, a set value gets an As Shot button that puts it back, and the control channel reads
+/// and sets the same values through `clip.sourceSettings`.
+#[test]
+fn source_settings_camera_raw_panel_sets_and_resets_the_clip_settings() {
+    use filmcraft_media::{DecodeUsing, Hundredths, RawGamma};
+    let mut d = Driver::new();
+    let id = add_r3d(&mut d, "DSC_0003.R3D", "/footage/DSC_0003.R3D");
+    d.exec("project.select", json!({"items": [id]}));
+    let develop = |d: &mut Driver| d.app().session.project.item(filmcraft_engine::project::ItemId(id)).unwrap().as_media().unwrap().interpret.develop;
+    let r = d.menu("clip.sourceSettings");
+    assert_eq!(r["dialog"], "sourceSettings", "{r}");
+    for key in [
+        "quality",
+        "decodeUsing",
+        "colorScience",
+        "colorSpace",
+        "gamma",
+        "blendType",
+        "blendBias",
+        "metadataCurves",
+        "creativeLut",
+        "cdl",
+        "iso",
+        "toneMap",
+        "rolloff",
+        "hdrPeakNits",
+        "chromaNoiseReduction",
+        "flashingPixel",
+        "kelvin",
+        "tint",
+        "exposure",
+        "updateRmd",
+        "useChanges",
+        "useSettings",
+        "ok",
+    ] {
+        assert!(d.has(&format!("sourceSettings.{key}")), "{key} missing: {:?}", d.ok("ui.elements", json!({"prefix": "sourceSettings."})));
+    }
+    let dialog = d.app().ui.extras.dialog.clone().unwrap();
+    assert_eq!((dialog.params["decodeUsing"].as_str(), dialog.params["gamma"].as_str()), (Some("project"), Some("bt1886")), "the project's standard");
+    assert!(!d.has("sourceSettings.kelvin.asShot"), "nothing is set yet, so nothing to put back");
+    // Decode Using ▸ Clip: the clip gets settings of its own, from the project's
+    d.ok("ui.set", json!({"menuDialog": {"decodeUsing": "clip"}}));
+    d.frames(3);
+    assert_eq!(develop(&mut d).map(|x| (x.decode_using, x.gamma)), Some((DecodeUsing::Clip, RawGamma::Bt1886)));
+    // type a colour temperature into the Color Temp field and OK: the clip develops at 4300 K
+    d.click("sourceSettings.kelvin");
+    d.ok("ui.key", json!({"key": "Cmd+A"}));
+    d.ok("ui.type", json!({"text": "4300"}));
+    d.ok("ui.key", json!({"key": "Enter"}));
+    d.frames(3);
+    assert!(d.has("sourceSettings.kelvin.asShot"), "a set value gets its As Shot button: {:?}", d.ok("ui.elements", json!({"prefix": "sourceSettings."})));
+    d.click("sourceSettings.ok");
+    assert_eq!(develop(&mut d).map(|x| (x.kelvin, x.iso, x.tint, x.exposure)), Some((Some(4300), None, None, None)));
+    // the control channel sets the rest; reopened, the dialog shows them set and puts them back
+    d.exec("clip.sourceSettings", json!({"iso": 3200, "tint": -12.5, "exposure": 0.5}));
+    assert_eq!(develop(&mut d).map(|x| (x.kelvin, x.iso, x.tint, x.exposure)), Some((Some(4300), Some(3200), Some(Hundredths(-1250)), Some(Hundredths(50)))));
+    d.menu("clip.sourceSettings");
+    for key in ["kelvin", "tint", "exposure"] {
+        assert!(d.has(&format!("sourceSettings.{key}.asShot")), "{key}");
+        d.click(&format!("sourceSettings.{key}.asShot"));
+        d.frames(2);
+        assert!(!d.has(&format!("sourceSettings.{key}.asShot")), "{key}");
+    }
+    d.click("sourceSettings.ok");
+    assert_eq!(develop(&mut d).map(|x| (x.kelvin, x.iso, x.tint, x.exposure)), Some((None, Some(3200), None, None)));
+    let r = d.exec("clip.sourceSettings", json!({}));
+    assert_eq!(r["develop"]["kelvin"], Value::Null, "{r}");
+    d.exec("edit.undo", json!({}));
+    assert_eq!(develop(&mut d).and_then(|x| x.kelvin), Some(4300), "one undo step");
+}
+
+/// Clip ▸ Source Settings… previews live: a changed control is applied to the
+/// clip while the dialog is open (the monitors develop the frame with it before OK), every change
+/// of one dialog shares one undo step, Cancel (and Escape) puts the opening values back and leaves
+/// no undo step, OK keeps them so one Undo restores the opening values, and a hostile value set
+/// over the control channel is refused with the error in the dialog, the clip untouched.
+#[test]
+fn source_settings_dialog_previews_live_and_cancel_or_undo_revert_in_one_step() {
+    use filmcraft_media::Hundredths;
+    let mut d = Driver::new();
+    let ocean = d.item_named("Ocean_Sunset.mp4");
+    let id = {
+        let app = d.app();
+        let mut p = (*app.session.project).clone();
+        let mut clip = p.item(filmcraft_engine::project::ItemId(ocean)).unwrap().as_media().unwrap().clone();
+        clip.media = filmcraft_engine::project::MediaRef::File { path: "/footage/DSC_0005.R3D".into() };
+        clip.info.video.as_mut().unwrap().codec = "R3D NE".into();
+        let id = p.add_item("DSC_0005.R3D", Label::Iris, ItemKind::Media(clip), None);
+        app.session.project = std::sync::Arc::new(p);
+        id.0
+    };
+    d.exec("project.select", json!({"items": [id]}));
+    d.exec("clip.sourceSettings", json!({"iso": 800}));
+    let develop = |d: &mut Driver| d.app().session.project.item(filmcraft_engine::project::ItemId(id)).unwrap().as_media().unwrap().interpret.develop;
+    let steps = |d: &mut Driver| d.app().session.history.undo.len();
+    let opened = (develop(&mut d), steps(&mut d));
+    // a drag of ISO and exposure: each change reaches the clip before OK, all of them one undo step
+    d.menu("clip.sourceSettings");
+    d.ok("ui.set", json!({"menuDialog": {"iso": 3200}}));
+    d.frames(3);
+    assert_eq!((develop(&mut d).unwrap().iso, steps(&mut d)), (Some(3200), opened.1 + 1), "applied live");
+    d.ok("ui.set", json!({"menuDialog": {"iso": 6400}}));
+    d.frames(3);
+    d.ok("ui.set", json!({"menuDialog": {"exposure": 1.5}}));
+    d.frames(3);
+    let x = develop(&mut d).unwrap();
+    assert_eq!((x.iso, x.exposure, steps(&mut d)), (Some(6400), Some(Hundredths(150)), opened.1 + 1), "one undo step for the dialog");
+    assert!(d.app().ui.extras.source_preview.is_some());
+    // a hostile value over the control channel (an ISO the decoder has not got; the number fields
+    // keep their values within the decoder's ranges themselves): refused, shown, the clip as it was
+    d.ok("ui.set", json!({"menuDialog": {"iso": 1250}}));
+    d.frames(3);
+    let dialog = d.app().ui.extras.dialog.clone().unwrap();
+    assert!(dialog.error.to_ascii_lowercase().contains("iso"), "{}", dialog.error);
+    assert_eq!((develop(&mut d).unwrap().iso, steps(&mut d)), (Some(6400), opened.1 + 1), "refused without applying");
+    d.ok("ui.set", json!({"menuDialog": {"iso": 12800, "kelvin": 4300}}));
+    d.frames(3);
+    assert_eq!((develop(&mut d).unwrap().iso, develop(&mut d).unwrap().kelvin), (Some(12800), Some(4300)));
+    assert!(d.app().ui.extras.dialog.clone().unwrap().error.is_empty(), "a good value clears the error");
+    // Cancel: the opening values are back and the history is as before the dialog
+    d.click("sourceSettings.cancel");
+    let extras = d.app().ui.extras.clone();
+    assert!(extras.dialog.is_none() && extras.source_preview.is_none(), "{:?} {:?}", extras.dialog, extras.source_preview);
+    assert_eq!((develop(&mut d), steps(&mut d)), opened, "Cancel reverts without a trace");
+    // Escape does the same
+    d.menu("clip.sourceSettings");
+    d.ok("ui.set", json!({"menuDialog": {"exposure": -2.0}}));
+    d.frames(3);
+    assert_eq!(develop(&mut d).unwrap().exposure, Some(Hundredths(-200)));
+    d.ok("ui.key", json!({"key": "Escape"}));
+    d.frames(3);
+    assert!(d.app().ui.extras.dialog.is_none());
+    assert_eq!((develop(&mut d), steps(&mut d)), opened, "Escape reverts too");
+    // OK keeps the previewed values as one undo step; Undo restores the opening values in one go
+    d.menu("clip.sourceSettings");
+    d.ok("ui.set", json!({"menuDialog": {"iso": 1600}}));
+    d.frames(3);
+    d.ok("ui.set", json!({"menuDialog": {"gamma": "srgb", "toneMap": "medium"}}));
+    d.frames(3);
+    d.click("sourceSettings.ok");
+    assert!(d.app().ui.extras.dialog.is_none() && d.app().ui.extras.source_preview.is_none());
+    let x = develop(&mut d).unwrap();
+    assert_eq!((x.iso, x.gamma.id(), steps(&mut d)), (Some(1600), "srgb", opened.1 + 1));
+    d.exec("edit.undo", json!({}));
+    assert_eq!((develop(&mut d), steps(&mut d)), (opened.0, opened.1), "one Undo takes the whole dialog back");
+    d.exec("edit.redo", json!({}));
+    assert_eq!(develop(&mut d).unwrap().iso, Some(1600));
+}
+
+/// The Camera Raw panel's buttons and the project's Camera RAW settings: Use Changes copies the
+/// shown clip's settings to every selected clip and Use Settings puts it back to what it records,
+/// both inside the dialog's one undo step (Cancel takes them back); Update RMD without a decoder
+/// says why in the dialog; File ▸ Project Settings ▸ Camera RAW changes what the clips that
+/// decode using the project develop with, a gamma chosen alone bringing its colour space along.
+#[test]
+fn camera_raw_buttons_and_the_project_settings_tab() {
+    use filmcraft_media::{DecodeUsing, RawColorSpace, RawDevelop, RawGamma, RawToneMap};
+    let mut d = Driver::new();
+    let a = add_r3d(&mut d, "A.R3D", "/footage/A.R3D");
+    let b = add_r3d(&mut d, "B.R3D", "/footage/B.R3D");
+    let develop = |d: &mut Driver, id: u64| d.app().session.project.item(filmcraft_engine::project::ItemId(id)).unwrap().as_media().unwrap().interpret.develop;
+    d.exec("clip.sourceSettings", json!({"items": [a], "iso": 1600, "gamma": "srgb", "toneMap": "high"}));
+    let opened = (develop(&mut d, a), develop(&mut d, b), d.app().session.history.undo.len());
+    d.exec("project.select", json!({"items": [a, b]}));
+    d.menu("clip.sourceSettings");
+    d.click("sourceSettings.useChanges");
+    d.frames(2);
+    assert_eq!(develop(&mut d, b), develop(&mut d, a), "B has A's settings");
+    assert_eq!(develop(&mut d, b).map(|x| (x.iso, x.gamma)), Some((Some(1600), RawGamma::Srgb)));
+    d.click("sourceSettings.useSettings");
+    d.frames(2);
+    let x = develop(&mut d, a).unwrap();
+    assert_eq!((x.decode_using, x.iso, x.tone_map), (DecodeUsing::Clip, None, RawToneMap::Clip), "as recorded");
+    assert_eq!(d.app().ui.extras.dialog.clone().unwrap().params["iso"], Value::Null, "the panel shows it");
+    // no decoder (the file does not exist): Update RMD says why, nothing changes
+    d.click("sourceSettings.updateRmd");
+    d.frames(2);
+    let error = d.app().ui.extras.dialog.clone().unwrap().error;
+    assert!(error.contains("RMD not written"), "{error}");
+    d.click("sourceSettings.cancel");
+    assert_eq!((develop(&mut d, a), develop(&mut d, b), d.app().session.history.undo.len()), opened, "Cancel takes it all back");
+    // the project's Camera RAW tab
+    let r = d.menu("file.projectSettings.cameraRaw");
+    assert_eq!(r["dialog"], "projectSettings", "{r}");
+    for key in
+        ["tab.cameraRaw", "cameraRaw.quality", "cameraRaw.colorSpace", "cameraRaw.gamma", "cameraRaw.toneMap", "cameraRaw.kelvin", "cameraRaw.standard", "ok"]
+    {
+        assert!(d.has(&format!("projectSettings.{key}")), "{key} missing: {:?}", d.ok("ui.elements", json!({"prefix": "projectSettings."})));
+    }
+    assert!(!d.has("projectSettings.cameraRaw.decodeUsing"), "Decode Using is a clip setting");
+    let mut cr = d.app().ui.extras.dialog.clone().unwrap().params["cameraRaw"].clone();
+    assert_eq!(cr["gamma"], "bt1886");
+    cr["gamma"] = json!("log3g10");
+    d.ok("ui.set", json!({"menuDialog": {"cameraRaw": cr}}));
+    d.frames(2);
+    assert_eq!(d.app().ui.extras.dialog.clone().unwrap().params["cameraRaw"]["colorSpace"], "redWideGamutRgb", "the gamma brought its colour space");
+    d.click("projectSettings.ok");
+    let st = d.app().session.project.settings.camera_raw;
+    assert_eq!((st.color_space, st.gamma), (RawColorSpace::RedWideGamutRgb, RawGamma::Log3G10));
+    assert_eq!(d.exec("media.colorInfo", json!({"item": b}))["effective"], "log3g10-rwg", "B decodes using the project");
+    assert_eq!(d.exec("media.colorInfo", json!({"item": a}))["effective"], "linear709", "A has settings of its own (sRGB, Rec. 709)");
+    d.exec("edit.undo", json!({}));
+    assert_eq!(d.app().session.project.settings.camera_raw, RawDevelop::STANDARD);
+}
diff --git a/docs/adr/0001-platform-ffi.md b/docs/adr/0001-platform-ffi.md
index bf5a987f46..3460bfb637 100644
--- a/docs/adr/0001-platform-ffi.md
+++ b/docs/adr/0001-platform-ffi.md
@@ -157,3 +157,28 @@ VA-API's. The rules above hold: `libnvcuvid.so.1` and `libcuda.so.1` are loaded
 the one `nvenc::device` already retains for encoding. Unlike VA-API, NVDEC parses the stream
 itself; its callbacks run inside our parse call and never unwind into C (they record the first
 error and return 0, and the decoder then fails over to software like any other backend).
+
+## Addendum: the R3D NE helper's shared frame memory (proposed, with ADR 0002)
+
+The optional `red-sdk` backend ([ADR 0002](0002-optional-vendor-sdk-helpers.md), proposed;
+[r3d-ne.md](../r3d-ne.md)) keeps RED's decoder in a helper process. Its decoded frames reach the
+app through memory the helper creates and the app maps read-only, instead of two copies of every
+frame through a pipe: a named, pagefile-backed section on Windows, a POSIX shared memory object on
+Linux. The rules above hold:
+
+- Only `red::shm` carries `#[allow(unsafe_code)]` (`OpenFileMappingW`, `MapViewOfFile`,
+  `UnmapViewOfFile`, `CloseHandle` through the `windows` crate; `shm_open`, `mmap`,
+  `munmap`, `shm_unlink` through `libc`, an optional dependency of the feature, MIT OR Apache-2.0).
+  The rest of the backend (the protocol, the slot leases, the repack) is safe code.
+- Its surface (used only inside `red`) is safe and the same on both platforms (`Mapping::open`, a bounds-checked
+  `slice`, unmap on drop); every `unsafe` block says why it is sound. On Linux `open` checks the
+  object's size before mapping, since reading a mapping past the end of a file faults.
+- The slices are over memory another process can write. The protocol hands a slot to the app with
+  its reply and takes it back only after the app's `F`; a helper that exits leaves the pages valid
+  (the memory lives while the mapping does); the readers treat every byte as untrusted input and
+  validate it. No Rust invariant depends on the other process keeping to the protocol.
+- The module adds no callbacks into Rust, and nothing of it is compiled without the feature.
+
+There is no software decoder to fall back to (R3D NE has no public specification); without the
+feature, or when the helper cannot start, a clip is unsupported or offline media with a message,
+never a crash.
diff --git a/docs/adr/0002-optional-vendor-sdk-helpers.md b/docs/adr/0002-optional-vendor-sdk-helpers.md
new file mode 100644
index 0000000000..cd6e20bc4c
--- /dev/null
+++ b/docs/adr/0002-optional-vendor-sdk-helpers.md
@@ -0,0 +1,91 @@
+# ADR 0002: optional vendor SDK decoders in a helper process
+
+- **Status:** proposed (for the owner decision asked for in #383; not in effect until accepted)
+- **Issues:** #383 (vendor SDK bridges for camera RAW), #1111 and #1415 (Nikon and RED raw);
+  sibling proposal #1954 (Blackmagic RAW through a runtime loaded at run time)
+- **First user:** the `red-sdk` feature of `crates/platform` ([r3d-ne.md](../r3d-ne.md))
+
+## Context
+
+FilmCraft's rules are pure Rust in the product, clean-room codecs from public specifications,
+permissive dependencies, and `unsafe` only in `crates/platform` for OS media APIs
+([ADR 0001](0001-platform-ffi.md), [AGENTS.md](../../AGENTS.md) §0 item 3, §2). Some camera RAW
+formats have no public specification: R3D and R3D NE (RED, Nikon ZR), BRAW (#290), ProRes RAW
+(#345). Their vendors' SDKs are the only decoders, and decoding them otherwise would mean reverse
+engineering, which the project does not do. Under the current rules those formats cannot be opened
+at all; editors who shoot them transcode first.
+
+#1954 proposes one shape for this: load a vendor's runtime library at run time, in the editor's
+process, through FFI in `crates/platform`. That fits a vendor that ships a runtime with a stable,
+COM-style C ABI and headers under a permissive licence. RED's SDK is different: a C++ API, a static
+library that has to be linked at build time, runtime libraries beside it, and an agreement that
+keeps its headers and libraries out of anyone else's distribution. Linking it into the editor would
+mean C++ FFI into the app process, and the SDK's threads, memory and crashes inside it.
+
+## Decision (proposed)
+
+An optional decoder built on a vendor SDK is acceptable when all of the following hold. Anything
+that does not meet them needs a decision of its own.
+
+1. **Off by default.** It is a Cargo feature named after the SDK (`red-sdk`), never in any crate's
+   default features, never enabled by CI, `cargo xtask ci`, the release packaging or any official
+   artefact. Without it none of the backend's code is compiled.
+2. **Built only against an SDK the user supplies.** An environment variable names the user's own
+   extraction of the SDK (`RED_R3D_SDK_DIR`); without it the feature build stops with a message
+   saying so, and on an unsupported target with a message naming the supported ones. Nothing is
+   downloaded, and no SDK file (headers, libraries, documentation, sample code, LUTs) and no camera
+   footage is ever committed or bundled.
+3. **The vendor's code runs in a separate helper process.** The adapter is an original program
+   compiled against the SDK by the crate's build script (in the SDK's own language: one C++ source
+   file for RED's, the only part of the repository that is not Rust, and compiled only with the
+   feature). The editor never links or loads the vendor's libraries. It starts the helper, talks
+   to it over a versioned, framed protocol that both sides check, and treats every reply as hostile
+   input under the never-crash rules: bounded sizes and counts, deadlines, a watchdog. A crash, hang
+   or refusal in the helper becomes a media error for the frames concerned and a helper started
+   again, never a crash or a frozen window.
+4. **`unsafe` stays under ADR 0001.** The backend lives in `crates/platform` (layer L5). Its only
+   `unsafe` may be one module that maps the helper's decoded frames read-only (shared memory, through
+   the official `windows` bindings or `libc`), with ADR 0001's containment: `#[allow(unsafe_code)]`
+   on that module alone, a `// SAFETY:` comment on every block, a safe API, no callbacks into Rust,
+   and no Rust invariant that depends on the other process keeping to the protocol. Everything else,
+   the protocol included, is safe Rust.
+5. **The default build is unchanged.** `cargo fmt`, `cargo clippy --workspace --all-targets -D
+   warnings`, the workspace tests, `cargo xtask layers`, `assets` and `wasm` pass and behave the same
+   without the feature; no default-build test needs the SDK; the layering is unchanged. A file the
+   feature decodes is still recognised without it and reported as unsupported with a message naming
+   the feature, without reading the file into memory. Tests that need the SDK compile only with the
+   feature, and tests that need footage are `#[ignore]` behind an environment variable naming the
+   user's own clip.
+6. **What the editor builds on stays generic.** Model changes the backend needs (a development
+   settings model, a frame format, a playback hint) are written for any camera RAW source, tested
+   without the SDK, and useful to a public-spec decoder such as DNG or CinemaDNG.
+7. **Evidence comes with the change.** CI cannot build the feature (it has no SDK), so a pull request
+   that touches such a backend carries its author's runs of the feature's clippy, unit tests and
+   real-clip tests on every platform it supports, and the backend's document records measurements
+   by hardware class.
+8. **The project ships no SDK-enabled build.** Publishing the adapter's source is reviewed against
+   the vendor's agreement in the backend's document. Distributing a build with the SDK linked in
+   is a separate decision, which would have to meet the vendor's conditions (for RED: an end-user
+   licence for the helper and RED's libraries, a private install directory, notices kept, no use of
+   RED's marks). Until such a decision, users build the feature locally against their own SDK.
+
+## Alternatives considered
+
+- **Public-spec formats only.** Camera RAW without a specification stays out of FilmCraft; such
+  backends live in forks. The simplest rule, and the default if this record is rejected.
+- **A runtime loaded in the editor's process** (#1954). Suits a vendor runtime with a C ABI and
+  permissively licensed headers, and avoids a process hop. For a C++ SDK linked at build time it
+  would put C++ FFI and the vendor's threads and crashes in the editor. The two shapes can coexist;
+  the owners may prefer one pattern for every vendor SDK, and this record would then follow it.
+- **An out-of-tree plug-in protocol.** The cleanest separation, but it asks the project to design and
+  keep a stable decoder-plug-in interface before there is a second user of it.
+
+## Consequences
+
+- People who shoot R3D NE can build FilmCraft that edits it natively, with the SDK they downloaded;
+  everyone else is unaffected.
+- The repository gains one non-Rust source file and a build script path behind the feature, and the
+  platform crate one more `unsafe` module. Changes to them need FFI-level review.
+- CI cannot verify the feature. Reviews rest on the author's runs, and a regression in it can go
+  unnoticed until someone with the SDK builds it.
+- Users of the feature build from source; the project's releases do not change.
diff --git a/docs/architecture.md b/docs/architecture.md
index 3056a7e371..8edbd2f015 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -343,8 +343,15 @@ file ──► codecs (MP4/MOV, MKV, audio)        demux + decode, GOP-aware see
   `filmcraft_media::cancel`) jobs for frames the playhead has passed; a new on-screen frame
   replaces the one asked for before (scrubbing); Stop cancels the prefetch. When frames cost more
   than the workers can render in real time (CPU effects), it starts only frames that can still be
-  on time and spaces them evenly (`playback_plan`). When decoding is what falls behind, each job
-  tells its sources which frames are already late (`filmcraft_media::cancel::with_catch_up`: the
+  on time and spaces them evenly (`playback_plan`). A source whose frames decode independently
+  and slower than the sequence rate (camera RAW through a pipelined decoder; it says so with
+  `MediaSource::frame_delivery`, and reports its own throughput when it measures one) gets a plan
+  of its own (`throughput_plan`): m of every n frames (three of four, every third…), what the
+  source delivers against the sequence rate and what the workers sustain (each holds a job for
+  the source's latency), chosen with hysteresis and anchored so the set does not shift with the
+  playhead, started its latency plus slack ahead, and a running job it has in hand is left to
+  finish a few frames behind the playhead (`LATE_GRACE`) and shown late rather than cancelled.
+  When decoding is what falls behind, each job tells its sources which frames are already late (`filmcraft_media::cancel::with_catch_up`: the
   playhead's distance, two seconds for a scrubbed-to frame), and the GOP cache skips
   non-reference pictures of late frames on the way (`VideoDecoder::is_disposable`, H.264
   `nal_ref_idc` 0 and HEVC sub-layer non-reference pictures; the wanted frame decodes exactly as
diff --git a/docs/codec-parity.md b/docs/codec-parity.md
index d3a79679d8..9a9deb0715 100644
--- a/docs/codec-parity.md
+++ b/docs/codec-parity.md
@@ -31,7 +31,8 @@ contents). Percent is ready-for-real-work for that codec: depth, profiles, speed
 | MPEG-4 Part 2 | decode | none | none | — | 0% | 10–20 h |
 | WMV / VC-1 | decode, encode on Windows | none | none | — | 0% | 20–35 h |
 | CinemaDNG / DNG | decode | own decoder (#1065): uncompressed and lossless JPEG raw, Bayer and LinearRaw, DNG colour model to linear Rec. 709 floats; folders of `.dng` frames as one clip with the CinemaDNG frame rate and time code; **no** lossy / JPEG XL DNG, MXF-wrapped CinemaDNG, opcode lists or Raw settings UI | — | raw samples bit-exact (and identical to LibRaw's on our synthetic files), processing against hand-computed values | 60% | 10–20 h |
-| Camera RAW: R3D (incl. R3D NE), ARRIRAW, Sony RAW / X-OCN, Canon RAW, BRAW (plug-in) | decode (vendor SDKs) | none (#383) | — | — | 0% | 50–90 h for the ones with public specs; vendor-SDK-only formats need an owner decision |
+| Nikon ZR R3D NE | decode (vendor SDK) | optional `red-sdk` build feature, Windows x64 / Linux x64, against an SDK the user supplies (proposed, #383, [ADR 0002](adr/0002-optional-vendor-sdk-helpers.md)): RED's SDK in a helper process, IPP2 development and the Camera Raw panel (Source Settings, project Camera RAW settings, RMD sidecars), audio; default builds recognise `.r3d` and say which feature they need ([r3d-ne.md](r3d-ne.md)) | — | real-clip tests opt-in (`RED_R3D_TEST_CLIP`), the protocol and the colour pipeline in unit tests | 60% with the feature (6K plays a planned subset at half resolution; export decode-bound), 0% in default builds | classic spanned R3D, macOS |
+| Camera RAW: classic RED R3D, ARRIRAW, Sony RAW / X-OCN, Canon RAW, BRAW (plug-in) | decode (vendor SDKs) | none (#383) | — | — | 0% | 50–90 h for the ones with public specs; vendor-SDK-only formats need an owner decision |
 | GIF (animated) | encode | decode as still | encode | — | 80% | — |
 | Uncompressed / v210 / RGB | decode + encode (AVI, MOV) | partial (MOV PCM-style video not verified) | none | — | 30% | 8–15 h |
 
diff --git a/docs/control-protocol.md b/docs/control-protocol.md
index 1dfd6498bf..d4a1acddfc 100644
--- a/docs/control-protocol.md
+++ b/docs/control-protocol.md
@@ -171,7 +171,38 @@ the result's `sourceInClamped: [ids]` (present with `keepSourceIn`) names the cl
 
 **Colour** (`crates/engine/src/color.rs`): `sequence.colorSettings {workingSpace: rec709|rec2100-pq|
 rec2100-hlg, wideGamut, autoToneMap}`, `clip.interpretFootage {items?, colorSpace?: auto|, pixelAspect?: [num, den]|file}`,
-`color.spaces`, `media.colorInfo {item}`; LUTs: `lut.import {path, name?}`, `lut.list`,
+`color.spaces`, `media.colorInfo {item}`; camera RAW development (the Camera Raw panel,
+[r3d-ne.md § Source Settings](r3d-ne.md#source-settings-the-camera-raw-panel)):
+`clip.sourceSettings {item?|items?, decodeUsing?: clip|project, quality?: project|fullPremium|halfPremium|halfGood|quarterGood|eighthGood|sixteenthGood,
+colorSpace?: rec709|rec2020|p3d65|redWideGamutRgb, gamma?: linear|bt1886|srgb|log3g10|st2084|hlg, toneMap?: clip|none|low|medium|high,
+rolloff?: clip|none|hard|medium|soft|verySoft, hdrPeakNits?: n|null, chromaNoiseReduction?: bool|null, flashingPixel?: off|mild|medium|strong|null,
+metadataCurves?: bool, creativeLut?: bool, cdl?: bool, iso?: n|null, kelvin?: n|null, tint?: f|null, exposure?: f|null, asShot?: bool,
+useChanges?: bool, useSettings?: bool, updateRmd?: bool, preview?: bool, commit?: bool, revert?: bool}` reports the panel's
+controls (`settings: [{id, kind: choice|iso|number|bool|fixed, label, value, enabled, column: 1–3, note?, asShot?, recorded?,
+options?|min/max/step/decimals/suffix}]`), `develop` (the settings in effect, `decodeUsing` included), `own` (the clip's own),
+`project` (the project's), `asShot` and `clip` (what the clip, or its RMD sidecar, records: recorded output choices, LUT, CDL,
+curves, `hdrx`, `rmd` path when one is in effect) from the open decoder, and with parameters sets them on the selected camera
+RAW clips (one undo step). A development setting given for a clip that decodes using the project's settings gives it settings
+of its own, starting from the project's; `decodeUsing: project` drops them. A colour space or gamma given alone brings a
+fitting other one along; both given must go together (display curves with Rec. 709 / Rec. 2020 / P3 D65, Log3G10 with
+REDWideGamutRGB, ST 2084 and HLG with Rec. 2020, linear with any). ISO is one of `filmcraft_media::RAW_ISO_VALUES`
+(50–25600), `kelvin` 1700–100000, `tint` −100..100, `exposure` −8..8 stops (two decimals kept), `hdrPeakNits` 100–10000;
+`null` puts one back to as recorded, `asShot: true` the four RAW adjustments; anything else is refused and nothing changes.
+`useChanges` copies the first clip's settings to every other clip in `items`, `useSettings` puts the clips back to what they
+record, both in the call's undo step; `updateRmd` writes each clip's settings into its RMD sidecar beside it (not an undo
+step; `rmdWritten: [{item, path}]`, `rmdErrors`). The colour interpretation follows the development (`media.colorInfo`'s
+`effective`; Rec. 709 needs none). `file.projectSettings.cameraRaw {quality?, colorSpace?, gamma?, …, standard?: bool}` sets
+the project's Camera RAW settings (one undo step; `standard: true` = Rec. 709, BT.1886) and reports them with `clips`, the
+number of RAW clips that decode using them. The dialog's ids are `sourceSettings.` for every control (with `.asShot`
+buttons once a value is set), `sourceSettings.updateRmd|useChanges|useSettings`, `sourceSettings.ok|cancel`; Project
+Settings' tab `projectSettings.tab.cameraRaw` has `projectSettings.cameraRaw.` and `projectSettings.cameraRaw.standard`.
+The dialog previews live: every change is applied at once with `preview: true`, which merges all the previews of one dialog
+into one undo step (the result's `preview` says one is open); OK sends the final values with `commit: true` (the step stays
+as one Undo), Cancel and Escape send `revert: true` (the clips return to the values the dialog opened with and the history
+is as before; `reverted` says whether a step was open). The same flow works over the control channel, `items` pinning the
+clips; a hostile value in a preview is refused with an error and applies nothing. Other media report `settings: []`, and
+a build without the `red-sdk` feature reports `.r3d` files as unsupported media whose message names the feature. LUTs:
+`lut.import {path, name?}`, `lut.list`,
 `lut.remove {id}`, `lut.export {lut, path, format?}`, `lumetri.setInputLut` / `lumetri.setLook
 {clip?, lut: lib:|builtin:|"", path?}`, `lumetri.setSection {clip?, section, on?}`,
 `lumetri.applyMatch {clip?, referenceTime|referenceFrame|referenceTimecode, faceDetection?}`. Without
diff --git a/docs/r3d-ne.md b/docs/r3d-ne.md
new file mode 100644
index 0000000000..3ec084e441
--- /dev/null
+++ b/docs/r3d-ne.md
@@ -0,0 +1,388 @@
+# Nikon ZR R3D NE (optional, through RED's SDK)
+
+**Status:** optional, off by default, Windows x64 and Linux x64. Proposed under
+[ADR 0002](adr/0002-optional-vendor-sdk-helpers.md) for #383; nothing here applies until the
+project owners accept it.
+
+R3D NE, the RAW format of Nikon's ZR camera, has no public specification. RED's R3D SDK is the
+only decoder. FilmCraft can use it when you supply the SDK yourself and build with the `red-sdk`
+feature: the SDK is compiled into a small helper program, `filmcraft-red-helper`, that runs as a
+separate process; the editor itself stays pure, safe Rust apart from one module that maps the
+helper's frame memory read-only. Default builds do not need, link, download or ship anything of
+RED's, and a `.r3d` file opened in them reports which build feature it needs.
+
+This is not a clean-room implementation of REDCODE. Read [Licensing](#licensing) before you pass
+an SDK-enabled build to anyone.
+
+Contents: [Setup](#setup) · [Behaviour](#behaviour) · [Source Settings](#source-settings-the-camera-raw-panel) ·
+[Limits](#limits) · [The helper protocol](#the-helper-protocol-fcrd9) · [Tests](#tests) ·
+[Measurements](#measurements) · [Licensing](#licensing)
+
+## Setup
+
+You need Rust 1.95 or newer, a C++17 compiler (the Visual Studio C++ build tools on Windows,
+`g++` and `uuid-dev` on Linux), and RED R3D SDK 9.2.1, downloaded from RED under RED's own
+agreement and extracted somewhere outside the source tree. `RED_R3D_SDK_DIR` names that folder;
+it must contain `Include`, `Lib` and `Redistributable`.
+
+```bash
+export RED_R3D_SDK_DIR=
+cargo run -p filmcraft --features red-sdk
+# inspect a clip without a window
+cargo run -p filmcraft-cli --features red-sdk -- probe .R3D
+```
+
+(PowerShell: `$env:RED_R3D_SDK_DIR = ''`.)
+
+`crates/platform/build.rs` compiles `crates/platform/src/red/helper.cpp`, an original C++ program,
+against the SDK's headers and static library, and copies the SDK's four runtime libraries
+(`REDR3D`, `REDDecoder`, `REDCuda`, `REDOpenCL`) beside the helper in Cargo's build output.
+Without `RED_R3D_SDK_DIR` the feature build stops with a message saying so; on any other target it
+stops with "red-sdk currently supports Windows x64 (MSVC) and Linux x64 (GNU) only". Nothing of the
+SDK is ever committed: no headers, libraries, documentation, sample code or footage.
+
+| | Windows x64 (`x86_64-pc-windows-msvc`) | Linux x64 (`x86_64-unknown-linux-gnu`) |
+| --- | --- | --- |
+| Compiler | MSVC `/std:c++17 /EHsc /MD /O2` | the system C++ compiler, `-std=c++17 -O2 -fPIC -pthread` |
+| SDK library | `Lib/win64/R3DSDK-2017MD.lib` | `Lib/linux64/libR3DSDKPIC-cpp11.a`, `-ldl -luuid` |
+| Runtime libraries | `Redistributable/win/*-x64.dll` | `Redistributable/linux/*-x64.so` |
+| Helper start | hidden (`CREATE_NO_WINDOW`) | runtime directory first on `LD_LIBRARY_PATH`; SIGPIPE ignored |
+| Reply pipe | a named pipe the helper creates | the helper's stdout |
+| Frame memory | a named, pagefile-backed section | a POSIX shared memory object (`/dev/shm`) |
+
+On Linux, SDK 9.2.1's `FinalizeSdk()` faults inside `REDR3D-x64.so` once the CUDA decoder has
+processed a frame (deterministic; CPU-only sessions are fine), so a GPU helper on Linux skips it at
+exit. The process ends right after.
+
+**Finding the helper.** The app looks for the helper beside its own executable, then in its Cargo
+build directory. `FILMCRAFT_R3D_HELPER` names another helper and `RED_R3D_LIBRARY_PATH` another
+directory for the runtime libraries; both select executable code, so point them only at files you
+trust. Keep the SDK's header, static library and runtime libraries at the same version, and
+rebuild the app and the helper together (see [the protocol](#the-helper-protocol-fcrd9)).
+
+**GPU.** Image processing runs on the GPU: CUDA when available, else OpenCL. RED recommends 12 GB
+of video memory; 10 GB works at half resolution with the default pools. `FILMCRAFT_R3D_PROCESSING=cpu`
+forces the CPU for comparisons; it is a diagnostic, not a setting.
+
+**Logs.** The app log records each helper's start (process id, GPU or CPU, path), its readiness
+(SDK start-up time, processing device, slots, threads), every failure with its reason, idle
+closes and each exit status. The helper's own stderr is forwarded line by line as warnings.
+`FILMCRAFT_R3D_TIMING=1` adds per-frame timings (submit to callback, reply write, slot hold, the
+app's round trip and conversion) at debug level (`RUST_LOG=filmcraft_platform::red=debug`). More
+than two helper processes alive at once is a bug.
+
+## Behaviour
+
+**Import.** `.r3d` files (any case) import through every import path: File ▸ Import, the Media
+Browser, drag and drop, the CLI and the control channel. Each Nikon file is one clip; consecutive
+file numbers are never joined. Opening reads the clip through a local file path without reading the
+file into memory; memory-only and browser readers answer "unsupported". Metadata (size, rational
+frame rate, duration, start timecode, audio layout, the recorded lens squeeze, the as-shot
+development) comes from the helper and is range-checked before the clip is accepted.
+
+**Picture.** RED's IPP2 pipeline develops the frame on the GPU. The app receives linear light as
+RGBA half floats (`PixelData::RgbaF16`), which the GPU compositor uploads as they are and the CPU
+paths widen to f32 on demand. Seeking addresses frames directly; times before the first frame clamp
+to it and times past the end to the last frame.
+
+**Sound.** Native 32-bit float audio keeps samples outside ±1; integer audio arrives as the SDK's
+big-endian, MSB-aligned samples. Both become planar f32, resampled linearly when the sequence rate
+differs. Requests outside the clip are silence.
+
+**Preview sizes.** A preview decodes at the smallest of full, half, quarter, eighth and sixteenth
+resolution that still covers it, never larger than the clip's decode quality allows; full-size
+requests (an export, a monitor at 100 %) always decode at full resolution. The SDK rounds a
+dimension that does not divide down: a 6048×3402 clip decodes at 3024×1701, 1512×850, 756×425 and
+378×212. That rule was measured (decoding into an oversized buffer filled with a sentinel and
+scanning what changed, on the CPU and GPU paths), because the headers do not state it and the
+helper's buffer and the app's pixel count must agree to the byte:
+
+| Mode | Exact fraction | Written |
+| --- | --- | --- |
+| Half (Premium and Good) | 3024×1701 | 3024×1701 |
+| Quarter Good | 1512×850.5 | 1512×850 |
+| Eighth Good | 756×425.25 | 756×425 |
+| Sixteenth Good | 378×212.625 | 378×212 |
+
+Rows are packed at `floor(W / d) × 6` bytes (the GPU job's row pitch is ignored when larger), and
+nothing is written past the frame. The helper sizes each buffer that way and writes a half-float
+NaN into the last pixel before each decode, so a decode that wrote less is rejected by the app's
+non-finite check instead of shown. Re-run this measurement when the SDK version changes.
+
+**Two helpers serve every clip.** The SDK runs in at most two processes, whatever the number of
+clips:
+
+- The **GPU helper** decodes the frames monitors, playback and export ask for, into five shared
+  frame slots. It starts on the first such frame, and a clip is opened in it (~6 ms inside the
+  running SDK) the first time that clip is asked for a frame. A cut in a timeline of a hundred clips
+  is a clip already open, not a process start.
+- The **CPU helper** serves audio, thumbnails (Project panel, Media Browser, timeline), the metadata
+  probe at import, and frames while the GPU helper is unavailable. It runs four threads fed from
+  two queues, audio first, so a thumbnail never makes the sound wait.
+- **Still frames** (playback stopped: a scrub, a click on the ruler, a paused monitor) at a quarter
+  of the size or smaller are decoded by the CPU helper, which answers an eighth in ~35 ms where one
+  frame through the GPU pipeline takes ~160–200 ms at any size. Such a frame also starts or keeps the
+  GPU helper warm, so playback after a scrub starts without dropped frames. Playback, export and
+  full or half still frames stay on the GPU. The hint comes from the frame server
+  (`filmcraft_media::cancel::with_still`); other sources ignore it.
+- A GPU helper that fails to become ready (no CUDA device, a full GPU, a missing runtime library)
+  gets one warning naming the reason; frames then come from the CPU helper and the next GPU start
+  waits for a back-off that doubles from two seconds to a minute and clears on success.
+- A helper that fails or exits is started again by the next request, and the frames that were in
+  flight are asked again of the new helper once. A helper idle for two minutes (GPU) or one minute
+  (CPU) is closed and gives its memory back; an orderly close lets it exit by itself with status 0,
+  so the log tells a failed helper from a closed one.
+- An error answers its own request only (an unreadable clip, a refused setting); the helper serves
+  the other clips on. A native crash, a closed pipe, an invalid reply or a request unanswered for
+  120 seconds fails the helper and becomes a media error for the requests in flight, which the media
+  pool shows as damaged media until frames arrive again. Quitting leaves no helper process and, on
+  Linux, nothing under `/dev/shm`.
+
+**Playback.** The source tells the frame server that its frames decode independently and what rate
+the helper sustains (`MediaSource::frame_delivery`), and the server plans which frames to ask for
+(`throughput_plan`, [architecture.md](architecture.md)): m of every n, the densest the measured
+capacity covers, so a clip that decodes a little slower than the sequence rate plays an even subset
+of its frames instead of freezing. That planner is generic and lives in its own change.
+
+## Source Settings: the Camera Raw panel
+
+Clip ▸ Source Settings… on a camera RAW clip is a Camera Raw panel in three columns
+(`clip.sourceSettings`; every control has the automation id `sourceSettings.`). File ▸ Project
+Settings ▸ Camera RAW… holds the same controls for the project (`file.projectSettings.cameraRaw`),
+without Decode Using and with a Standard button. The commands and their parameters are in
+[control-protocol.md](control-protocol.md).
+
+**Decode Using.** A clip develops with the project's Camera RAW settings (Decode Using ▸ Project; its
+development controls are greyed out) or with settings of its own (Decode Using ▸ Clip, starting from
+the project's). A new project's settings are the standard: IPP2 into Rec. 709 with the BT.1886 curve
+and the SDK's default output tone map and highlight roll-off (Medium and Soft in SDK 9.2.1, which the
+helper reports and a real-clip test checks), as shot. The project stores a clip's settings only once
+it has its own (`interpret.develop`) and the project's in `settings.camera_raw` (project schema v16;
+a project from before reads as scene-linear Rec. 709, what such clips decoded with, so nothing
+changes picture on opening).
+
+**Column 1: decode and colour pipeline.**
+
+- *Decode Quality*: Use project setting, Full Res. Premium, Half Res. Premium, Half Res. Good, Quarter /
+  Eighth / Sixteenth Res. Good. It caps the size a preview decodes at (a half-size preview at Quarter
+  decodes a quarter and is scaled up); a full-size request always decodes at full resolution,
+  premium, so the setting never lowers an export.
+- *Color Science*: IPP2 (shown, fixed: the SDK develops R3D NE with IPP2 only).
+- *Color Space* and *Gamma Curve*: Rec. 709, Rec. 2020, P3 D65 and REDWideGamutRGB; Linear, BT.1886,
+  sRGB, Log3G10, ST 2084 (PQ) and Hybrid Log-Gamma: the IPP2 values the compositor can interpret.
+  Display curves go with the display spaces, Log3G10 with REDWideGamutRGB, ST 2084 and HLG with
+  Rec. 2020, Linear with any; choosing one that does not fit the other brings a fitting one along.
+- *Blend Type* and *Blend Bias* (HDRx): greyed out with the reason. R3D NE records no HDRx track.
+- *Apply Metadata Curves*, *Apply Creative LUT* (greyed out on a clip that carries none) and *Apply
+  CDL*: what the clip or its RMD sidecar records. They act on a display or log curve only; the SDK's
+  linear half-float output ignores curves and CDL.
+
+**Column 2.** *ISO* (one of the SDK's 28 values, 50–25600), *Output Tone Map* and *Highlight Roll Off*
+(for a display curve), *HDR Peak Nits* (ST 2084 only, 100–10000 within the SDK's limits), *Chroma
+Noise Reduction* and *Flashing Pixel Adjust*.
+
+**Column 3.** *Color Temp* (1700–100000 K), *Tint* (−100..100) and *Exposure Adjust* (−8..8 stops, two
+decimals). With ISO these are the RAW adjustments IPP2 applies before development. Each shows the
+clip's recorded value (as shot) until set, and its As Shot button puts it back. A value outside the
+SDK's lists or limits is refused, not clamped. The colour effect's own temperature, tint and exposure
+still act after development, in the working space.
+
+**Buttons.** *Update RMD* writes the clip's settings into its RMD sidecar beside the clip, and only
+when pressed (opening, previewing or saving never writes one); it is not an undo step, since it
+changes a file. *Use Changes* applies this clip's settings to every selected clip. *Use Settings* puts
+the clip back to what it records. When a clip has an RMD sidecar, its recorded values are the
+sidecar's, at import too, and the panel names the sidecar in effect.
+
+**Live preview.** Every change is applied while the dialog is open, so the monitors develop the
+current frame as a control is dragged; OK keeps the values, Cancel or Escape puts back the ones the
+dialog opened with, and one dialog is one undo step however many changes it made. During a drag at
+most one change is outstanding: the next waits until the Program monitor shows the last (or a second
+passes). The dialog does this through the command alone (`preview`, `commit`, `revert`), so the
+control channel drives the same flow. A change reaches the open helper through the media pool; no
+second decoder is started.
+
+**Colour pipeline.** Every frame the source hands over is linear light in the developed primaries:
+the Linear curve as the SDK's half floats; every other curve decodes as 16-bit code values (the SDK's
+half-float output ignores the gamma curve and with it the IPP2 output transform), which the app
+linearises with the compositor's own curve for that gamma into half floats. The compositor's
+interpretation follows from the development and is derived, not stored
+(`Project::color_space_override`): Rec. 709 frames are plain linear Rec. 709 (the fast path);
+Rec. 2020 and P3 D65 are those spaces; ST 2084 and HLG are Rec. 2100 PQ and HLG; REDWideGamutRGB is
+RED Log3G10 / REDWideGamutRGB (`filmcraft_color`, from RED's published white paper). An Interpret
+Footage choice of your own wins. A test takes a mid-grey patch through every offered curve and space,
+in a Rec. 709 and a Rec. 2100 PQ sequence, and gets 18 % back (and a BT.1886 grey reaches the monitor
+as the code value the decoder wrote). Proxies keep the look they were made with.
+
+## Limits
+
+- Windows x64 and Linux x64 only; no macOS, no browser build. Single-file Nikon R3D NE only: classic
+  RED spanned clips (`.RDC` folders), N-RAW (`.NEV`), HDRx blending and R3D encoding are not
+  implemented.
+- R3D NE decompresses on the CPU (SDK 9.2.1 has no GPU decompression for it; the GPU does the image
+  processing), so throughput follows the CPU, and anything else using the CPU, the editor's own
+  threads included, shows up in it. 6K is not realtime at full or half resolution; quarter
+  resolution and proxies are (Clip ▸ Proxy ▸ Create Proxies, then View ▸ Toggle Proxies; exports still
+  read the original).
+- Export of 6K is decode-bound (about 14–15 frames per second rendering 6K to 1080p on a current
+  desktop GPU): the export path asks for one frame at a time.
+- Only the colour spaces and curves the compositor interprets are offered (no Log3G12, REDlogFilm,
+  gamma 2.2 / 2.6 or the legacy curves). Nikon clips carry no creative LUT; import a `.cube` for a
+  look.
+- Bounds: 8192 pixels per side, 33,554,432 pixels per frame, 16 audio channels, sample rates up to
+  384 kHz, 1,048,576 audio samples per request, 64 queued video requests in the helper; non-finite
+  decoded values are rejected. Callers wait (cancellably) for a free slot; a wait of 120 seconds fails
+  that request.
+
+## The helper protocol (FCRD9)
+
+The app and the helper speak one protocol version, FCRD9: the first field of the helper's start-up
+reply and of every clip's metadata. Each side checks the other's; a helper from another build is
+refused with an error naming the mismatch, and a request with the wrong field count is refused by the
+helper. The rules below are the contract both sides test.
+
+**Process.** The helper takes three arguments: the SDK runtime directory, the name of its reply pipe
+(which also names its frame memory; letters, digits and dashes, at most 64, chosen by the app from its
+process id, a serial and a random nonce) and the size of a frame slot in bytes (0 for the CPU helper;
+the GPU helper starts with a 6K frame's size and the app starts a helper with larger slots for a
+larger clip). Its start-up reply is `FCRD9   ` and its processing
+device.
+
+**Requests** arrive on its stdin, one per line:
+
+| Request | Meaning |
+| --- | --- |
+| `O   ` | open the file at `path` (the rest of the line, UTF-8) as handle `clip`; answers its metadata or an error |
+| `C ` | close it (no reply; a frame in flight keeps it until written) |
+| `V                 ` | decode a frame with those development settings |
+| `W    … ` | write those settings into the clip's RMD sidecar, read the clip's settings again and answer its metadata (CPU helper only, on Update RMD) |
+| `A    ` | decode audio |
+| `F ` | the app has finished with a shared frame slot |
+
+Request ids are nonzero, increasing `u64`s; clip handles are nonzero and unique across both helpers.
+The development fields are small integers the helper range-checks against the SDK's limits (anything
+else, or a field too many or too few, is refused and never applied): `half` 0 premium, 1 good; `space`
+0 Rec. 709, 1 Rec. 2020, 2 P3 D65, 3 REDWideGamutRGB; `gamma` 0 linear (half floats), 1 BT.1886,
+2 sRGB, 3 Log3G10, 4 ST 2084, 5 HLG (16-bit code values), and the two must go together as
+`RawDevelop::supports` says; `tonemap`, `rolloff`, `peak` and `flashing` 0 for as recorded, else a
+value; `options` a flag word (chroma noise reduction set and its value, metadata curves, creative LUT,
+CDL); `raw` a flag word saying which of ISO, Kelvin, tint and exposure are set (a clear bit means the
+recorded value and its field must be 0), `tint` and `exposure` in hundredths. Each request develops
+its own copy of the clip's recorded settings, held with its slot until the frame is written, so
+requests in flight with different settings or clips never share one.
+
+**Replies** go down the reply pipe, each with a 24-byte little-endian header: status `u32`, request
+id `u64`, payload length `u64`, slot `u32`.
+
+- Status 0: exactly `length` payload bytes follow (metadata, audio, and video from a CPU-only helper).
+- Status 1: an error message follows; it answers that request only.
+- Status 2: nothing follows; the frame is the first `length` bytes of shared slot `slot`.
+- Id 0 after start-up is a fatal error of the helper's own, after which it exits.
+
+Replies may arrive out of order; the app routes them by id. A slot or length beyond the declared
+geometry, a slot reply from a helper with no slots, or an unknown or duplicate id fails the helper. A
+clip's metadata begins `FCRD9`, then its dimensions, timing and audio layout and the helper's slot
+count and slot size; then a timecode line, a backend line, the as-shot line (`ISO Kelvin Tint
+ExposureAdjust`, from the RMD sidecar when one is present), a line of 28 numbers of what else the clip
+records (recorded colour space, gamma, tone map and roll-off; HDR peak; noise reduction; flashing
+pixel; creative LUT; CDL; curves; video track count; whether an RMD is in effect; the SDK's HDR peak
+range and default tone map and roll-off; the CDL values), the RMD path, and the recorded lens squeeze.
+The app range-checks every number and refuses the clip otherwise.
+
+**Shared frames.** A GPU helper creates its frame memory before its first reply: `slots × slot_bytes`,
+where a slot holds a full-resolution frame (width × height × 6 bytes) rounded up to whole 4 KiB pages,
+created exclusively and never resized. The SDK decodes straight into a slot. The app maps the memory
+read-only, exactly as large as the metadata says (on Linux it checks the object's size first, since a
+mapping past the end of a file faults when read), and repacks each frame straight out of its slot.
+A slot is the SDK's from submission to its callback, the app's from the reply to its `F`, and free in
+between. The app holds each slot reply as a lease whose drop sends `F`, wherever the reply ends up (a
+cancelled caller's dropped channel included), so a cancelled frame is completed and discarded, never
+mistaken for another. A slot the app has held for 120 seconds, or an `F` for a slot that is not the
+app's, is a fatal helper error. On Linux the app removes the object's name once it holds the mapping,
+and on every failed start-up, so nothing is left under `/dev/shm`.
+
+**The app side.** The app keeps at most the helper's slot count plus one video request outstanding;
+further callers wait for a slot, cancellably, so a frame playback has moved past is never sent. It
+holds no source lock and not the helper's stdin while it waits for a reply or converts samples.
+Every submitted request keeps a 120-second deadline after cancellation; a watchdog kills a stalled
+helper and wakes the remaining callers. The frame memory is the backend's only `unsafe` code
+(`red::shm`, under [ADR 0001](adr/0001-platform-ffi.md)'s rules): a safe API (open, bounds-checked
+slices, unmap on drop) over memory another process can write, read only as bytes that are validated
+before use, so no Rust invariant depends on the helper keeping to the protocol.
+
+**Knobs** for measurement and small GPUs, not user settings: `FILMCRAFT_R3D_SLOTS` (1–8, default 5),
+`FILMCRAFT_R3D_GPU_FRAMES` (the SDK's GPU streams, 1–8, default 3), `FILMCRAFT_R3D_HOST_POOL_MB` and
+`FILMCRAFT_R3D_GPU_POOL_MB` (the SDK's pool limits, default 8192 and 6144; 4096 for a GPU that others
+already fill), `FILMCRAFT_R3D_CPU_THREADS` (1–16, default 4) and `FILMCRAFT_R3D_STILL_CPU_DIVISOR`
+(0, 1, 2, 4, 8 or 16, default 4: the smallest still frame the CPU helper decodes; 0 keeps them on the
+GPU). The SDK's image-thread count stays at its default: one image thread serialises the whole SDK
+pipeline however many frames are submitted.
+
+## Tests
+
+Default builds compile none of the backend; their tests cover what it builds on (half-float frames
+against every bit pattern, Log3G10 against RED's published values, `.r3d` recognition without
+reading the file, the Camera RAW model and commands with stand-in sources, the mid-grey round trip
+through the compositor, the panel headless). With the feature, the backend's unit tests need no
+footage (framing, metadata, the rounding rule, repack and linearisation, PCM, dispatch, deadlines,
+back-off, the shared memory). The real-clip tests are `#[ignore]` unless `RED_R3D_TEST_CLIP` names a
+clip of your own; they read it in place and write only into a temporary folder. Several count
+processes, so run them alone; the ones that start the GPU helper need an NVIDIA GPU:
+
+```bash
+cargo test -p filmcraft-platform --features red-sdk --lib red::
+RED_R3D_TEST_CLIP=.R3D cargo test -p filmcraft-platform --features red-sdk --lib red:: -- --include-ignored --test-threads=1
+```
+
+CI cannot build the feature, so a change to this backend carries its author's runs on both platforms.
+
+## Measurements
+
+Two hardware classes: **A** a 16-core desktop CPU with a 16 GB current-generation NVIDIA GPU
+(Windows 11), **B** a 16-core desktop CPU with a 10 GB previous-generation NVIDIA GPU (Ubuntu 24.04);
+Nikon ZR clips at 6048×3402 and 3984×2240, release builds.
+
+| | A | B |
+| --- | --- | --- |
+| SDK, 6K at half resolution, 1 / 4 / 5 frames in flight | 4.2 / 24.3 / – fps | 4.8 / 24.4 / 25.1 fps |
+| Playback at half: 6K 23.976p | every frame | every frame |
+| Playback at half: 6K 59.94p | two of five frames (22.5–24 fps) | – |
+| Playback at half: 4K 119.88p / 6K 59.94p at quarter | every frame | – |
+| One frame (paused monitor, an eighth): CPU / GPU helper | 34 / ~200 ms | 28–36 / 160–178 ms |
+| Scrub: playhead jump to picture (median) | 44–48 ms | 48–58 ms |
+| Source Settings change to the monitor | 62–67 ms | 120–240 ms |
+| Export, 6K timeline to 1080p | ~14–15 fps (decode-bound) | – |
+
+R3D NE decompresses on the CPU, so these follow the CPU. GPU and CPU processing of a frame differ by
+a mean linear error of 0.0005. B draws its window in software, so its UI figures are pessimistic and
+its high-frame-rate playback (–) was not measured.
+
+## Licensing
+
+What follows is a contributor's reading of RED's *SDK License Agreement* shipped with R3D SDK 9.2.1,
+not legal advice. The decision is the project owners' (#383).
+
+- **The licence is personal.** It is a non-exclusive, non-transferable click-through between RED and
+  whoever downloads the SDK. Its rights are that person's, not the project's: everyone who builds the
+  feature downloads the SDK and accepts the agreement themselves.
+- **The adapter source** (`helper.cpp`, `build.rs`, `red/*.rs`) is original code that calls the SDK's
+  public API. The agreement forbids distributing the SDK's own parts (its source, sample code,
+  headers, documentation, libraries and object code) and claims ownership of the SDK only; it does not
+  plainly forbid publishing code of one's own that uses it. Two clauses make it a judgement rather than
+  a clear yes: one forbids programs that expose or disclose "Header File Information", and one treats
+  the whole SDK as confidential except what is public through no fault of the developer. The adapter
+  necessarily names the API it calls (class names, enumerators, metadata keys), which also appear in
+  RED's public sample code. To keep that exposure minimal, no header or documentation text is copied
+  or paraphrased into the adapter, no enumerator values, structure layouts or sample code are
+  reproduced, the adapter compiles only against an SDK the user supplies, and this document describes
+  observed behaviour rather than quoting RED's documentation.
+- **Shipping an SDK-enabled build** (the helper with RED's static library linked in, plus RED's runtime
+  libraries) is allowed by the agreement only under conditions an MIT OR Apache-2.0 release does not
+  meet on its own: an end-user licence with the agreement's terms for the helper and RED's libraries,
+  the runtime libraries unmodified and installed in a private directory beside the helper, RED's
+  notices intact, no headers, documentation or static libraries shipped, no use of RED's name or marks
+  to market the build, and the distributor's own obligations (support, indemnity, export law). **This
+  change proposes none of that.** Official FilmCraft builds and releases stay as they are; users build
+  the feature locally against their own SDK.
+- The app binary contains no RED code (the SDK lives in the helper process) and stays under the
+  project's licence. The SDK's readme says it was developed with Kakadu software; a distributor's
+  third-party notices would repeat that.