RFC / investigation: transport-neutral live media exchange for Craft apps (future StreamCraft use case) #230

Summary

I'd like to propose a small, exploratory cross-app media-exchange contract as a possible future capability of the Craft ecosystem, not a new media engine or a request to start implementation now.

This grows out of the StreamCraft live-production proposal: its most useful ecosystem-level prerequisite may be the ability for FilmCraft, EffectCraft, SoundCraft and a future live switcher to hand off timestamped video/audio without imposing one transport, one codec, or one application engine.

For now, "CraftLink" is a working name for the idea, not a proposed new network protocol. I'd prefer to reuse existing protocols and the team's existing shared crates wherever possible.

PR / architecture audit before this proposal (2026-10-22)

I checked the relevant work to avoid asking for a second implementation of something already underway:

• Merged: craft-libs #182 introduced craft-media for bitstream, ISOBMFF and Matroska; its codecs remain in the consuming apps. Open: #196 follows up with validation and mux/demux fixes.
• Open: craft-libs #15 proposes exact rational media time/timecode. This should not be duplicated; note that a media timeline position, a monotonic presentation deadline, and a network/PTP clock are different concepts even when mapped together.
• Draft adoption: FilmCraft #2435 and SoundCraft #776 move app-local container/bitstream code onto craft-media.
• Open and separate in scope: craft-libs #160 proposes runtime/GPU provider/worker infrastructure, not an established live media transport ABI. Reuse or align with it if accepted.
• Open: EffectCraft #1840 tackles hardware H.264/HEVC paths. FilmCraft #2691 addresses VA-API AV1 hardware decode; #2657 addresses VideoToolbox AV1 decode and other formats. None by themselves establish a cross-app live frame/transport contract.
• FilmCraft ↔ EffectCraft tracking #1024 already discusses one frame-request contract, initially via a process, later potentially via in-process library linking. This should be a primary design input, not a competing Dynamic Link proposal.
• craft-libs #2 and the maintainer response closing #94 favor small, evidenced extractions with real consumers, not a large common framework designed ahead of demand. I agree.

The narrower gap to investigate

A renderer, camera source, frame server, and live mixer all need to agree on things like:

1. Source discovery and capabilities: source identity, availability, pixel/sample formats, frame rate, color primaries/transfer/matrix/range, HDR metadata, alpha, audio channels/layout, and whether a device can actually HW encode/decode a specific codec/profile/bit-depth/chroma combination.
2. Presentation and lifetime: timestamp/timebase, clock domain, deadline vs media position, frame ordering, buffer ownership, invalidation, GPU fences and release semantics; bounded back-pressure and cancellation.
3. Different delivery paths behind a small interface:
   • In-process renderer/frame-request API (especially EffectCraft → FilmCraft).
   • Local IPC/shared memory or platform-specific zero-copy GPU sharing where supported, with an explicit copy fallback (e.g. DMA-BUF, IOSurface, Windows shared D3D textures as implementation options, not portable assumptions).
   • Network media sources using existing standards: Open Media Transport/VMX for compatible low-latency LAN workflows; WebRTC with AV1/H.264 where supported for phones; SRT/RIST/RTP as appropriate; later standards-based ST 2110 paths for professional facilities.
4. Status and observability: negotiated mode, measured end-to-end delay/jitter, drops, reconnect state, active adapter and why fallback occurred.

A receiver should be able to ask for frames by time or subscribe to live frames without requiring a uniform CPU RGBA copy. Encoded packets, CPU buffers, and GPU surfaces are different resource kinds with different ownership and sync rules.

This is not a proposal to put control-plane permissions, AI agents, PTZ/tally business logic, or a broadcast video mixer inside craft-media. Those belong above the media boundary. Any common control/agent API should be separately evaluated.

Why AV1 matters, and why it should not be mandatory

Increasing hardware AV1 encode/decode availability makes AV1 useful for wireless/mobile and bandwidth-constrained sources. But capability is platform-, profile-, bit-depth- and device-specific, and AV1 is not the best answer for every low-latency/4:2:2/alpha/key-fill broadcast path.

I would favor capability negotiation and codec-pluggable adapters, rather than inventing a new AV1 wire protocol. AV1 already has an AOMedia RTP payload specification. OMT already addresses another part of the problem with its open VMX-based LAN protocol. Any protocol/codec extension should only be considered after a measurable incompatibility with established standards is demonstrated.

Suggested first experiment — no suite-wide refactor

Could we answer three empirical questions before promoting any new shared crate?

1. Local frame exchange: EffectCraft renders a timestamped composition frame for FilmCraft using the existing #1024 frame-request direction. Compare out-of-process IPC against in-process and, where possible, GPU-resource exchange. Record copies, CPU/GPU time, latency, memory and sandbox constraints.
2. LAN camera path: try two phone/camera streams and test an open transport (e.g. WebRTC AV1 when both ends have the requisite hardware, with H.264 fallback), plus OMT/VMX on suitable LAN hosts. Measure glass-to-glass latency, packet-loss recovery, lip sync, bandwidth, encode/decode utilization and alpha support.
3. Reliability contract: establish what happens on late/missing frames, a dead producer, client crash, profile mismatch, GPU device loss and source reconnection. A future StreamCraft Program feed must continue independently of a producer GUI or AI service.

If one small frame/time/capability contract proves useful for at least two existing apps, extract just that reusable piece into craft-libs. Keep protocol adapters and broadcast-domain operations outside until real consumers justify broader sharing.

Questions for maintainers

• Would you prefer this investigation to live under FilmCraft #1024 for now, with only proven cross-app primitives later proposed to craft-libs?
• Is a small shared frame metadata / capability / ownership contract something you'd eventually want in craft-media, or should that contract stay app-local until the process-vs-library decision in #1024?
• Should local IPC/GPU-sharing and network transport remain separate adapters behind one optional interface, or is even that abstraction premature?

No code or new crate is being requested here. I'm primarily hoping to avoid a future StreamCraft becoming dependent on duplicated, app-specific video plumbing while respecting existing ongoing work.