PicView Motion Photo (Live Photo) Support - #374
Open
reflyable wants to merge 3 commits into
Open
Conversation
- Detector: limit Samsung trailer scan to JPEG, truncate the JPEG XMP packet at its end tag, validate sidecar candidates with an ftyp header check, and bound the Item:Length lookup to its own Directory item so a sibling's length is never mistaken for the video length. - Extractor: serve embedded videos via the new read-only seekable FileSliceStream window instead of copying them into a MemoryStream. - Playback: single-copy frame pipeline using a 3+1 unmanaged buffer pool with the buffer index as picture cookie, copying straight into the WriteableBitmap; frames drop via an overflow buffer when the UI lags. - Fix the zoom flash when starting playback: the video surface is only revealed once the first decoded frame is ready, and Stop() now clears the frame bitmap so stale frames can never render. - Prefer hardware decoding via --avcodec-hw=any (automatic software fallback). - Tests: 36 motion photo tests green (new FileSliceStreamTests, sidecar ftyp rejection, HEIC Samsung-scan skip, sibling Item:Length cases).
…m libvlc plugins; verify Native AOT. - Localize the play badge with a new MotionPhoto translation key (LanguageModel, TranslationViewModel, all 22 language JSON files). - Append a localized (Motion Photo) marker to the window/tab titles when the current or secondary image carries a video. - Give the gallery a motion photo badge: IsMotionPhoto on GalleryItemViewModel, detection in the parallel thumbnail loop and the file-watcher new-item path, play-triangle overlay on thumbnails. - Support side-by-side playback: MotionPhotoView is now model-driven and each PicBox hosts its own overlay inside the zoom/rotation transform, aligning exactly with its image rect. Playback is mutually exclusive, the zoom lock covers both views, and ZoomPanControl's pan handler skips Button sources so the badge stays clickable while zoomed. - Trim the publish output via Build/Trim-LibVLCPlugins.ps1 (wired into the Win32 build scripts): libvlc drops ~279 MB to ~23 MB by removing sibling architecture folders the NuGet package always copies, link-time *.lib, lua/hrtfs data, and all but the 22 plugin DLLs playback loads (whitelist from -vv module logs, verified through software and 1080p hardware decode). Publish-root *.pdb (~264 MB, incl. the >150 MB native PicView.pdb) is now removed at publish time, which also keeps the CI portable artifacts free of debug symbols. - Verify Native AOT: PublishAot spike plays mp4 via StreamMediaInput + software callbacks to EndReached; full-app AOT publish succeeds with no LibVLCSharp warnings and launches cleanly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds motion photo (Live Photo) viewing to PicView. Still images that carry a short video clip are displayed as usual, with a "Motion Photo" badge overlay that plays the clip once and then freezes back onto the cover (no looping, mirroring Google Photos / JarkViewer behavior). Any failure at any stage degrades gracefully to the plain still image.
The specific code is written with AI assistance. I understand the overall idea and general implementation, and I am happy to make any necessary changes or refactoring.
Supported sources:
Item:Lengthof theItem:Semantic="MotionPhoto"entry in XMPContainer:Directory(lookup bounded to the correct Directory item)MicroVideoOffsetMotionPhoto_Datamarker near the file tail.mov/.mp4, validated with an ftyp header check.livpcontainerImplementation in brief:
PicView.Core(AOT-safe, zero video dependencies): byte-levelMotionPhotoDetector(XMP string search — no XML parsing, vendor-namespace tolerant) andMotionPhotoExtractor(ftyp box correction with a ±8KB window for vendor trailing blocks à la DJI). Embedded videos are served through the new read-only, seekableFileSliceStreamwindow over the source file — no in-memory copy, playback starts immediately;.livpentries are decompressed in memory via the existing SharpCompress dependency. No temp files anywhere on the happy path.PicView.Avalonia: LibVLCSharp plays the extracted stream via software video callbacks, so a single implementation works on Windows, X11, Wayland and macOS (no native window embedding) and the video follows zoom/rotation. A 3+1 unmanaged buffer pool hands frames to the UI thread, which copies them straight into aWriteableBitmap(one memcpy per frame, overflow buffer drops frames instead of tearing). Hardware decoding is preferred via--avcodec-hw=anywith automatic software fallback. Windows native libs ship viaVideoLAN.LibVLC.Windows; Linux/macOS use the system libvlc and degrade gracefully when it is missing.ImageType.MotionPhoto+ImageModel.MotionPhoto(following theTiffNavigationprecedent), detection hooked intoGetImageModelafter the JPEG/HEIC branches, one playback overlay per image (main + side-by-side secondary, each aligned to its own image rect inside the zoom/rotation transform), Space = play/pause, Esc = stop, two new settings (UIProperties.AutoPlayMotionPhotosdefault off,MuteMotionPhotosdefault on),FileSaverHelperkeeps the video when the file is copied as-is. The badge is localized (newMotionPhotokey in all 22 language files), the window/tab title gets a "(Motion Photo)" marker, and gallery thumbnails show a play-triangle badge.Bug fixes included (found during development):
UpdateImage.ChangeImagenotifications (R3 callbacks are not guaranteed to be on the UI thread; the badge was unclickable).AltButtonsPaneltransparent overlay was swallowing all clicks in the top 150px of the image area (pre-existing interaction dead zone; fixed viaIsHitTestVisiblebinding).Stop()clears the frame bitmap.ZoomPanControl's pan handler no longer swallows badge clicks while zoomed (skipsButtonevent sources).The full design document (detection/extraction design, state machine, degradation matrix, AOT risk assessment, file inventory) is embedded at the end of this description.
Known limitations / not yet done
Motivation and Context
Phones from Google, Samsung, Xiaomi, OPPO, DJI and Apple all capture short video clips alongside still photos by default (Motion Photo / Live Photo). Today PicView shows these as plain stills and the live moment is invisible — worse, a folder of iPhone exports (
IMG_1234.HEIC+IMG_1234.MOV) looks like duplicated content. This PR surfaces the embedded video in the least intrusive way: the still image pipeline is untouched, the video is extracted on demand only, and every failure path falls back to the still image.#337
How Has This Been Tested?
PicView.Tests\MotionPhoto\(xUnit.v3), all green — detector (new/legacy XMP forms, attribute reordering, sibling-itemItem:Lengthrejection, Samsung marker incl. HEIC skip, sidecar precedence + ftyp validation,.livp), extractor (ftyp window correction, DJI-style trailer shift, corrupt input, livp cover) andFileSliceStream(slice read/seek/length clamping/async read). Fixtures are synthesized in code; no real photos committed.StreamMediaInputplayback →EndReached, passing on Windows 11 x64 and Linux (WSL, JIT self-containedlinux-x64build); software video-callback smoke verified per-frame BGRA delivery on both.PublishAot-published spike console (same LibVLCSharp call pattern:StreamMediaInput+ video callbacks +--avcodec-hw=any) plays toEndReached(61 frames); the full Win32 appdotnet publish(Native AOT) succeeds with zero LibVLCSharp trim/AOT warnings, and the artifact launches on a synthetic motion photo without crashing.*.lib/lua/hrtfsalso removed), via JIT software decode, the 1080p d3d11va/dxva2 hardware-decode path, and the AOT-published spike.PicView.CoreandPicView.Avaloniabuild clean; the full test suite matches the pre-existing baseline (11 failures in unrelated Navigation/FileWatcher/Tiff/Archive test infrastructure — nullImageIteratorctor argument, xUnit discovery-time version conflict — reproducible without these changes). All 22 language JSON files parse cleanly.net11.0(LangVersion preview), LibVLCSharp 3.10.1, VideoLAN.LibVLC.Windows 3.0.23.1. macOS runtime untested (see limitations).Screenshots (if appropriate):
2026-08-15.00-01-06.mp4
Types of changes
Checklist:
Design document
Click to expand: Motion Photo Support — Design Document
PicView Motion Photo (Live Photo) Support — Design Document
0. Implementation Status (2026-08-13)
Completed (Phase 1 + Phase 3 Windows part):
ImageType.MotionPhoto/ImageModel.MotionPhoto/.livpsupported extensionGetImageModeldetection hook (after the JPEG/HEIC branches + dedicated livp branch, before Magick Ping)libvlc\win-x64+win-arm64, 962 dlls) +Core.Initializepathlinux-x64publish + WSL launch verificationPicView.Avalonia.Linux\bin\x64\Release\net11.0\linux-x64\publish)Implementation deviations:
LibVLCSharp.Avaloniapackage. That package is compiled against Avalonia 11 (PicView uses Avalonia 12; theNativeControlHostAPI survived but is unvalidated by that package). Instead the video surface is implemented in ~100 lines, depending only on theLibVLCSharpcore package.MediaPlayer.SetVideoFormatCallbacks+SetVideoCallbacksmake libvlc hand back BGRA32 frames, which are written into aWriteableBitmapand rendered byMotionPhotoVideoSurfaceinside the Avalonia compositor. Native window embedding (X11 child windows / NSView / HWND) was dropped: libvlc 3.x has no publicwl_surfaceembedding API, so native embedding is impossible on Wayland; software callbacks natively support Wayland/X11/Windows/macOS, follow zoom/rotation, and have no airspace issues. Audio is still handled by libvlc's own output (governed by MuteMotionPhotos). Cost: one memcpy per frame (~8MB at 1080p — negligible for 3-second clips).UIProperties.AutoPlayMotionPhotos/MuteMotionPhotos(auto-play off by default, muted by default).pacman -S vlc/apt install vlc), and LibVLCSharp's default search locateslibvlc.so.5; when not installed, the feature degrades gracefully. The Linux deliverable is currently a JIT self-contained publish (AOT cannot cross-compile from Windows).Integration defects found and fixed (discovered empirically):
UpdateImage.ChangeImageare not guaranteed to be on the UI thread (R3EveryValueChangedcallbacks fire on the thread that changed the property). Setting Avalonia visual properties (IsVisible) directly in the callback throwsInvalidOperationException, which the R3 subscription error handler silently swallows — the symptom being "clicking the badge does nothing". Fix:ImageViewer.UpdateMotionPhotomarshals viaDispatcher.UIThread.CheckAccess()/Post. Any new UI logic hooked intoChangeImagemust be marshalled the same way.AltButtonsPanel(the hidden-UI hover zone: top 150px,Background=Transparent,Opacity=0, ZIndex=9) — per Avalonia's hit-testing rules (Opacity=0 + transparent background still participate in hit testing, higher ZIndex wins) — swallowed every click in the top part of the image area; the badge was visible but unclickable. Fix:AltButtonsPanel.IsHitTestVisible="{Binding !IsUIShown.Value}"— it no longer intercepts in normal UI mode (which also fixes the pre-existing interaction dead zone in the top 150px of the image), while the hidden-UI hover-to-reveal behavior for the Alt buttons is unchanged. Note: in hidden-UI mode the top region still belongs to the Alt hover zone; use the Space key for playback there.Remaining risks:
dotnet publish(ILC); if needed, keep the LibVLCSharp assembly viaTrimmerRootDescriptor.Changelog (2026-08-14): detection/extraction/playback optimizations + playback flash fix
Detector (
MotionPhotoDetector.cs).jpg/.jpeg: HEIC files no longer get their tail (up to 32MB) scanned — previously every HEIC without motion XMP had its entire file tail read, a visible waste during preloading.ReadJpegXmpPackettruncates at</x:xmpmeta>: the remaining ~1MB of JPEG binary data in the scan window is no longer converted into a string, saving one large allocation and speeding up the subsequent string searches..mov/.mp4must start with a valid 8-byte ftyp box signature, so same-named non-video files (downloads, export leftovers) are no longer misdetected.Item:Lengthis now bounded to the same Directory item: the newFindItemEndbounds the search at the earliest of/>,</Container:Item>and the next<Container:Item; the backward search is likewise bounded to the same tag. The still-image item'sItem:Lengthcan no longer be mistaken for the video length (the unbounded ±1024-char fallback was removed; unconventional XMP naturally falls through to the MicroVideo/Samsung/sidecar branches).Extractor (
MotionPhotoExtractor.cs+ newMotionPhoto\FileSliceStream.cs)MemoryStreamup to 256MB): after ftyp location, the extractor returns the newFileSliceStream— a read-only, seekable, bounds-clamped window over the source file. Playback starts immediately with zero extra memory; the Sidecar/Livp paths are unchanged.Playback (
MotionPhotoView.axaml.cs/MotionPhotoVideoSurface.cs/MotionPhotoService.cs)WriteableBitmapviaBuffer.MemoryCopy(unsafe;AllowUnsafeBlocksalready enabled), eliminating one full-frameMarshal.Copyand the managed intermediate array per frame. When the UI falls more than 3 frames behind, decoding goes into the overflow buffer and the frame is dropped — no tearing, no lock contention (onlyInterlocked+_frameLock).VideoSurfacewas made visible beforePlay(), andStop()never cleared the bitmap, so the surface first rendered the stale frame from the previous session; when the previous photo's video had a different resolution/aspect ratio this appeared as a "sudden zoom-in then restore". Fix: the surface only becomes visible once the first frame has been written into the bitmap, andStop()now callsVideoSurface.Clear(). The transition cover → first frame → playback is now seamless.--avcodec-hw=anyto the LibVLC init options (auto-selects dxva2/d3d11va/vaapi/videotoolbox per platform, automatic software fallback; frames are still downloaded to CPU memory for the software callbacks — for 3-second clips the gain is mainly lower decode CPU usage).Tests: 28 → 36, all green. New: sidecar non-video-header rejection, HEIC skips the Samsung scan, sibling-item
Item:Lengthnot picked up,FileSliceStreamTests(slice read/seek/length clamping/async read, 5 tests); 2 sidecar tests now use valid MP4 headers. The full suite has 12 additional failures, all pre-existing issues unrelated to this feature (Navigation/FileWatcher/Tiff test infrastructure: nullImageIteratorctor argument, xUnit discovery-time version conflicts, etc.).File inventory update: Core adds
MotionPhoto\FileSliceStream.cs; Tests addsMotionPhoto\FileSliceStreamTests.cs.Changelog (2026-08-14, batch 2): localization/title/side-by-side/gallery + plugin trim + AOT verification
Localization
MotionPhototranslation key:LanguageModel+ all 22 language JSON files +TranslationViewModel; the badge text and tooltip now bindParentWindowContext.Translation.MotionPhoto.Valueinstead of a hardcoded English string.Title marker
TabViewModel.UpdateTabTitle()appends a localized" (Motion Photo)"suffix to all title variants when the current image (or either image in side-by-side mode) is a motion photo.Side-by-side playback
MotionPhotoViewrefactored to be model-driven (OnImageChanged(ImageModel?)); eachPicBoxis wrapped in aGridhosting its own overlay (main + secondary withAllowAutoPlay=false). Overlays live inside the zoom/rotation transform: exact per-image alignment, video letterboxes into the image rect. Playback is mutually exclusive; the zoom lock engages while either view plays.ZoomPanControl's tunneled PointerPressed handler skipsButtonsources so the badge stays clickable while zoomed.Gallery thumbnail badge
GalleryItemViewModel.IsMotionPhoto(reactive); detection runs per item insideGalleryLoader's parallel thumbnail loop and in theFileWatcherServicenew-file path; thumbnails show a play-triangle badge (hit-test transparent). Known limitation: HEIC-embedded motion photos are not badged in the gallery (no Magick XMP in the thumbnail pipeline).Neighbor pre-extraction — closed as obsolete: with
FileSliceStream, extraction is a bare file open; livp stays on-demand to avoid caching decompressed videos.libvlc plugin trim
-vvmodule logs: 323 → 23 dlls, 96.3 MB → 23.3 MB; playback-verified (software + 1080p hardware decode paths). NewBuild\Trim-LibVLCPlugins.ps1wired into the Win32 build scripts.Native AOT verification — passed, no TrimmerRootDescriptor
PublishAotspike (StreamMediaInput + SetVideoCallbacks +--avcodec-hw=any) plays toEndReached, 61 frames. Full-app AOT publish succeeds with zero LibVLCSharp warnings; artifact launches on a synthetic motion photo without crashing.Tests: full suite of 198, 11 failures identical to the pre-change baseline (test-infrastructure isolation issues); all 22 language JSON files parse cleanly.
Changelog (2026-08-14, batch 3): deep publish-output trim
Analyzing the x64 AOT publish output (637.8 MB) surfaced more waste, now handled by the rewritten
Build\Trim-LibVLCPlugins.ps1(-LibVlcRoot+-TargetArch) and the Win32 build scripts:win-x86,win-arm64in an x64 publish): −178.4 MB — the NuGet package copies all three even for RID-specific publishes.PicView.pdb156.8 +libSkiaSharp.pdb84.5 +libHarfBuzzSharp.pdb22.2 + managed): −264 MB, now deleted right after publish in all three build scripts. This also fixes the CI portable artifacts, which uploaded before the pdb-removal step and shipped the symbols (only the installer was clean).*.libimport libs,lua\scripts,hrtfs\data: −1.7 MB (link-time only / serving removed plugins and filters).Net: libvlc 279.3 → 23.2 MB (22 plugins kept); estimated x64 publish output 637.8 → ~117.7 MB. Verified playing to
EndReachedvia JIT software decode, 1080p hardware decode, and the AOT spike on the fully trimmed set.1. Goals and Scope
Support the following motion photo sources in PicView (covering all three categories JarkViewer supports):
Item:Lengthof theItem:Semantic="MotionPhoto"entry in XMPContainer:DirectoryGCamera:MicroVideoOffsetMotionPhoto_Datamarker near the file tail.mov/.mp4file.livpcontainerPlayback behavior aligned with JarkViewer: show the still cover by default → play the embedded video once when triggered → freeze back onto the cover when finished (no looping).
Any failure at any stage degrades gracefully to a plain still image — a broken file must never crash the viewer.
2. PicView Current-State Inventory (verified via DeepWiki + source)
2.1 Loading pipeline
Key facts (all file paths verified):
ImageTypeenum (PicView.Core\ImageDecoding\ImageType.cs) currently has 5 values:Invalid, AnimatedGif, AnimatedWebp, Bitmap, Svg.ImageModel(PicView.Core\Models\ImageModel.cs) carries the bitmap asobject? Image; there is an existing precedent for attached data:TiffNavigationInfo? TiffNavigation(multi-page TIFF). Motion photo metadata is attached following this pattern.GetImageModel.GetImageModelAsync(PicView.Avalonia\ImageHandling\GetImageModel.cs) dispatches onmagickImage.Format: unrotated JPEG goes through theProcessSkBitmapAsyncfast path; HEIC goes through the default branchProcessNonStandardImageAsync.ImageAnalyzer(PicView.Core\ImageDecoding\ImageAnalyzer.cs) already has a precedent for "cheap binary detection" (Ping frame count, GIF tail-byte check) — a motion photo detector fits this style.Xmp,Exiv2,GCamera,video,MediaPlayer). All metadata goes through Magick.NET;IXmpProfile(magickImage.GetXmpProfile()) can reuse the existing Magick.NET dependency with zero new additions..livp(zip) decompression needs zero new dependencies.TempFileManager(PicView.Core\FileHandling\TempFileManager.cs) provides the temp-file convention (URL downloads and base64 drags both use it); it can serve as a fallback path if playback fails.PicBox→IGifInstance(frame timing/decoding) →CustomVisualHandler(driven by compositor clockCompositionNow, output viaDrawBitmap). GIF/WebP always useIterationCount.Infinite.Views\UC\ImageViewer.axaml):ZoomPanControl > AutoScrollViewer > MainBorder > DockPanel > PicBox(MainImage) + PicBox(SecondaryImage); the outerMainPanelstacks overlays (HoverBar, zoom preview, etc.) via ZIndex.UpdateImage.ChangeImage(PicView.Avalonia\Navigation\UpdateImage.cs) only special-casesImageType.Svg; animation startup is triggered entirely by PicBox itself based on the boundImageType+CurrentFileInfo.2.2 Key constraints
PublishAot=true+Trimming=full+IsAotCompatible. LibVLCSharp is not officially AOT-validated (heavy P/Invoke + delegate callbacks) — this is the biggest risk of the plan and requires isolation measures plus fallback plans (see section 8).x64;arm64only, matching thewin-x64/win-arm64runtimes provided byVideoLAN.LibVLC.Windows.MaxDegreeOfParallelism = ProcessorCount - 1), so detection logic must be cheap and thread-safe (no shared mutable state).3. JarkViewer → PicView Concept Mapping
ImageAsset{format, primaryFrame, frames[]}unified assetImageModel{Image, ImageType, MotionPhotoInfo}: cover = theImagebitmap; video frames are not pre-decoded, only "slice coordinates" are storedGetXmpProfile()string search (no XML parsing)locateMotionPhotoVideoftyp-box correctionMotionPhotoExtractorStreamMediaInput(LibVLCSharp) plays from a seekable stream, zero temp filesdelayRemaintimed playbackformat=Stillfreezes on the coverMediaPlayer.EndReached→ hide the video layer → freeze on the coverImageType.BitmapCore architectural difference: JarkViewer decodes the whole video into
cv::Mat[]with FFmpeg and plays it with manual timing (Live Photo videos are short, ~3 seconds — feasible but memory-heavy, no audio). With LibVLCSharp, PicView does not decode frames in managed code; the video byte stream is handed directly to libvlc: low memory usage, hardware decoding, native audio support. The byte-handling experience from the detection/extraction layer (XMP search, ftyp correction, fault-tolerant degradation) is fully inherited.4. Overall Architecture
Dependency direction is strictly one-way: Core knows nothing about LibVLCSharp; the Avalonia layer hands the
Streamextracted by Core to libvlc.5. Detection and Extraction Design (PicView.Core)
5.1 Data model
ImageModelgainspublic MotionPhotoInfo? MotionPhoto { get; set; }(following theTiffNavigationprecedent).5.2 Detection order (
MotionPhotoDetector.TryDetect(FileInfo, string? xmpPacket) → MotionPhotoInfo?).livpextension →LivpContainer(record only; bytes are decompressed on demand by the Extractor).MotionPhotonearItem:Semantic→ take theItem:Lengthnumber within the same Directory item (bounded lookup, so the still-image item's length is never picked up);MicroVideoOffset(or the Offset near theMicroVideoversion fields) → take the number;fileLength - videoLength.MotionPhoto_Datain a tail window; if found, the video starts right after the marker..mov→.mp4in order; the candidate must start with a valid ftyp box header.null, treat as a regular still image.XMP packet source: prefer Magick.NET
magickImage.GetXmpProfile()(GetImageModelAsyncalready has a pingedmagickImage, zero extra cost; verified that Ping surfaces the XMP profile). HEIC XMP lives in the meta box and is also obtained via Magick; if Magick cannot obtain XMP for some carrier, fall back to a self-implemented JPEG APP1 segment scan (pure byte code, AOT-safe), which truncates the packet at</x:xmpmeta>.5.3 Extraction (
MotionPhotoExtractor)ftyp box validation:
size(4B, big-endian, ≥8 and ≤ remaining length) + "ftyp"(4B ASCII), identical to JarkViewer's box-size validation logic.5.4 Loading-pipeline hook (
GetImageModel.cs)Hook detection uniformly after the
switchinGetImageModelAsync(without intruding into individual branches):imageModelLoaderdelegate); detection only does Ping-level XMP reads + a tail-window scan on JPEG, sub-millisecond per call, acceptable under parallel preloading.PreLoaderhits a motion photo, pre-extract neighboring items' video bytes in the background (typically 2–5MB each, memory impact needs evaluation, Phase 4).6. Playback Design (PicView.Avalonia + LibVLCSharp)
6.1 New packages
LibVLCSharp(theLibVLCSharp.AvaloniaVideoView package was evaluated but not adopted, see deviations)VideoLAN.LibVLC.Windows(win-x64/win-arm64 native libs + plugins)pacman -S vlc/apt install vlc)Size control: the full libvlc plugin set exceeds 100MB. At publish time, trim
libvlc/pluginsto keep only:demux/mp4(mov),codec/avcodec(h264/hevc),packetizer/avcodec,swscale/d3d11video output, the audio output chain. Determine the trim list empirically during the Phase 2 spike (use-vvlogs at libvlc startup to observe which modules actually load).6.2 Rendering approach selection
LibVLCSharp.Avalonia's
VideoViewis aNativeControlHost(native window handle embedding). This leads to two approaches:MediaPlayer.SetVideoCallbacksreturns frames → copy intoWriteableBitmap→ render in the Avalonia compositorwl_surfaceembedding API)--avcodec-hw=any)Conclusion: Approach B adopted — a single implementation covers Windows/X11/Wayland/macOS, the video follows zoom/rotation, and there are no airspace issues. The frame pipeline uses a 3+1 unmanaged buffer pool: the lock callback hands out buffers and returns the index as the picture cookie; the UI thread copies the addressed buffer straight into the
WriteableBitmap(single copy per frame, no managed intermediate array); when the UI lags, decoding goes into an overflow buffer and the frame is dropped instead of tearing.6.3 Stream playback (inheriting the "zero temp files" principle)
Stream/Media/StreamMediaInputlifetimes are managed by the playback session; they areDisposed together afterStop/ image change /EndReached.TempFileManager.GetNewTempFilePath("xxx.mp4")and play by path (existing project convention).6.4 Playback state machine
IterationCount.Infinite— a motion photo plays once and freezes (JarkViewer behavior); replay is explicitly user-triggered.Stop()clears the bitmap — a stale frame from a previous session can never flash up (previously perceived as a "zoom-in then restore").MediaPlayer.Mute.MotionPhotoServicelazily loads a singleLibVLCinstance (--no-video-title-show,--avcodec-hw=any, etc.), released at app exit; each tab owns its ownMediaPlayerbut shares theLibVLC.6.5 UI integration
<mp:MotionPhotoView ZIndex="3" />insideMainPanelinImageViewer.axaml(an independent layer above PicBox; while playing it covers the cover image's displayed area).MotionPhotoView:MotionPhotoVideoSurface+ corner badge button ("Motion Photo" badge, shown in cover state, click to play,Cursor="Hand").UpdateImage.ChangeImagehook: forImageType.MotionPhotoset the coverImageas usual and notify the motion photo view (stop any running playback, show the badge, trigger playback directly if auto-play is enabled). On image change the previous playback must be stopped and hidden. The call may come from a background thread, so it must be marshalled to the UI thread.SecondaryImagecould be supported the same way (twoMediaPlayerinstances); not implemented yet.6.6 Existing code that must be updated in sync
FileSaverHelper: the existingAnimatedGif/AnimatedWebpbranches are TODOs; after adding theMotionPhotovalue, saving must either go through "still cover" or "copy the file as-is" (the latter implemented — copying the source file preserves the video).switch (ImageType)in the repo to confirm the new enum value's default behavior is safe (printing, copying, gallery thumbnails, etc. can all treat it as Bitmap).UIProperties.AutoPlayMotionPhotos(default false),UIProperties.MuteMotionPhotos(default true).7. Graceful Degradation Matrix (JarkViewer lesson: never let a broken file crash the viewer)
ImageType.Bitmap, plain still imageEncounteredError)DebugHelper.LogDebug+ degrade to still image, never rethrow8. Native AOT Risk Assessment and Mitigation
Risk: LibVLCSharp (3.x) does not declare AOT compatibility; it uses P/Invoke, delegate callbacks,
delegate* unmanaged, etc.Trimming=fullmay trim required members; event-callback marshalling under Native AOT needs empirical testing.Mitigations:
PicView.Avalonia\MotionPhoto\and the platform startup projects; Core stays clean (Core's AOT compatibility is unaffected).dotnet publish -r win-x64, empirically test the four things: play/pause/EndReached/StreamMediaInput. If any fails, switch to a fallback plan — avoid late rework.TrimmerRootDescriptor(rd.xml) to keep the wholeLibVLCSharpassembly.libvlcC API wrapper (a dozen[LibraryImport]declarations, AOT-friendly), keeping the software video callbacks (libvlc_video_set_callbacksand friends), which need no window embedding at all.9. Test Plan (PicView.Tests, xUnit.v3)
PicView.Tests\MotionPhoto\(36 tests):TryDetectFromXmp_NewStandard...TryDetectFromXmp_MicroVideoOffset...GCamera:MicroVideoOffsetvariants (namespace churn: GCamera/OpCamera)TryDetectFromXmp_MotionItemWithoutLength_IgnoresSiblingLengthItem:Lengthis never picked upTryDetect_SamsungTrailer...MotionPhoto_Datamarker at the file tail (JPEG); skipped for HEICTryDetect_Sidecar...TryDetect_NoMotionPhotoData_ReturnsNullReadJpegXmpPacket...FindFtypStart.../ExtractEmbedded...ExtractAsync_Sidecar/LivpContainer...ExtractLivpCoverToTempFileAsync...FileSliceStreamTestsFixtures are synthesized in code (never commit real copyrighted photos). The playback layer (LibVLC) is not unit-tested; it is covered by the E2E smoke run (ffmpeg-generated mp4 → synthetic motion photo → extraction → StreamMediaInput playback → EndReached, verified passing) and manual acceptance.
10. File Inventory
PicView.Core (4 new, 2 modified)
MotionPhoto\MotionPhotoInfo.cs(new)MotionPhoto\MotionPhotoDetector.cs(new)MotionPhoto\MotionPhotoExtractor.cs(new)MotionPhoto\FileSliceStream.cs(new)ImageDecoding\ImageType.cs(+MotionPhoto)Models\ImageModel.cs(+MotionPhotoproperty)PicView.Avalonia (4 new, several modified)
MotionPhoto\MotionPhotoService.cs(new, LibVLC singleton)MotionPhoto\MotionPhotoVideoSurface.cs(new, frame surface control)MotionPhoto\MotionPhotoView.axaml(.cs)(new, video surface + badge + state machine)ImageHandling\GetImageModel.cs(detection hook)Navigation\UpdateImage.cs(MotionPhoto hook)Views\UC\ImageViewer.axaml(.cs)(playback overlay + thread marshalling)Views\Main\MainView.axaml(AltButtonsPanel hit-test fix)Input\MainKeyboardShortcuts.cs(Space/Esc during playback)FileSystem\FileSaverHelper.cs(new enum value handling)PicView.Avalonia.csproj(+LibVLCSharp)Platform projects: Win32 csproj +
VideoLAN.LibVLC.Windowsnative package; theBuild\installer scripts should include the trimmed libvlc plugins (pending).Tests: the whole
PicView.Tests\MotionPhoto\suite (36 tests, incl.FileSliceStreamTests.cs).11. Implementation Phases
12. Key Lessons Inherited from JarkViewer
Item:Lengthlookup bounded to the correct Directory item.FileSliceStreamfile-window playback, zero temp files (TempFileManager only as a fallback and for the livp cover).