-
Notifications
You must be signed in to change notification settings - Fork 250
feat(skills): add demo-video skill for narrated video assembly #2404
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 13 commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
efd610c
feat(skills): add demo-video skill for narrated video assembly
4341272
Merge branch 'main' into feat/demo-video-skill
rezatnoMsirhC c96b614
fix(skills): normalize demo-video copyright headers
e97818f
docs(skills): prioritize Azure AI Speech neural voices for demo-video…
3149051
test(skills): add demo-video eval coverage
c655057
fix(skills): use full words in demo-video eval patterns
1964f1d
fix(skills): address demo-video PR review feedback
c136b9c
fix(skills): update demo-video test call sites for dropped param
6fbe8ab
Merge branch 'main' into feat/demo-video-skill
WilliamBerryiii bf7e055
Merge branch 'main' into feat/demo-video-skill
jkim323 fea2e90
Merge remote-tracking branch 'origin/main' into feat/demo-video-skill
9c176b3
fix(skills): address demo-video review feedback on fps, type, and fuz…
a5b38c8
fix(skills): harden demo-video manifest validation and formatting
76e76fd
Merge branch 'main' into feat/demo-video-skill
auyidi1 a9d254a
fix(collections): regenerate hve-core-all markdown for demo-video and…
d51fa29
Merge branch 'main' into feat/demo-video-skill
auyidi1 c0b6597
Potential fix for pull request finding
auyidi1 f9af015
Potential fix for pull request finding
auyidi1 c30c7b8
Potential fix for pull request finding
auyidi1 cc499d4
Merge branch 'main' into feat/demo-video-skill
auyidi1 fcf59cd
Merge branch 'main' into feat/demo-video-skill
auyidi1 2f2d912
Merge branch 'main' into feat/demo-video-skill
auyidi1 dcf95b7
Merge branch 'main' into feat/demo-video-skill
WilliamBerryiii e3b9bcd
fix(skills): address demo-video review feedback
3c1f6fd
Merge origin/main into feat/demo-video-skill
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| --- | ||
| name: demo-video | ||
| description: 'Assemble ordered frames or clips with narration into a narrated MP4 via FFmpeg' | ||
| license: MIT | ||
| compatibility: 'Requires FFmpeg on PATH' | ||
| metadata: | ||
| authors: "microsoft/hve-core" | ||
| spec_version: "1.0" | ||
| last_updated: "2026-07-09" | ||
| --- | ||
|
|
||
| # Demo Video Assembly Skill | ||
|
|
||
| This skill assembles a narrated demo video from ordered visual segments and matching narration audio. It is designed for first-pass walkthrough videos that combine captured prototype frames or clips with per-segment voiceover WAV files. | ||
|
|
||
| ## Overview | ||
|
|
||
| The workflow takes a manifest that describes each segment, resolves the visual source, and uses FFmpeg to render each segment into a normalized video clip before concatenating them into a final MP4. The narration track is muxed from WAV files so the output can be reviewed as a polished walkthrough without requiring a separate video-editing tool. | ||
|
|
||
| ## Manifest Schema | ||
|
|
||
| Use a `segments.yml` manifest with an ordered list of segments. Each entry describes a visual source and the narration audio to combine for that portion of the video. | ||
|
|
||
| ```yaml | ||
| segments: | ||
| - type: frame | ||
| visual: ./frames/intro.png | ||
| narration: ./audio/intro.wav | ||
| duration: 4.5 | ||
| - type: clip | ||
| clip: ./clips/interaction.mp4 | ||
| narration: ./audio/interaction.wav | ||
| ``` | ||
|
|
||
| ### Segment fields | ||
|
|
||
| * `type` identifies whether the segment is a still image (`frame`) or a motion clip (`clip`) | ||
| * `visual` points to an image file for a frame segment | ||
| * `clip` points to a motion clip file for a clip segment | ||
| * `narration` points to the WAV file generated from narration text (the script also accepts `narration_wav` as an alias) | ||
| * `duration` is optional and overrides the inferred duration when you want a fixed segment length | ||
|
|
||
| ## Quick Start | ||
|
|
||
| Use the bash or PowerShell wrappers to invoke the assembler from the skill directory. | ||
|
|
||
| ```bash | ||
| scripts/assemble-video.sh --manifest examples/segments.yml --output ./output/demo.mp4 | ||
| ``` | ||
|
|
||
|
auyidi1 marked this conversation as resolved.
|
||
| ```powershell | ||
| scripts/Invoke-AssembleVideo.ps1 -ManifestPath examples/segments.yml -OutputPath ./output/demo.mp4 | ||
| ``` | ||
|
|
||
| ## Parameters Reference | ||
|
|
||
| The assembly step accepts the following high-level controls: | ||
|
|
||
| * `--manifest` or `-ManifestPath` selects the YAML manifest to process | ||
| * `--output` or `-OutputPath` sets the destination MP4 path | ||
| * `--fps` or `-Fps` controls the output frame rate for rendered segments | ||
| * `--resolution` or `-Resolution` controls the output width and height in the form `WIDTHxHEIGHT` | ||
| * `duration` per segment lets you override the inferred length when narration timing is known in advance | ||
|
|
||
| ## Narration Quality | ||
|
|
||
| Narration quality is the single biggest driver of how polished the final video feels. Prioritize neural voices from **Azure AI Speech (part of Azure AI Foundry)** through the `tts-voiceover` skill for any video you intend to share. | ||
|
|
||
| * **Recommended:** Use the `tts-voiceover` skill backed by Azure AI Speech neural voices (for example `en-US-Andrew:DragonHDLatestNeural` or `en-US-Jenny:DragonHDLatestNeural`). These produce natural, presentation-grade narration and are the default for shareable output. | ||
| * **Fallback only:** Offline open-source engines such as `espeak-ng` require no credentials but sound noticeably robotic. Treat them as a no-network smoke-test fallback, not a delivery format. Regenerate narration with Azure AI Speech before publishing. | ||
|
|
||
| See the `tts-voiceover` skill for the neural voice catalog, `--voice` and `--rate` controls, and Azure authentication (Entra ID or key). | ||
|
|
||
| ## Reuse Bridge | ||
|
|
||
| This skill is intentionally designed to fit into the existing media workflow: | ||
|
|
||
| * `tts-voiceover` provides the narration WAV files that this skill muxes into the final output; prefer its Azure AI Speech neural voices for production-quality narration | ||
| * `vscode-playwright` provides the frame-capture source for prototype walkthroughs and screen-based demos | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| FFmpeg and ffprobe must be available on your PATH. | ||
|
|
||
| ### Linux | ||
|
|
||
| ```bash | ||
| sudo apt update && sudo apt install ffmpeg | ||
| ``` | ||
|
|
||
| ### macOS | ||
|
|
||
| ```bash | ||
| brew install ffmpeg | ||
| ``` | ||
|
|
||
| ### Windows | ||
|
|
||
| ```powershell | ||
| winget install FFmpeg.FFmpeg | ||
| ``` | ||
|
auyidi1 marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| title: Demo Video Examples | ||
| description: Example manifest and fixture setup for the demo-video skill | ||
| author: Microsoft | ||
| ms.date: 2026-07-09 | ||
|
Copilot marked this conversation as resolved.
Outdated
|
||
| ms.topic: reference | ||
| keywords: | ||
| - video | ||
| - ffmpeg | ||
| - examples | ||
| estimated_reading_time: 2 | ||
| --- | ||
|
|
||
| ## Demo Video Example Assets | ||
|
|
||
| This example manifest is designed to run with small fixture files that are kept out of source control. Create the following files locally before running the wrapper: | ||
|
|
||
| - examples/fixtures/intro.png | ||
| - examples/fixtures/intro.wav | ||
| - examples/fixtures/interaction.mp4 | ||
| - examples/fixtures/interaction.wav | ||
|
|
||
| The fixtures can be generated from any short local assets, such as: | ||
|
|
||
| - a small PNG frame exported from a prototype screenshot | ||
| - a short WAV narration clip produced by the tts-voiceover skill (prefer Azure AI Speech neural voices for natural-sounding narration; offline open-source TTS is a no-credential fallback only) | ||
| - a short MP4 clip exported from a local browser recording or sample video | ||
|
|
||
| *🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.* | ||
11 changes: 11 additions & 0 deletions
11
.github/skills/experimental/demo-video/examples/segments.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| output: ./examples/output/demo.mp4 | ||
| resolution: 1280x720 | ||
| fps: 24 | ||
| segments: | ||
| - type: frame | ||
| visual: ./examples/fixtures/intro.png | ||
| narration: ./examples/fixtures/intro.wav | ||
| duration: 3.0 | ||
| - type: clip | ||
| clip: ./examples/fixtures/interaction.mp4 | ||
| narration: ./examples/fixtures/interaction.wav | ||
|
auyidi1 marked this conversation as resolved.
Outdated
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| [project] | ||
| name = "demo-video-skill" | ||
| version = "0.0.0" | ||
| requires-python = ">=3.11" | ||
| dependencies = [ | ||
| "pyyaml>=6.0", | ||
| ] | ||
|
|
||
| [dependency-groups] | ||
| dev = [ | ||
| "pytest>=9.0", | ||
| "pytest-cov>=5.0", | ||
| "pytest-mock>=3.14", | ||
| "ruff>=0.15", | ||
| ] | ||
| fuzz = [ | ||
| "atheris>=3.0", | ||
| ] | ||
|
|
||
| [tool.pytest.ini_options] | ||
| testpaths = ["tests"] | ||
| pythonpath = ["scripts"] | ||
| python_files = ["test_*.py", "fuzz_harness.py"] | ||
|
|
||
| [tool.ruff] | ||
| line-length = 88 | ||
| target-version = "py311" | ||
|
|
||
| [tool.ruff.lint] | ||
| select = ["E", "F", "I", "W"] |
102 changes: 102 additions & 0 deletions
102
.github/skills/experimental/demo-video/scripts/Invoke-AssembleVideo.ps1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| #!/usr/bin/env pwsh | ||
| # Copyright (c) 2026 Microsoft Corporation. All rights reserved. | ||
| # SPDX-License-Identifier: MIT | ||
| # | ||
| # Invoke-AssembleVideo.ps1 | ||
| # Wrapper for assemble_video.py that resolves the skill's Python environment | ||
| # and delegates FFmpeg video assembly. | ||
| #Requires -Version 7.4 | ||
|
|
||
| <# | ||
| .SYNOPSIS | ||
| Assembles a narrated MP4 from image/clip segments and narration WAV files. | ||
|
|
||
| .DESCRIPTION | ||
| Resolves the demo-video skill's Python environment with uv and forwards | ||
| the supplied manifest and output arguments to assemble_video.py. | ||
|
|
||
| .PARAMETER ManifestPath | ||
| Path to the YAML manifest describing the visual segments. | ||
|
|
||
| .PARAMETER OutputPath | ||
| Destination MP4 path for the assembled video. | ||
|
|
||
| .PARAMETER Fps | ||
| Frame rate used when rendering each segment. | ||
|
|
||
| .PARAMETER Resolution | ||
| Output resolution in WIDTHxHEIGHT format. | ||
|
|
||
| .EXAMPLE | ||
| ./Invoke-AssembleVideo.ps1 -ManifestPath examples/segments.yml -OutputPath ./output/demo.mp4 | ||
| #> | ||
|
|
||
| [CmdletBinding()] | ||
| param( | ||
| [Parameter(Mandatory = $false)] | ||
| [string]$ManifestPath, | ||
|
|
||
| [Parameter(Mandatory = $false)] | ||
| [string]$OutputPath, | ||
|
|
||
| [Parameter(Mandatory = $false)] | ||
| [int]$Fps, | ||
|
|
||
| [Parameter(Mandatory = $false)] | ||
| [string]$Resolution | ||
| ) | ||
|
|
||
| $ErrorActionPreference = 'Stop' | ||
|
|
||
| $ScriptDir = $PSScriptRoot | ||
| $SkillRoot = Split-Path $ScriptDir | ||
| $VenvDir = Join-Path $SkillRoot '.venv' | ||
|
|
||
| function Test-UvAvailability { | ||
| if (-not (Get-Command -Name 'uv' -ErrorAction SilentlyContinue)) { | ||
| throw "uv is required but was not found on PATH. See https://docs.astral.sh/uv/getting-started/installation/ (for example: 'winget install astral-sh.uv' on Windows, or 'curl -LsSf https://astral.sh/uv/install.sh | sh' on Linux/macOS)." | ||
| } | ||
| } | ||
|
auyidi1 marked this conversation as resolved.
|
||
|
|
||
| function Initialize-PythonEnvironment { | ||
| Write-Verbose 'Syncing Python environment via uv...' | ||
| & uv sync --directory $SkillRoot | ||
| if ($LASTEXITCODE -ne 0) { | ||
| throw 'uv sync failed' | ||
| } | ||
| } | ||
|
|
||
| function Get-VenvPythonPath { | ||
| if ($IsWindows) { | ||
| return Join-Path $VenvDir 'Scripts/python.exe' | ||
| } | ||
| return Join-Path $VenvDir 'bin/python' | ||
| } | ||
|
|
||
| if ($MyInvocation.InvocationName -ne '.') { | ||
| try { | ||
| Test-UvAvailability | ||
| Initialize-PythonEnvironment | ||
|
|
||
| $python = Get-VenvPythonPath | ||
| if (-not (Test-Path $python)) { | ||
| throw "Python not found at $python" | ||
| } | ||
|
|
||
| $script = Join-Path $ScriptDir 'assemble_video.py' | ||
| $PythonArgs = @() | ||
| if ($ManifestPath) { $PythonArgs += '--manifest', $ManifestPath } | ||
| if ($OutputPath) { $PythonArgs += '--output', $OutputPath } | ||
| if ($PSBoundParameters.ContainsKey('Fps')) { $PythonArgs += '--fps', $Fps } | ||
| if ($Resolution) { $PythonArgs += '--resolution', $Resolution } | ||
|
|
||
| & $python $script @PythonArgs | ||
| if ($LASTEXITCODE -ne 0) { | ||
| throw "assemble_video.py exited with code $LASTEXITCODE" | ||
| } | ||
| } | ||
| catch { | ||
| Write-Error -ErrorAction Continue "Invoke-AssembleVideo.ps1 failed: $($_.Exception.Message)" | ||
| exit 1 | ||
| } | ||
| } | ||
50 changes: 50 additions & 0 deletions
50
.github/skills/experimental/demo-video/scripts/assemble-video.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| #!/usr/bin/env bash | ||
| # Copyright (c) 2026 Microsoft Corporation. All rights reserved. | ||
| # SPDX-License-Identifier: MIT | ||
| # | ||
| # assemble-video.sh | ||
| # Wrapper for assemble_video.py that resolves the skill's Python environment | ||
| # and delegates FFmpeg video assembly. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| SKILL_ROOT="$(dirname "${SCRIPT_DIR}")" | ||
|
|
||
| err() { | ||
| printf "ERROR: %s\n" "$1" >&2 | ||
| exit 1 | ||
| } | ||
|
|
||
| test_uv_availability() { | ||
| if ! command -v uv &>/dev/null; then | ||
| err "uv is required but was not found on PATH. Install with: curl -LsSf https://astral.sh/uv/install.sh | sh" | ||
| fi | ||
| } | ||
|
|
||
| initialize_python_environment() { | ||
| echo "Syncing Python environment via uv..." | ||
| uv sync --directory "${SKILL_ROOT}" | ||
| echo "Environment synchronized." | ||
| } | ||
|
|
||
| get_venv_python_path() { | ||
| echo "${SKILL_ROOT}/.venv/bin/python" | ||
| } | ||
|
|
||
| main() { | ||
| local python | ||
| local -a passthrough_args=("$@") | ||
|
|
||
| test_uv_availability | ||
| initialize_python_environment | ||
|
|
||
| python="$(get_venv_python_path)" | ||
| if [[ ! -x "${python}" ]]; then | ||
| err "Python not found at ${python}." | ||
| fi | ||
|
|
||
| "${python}" "${SCRIPT_DIR}/assemble_video.py" "${passthrough_args[@]}" | ||
| } | ||
|
|
||
| main "$@" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.