Skip to content

feat(skills): add demo-video skill for narrated video assembly - #2404

Merged
Alain Uyidi (auyidi1) merged 25 commits into
mainfrom
feat/demo-video-skill
Aug 6, 2026
Merged

feat(skills): add demo-video skill for narrated video assembly#2404
Alain Uyidi (auyidi1) merged 25 commits into
mainfrom
feat/demo-video-skill

Conversation

@auyidi1

Copy link
Copy Markdown
Contributor

Summary

Adds an experimental demo-video skill that assembles ordered frames or clips with per-segment narration into a narrated MP4 via FFmpeg. It closes the video-assembly gap in the Design Thinking demo workflow by bridging existing skills: tts-voiceover (narration WAVs) and vscode-playwright (frame capture).

What's included

  • SKILL.md — overview, segments.yml manifest schema, quick start, parameters reference, and the reuse bridge to tts-voiceover / vscode-playwright, plus FFmpeg prerequisites.
  • scripts/assemble_video.py — the assembler CLI (--manifest, --output, --fps, --resolution, -v/--verbose). Validates the manifest, resolves each segment's visual (frame image or clip) and narration WAV, derives per-segment duration from the narration length via ffprobe (overridable with duration:), renders normalized per-segment MP4s, and concatenates them. Uses subprocess.run with list args and no shell=True.
  • scripts/assemble-video.sh + scripts/Invoke-AssembleVideo.ps1 — dual bash/PowerShell wrappers mirroring the video-to-gif convention; the bash wrapper runs uv sync and forwards to the Python CLI.
  • examples/ — an example segments.yml manifest and a README describing the local fixture setup (fixtures are intentionally kept out of source control).
  • Testspytest suite (parse/validation, duration default+override, frame/clip branches, error paths, subprocess safety), an Atheris fuzz harness with corpus fixtures, and a Pester helper test.
  • Registration — added to the experimental and hve-core-all collections with regenerated plugin outputs.

Manifest schema

output: ./output/demo.mp4
resolution: 1280x720
fps: 24
segments:
  - type: frame
    visual: ./assets/intro.png
    narration: ./assets/intro.wav
    duration: 3.0
  - type: clip
    clip: ./assets/interaction.mp4
    narration: ./assets/interaction.wav

Validation

  • npm run validate:skills — pass
  • npm run test:py — 13 passed
  • npm run lint:collections-metadata / plugin:validate — pass
  • npm run lint:frontmatter — pass (demo-video files clean)
  • npm run validate:copyright — pass

Notes / follow-ups

  • Runtime dependency: FFmpeg + ffprobe must be on PATH. The end-to-end smoke run was not executed here because FFmpeg is not installed in the dev container.
  • The demo-setup orchestration (PR feat(skills): add demo-setup skill to design-thinking collection #1623) that would auto-generate the demo script and capture prototype frames is pending; until it merges, the DT → demo-video wiring is manual.

Checklist

  • Reviewed and validated by a qualified human reviewer

Add an experimental demo-video skill that assembles ordered frames or
clips with per-segment narration into a narrated MP4 via FFmpeg. Closes
the video-assembly gap in the Design Thinking demo workflow by bridging
tts-voiceover narration and vscode-playwright frame capture.

- Add SKILL.md with manifest schema, quick start, and reuse bridge
- Add Python assembler (assemble_video.py) with segments.yml validation,
  ffprobe-derived durations, and safe list-arg subprocess calls
- Add bash and PowerShell wrappers mirroring the video-to-gif convention
- Add pytest suite, Atheris fuzz harness, corpus fixtures, and Pester
  helper tests
- Register the skill in the experimental and hve-core-all collections and
  regenerate plugin outputs
@auyidi1
Alain Uyidi (auyidi1) requested a review from a team as a code owner July 6, 2026 23:24
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Scorecard details
PackageVersionScoreDetails
pip/atheris 3.1.0 🟢 6.1
Details
CheckScoreReason
Packaging⚠️ -1packaging workflow not detected
Maintained⚠️ 23 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 2
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
SAST⚠️ 0no SAST tool detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Binary-Artifacts🟢 10no binaries found in the repo
Code-Review🟢 7Found 23/30 approved changesets -- score normalized to 7
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing🟢 10project is fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Security-Policy🟢 10security policy file detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
pip/colorama 0.4.6 UnknownUnknown
pip/coverage 7.15.0 UnknownUnknown
pip/iniconfig 2.3.0 UnknownUnknown
pip/packaging 26.2 UnknownUnknown
pip/pluggy 1.6.0 UnknownUnknown
pip/pygments 2.20.0 UnknownUnknown
pip/pytest 9.1.1 UnknownUnknown
pip/pytest-cov 7.1.0 UnknownUnknown
pip/pytest-mock 3.15.1 UnknownUnknown
pip/pyyaml 6.0.3 UnknownUnknown
pip/ruff 0.15.20 UnknownUnknown
pip/tomli 2.4.1 UnknownUnknown

Scanned Files

  • .github/skills/experimental/demo-video/uv.lock

@codecov-commenter

Codecov Comments Bot (codecov-commenter) commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.23622% with 101 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.96%. Comparing base (197afb8) to head (3c1f6fd).

Files with missing lines Patch % Lines
.../experimental/demo-video/scripts/assemble_video.py 68.30% 71 Missing ⚠️
...mental/demo-video/scripts/Invoke-AssembleVideo.ps1 0.00% 30 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2404      +/-   ##
==========================================
+ Coverage   83.10%   85.96%   +2.85%     
==========================================
  Files         164       97      -67     
  Lines       22254    10530   -11724     
  Branches       29       29              
==========================================
- Hits        18495     9052    -9443     
+ Misses       3756     1475    -2281     
  Partials        3        3              
Flag Coverage Δ
docusaurus 95.07% <ø> (ø)
pester 86.22% <0.00%> (-0.27%) ⬇️
pytest 68.30% <68.30%> (-11.81%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...mental/demo-video/scripts/Invoke-AssembleVideo.ps1 0.00% <0.00%> (ø)
.../experimental/demo-video/scripts/assemble_video.py 68.30% <68.30%> (ø)

... and 70 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Eval Execution

Status: Passed — no merge-blocking failures (4 advisory assertion failure(s) present)

  • Artifacts evaluated: 1
  • Specs run: 1
  • Assertions passed: 5
  • Assertions failed (blocking): 0
  • Assertions failed (advisory): 4
  • Failed specs (merge-blocking): 0
Artifact Kind Status Specs Passed Failed (blocking) Failed (advisory)
demo-video skill ⚠️ advisory-fail 1 5 0 4

Legend — ✅ clean · ⚠️ advisory failures only (non-blocking) · ⏭️ skipped · ❌ merge-blocking failure

Only Failed specs (merge-blocking) gates this PR. Advisory assertion failures are signal-quality checks captured during iteration; review them, but they do not block merge and may be acceptable.

Use the canonical "Copyright (c) 2026 Microsoft Corporation. All rights reserved." form so Copyright Header Validation passes on PR #2404.
… narration

Recommend the tts-voiceover skill's Azure AI Speech (Azure AI Foundry) neural voices as the default for shareable narration and mark offline open-source TTS as a no-credential fallback only.
Add knowledge, tool-trigger, and bleed-detection stimuli for the demo-video skill so the eval stimulus-presence gate passes on PR #2404.
Replace the truncated 'narrat' stem with 'narration|narrated' so Spell Check passes on PR #2404.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new experimental demo-video skill to assemble ordered prototype frames/clips and per-segment narration WAVs into a single narrated MP4 using FFmpeg, and registers the skill across collections, plugins, and behavior evals.

Changes:

  • Added the demo-video skill package (docs, Python assembler CLI, bash/PowerShell wrappers, examples, and tests including a fuzz harness).
  • Registered demo-video in experimental and hve-core-all collections and regenerated plugin READMEs.
  • Extended skill behavior conformance eval coverage with demo-video knowledge/tool-trigger/bleed-detection stimuli.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
plugins/hve-core-all/README.md Adds demo-video to the packaged skills list
plugins/experimental/README.md Adds demo-video to the experimental skills list
evals/behavior-conformance/skill-behavior.eval.yaml Adds 3 new demo-video behavior-conformance stimuli and updates total count
collections/hve-core-all.collection.yml Registers demo-video skill in the hve-core-all collection
collections/hve-core-all.collection.md Lists demo-video in the hve-core-all collection documentation table
collections/experimental.collection.yml Registers demo-video skill in the experimental collection
collections/experimental.collection.md Lists demo-video in the experimental collection documentation table
.github/skills/experimental/demo-video/SKILL.md Documents the skill purpose, manifest schema, usage, and prerequisites
.github/skills/experimental/demo-video/pyproject.toml Declares Python dependencies + pytest/ruff config for the skill
.github/skills/experimental/demo-video/uv.lock Pins the Python environment for the skill
.github/skills/experimental/demo-video/scripts/assemble_video.py Implements the manifest-driven FFmpeg assembly CLI
.github/skills/experimental/demo-video/scripts/assemble-video.sh Bash wrapper to uv sync and run the Python assembler
.github/skills/experimental/demo-video/scripts/Invoke-AssembleVideo.ps1 PowerShell wrapper to uv sync and run the Python assembler
.github/skills/experimental/demo-video/tests/test_assemble_video.py Pytest coverage for manifest parsing/validation and ffmpeg subprocess safety
.github/skills/experimental/demo-video/tests/fuzz_harness.py Adds an Atheris/pytest-compatible fuzz harness for the manifest validator
.github/skills/experimental/demo-video/tests/DemoVideoHelpers.Tests.ps1 Adds Pester tests for wrapper parameter/argument forwarding
.github/skills/experimental/demo-video/tests/corpus/valid-manifest.yml Adds fuzz/test corpus fixture for a valid manifest shape
.github/skills/experimental/demo-video/tests/corpus/unknown-key.yml Adds fuzz/test corpus fixture for unknown-key validation
.github/skills/experimental/demo-video/tests/corpus/empty-segments.yml Adds fuzz/test corpus fixture for empty segments validation
.github/skills/experimental/demo-video/examples/segments.yml Adds an example segments manifest
.github/skills/experimental/demo-video/examples/README.md Documents how to set up local fixtures for the example manifest
Comments suppressed due to low confidence (1)

.github/skills/experimental/demo-video/examples/README.md:30

  • The file ends with a whitespace-only line (trailing spaces). This tends to fail markdownlint/formatters; remove the trailing spaces/extra blank line at EOF.
*🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.*

Comment thread .github/skills/experimental/demo-video/scripts/Invoke-AssembleVideo.ps1 Outdated
Comment thread .github/skills/experimental/demo-video/scripts/assemble_video.py Outdated
Comment thread .github/skills/experimental/demo-video/scripts/assemble_video.py
Comment thread .github/skills/experimental/demo-video/examples/README.md Outdated
Comment thread .github/skills/experimental/demo-video/scripts/assemble_video.py
Comment thread .github/skills/experimental/demo-video/SKILL.md
- Require PowerShell 7.4 in the wrapper to match repo conventions
- Make uv install hint platform-neutral (winget/curl)
- Remove dead subprocess.CalledProcessError handler
- Emit debug logging so --verbose has observable effect
- Drop unused manifest_path parameter from _validate_manifest
- Use H2 heading in examples README (title in frontmatter)
- Fix Pester fallback repo-root walk (5 levels)
- Remove stale 'planned' qualifier in SKILL.md
@github-actions github-actions Bot mentioned this pull request Jul 9, 2026
The _validate_manifest signature no longer takes manifest_path; update the pytest and fuzz-harness call sites accordingly.

@bindsi Marcel Bindseil (bindsi) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alain Uyidi (@auyidi1) if you don´t mind you can add the additional context from our chat yesterday to the readme as well. Thanks for this great skill

Jamie Kim (jkim323) and others added 2 commits July 10, 2026 21:57
# Conflicts:
#	collections/hve-core-all.collection.md
#	evals/behavior-conformance/skill-behavior.eval.yaml
#	plugins/hve-core-all/README.md

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the demo-video skill — the assembler/wrapper design is clean, the subprocess handling is injection-safe (list args, no shell=True, yaml.safe_load), and the repo Python-skill conventions (ruff/pytest/fuzz/uv.lock, dual wrappers, copyright headers) are all met. A few items I didn't see raised elsewhere on the PR:

  • Blocking: the shipped examples/segments.yml uses ./examples/fixtures/... paths, but the assembler resolves relative paths against the manifest's own directory (examples/), so they expand to examples/examples/fixtures/... and don't match the fixture layout examples/README.md documents — the flagship example fails on first run. Changing the manifest paths to ./fixtures/... (and output: ./output/demo.mp4) fixes it; please verify end-to-end since the PR notes the smoke run was skipped.
  • The npm run test:py 13 passed figure in the PR body isn't reproducible — the two discoverable pytest files contain 11 test functions; please re-run and update the count.
  • tests/DemoVideoHelpers.Tests.ps1 doesn't mirror its system under test (Invoke-AssembleVideo.ps1); renaming to Invoke-AssembleVideo.Tests.ps1 matches the Pester convention.
  • Minor: SKILL.md's Manifest Schema omits the top-level output/resolution/fps keys the code and example use; an empty-string narration skips the narration_wav fallback; and _validate_manifest returns the segments list redundantly.

Note

This review was AI-assisted and should be independently validated by a qualified human reviewer before acting on any finding.

auyidi added 2 commits July 13, 2026 18:47
…z scope

Validate effective frame rate is positive after merging CLI and manifest values, enforce that a declared segment type matches its visual or clip source, and narrow fuzz suppression to ManifestError so unexpected crashes surface. Adds tests for fps and type validations.
Coerce unexpected manifest keys to str when building error messages so non-string YAML keys raise ManifestError instead of a TypeError that crashes the Atheris fuzz target. Adds a regression test and applies ruff formatting.
Comment thread .github/skills/experimental/demo-video/examples/README.md

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dependency Review — demo-video Skill

This PR introduces a new Python skill with the following dependencies. No existing workflow, devcontainer, or copilot-setup-steps.yml files are modified, so SHA-pinning and environment-sync checks are not applicable.

Dependencies Introduced

Package Resolved Group License Notes
pyyaml 6.0.3 runtime MIT YAML manifest parsing — necessary, no alternative in stdlib
pytest 9.1.1 dev MIT Standard test runner
pytest-cov 7.1.0 dev MIT Coverage reporting
pytest-mock 3.15.1 dev MIT Mock support for pytest
ruff 0.15.20 dev MIT Linter — consistent with other skills
atheris 3.1.0 fuzz Apache-2.0 OSSF fuzz harness — consistent with repo convention
Transitive (colorama, coverage, iniconfig, packaging, pluggy, pygments, tomli) indirect MIT/BSD/Apache All permissive, no concerns

Safety Checks

Check Result
License compatibility (MIT project) ✅ All permissive (MIT, Apache-2.0, BSD-3-Clause)
New dependencies justified pyyaml is necessary for manifest parsing; dev/fuzz deps follow repo convention
Duplicates existing functionality ✅ No duplication detected
GitHub Actions SHA pinning ✅ No workflow files modified
Devcontainer / setup alignment ✅ Neither .devcontainer/ nor copilot-setup-steps.yml touched
Known vulnerabilities ✅ No known CVEs for these versions

All dependency safety checks passed. No action required from this review.

Generated by Dependabot PR Review for #2404 · 39.3 AIC · ⌖ 7.05 AIC · ⊞ 8.5K

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dependency Review — demo-video skill

Scope: pyproject.toml + uv.lock added under .github/skills/experimental/demo-video/


Dependencies introduced

Package Version constraint Group License Notes
pyyaml >=6.0 runtime MIT Actively maintained; widely used
pytest >=9.0 dev MIT Standard test framework
pytest-cov >=5.0 dev MIT Coverage plugin
pytest-mock >=3.14 dev MIT Mock plugin
ruff >=0.15 dev MIT Linter/formatter
atheris >=3.0 fuzz Apache-2.0 OSS fuzz harness; Apache-2.0 is MIT-compatible

Indirect lock-file entries (colorama, coverage, iniconfig, packaging, pluggy, pygments, tomli) are all MIT, BSD, or Apache-2.0 — all compatible with the project's MIT license.


Safety check results

  • License compatibility — all dependencies (runtime and dev) carry MIT, BSD, or Apache-2.0 licenses, compatible with the project's MIT license.
  • No duplicationpyyaml is the only runtime dependency; no existing repo-level dependency provides YAML parsing for this skill's use case.
  • SHA/hash pinninguv.lock pins every wheel and sdist by SHA-256 hash. No GitHub Actions references were modified in this PR.
  • Environment synchronization — no changes to .devcontainer/ or copilot-setup-steps.yml; synchronization concern is N/A.
  • Vulnerability posture — all packages are well-established, actively maintained, and carry no known vulnerability signals. atheris is correctly isolated in the fuzz dependency group (no macOS wheels) per repo convention.
  • uv.lock committed — lock file is present alongside pyproject.toml, satisfying Dependabot resolution requirements.

All dependency safety checks passed. No findings require changes.

Generated by Dependabot PR Review for #2404 · 42.7 AIC · ⌖ 8.7 AIC · ⊞ 8.5K

… tts-voiceover

The main merge left collections/hve-core-all.collection.md out of sync with its YAML source: the generated markdown retained the demo-video entry but dropped the tts-voiceover row that the YAML still lists. Regenerate via plugin:generate so the table includes both skills and prevent a merge-time regression that would remove tts-voiceover from main.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 3 comments.

Comment thread .github/skills/experimental/demo-video/scripts/assemble_video.py
Comment thread .github/skills/experimental/demo-video/scripts/assemble_video.py
Comment thread .github/skills/experimental/demo-video/examples/README.md Outdated
Alain Uyidi (auyidi1) and others added 3 commits July 15, 2026 09:28
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment thread .github/skills/experimental/demo-video/examples/segments.yml Outdated
Comment thread .github/skills/experimental/demo-video/SKILL.md

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewing at 2f2d912 — the type/source enforcement, --fps validation, non-string-key handling, and tighter fuzz scope from the follow-up commits all look good and are covered by new tests. I'm not repeating the items already open on the PR (the examples/segments.yml path-doubling thread and the Pester filename / SKILL.md top-level-keys notes from Chris Montazer (@rezatnoMsirhC) still apply and remain unresolved — the example still fails on first run). A few net-new items I didn't see raised elsewhere:

  • Please verify (from the concat change): the final concat now uses -c copy (per the earlier suggestion) instead of re-encoding. With fractional per-segment durations and fixed-size AAC frames, stream-copy concat can drift A/V sync or leave gaps at segment seams that the previous re-encode masked. Since FFmpeg isn't in the dev container and the smoke run is still skipped, this primary output path is untested — please confirm the final MP4 stays in sync end-to-end, or keep a re-encode fallback for the concat step.
  • Add a shipped-example regression test: a test that loads examples/segments.yml and asserts the resolved fixture paths match the README layout would guard the open path-doubling issue from recurring.
  • PR body count: npm run test:py 13 passed isn't reproducible — there are now 14 discoverable test functions for this skill (12 in test_assemble_video.py + 2 in fuzz_harness.py). Please re-run and update the figure (or note it''s a repo-wide aggregate).
  • Minor: an empty-string narration: still skips the narration_wav fallback (only None/absent falls back); and _validate_manifest returns the segments list a second time even though the config dict it returns already contains segments.

Also note the branch is currently BEHIND main and the four approvals predate the still-open 07-15 findings, so a fresh look after the fixes would be worthwhile.

Note

This review was AI-assisted and should be independently validated by a qualified human reviewer before acting on any finding.

- Correct example manifest paths to resolve relative to examples/ (./fixtures, ./output)
- Document top-level output/resolution/fps manifest keys in SKILL.md
- Rename Pester test to Invoke-AssembleVideo.Tests.ps1 to match script under test
Resolve conflicts by adopting the marketplace snapshot architecture:
- Delete collections/ and plugins/ trees (removed on main via PR #2577)
- Register skills/experimental/demo-video in .github/plugin/marketplace.json
  (experimental and hve-core-all plugins: skills arrays + componentMaturity)
- Resolve skill-behavior.eval.yaml stimulus Total to 133
- Generate docs/reference/skills/experimental/demo-video.md and refresh index/position pages
@auyidi1
Alain Uyidi (auyidi1) merged commit 620854c into main Aug 6, 2026
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants