Nova Motion

One prompt. One pipeline. Scripted, generated, narrated, and rendered into a finished, platform-ready video.

What is Nova Motion?

Nova Motion is an end-to-end AI video generation engine designed to transform simple text prompts and concepts into production-grade, platform-ready videos. Built on Next.js, Remotion, and an asynchronous Express render pipeline, Nova Motion orchestrates story writing, visual generation, voiceover synthesis, kinetic captioning, and final video rendering into one continuous automated workflow.

The platform provides a suite of creation engines tailored for distinct video formats, including AI storyboards, stock footage shorts, typography slides, 3D motion graphics, text-to-video B-rolls, micro drama narratives, UGC video ads, concept-to-video campaigns, Luma Ray 3.2 cinematic clips, Vox-style paper-collage explainers, Zack D Films-style 3D curiosity shorts, AI comic / anime drama episodes, stickman explainers, and whiteboard-style drawing explainers.

Media assets, voiceover tracks, and final rendered MP4 videos are managed through a unified storage pipeline, with support for local disk storage as well as cloud-native object storage such as DigitalOcean Spaces and Cloudflare R2.

Highlights

17 Modes, One API AI storyboards, stock shorts, typography slides, motion graphics, AI text-to-video, micro drama, UGC ads, agentic video, Luma Ray 3.2, Vox collage explainers, Zack D-style 3D curiosity shorts, comic drama episodes, stickman explainers, and whiteboard animation explainers - all behind a single endpoint.	Agentic Pipelines LLMs write scripts, cast characters, plan storyboards, and direct every shot inside the job queue.	Kinetic Captions ElevenLabs / Deepgram TTS narration with word-level subtitles synced to the voiceover.
AI-Generated Assets Seedream collage posters, Seedance text/image-to-video clips, Lyria background music, and gpt-image illustrations.	Async by Design Submit once, poll `GET /api/videos/{jobId}`, download the finished mp4.	Cloud-Native Storage Local disk by default; DigitalOcean Spaces or Cloudflare R2 when configured.

Tech Stack

Category	Technologies
Frontend	Next.js 14, React, TypeScript, Tailwind CSS, shadcn/ui
Video rendering	Remotion, Express.js render queue, headless Chromium
LLMs	OpenAI GPT-4o-mini, WaveSpeed LLM
Image generation	OpenAI gpt-image-2, WaveSpeed Seedream
Video generation	WaveSpeed Seedance, Luma Ray 3.2, Veo 3.1, Grok Video, Happy Horse
Music	WaveSpeed Lyria
TTS / STT	ElevenLabs, Deepgram (Aura TTS, Nova-2 STT)
Stock media	Pexels, Pixabay
Storage	Local disk, DigitalOcean Spaces, Cloudflare R2 (S3-compatible)

Current Outputs

See Nova Motion in action - click a preview to watch the full video.

AI Storyboard	Text to Video
Stock Video	Stock Image

Studio Preview

Project Structure

├── src/                     # Next.js API gateway + Remotion compositions + studio UI
│   ├── app/                 #   API routes (POST /api/videos, GET /api/videos/:jobId, /ugc)
│   ├── lib/                 #   Generation pipelines (vox, wavespeed, luma, ugc, ...)
│   ├── remotion/            #   Remotion compositions (WavespeedVideo, ...)
│   └── components/          #   Studio UI components
├── render-server/           # Express queue rendering backend (headless Chromium)
│   ├── server.ts            #   POST /render/* endpoints
│   ├── renderer.ts          #   Pipeline dispatch + Remotion rendering
│   ├── queue.ts             #   Job queue state
│   └── storage.ts           #   Local / S3 storage layer
├── shared/                  # Shared request & response schemas
├── public/                  # Static assets (incl. the Nova Motion logo)
├── .env.local.example       # Every environment variable documented
└── package.json

Video Generation Models (Compositions)

1. AI Storyboard Video (`videoType: "AIStoryboardVideo"`)

Produces highly engaging story videos in multiple aspect ratios (9:16, 16:9, 1:1) using gpt-image-2 image generation.

GPT-4o-mini: Writes a narrative story script and detailed visual scene descriptions.
gpt-image-2: Generates high-resolution illustrations for each scene.
Deepgram Aura TTS & Nova-2 STT: Synthesizes natural narration and transcribes word-level offsets for kinetic subtitle alignments.

2. Stock Video Short (`videoType: "StockVideo"`)

Generates videos in multiple aspect ratios (9:16, 16:9, 1:1) by sourcing stock clips from Pexels.

GPT-4o-mini: Writes the script and outputs context-based search keywords.
Pexels API: Fetches relevant stock video loops.
Background Music: Low-volume background tracks mixed under the voiceover.
Deepgram TTS + STT: Narrates and maps word captions.

3. Stock Image Short (`videoType: "StockImage"`)

Generates videos in multiple aspect ratios (9:16, 16:9, 1:1) by sourcing stock images from Pixabay.

GPT-4o-mini: Writes the script and outputs context-based search keywords.
Pixabay API: Fetches relevant stock images.
Background Music: Low-volume background tracks mixed under the voiceover.
Deepgram TTS + STT: Narrates and maps word captions.

4. Typography/Layout Slide Videos (`videoType: "SocialMedia" | "Explainer" | "General" | "TextAnimation"`)

Clean layouts using modern typographic animations and styles.

Claude/OpenAI: Writes structured JSON scripts defining slide colors, text, and timing.
Deepgram: Overlays audio voiceovers.

5. Motion Graphics & Data Visualizations (`videoType: "MotionGraphics"`)

Produces premium, highly animated visual components using a dynamic, unified JSON slide renderer.

OpenAI (GPT-4o-mini): Generates a complete storyboard containing glitch/neon/wave text, bar/pie/line charts, growth metrics, and dynamic background layouts in strict JSON.
Deepgram & Background Music: Synthesizes custom TTS narration for each slide and overlays background audio tracks.

6. AI Text-to-Video (`videoType: "TextToVideo"`)

Generates a full AI B-roll video: creates the script, narrates it, then generates each B-roll segment with WaveSpeed's Seedance text-to-video model.

OpenAI (GPT-4o-mini): Writes a short facts-style script from the topic.
Deepgram/ElevenLabs TTS: Narrates the script and returns native word-level timestamps.
WaveSpeed Seedance: Generates an AI video clip per timed B-roll segment using the segment's visual keywords. Resolution via WAVESPEED_VIDEO_RESOLUTION (default 480p), clip length via WAVESPEED_VIDEO_DURATION (default 5s, 3–10s); segments are sized to match the clip length so nothing is wasted.
WaveSpeed Lyria: Generates a cinematic background music bed (fallback to stock tracks).
Pexels Fallback: If WaveSpeed clip generation fails for a segment, a matching Pexels stock clip is used instead.
Kinetic Captions: The narration words are highlighted in sync with the voiceover.

7. Micro Drama (`videoType: "MicroDrama"`)

A full agentic story-to-video workflow where the AI acts as screenwriter, casting director, storyboard artist, and cinematographer.

WaveSpeed LLM (WAVESPEED_LLM_MODEL, default deepseek/deepseek-v4-flash): Writes the story from an idea (or consumes a supplied script), extracts consistent characters, writes per-scene scripts, and designs a shot-by-shot storyboard (visual + motion + audio descriptions).
WaveSpeed Seedream T2I: Generates character portraits (WAVESPEED_PORTRAIT_MODEL) and a scene first-frame image per shot (WAVESPEED_FRAME_MODEL).
WaveSpeed Seedance I2V (WAVESPEED_I2V_MODEL): Animates each shot's first-frame into a clip (default 5s, 720p) with native audio (ambient SFX — no TTS voiceover).
Two modes: pass only idea (the AI writes the whole story) or pass a script (the raw script is used directly, skipping the screenwriter).
Async: the request returns a jobId immediately; the whole pipeline runs inside the render job and is polled via GET /api/videos/{jobId}.

8. AI UGC Ad (`videoType: "UGC"`)

An AI UGC video ad studio utilizing high-performance image-to-video and text-to-video models with custom reference avatar inputs.

Model picker: Veo 3.1 (google/veo3.1/*), Seedance 2 (bytedance/seedance-2.0/*), Grok Video (x-ai/grok-imagine-video/*), and Happy Horse 1 (alibaba/happyhorse-1.0/*), each with text-to-video and image-to-video endpoints. Default model via UGC_DEFAULT_MODEL (default seedance-2).
T2V vs I2V: with no reference images, the model's text-to-video endpoint is used; with reference image(s), the image-to-video endpoint animates the image (e.g. an actor face) with native audio from the script.
Reference images: upload up to 7 images via POST /api/upload (returns a hosted URL) and reference them inline in the script as @image1, @image2, etc.
Per-model controls: aspect ratio, duration, and resolution (Grok clips are 6 or 10s and limited to 16:9 / 1:1 / 9:16 per WaveSpeed's contract; no mode presets are available).
Multi-scene UGC: with multiScene: true the script is broken into 2-6 scenes by the WaveSpeed LLM following the UGC beat structure (Hook → Problem → Product → Demo → Outcome → CTA, kept honest — no fake testimonials/guarantees invented), a single TTS voiceover (ElevenLabs preferred, Deepgram fallback) is generated for the whole ad, each scene is generated as its own clip (I2V with a reference image to keep the same actor, T2V otherwise), and the clips are cut together into one timeline with kinetic captions — played by the UGC Remotion composition. voice selects the voiceover, targetDurationSec (10-60) hints the total length. Lip-sync (lipSync, default on) re-animates each clip's mouth to match the TTS via WaveSpeed sync/lipsync-2 (~$0.05/run + ~2 min per scene, non-fatal fallback; requires the TTS audio to be publicly reachable via RENDER_SERVER_BASE_URL). Requires ELEVENLABS_API_KEY or DEEPGRAM_API_KEY.
Async: the request returns a jobId immediately; the WaveSpeed generation runs inside the render job, the finished mp4 is downloaded into rendered-videos/, uploaded to S3/R2/Spaces when configured, and polled via GET /api/videos/{jobId}.
UI: a full studio lives at /ugc (model cards, image upload with previews, template library with 18 proven UGC scripts — meal planning, beauty, SaaS, pet, local service, etc. — that load into the script box, param pickers, generation-style toggle, live polling player).

9. Agentic Video Generator (`videoType: "AgenticVideoGenerator"`)

Executes an end-to-end concept-to-video production pipeline, combining agentic scripting, character casting, and high-resolution video models.

Production agents: WaveSpeed LLM develops a screenplay, extracts characters, plans scenes, and writes visual, camera, and sound direction.
Character and scene generation: WaveSpeed bytedance/seedream-v5.0-pro generates shot keyframes; verified video options are Seedance 2.0 (bytedance/seedance-2.0/image-to-video), Seedance 2.0 Fast (bytedance/seedance-2.0-fast/image-to-video), Veo 3.1 Fast Reference (google/veo3.1-fast/reference-to-video), MiniMax H3 Image-to-Video (minimax/h3/image-to-video), and MiniMax H3 Reference-to-Video (minimax/h3/reference-to-video).
Strict model execution: this mode has no Pexels, stock, or alternate-provider fallback. It validates the selected model's ratios, duration, resolution, and reference-image limits before submission. A provider failure fails the job with the provider error.
Talking scenes: when lipSync is enabled, ElevenLabs audio is sent to verified WaveSpeed wavespeed-ai/infinitetalk. No lip-sync fallback is used.
Assembly and export: the generated clips are assembled by Remotion, persisted through the existing local/S3/R2 storage layer, and returned through the standard async job API.
Platform presets: youtube, linkedin, and standard default to 16:9; instagram_reels and tiktok default to 9:16. An explicit aspectRatio overrides the platform preset.
Async: the endpoint returns a jobId immediately. Poll GET /api/videos/{jobId} for progress and the final URL.

Verified payload constraints:

Seedance I2V uses prompt, image, aspect_ratio, resolution, duration from 4-15 seconds, and generate_audio.
Veo Fast Reference uses prompt, 1-3 images, 16:9 or 9:16, 720p or 1080p, and fixed 8-second output.
MiniMax H3 I2V uses image, prompt, resolution: "2k", and duration from 5-15 seconds.
MiniMax H3 Reference uses prompt, 1-9 reference_images, supported aspect_ratio, resolution: "2k", and duration from 5-15 seconds.
Seedream V5 Pro uses prompt, aspect_ratio, resolution of 1k or 2k, and output_format of jpeg or png.
InfiniteTalk uses image, audio, optional prompt, resolution of 480p or 720p, and seed.

{
  "videoType": "AgenticVideoGenerator",
  "title": "AI Manufacturing Copilot",
  "brief": "Introduce an AI copilot that helps factory teams diagnose downtime and improve throughput.",
  "targetAudience": "Manufacturing executives",
  "durationSeconds": 45,
  "language": "English",
  "tone": "Professional and optimistic",
  "keyMessages": ["Faster diagnosis", "Safer decisions", "Measurable throughput"],
  "callToAction": "Book a factory assessment",
  "platform": "linkedin",
  "style": "Premium industrial documentary, realistic lighting"
}

10. Luma AI Ray 3.2 (`videoType: "Luma"`)

Implements every core capability of Luma's Ray 3.2 model in a single unified mode with capability resolution, multi-scene screenplay planning, extend chaining, and layered ElevenLabs voiceover + kinetic captions.

All Ray 3.2 capabilities: Text-to-Video, Image-to-Video (start/end frames & multi-keyframes up to 64 guide anchors), Seamless Loop, Forward Extend (start_frame: { generation_id }), Video Editing (video_edit with adhere_1-3, flex_1-3, reimagine_1-3), and Video Reframing (video_reframe with target aspect ratio and position controls).
Scenario templates: Preconfigured prompts for UGC Product Posts, Product Showcase Ads, Product Launch Reveals, Real Estate Walkthroughs, Event Promos, Educational Tutorials, and Social Media Reels.
Extending vs Joining: Seamlessly extends continuous takes using Luma generation_id chaining (transition: "extend") or sequences distinct shots on the Remotion timeline (transition: "cut").
Audio Layering: Ray 3.2 generates silent video clips; the pipeline synthesizes narration with ElevenLabs TTS, extracts word timestamps, and overlays animated kinetic captions on the rendered video.
Async: Returns a jobId immediately. Poll GET /api/videos/{jobId} for render status.

{
  "videoType": "Luma",
  "prompt": "Show a futuristic smartwatch with holographic health metrics in action",
  "useCase": "product_ad",
  "aspectRatio": "16:9",
  "resolution": "720p",
  "duration": "5s",
  "generateAudio": true,
  "voice": "EXAVITQu4vr4xnSDxMaL"
}

11. Vox Collage Explainer (`videoType: "VoxVideo"`)

A Vox-style paper-collage explainer generator producing torn-paper collage posters and animated clips.

Beat map: A hook-led, arc-driven narrative structure (6 arcs) is written by the WaveSpeed LLM, with per-beat headlines, narration, scene specs, and element motion.
Collage posters: Each beat is generated as a finished torn-paper collage poster by Seedream using a strict 5-part Vox prompt formula (style block, scene, background, headline banner, technical) across 4 visual themes (American Retro, Swiss Modern, Punk Zine, Chinese Ink).
Motion: Each poster is animated into a clip by Seedance image-to-video using the beat's camera move + element-motion prompt.
Voice & captions: ElevenLabs (preferred) / Deepgram TTS narration with word-level kinetic captions, plus optional Lyria background music — all assembled by the WavespeedVideo composition.
Async: Returns a jobId immediately. Poll GET /api/videos/{jobId} for render status.

{
  "videoType": "VoxVideo",
  "prompt": "A brief history of coffee",
  "theme": "american-retro",
  "arc": "timeline",
  "targetDurationSeconds": 30,
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

12. Zack D Films 3D Short (`videoType: "ZackDVideo"`)

A Zack D Films-style stylized 3D curiosity short — one topic in, a scroll-stopping animated explainer out.

Curiosity-loop script: The WaveSpeed LLM writes a hook-led beat map where every beat opens a question the viewer needs answered (myth-buster and "what if" formulas), with 2 shots per beat (wide establishing + macro cross-section cutaway) and alternating camera moves.
Character consistency sheets: Orthographic turnaround sheets (Seedream, front/side/¾ views for characters; cross-section cutaways for organs/assets) are generated first and anchored into every keyframe prompt so identity stays locked across shots.
Keyframes → motion clips: Each shot is rendered as a 3D keyframe (plasticine clay subsurface-scattering style block, macro 85mm lighting formula) then animated by image-to-video (Seedance 2.0 by default, ZACK_D_I2V_MODEL=google/veo3.1/image-to-video to switch).
Signature editing: Impact zooms (slow push-in on every "B" shot) and the fade / wipe / slide / circle-open transition cycle are rendered natively by the ZackDVideo Remotion composition — no ffmpeg required.
Voice & captions & music: ElevenLabs (preferred) / Deepgram TTS with word-level kinetic captions, plus optional Lyria background music.
Async: Returns a jobId immediately. Poll GET /api/videos/{jobId} for render status.

{
  "videoType": "ZackDVideo",
  "prompt": "What happens when you swallow gum?",
  "targetDurationSeconds": 30,
  "sceneCount": 4,
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

13. AI Comic Drama Episode (`videoType: "ComicDramaVideo"`)

An AI comic / anime drama generator (AIComicBuilder replication) — one premise or raw script in, a fully animated story episode out.

Story plan: The WaveSpeed LLM parses any premise or raw script into a short episode: it extracts recurring characters with vivid visual descriptions, auto-detects the art style (anime / manga / comic book / 3D / realistic), and writes a continuity-chained shot list where each shot carries a FIRST and LAST keyframe description, camera direction, and 0-2 dialogue lines.
4-view character sheets: Each character gets an orthographic reference sheet (front / ¾ / side / back) rendered by Seedream in the detected style, with the character's name printed on the sheet for identity anchoring.
Reference-locked keyframes: The sheets are attached as real reference images to Seedream v4 Edit ($0.027/image, up to 10 refs) — and each shot's LAST keyframe receives its own FIRST frame as the primary visual reference — so identity, outfits and environments are visually locked instead of merely described (falls back to text-to-image if the edit model fails; override via COMIC_KEYFRAME_EDIT_MODEL).
First/last keyframe pairs: Every shot is generated twice — an opening frame anchored on the character sheets (and chained from the previous shot's end state) plus a closing frame locked to the opening frame's environment, lighting and outfits, with only pose/expression/position changed.
Start/end interpolation clips: The keyframe pair is animated by Wan 2.2 image-to-video using its last_image parameter ($0.10/5s; switch to wavespeed-ai/wan-flf2v via COMIC_I2V_MODEL for the dedicated first-last-frame model).
Dialogue & subtitles: Per-shot dialogue is voiced by ElevenLabs (preferred) / Deepgram TTS with word-level timestamps, rendered as comic-style caption strips; optional Lyria background music.
Comic assembly: Hard cuts with white panel-flash transitions and a slamming title card are rendered natively by the ComicDramaVideo Remotion composition — no ffmpeg required.
Async: Returns a jobId immediately. Poll GET /api/videos/{jobId} for render status.

{
  "videoType": "ComicDramaVideo",
  "prompt": "A shy ramen shop cat secretly defends the neighborhood from spirits at night",
  "targetDurationSeconds": 30,
  "sceneCount": 6,
  "artStyle": "anime",
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

14. Stickman Explainer Short (`videoType: "StickmanExplainerVideo"`)

A Stickman-Studio-style educational explainer generator — one topic in, a narrated stickman animation out. The cheapest mode in the suite: slideshow mode costs ~$0.05/video.

Viral storyboard: The WaveSpeed LLM turns any topic into a fast-paced scene plan following viral Shorts rules — scene 1 is always a hook (shocking fact / weird question / extreme visual), narration stays punchy and conversational, and the stickman performs exaggerated physical comedy.
One canonical character reference: A single T-pose stickman reference sheet (Seedream) anchors the whole video's identity.
Reference-locked scenes: Every scene image is generated by Seedream v4 Edit with the character reference attached as a real image input ($0.027/image; falls back to text-to-image; override via STICKMAN_IMAGE_EDIT_MODEL).
Slideshow mode (default, near-free): Scene stills are animated natively in Remotion with an alternating Ken Burns zoom/pan (the reference project's ffmpeg zoompan, re-implemented as spring math) — zero video-generation cost.
Animated mode: Each scene becomes a Wan 2.2 I2V clip ($0.10/5s; override via STICKMAN_I2V_MODEL) that plays the scene's action with cartoon motion while preserving the flat line-art world.
Narration & captions: Per-scene narration voiced by ElevenLabs (preferred) / Deepgram TTS with word-level timestamps, rendered as lower-third white-on-black-stroke captions; optional Lyria background music.
Async: Returns a jobId immediately. Poll GET /api/videos/{jobId} for render status.

{
  "videoType": "StickmanExplainerVideo",
  "prompt": "How black holes work",
  "targetDurationSeconds": 40,
  "sceneCount": 5,
  "animation": "slideshow",
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

15. Whiteboard Animation Explainer (`videoType: "WhiteboardVideo"`)

A Storyboard-AI-style whiteboard drawing explainer — any topic becomes a narrated whiteboard animation with hand-drawn line-art scenes, optional AI motion clips, and optional SAM3 Video object segmentation. Replicates the full Storyboard-AI pipeline (director.py + image_prompt_tool.py + SAM3 + OpenCV drawing) entirely on the WaveSpeed stack.

LLM Director Agent: DeepSeek (via WaveSpeed LLM) turns a topic into a scene-by-scene whiteboard script with narration, image prompts following the Nano Banana formula (whiteboard line-art, thick black outlines, marker strokes, off-white surface), and optional object lists for segmentation.
Seedream whiteboard images: Each scene gets a unique Seedream-generated whiteboard line-art image ($0.027/image; falls back to text-to-image; override via WHITEBOARD_IMAGE_MODEL).
Slideshow mode (default, near-free): Scene stills are animated natively in Remotion with alternating Ken Burns zoom/pan — zero video-generation cost.
Animated mode: Each scene becomes a Seedance I2V clip ($0.06/scene; override via WHITEBOARD_I2V_MODEL) that plays the scene's action with cartoon motion while preserving the whiteboard line-art style.
Optional SAM3 Video segmentation: When animation: "animated", each clip can be passed through WaveSpeed's SAM3 Video API to isolate key objects — the closest equivalent to Storyboard-AI's multi-pass per-object OpenCV drawing effect ($0.05/5s billed).
Narration & captions: Per-scene narration voiced by ElevenLabs (preferred) / Deepgram TTS with word-level timestamps, rendered as lower-third captions; optional Lyria background music.
Async: Returns a jobId immediately. Poll GET /api/videos/{jobId} for render status.

{
  "videoType": "WhiteboardVideo",
  "prompt": "How photosynthesis works",
  "targetDurationSeconds": 40,
  "sceneCount": 5,
  "animationStyle": "slideshow",
  "tone": "curious and energetic",
  "language": "English",
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

Quick Start (Local Development)

1. Install Dependencies

npm install

2. Set Up Environment Variables

Create a .env.local file in the root of the project (copy from .env.local.example):

# ============ AI API KEYS (Required) ============
# AIStoryboardVideo (gpt-4o-mini + gpt-image-2) + Luma LLM fallback
OPENAI_API_KEY=your-openai-api-key
# Primary TTS (native word timestamps) — used by almost every mode
ELEVENLABS_API_KEY=your-elevenlabs-api-key
# Fallback TTS + STT word timestamps when ElevenLabs is absent
DEEPGRAM_API_KEY=your-deepgram-api-key
# StockVideo mode (Pexels stock loops)
PEXELS_API_KEY=your-pexels-api-key
# StockImage mode (Pixabay stock images)
PIXABAY_API_KEY=your-pixabay-api-key
# TextToVideo, MicroDrama, UGC modes (Seedance/Seedream/Lyria) + Luma LLM fallback
WAVESPEED_API_KEY=your-wavespeed-api-key
# Luma mode (Ray 3.2 video generation)
LUMA_AGENTS_API_KEY=your-luma-agents-api-key

# ============ Render Server ============
RENDER_SERVER_URL=http://localhost:3001
RENDER_SERVER_BASE_URL=http://localhost:3001
RENDER_SERVER_SECRET=your-secret-string
RENDER_SERVER_PORT=3001
RENDER_CONCURRENCY=2
NEXT_PUBLIC_APP_URL=http://localhost:3000

# ============ WaveSpeed model defaults (optional) ============
WAVESPEED_VIDEO_MODEL=bytedance/seedance-v1-pro-fast/text-to-video
WAVESPEED_VIDEO_RESOLUTION=480p
WAVESPEED_VIDEO_DURATION=5
WAVESPEED_LLM_MODEL=deepseek/deepseek-v4-flash
WAVESPEED_IMAGE_MODEL=bytedance/seedream-v4.5
WAVESPEED_PORTRAIT_MODEL=bytedance/seedream-v4.5
WAVESPEED_FRAME_MODEL=bytedance/seedream-v4.5
WAVESPEED_I2V_MODEL=bytedance/seedance-2.0/image-to-video
UGC_DEFAULT_MODEL=seedance-2

# ============ Luma LLM defaults (optional) ============
LUMA_LLM_URL=https://llm.wavespeed.ai/v1/chat/completions
LUMA_LLM_MODEL=deepseek/deepseek-v4-flash

# ============ Storage (optional — local disk fallback) ============
# DigitalOcean Spaces
SPACES_ENDPOINT=https://nyc3.digitaloceanspaces.com
SPACES_KEY=your-do-access-key
SPACES_SECRET=your-do-secret-key
SPACES_BUCKET_NAME=your-bucket-name
SPACES_PUBLIC_URL=https://your-custom-cdn.com
# Cloudflare R2 (alternative)
R2_ACCOUNT_ID=your-r2-account-id
R2_ACCESS_KEY_ID=your-r2-access-key-id
R2_SECRET_ACCESS_KEY=your-r2-secret-access-key
R2_BUCKET_NAME=your-bucket-name
R2_PUBLIC_URL=https://your-account-id.r2.dev/your-bucket-name
R2_DELETE_LOCAL_AFTER_UPLOAD=false

A complete reference with per-key comments and setup links lives in .env.local.example. Each API key unlocks the modes it documents above; without a key the corresponding mode returns an error at submission time.

3. Start the Render Server

The rendering backend handles the queue and runs chromium in headless mode to render the video files.

npm run render-server

4. Start the Next.js API Gateway

In a new terminal:

npm run dev

API Documentation

1. Generate Video (`POST /api/videos`)

Submits a prompt to generate and render a new video.

Request Parameters

All endpoints support the following root payload fields:

prompt (string, required): The core instruction, script, topic, or concept to generate.
videoType (string, required): The video template mode. Must be one of:
- "AIStoryboardVideo": AI Storyboard mode (gpt-image-2 images + voiceover).
- "StockVideo": Pexels stock footage mode (stock video loops + voiceover).
- "StockImage": Pixabay stock image mode (stock images + voiceover).
- "SocialMedia": Kinetic typographic slide style (quotes/shorts).
- "Explainer": Multi-step layout slides supporting step numbers.
- "General": Simple slide layout transitions.
- "TextAnimation": Words with active highlighting animations.
- "MotionGraphics": Dynamic motion graphics, charts, and interactive UI component simulations.
- "TextToVideo": AI-generated B-roll from WaveSpeed Seedance + TTS voiceover (Text-To-Video-AI replication).
- "MicroDrama": Full agentic pipeline — AI story, characters, storyboard, and Seedance I2V clips (Open-AI-Micro-Drama-Generator replication).
- "UGC": AI UGC ad studio — script + optional reference images → Veo/Grok/Seedance/Happy Horse clip with native audio (Open-AI-UGC replication via WaveSpeed).
- "AgenticVideoGenerator": Concept to screenplay, casting, storyboard, AI scenes, audio, and platform-ready video.
- "Luma": Unified Luma Ray 3.2 mode — text-to-video, image-to-video, loop, extend, video edit, reframe + TTS voiceover & kinetic captions.
- "VoxVideo": Vox-style paper-collage explainer — LLM beat map → Seedream collage posters → Seedance animated clips + TTS voiceover, captions & music.
- "ZackDVideo": Zack D Films-style 3D curiosity short — curiosity-loop script → character turnaround sheets → keyframes → I2V clips + impact zooms, transitions, captions & music.
- "ComicDramaVideo": AI comic / anime drama episode — story plan → 4-view character sheets → first/last keyframe pairs → interpolated I2V clips + dialogue TTS & comic subtitles.
- "StickmanExplainerVideo": Stickman-Studio-style educational short — hook storyboard → stickman character reference → reference-locked scene images → Ken Burns slideshow or animated I2V clips + narrator TTS & captions.
aspectRatio (string, optional): Target video layout format. Must be one of:
- "9:16": Portrait (mobile vertical) - defaults to 1080x1920.
- "16:9": Landscape (widescreen desktop) - defaults to 1920x1080.
- "1:1": Square (social feed block) - defaults to 1080x1080.
durationSec (number, optional): Target video duration (under 45 seconds is recommended).
topic (string, optional): Context topic used to direct scriptwriting style.
voice (string, optional): Deepgram voice key to customize narrator voice (e.g., "aura-2-aries-en", "aura-2-arcas-en", "aura-2-luna-en").
style (object, optional): Global branding override parameters:
- primaryColor (string): Background or primary element hex code (e.g. "#020617").
- secondaryColor (string): Accent highlight hex code (e.g. "#38bdf8").
- textColor (string): Slide foreground text hex code (e.g. "#ffffff").

Endpoint A: AI Storyboard Video (`videoType: "AIStoryboardVideo"`)

Produces animated vertical or landscape shorts using gpt-image-2 base64 illustration generation and Deepgram TTS voiceover.

Example Payload:

{
  "prompt": "The mystery of the Oak Island money pit",
  "videoType": "AIStoryboardVideo",
  "topic": "History",
  "aspectRatio": "16:9",
  "voice": "aura-2-aries-en"
}

Endpoint B: Stock Video Short (`videoType: "StockVideo"`)

Produces shorts using context-matched Pexels stock video footage, Deepgram TTS voiceover, and background music overlays.

Example Payload:

{
  "prompt": "Top 3 healthy habits for programmers",
  "videoType": "StockVideo",
  "topic": "Health",
  "aspectRatio": "16:9",
  "voice": "aura-2-arcas-en"
}

Endpoint C: Stock Image Short (`videoType: "StockImage"`)

Produces shorts using context-matched Pixabay stock images, Deepgram TTS voiceover, and background music overlays.

Example Payload:

{
  "prompt": "Why reading daily is key to success",
  "videoType": "StockImage",
  "topic": "Education",
  "aspectRatio": "9:16",
  "voice": "aura-2-aries-en"
}

Endpoint D: Typographic Slide Videos (`videoType` Options: `"SocialMedia" | "Explainer" | "General" | "TextAnimation"`)

Outputs animated text slides with dynamic TTS voiceover. Layout automatically adapts font dimensions to the target aspect ratio.

Example Payload:

{
  "prompt": "A quote by Steve Jobs about design",
  "videoType": "SocialMedia",
  "durationSec": 15,
  "aspectRatio": "1:1",
  "style": {
    "primaryColor": "#0f172a",
    "textColor": "#38bdf8"
  }
}

Endpoint E: Motion Graphics & Data Visualizations (`videoType: "MotionGraphics"`)

Generates highly animated technical slides with data visualization layouts (bar charts, pie charts) and animated typography (badges, glitch text, large stats) driven by a self-contained JSON schema.

Example Payload:

{
  "prompt": "Show a comparison of the top 3 programming languages in 2026",
  "videoType": "MotionGraphics",
  "topic": "Programming",
  "aspectRatio": "9:16",
  "voice": "aura-2-aries-en"
}

Endpoint F: AI Text-to-Video (`videoType: "TextToVideo"`)

Generates a facts-style short using WaveSpeed Seedance AI clips for every timed B-roll segment, WaveSpeed Lyria background music, and kinetic word-level captions synced to the TTS voiceover.

Requires WAVESPEED_API_KEY (add to .env.local). Uses the WAVESPEED_VIDEO_MODEL model (default: bytedance/seedance-v1-pro-fast/text-to-video).
Async: Unlike other modes, the request returns immediately with a jobId. The script → TTS → WaveSpeed clips → music pipeline runs inside the render job on the render server; poll GET /api/videos/{jobId} for progress.

Example Payload:

{
  "prompt": "Weird facts you don't know about the deep ocean",
  "videoType": "TextToVideo",
  "topic": "Marine Biology",
  "aspectRatio": "9:16",
  "voice": "aura-2-aries-en"
}

Endpoint G: Micro Drama (`videoType: "MicroDrama"`)

Generates a micro-drama video through the full agentic pipeline. Pass just an idea (story is written for you) or include a script to use your own text directly. Optional style guides the visual style and requirement adds extra constraints (e.g. runtime, mood).

Requires WAVESPEED_API_KEY. Uses WAVESPEED_LLM_MODEL (default deepseek/deepseek-v4-flash), WAVESPEED_PORTRAIT_MODEL/WAVESPEED_FRAME_MODEL (default bytedance/seedream-v4.5), and WAVESPEED_I2V_MODEL (default bytedance/seedance-2.0/image-to-video).
Async: returns a jobId immediately; the story → characters → scene scripts → storyboard → portraits → frames → I2V clips → render pipeline runs inside the job. Each AI clip takes ~1-2 minutes, so poll GET /api/videos/{jobId} until status is completed.

Example Payload (idea only):

{
  "videoType": "MicroDrama",
  "idea": "A detective in a rainy city discovers his partner was the mastermind all along",
  "style": "Noir thriller, cold color grade, dramatic shadows",
  "requirement": "Keep the story under 45 seconds with a twist ending",
  "aspectRatio": "16:9"
}

Example Payload (with script — script2video mode):

{
  "videoType": "MicroDrama",
  "idea": "A detective in a rainy city discovers his partner was the mastermind all along",
  "script": "INT. PRECINCT - NIGHT\nDetective Mara stares at the evidence board...",
  "style": "Noir thriller, cold color grade, dramatic shadows"
}

Endpoint H: AI UGC Ad (`videoType: "UGC"`)

Generates a UGC ad through WaveSpeed using any of the studio models (Veo 3.1, Seedance 2, Grok Video, Happy Horse 1). prompt is the script (reference images inline as @image1...). Optional images (hosted URLs from POST /api/upload) switches to image-to-video. model, aspectRatio, duration, and resolution mirror the studio controls.

Requires WAVESPEED_API_KEY. Default model via UGC_DEFAULT_MODEL (default seedance-2).
Multi-scene (multiScene: true): the WaveSpeed LLM breaks the script into 2-6 scenes, one TTS voiceover covers the whole ad (voice picks it, default auto), each scene is generated as its own clip (I2V with a reference image keeps the same actor), and the clips are assembled into a single timeline with kinetic captions. targetDurationSec (10-60) hints total length. Lip-sync (lipSync, default on) warps each clip's mouth to the TTS via WaveSpeed sync/lipsync-2; set lipSync: false to skip it for cost/speed. Requires ELEVENLABS_API_KEY or DEEPGRAM_API_KEY.
Async: returns a jobId immediately; the WaveSpeed generation runs inside the job and the finished mp4 is persisted to storage. Poll GET /api/videos/{jobId} until status is completed.
UI: full studio at /ugc — model cards, reference image upload, script box, param pickers, generation-style toggle, live polling player.

Example Payload (text-to-video):

{
  "videoType": "UGC",
  "prompt": "Hey! Here are 3 skincare mistakes you're making every morning. First, you're washing your face with hot water...",
  "model": "veo-3-1",
  "aspectRatio": "9:16",
  "duration": 8,
  "resolution": "1080p"
}

Example Payload (multi-scene):

{
  "videoType": "UGC",
  "prompt": "Tired of brittle nails? Here's the 3-step routine I swear by... grab this kit while it's 40% off.",
  "model": "seedance-2",
  "images": ["https://your-bucket.nyc3.digitaloceanspaces.com/images/actor.jpg"],
  "aspectRatio": "9:16",
  "resolution": "1080p",
  "multiScene": true,
  "voice": "pNInz6obpgDQGcFmaJgB",
  "targetDurationSec": 30
}

Example Payload (image-to-video with a reference face):

{
  "videoType": "UGC",
  "prompt": "@image1 here with the best hack for your morning routine. Trust me, this changes everything...",
  "model": "grok-video",
  "images": ["https://your-bucket.nyc3.digitaloceanspaces.com/images/abc123.jpg"],
  "aspectRatio": "9:16",
  "duration": 6
}

Exact JSON Request Payloads for All Video Modes

All requests must be sent as POST requests to: http://localhost:3000/api/videos

1. AI Storyboard Video (`AIStoryboardVideo`)

{
  "prompt": "Explain how black holes are formed in space",
  "videoType": "AIStoryboardVideo",
  "topic": "Space Science",
  "aspectRatio": "16:9",
  "voice": "aura-2-aries-en"
}

2. Stock Footage Video (`StockVideo`)

{
  "prompt": "Why drinking water in the morning improves focus",
  "videoType": "StockVideo",
  "topic": "Wellness & Health",
  "aspectRatio": "16:9",
  "voice": "aura-2-arcas-en"
}

3. Stock Image Video (`StockImage`)

{
  "prompt": "Why reading daily is key to success",
  "videoType": "StockImage",
  "topic": "Education",
  "aspectRatio": "9:16",
  "voice": "aura-2-aries-en"
}

4. Social Media Typography Slide (`SocialMedia`)

{
  "prompt": "A short piece of advice about starting a business today",
  "videoType": "SocialMedia",
  "aspectRatio": "9:16",
  "durationSec": 15,
  "style": {
    "primaryColor": "#020617",
    "textColor": "#38bdf8"
  }
}

5. Explainer Presentation Slide (`Explainer`)

{
  "prompt": "3 steps to write clean code",
  "videoType": "Explainer",
  "aspectRatio": "16:9",
  "durationSec": 20,
  "style": {
    "primaryColor": "#1e1b4b",
    "textColor": "#818cf8"
  }
}

6. General Slide Layout (`General`)

{
  "prompt": "A description of the scale of the solar system",
  "videoType": "General",
  "aspectRatio": "1:1",
  "durationSec": 15,
  "style": {
    "primaryColor": "#172554",
    "textColor": "#f8fafc"
  }
}

7. Text Animation / Kinetic Highlight (`TextAnimation`)

{
  "prompt": "A high energy quote about doing your best work",
  "videoType": "TextAnimation",
  "aspectRatio": "9:16",
  "durationSec": 12,
  "style": {
    "primaryColor": "#000000",
    "textColor": "#ffffff"
  }
}

8. Motion Graphics Video (`MotionGraphics`)

{
  "prompt": "Show a comparison of the top 3 programming languages in 2026",
  "videoType": "MotionGraphics",
  "topic": "Programming",
  "aspectRatio": "9:16",
  "voice": "aura-2-aries-en"
}

9. AI Text-to-Video (`TextToVideo`)

{
  "prompt": "Weird facts you don't know about the deep ocean",
  "videoType": "TextToVideo",
  "topic": "Marine Biology",
  "aspectRatio": "9:16",
  "voice": "aura-2-aries-en"
}

Requires WAVESPEED_API_KEY in .env.local. The B-roll model defaults to bytedance/seedance-v1-pro-fast/text-to-video and can be overridden with WAVESPEED_VIDEO_MODEL. This mode is async: the request returns a jobId immediately and the render server generates the whole pipeline (script → TTS → WaveSpeed clips → music) inside the job, reporting progress as it goes. Because each AI clip takes ~1-2 minutes to generate, the job stays in the rendering state for several minutes — poll GET /api/videos/{jobId} until status is completed.

Response:

{
  "success": true,
  "jobId": "a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6",
  "status": "queued",
  "createdAt": "2026-07-01T18:00:00.000Z"
}

10. Micro Drama (`MicroDrama`)

{
  "videoType": "MicroDrama",
  "idea": "A detective in a rainy city discovers his partner was the mastermind all along",
  "style": "Noir thriller, cold color grade, dramatic shadows",
  "requirement": "Keep the story under 45 seconds with a twist ending",
  "aspectRatio": "16:9"
}

Requires WAVESPEED_API_KEY in .env.local. Replicates the Open-AI-Micro-Drama-Generator pipeline: the WaveSpeed LLM writes the story (or uses a supplied script), extracts consistent characters, generates per-scene scripts, and designs a shot-by-shot storyboard; Seedream generates character portraits and scene first-frames; Seedance I2V animates each frame into a ~5s clip with native audio. This mode is async like TextToVideo — submit and poll GET /api/videos/{jobId}. Model defaults can be overridden with WAVESPEED_LLM_MODEL, WAVESPEED_PORTRAIT_MODEL, WAVESPEED_FRAME_MODEL, and WAVESPEED_I2V_MODEL.

Response (same shape as TextToVideo):

{
  "success": true,
  "jobId": "a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6",
  "status": "queued",
  "createdAt": "2026-07-01T18:00:00.000Z"
}

11. AI UGC Ad (`UGC`)

{
  "videoType": "UGC",
  "prompt": "Hey! @image1 here with the best hack for your morning routine. Trust me, this changes everything...",
  "model": "veo-3-1",
  "images": ["https://your-bucket.nyc3.digitaloceanspaces.com/images/abc123.jpg"],
  "aspectRatio": "9:16",
  "duration": 8,
  "resolution": "1080p"
}

Requires WAVESPEED_API_KEY in .env.local. Replicates the Open-AI-UGC studio: choose a model (veo-3-1 | seedance-2 | grok-video | happy-horse), optionally upload reference images, and pass a script that references them with @imageN. With images the model's image-to-video endpoint is used (I2V); without them, text-to-video (T2V). Native audio is generated by the video model (the AI actor speaks). Model defaults can be overridden with UGC_DEFAULT_MODEL. The finished mp4 is persisted to storage (S3/R2/Spaces when configured). Async — submit and poll GET /api/videos/{jobId}. A full studio UI is available at /ugc.

Response (same shape as TextToVideo):

{
  "success": true,
  "jobId": "a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6",
  "status": "queued",
  "createdAt": "2026-07-01T18:00:00.000Z"
}

12. Luma Ray 3.2 Video (`Luma`)

The Luma mode automatically detects the right Ray 3.2 endpoint based on the media attachments you provide, or you can set explicitOperation manually. If useCase is omitted, it defaults to "custom" and generates video directly.

Example A: Text-to-Video / Multi-Scene Ad (Scenario Picker)

{
  "videoType": "Luma",
  "prompt": "Create a high-energy product launch ad for a sleek espresso machine",
  "useCase": "product_launch",
  "aspectRatio": "16:9",
  "resolution": "720p",
  "duration": "5s",
  "generateAudio": true,
  "voice": "EXAVITQu4vr4xnSDxMaL"
}

Example B: Image-to-Video with Image Attachments (`referenceImages`)

Animates uploaded images as starting/guide keyframes:

{
  "videoType": "Luma",
  "prompt": "Animate this product photo with gentle steam rising and soft lighting movement",
  "referenceImages": ["https://your-cdn.com/product-photo.jpg"],
  "explicitOperation": "image_to_video",
  "aspectRatio": "9:16",
  "resolution": "1080p",
  "duration": "5s"
}

Example C: Video Editing / Restyling Uploaded Video (`sourceVideoUrl`)

Re-renders an uploaded video clip under a new prompt and edit strength (adhere_1-3, flex_1-3, reimagine_1-3):

{
  "videoType": "Luma",
  "prompt": "Transform this video into a moonlit cyberpunk scene with neon highlights",
  "sourceVideoUrl": "https://your-cdn.com/user-footage.mp4",
  "explicitOperation": "edit",
  "editStrength": "flex_2",
  "resolution": "720p"
}

Example D: Video Reframing / Outpainting (`sourceVideoUrl` + `aspectRatio`)

Outpaints an existing video into a new platform-native shape (e.g., 16:9 landscape $\rightarrow$ 9:16 vertical Reel):

{
  "videoType": "Luma",
  "prompt": "Extend the background into cinematic vertical portrait space",
  "sourceVideoUrl": "https://your-cdn.com/landscape-clip.mp4",
  "explicitOperation": "reframe",
  "aspectRatio": "9:16",
  "resolution": "720p"
}

Requires LUMA_AGENTS_API_KEY in .env.local. Dispatches to Luma's Ray 3.2 API (agents.lumalabs.ai). Automatically handles single-shot operations or multi-scene screenplay planning with extend chaining (start_frame: { generation_id }) and layered ElevenLabs TTS + kinetic word-level captions.
Default Behavior: When no useCase is passed, it defaults to "custom" (pure prompt generation). When media attachments (sourceVideoUrl / referenceImages) are attached without an explicitOperation, it auto-detects the right operation (edit for video files, image_to_video for images).
Async — returns a jobId immediately; poll GET /api/videos/{jobId} until completed.

Response:

{
  "success": true,
  "jobId": "a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6",
  "status": "queued",
  "createdAt": "2026-07-01T18:00:00.000Z"
}

13. Vox Collage Explainer (`VoxVideo`)

Replicates the Vox-style paper-collage explainer generator on top of the APIs this project already uses. A single topic flows through: LLM narrative beat map (hook-led, arc-driven, cadence every ~4-6s) → one Seedream collage poster per beat (strict 5-part Vox prompt formula + theme preset) → Seedance I2V animates each poster into a clip → ElevenLabs/Deepgram TTS voiceover with kinetic word captions → optional Lyria background music → assembled by the WavespeedVideo Remotion composition into a finished mp4.

{
  "videoType": "VoxVideo",
  "prompt": "A brief history of coffee",
  "theme": "american-retro",
  "arc": "timeline",
  "targetDurationSeconds": 30,
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

Themes (theme): swiss-modern · american-retro · punk-zine · chinese-ink (default american-retro).
Arcs (arc): hook_payoff · timeline · how_it_works · pas · bab · man_in_hole (default hook_payoff).
Requires WAVESPEED_API_KEY (posters + clips + music) and a TTS key (ELEVENLABS_API_KEY preferred or DEEPGRAM_API_KEY). The beat-map LLM uses VOX_LLM_URL/VOX_LLM_MODEL (defaults to the WaveSpeed LLM, fallback key OPENAI_API_KEY).
Async — returns a jobId immediately; poll GET /api/videos/{jobId} until completed.

Response:

{
  "success": true,
  "jobId": "a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6",
  "status": "queued",
  "createdAt": "2026-07-01T18:00:00.000Z"
}

14. Zack D Films 3D Short (`ZackDVideo`)

Replicates the zackd-director skill (Zack D Films-style curiosity shorts) entirely on the WaveSpeed stack. A single topic flows through: LLM curiosity-loop beat map (hook opens the loop in <3s, every beat opens a new question, final beat resolves it; 2 shots per beat with camera-move rotation) → Seedream character turnaround sheets (orthographic front/side/¾ views for characters, cross-section cutaways for organs/assets) → per-shot keyframe anchored on the sheet descriptions (5-part prompt formula: subject & action + character anchor + clay/SSS style block + macro lighting + dark studio background) → image-to-video clip (Seedance 2.0 by default) → ElevenLabs/Deepgram TTS voiceover with kinetic word captions → optional Lyria music → assembled by the ZackDVideo Remotion composition with impact zooms (zoom_impact shots get a slow push-in) and the fade/wipe/slide/circle-open transition cycle.

{
  "videoType": "ZackDVideo",
  "prompt": "What happens when you swallow gum?",
  "targetDurationSeconds": 30,
  "sceneCount": 4,
  "tone": "energetic",
  "language": "English",
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

Fields: sceneCount = number of curiosity-loop beats (2-8, each ≈ one narration line + 2 shots). prompt accepts a full script or facts dump — the LLM restructures it into the curiosity-loop format.
Requires WAVESPEED_API_KEY (sheets + keyframes + clips + music) and a TTS key (ELEVENLABS_API_KEY preferred or DEEPGRAM_API_KEY). The beat-map LLM uses ZACK_D_LLM_URL/ZACK_D_LLM_MODEL (defaults to the WaveSpeed LLM); set ZACK_D_I2V_MODEL=google/veo3.1/image-to-video to animate keyframes with Veo 3.1 instead of Seedance.
Async — returns a jobId immediately; poll GET /api/videos/{jobId} until completed.

Response:

{
  "success": true,
  "jobId": "a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6",
  "status": "queued",
  "createdAt": "2026-07-01T18:00:00.000Z"
}

15. AI Comic Drama Episode (`ComicDramaVideo`)

Replicates the AIComicBuilder pipeline entirely on the WaveSpeed stack. A single premise or raw script flows through: LLM story plan (character extraction, art-style auto-detection, continuity-chained shot list with first/last frame descriptions, camera directions and dialogue) → Seedream 4-view character sheets (front / ¾ / side / back grids with the name printed on the sheet) → per shot a first keyframe + last keyframe pair (the closing frame is locked to the opening frame's style/environment — only pose/expression/position changes; shot N+1 chains from shot N's end state) → start/end-frame interpolated I2V clip via Wan 2.2 last_image (or the dedicated wan-flf2v model) → per-shot dialogue TTS with word timestamps → optional Lyria music → assembled by the ComicDramaVideo Remotion composition with hard cuts, white panel-flash transitions and comic dialogue strips.

{
  "videoType": "ComicDramaVideo",
  "prompt": "A shy ramen shop cat secretly defends the neighborhood from spirits at night",
  "targetDurationSeconds": 30,
  "sceneCount": 6,
  "artStyle": "anime",
  "tone": "dramatic",
  "language": "English",
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

Fields: sceneCount = number of storyboard shots (2-10; each ≈ one keyframe pair + one 5s or 8s clip). artStyle: auto · anime · manga · comic_book · 3d_pixar · realistic_cinematic. prompt accepts a full raw script — the LLM parses characters and shots from it directly.
Requires WAVESPEED_API_KEY (sheets + keyframes + interpolation clips + music) and a TTS key (ELEVENLABS_API_KEY preferred or DEEPGRAM_API_KEY). The story-plan LLM uses COMIC_LLM_URL/COMIC_LLM_MODEL (defaults to the WaveSpeed LLM); keyframes are reference-locked via COMIC_KEYFRAME_EDIT_MODEL (default bytedance/seedream-v4/edit); set COMIC_I2V_MODEL=wavespeed-ai/wan-flf2v for the dedicated first-last-frame model.
Async — returns a jobId immediately; poll GET /api/videos/{jobId} until completed.

Response:

{
  "success": true,
  "jobId": "a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6",
  "status": "queued",
  "createdAt": "2026-07-01T18:00:00.000Z"
}

16. Stickman Explainer Short (`StickmanExplainerVideo`)

Replicates the Stickman Studio pipeline entirely on the WaveSpeed stack. A topic flows through: LLM storyboard (viral hook rules — scene 1 shocks, punchy conversational narration, exaggerated stickman physical comedy) → one Seedream T-pose character reference → per scene a reference-locked scene image (Seedream v4 Edit with the char ref attached; falls back to text-to-image) → animation via either slideshow mode (free alternating Ken Burns zoom/pan rendered natively in Remotion) or animated mode (one Wan 2.2 I2V clip per scene) → per-scene narrator TTS with word timestamps → optional Lyria music → assembled by the StickmanExplainerVideo composition with scene chips, lower-third captions and a marker-style title card.

{
  "videoType": "StickmanExplainerVideo",
  "prompt": "How black holes work",
  "targetDurationSeconds": 40,
  "sceneCount": 5,
  "animation": "slideshow",
  "tone": "curious and energetic",
  "language": "English",
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

Fields: sceneCount = number of scenes (2-10; scene 1 doubles as the hook). animation: slideshow (Ken Burns zoom on stills, ~$0.05/video total) or animated (Wan I2V clip per scene, ~$0.70/video). prompt accepts a topic, question, or raw explainer script.
Requires WAVESPEED_API_KEY (char ref + scene images + animated-mode clips + music) and a TTS key (ELEVENLABS_API_KEY preferred or DEEPGRAM_API_KEY). The storyboard LLM uses STICKMAN_LLM_URL/STICKMAN_LLM_MODEL (falling back to the ComicDrama/ZackD LLM vars); scene images are reference-locked via STICKMAN_IMAGE_EDIT_MODEL (default bytedance/seedream-v4/edit); set STICKMAN_I2V_MODEL to change the animated-mode clip model.
Async — returns a jobId immediately; poll GET /api/videos/{jobId} until completed.

Response:

{
  "success": true,
  "jobId": "b2c3d4e5-f6a7-8b9c-0d1e-2f3a4b5c6d7e",
  "status": "queued",
  "createdAt": "2026-07-01T19:00:00.000Z"
}

17. Whiteboard Animation Explainer (`WhiteboardVideo`)

Replicates the full Storyboard-AI whiteboard animation pipeline entirely on the WaveSpeed stack. A topic flows through: LLM Director Agent (DeepSeek via WaveSpeed LLM — generates scene-by-scene whiteboard scripts with narration, image prompts following the Nano Banana formula, and optional object lists) → per scene a Seedream whiteboard line-art image ($0.027/image; falls back to text-to-image) → animation via either slideshow mode (free alternating Ken Burns zoom/pan in Remotion) or animated mode (Seedance I2V clip per scene, ~$0.06/scene) → optional SAM3 Video segmentation to isolate key objects per clip ($0.05/5s billed; closest equivalent to Storyboard-AI's multi-pass OpenCV drawing) → per-scene narrator TTS with word timestamps → optional Lyria background music → assembled by the WhiteboardVideo composition (uses WavespeedVideo component) with scene chips, lower-third captions and a marker-style title card.

{
  "videoType": "WhiteboardVideo",
  "prompt": "How photosynthesis works",
  "targetDurationSeconds": 40,
  "sceneCount": 5,
  "animationStyle": "slideshow",
  "tone": "curious and energetic",
  "language": "English",
  "aspectRatio": "9:16",
  "generateAudio": true,
  "music": true
}

Fields: sceneCount = number of scenes (2-10; scene 1 doubles as the hook). animationStyle: slideshow (Ken Burns zoom on stills, ~$0.05/video total) or animated (Seedance I2V clip per scene, ~$0.42/video + optional SAM3). prompt accepts a topic, question, or raw explainer script.
Requires WAVESPEED_API_KEY (Seedream images + animated-mode clips + optional SAM3 Video + music) and a TTS key (ELEVENLABS_API_KEY preferred or DEEPGRAM_API_KEY). The Director Agent LLM uses WHITEBOARD_LLM_URL/WHITEBOARD_LLM_MODEL (falling back to the WaveSpeed LLM endpoint); scene images use WHITEBOARD_IMAGE_MODEL (default bytedance/seedream-v5.0-pro); animated-mode clips use WHITEBOARD_I2V_MODEL (default wavespeed-ai/seedance-1.0-pro); SAM3 Video uses wavespeed-ai/sam3-video.
Async — returns a jobId immediately; poll GET /api/videos/{jobId} until completed.

Response:

{
  "success": true,
  "jobId": "c3d4e5f6-a7b8-9c0d-1e2f-3a4b5c6d7e8f",
  "status": "queued",
  "createdAt": "2026-07-01T20:00:00.000Z"
}

API Modes: Generate vs. Direct Render

The endpoint POST /api/videos operates in two distinct modes depending on your payload structure:

A. Generate Mode (AI Auto-styling)

If you only supply a "prompt", OpenAI will dynamically write the script or timeline. You can pass optional style overrides to guide the AI's aesthetic choices.

Additionally, you can specify a consistent voice parameter to narrate the entire video. If omitted, the API will randomly pick a single voice and use it consistently for all scenes of that video.

Supported Voices: "aura-2-thalia-en", "aura-2-andromeda-en", "aura-2-arcas-en", "aura-2-aries-en"

{
  "prompt": "A quote by Steve Jobs about design",
  "videoType": "SocialMedia",
  "voice": "aura-2-thalia-en",     // Optional premium Aura-2 voice override
  "style": {
    "primaryColor": "#0f172a",    // Guide AI to use slate blue backgrounds
    "textColor": "#38bdf8"       // Guide AI to use light blue text
  }
}

B. Direct Render Mode (Full Custom Control)

If you want to design a custom editor interface where the user manually controls every slide (timing, background color, text color, individual animations, and voiceover audio files), you can submit a fully pre-defined script or timeline object. In this mode, the server skips OpenAI completely and renders the exact parameters provided:

{
  "videoType": "SocialMedia",
  "script": {
    "title": "Manual Quote",
    "durationSec": 10,
    "fps": 30,
    "width": 1080,
    "height": 1920,
    "scenes": [
      {
        "text": "Design is not just what it looks like.",
        "startSec": 0,
        "durationSec": 5,
        "bgColor": "#1e1b4b",
        "textColor": "#facc15",
        "fontSize": 64,
        "animation": "bounce",
        "audioUrl": "/assets-temp/scene-0.mp3"
      },
      {
        "text": "Design is how it works.",
        "startSec": 5,
        "durationSec": 5,
        "bgColor": "#020617",
        "textColor": "#38bdf8",
        "fontSize": 64,
        "animation": "typewriter",
        "audioUrl": "/assets-temp/scene-1.mp3"
      }
    ]
  }
}

C. Direct Render Mode for Motion Graphics (`timeline`)

For MotionGraphics, you provide a timeline object instead of a script object, specifying dynamic slides:

{
  "videoType": "MotionGraphics",
  "timeline": {
    "shortTitle": "Manual Motion Graphics",
    "slides": [
      {
        "durationFrames": 120,
        "background": {
          "type": "mesh",
          "from": "#0a0a0a",
          "to": "#1a1a2e"
        },
        "elements": [
          { "type": "badge", "text": "INTRODUCTION", "color": "#00ffd2", "delay": 0 },
          { "type": "title", "text": "AI Revolution", "animation": "slideUp", "color": "#ffffff", "delay": 15 },
          { "type": "subtitle", "text": "Changing the world", "color": "#a1a1aa", "delay": 30 }
        ]
      }
    ]
  }
}

2. Get Video Link / Job Status (`GET /api/videos/:jobId`)

Polls the render queue status. Once rendering is completed, it uploads the final .mp4 file to your S3 storage bucket and returns the public link.

Response (completed):

{
  "jobId": "a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6",
  "status": "completed",
  "progress": 100,
  "videoUrl": "https://your-spaces-bucket.nyc3.digitaloceanspaces.com/videos/a1b2c3d4-e5f6-7a8b-9c0d-e1f2a3b4c5d6.mp4",
  "createdAt": "2026-06-30T04:30:00.000Z",
  "startedAt": "2026-06-30T04:30:05.000Z",
  "completedAt": "2026-06-30T04:30:50.000Z"
}

Troubleshooting

Render server cannot connect

Ensure the render server is running on port 3001 (npm run render-server).
If deploying, verify your Cloudflare Tunnel URL and ensure RENDER_SERVER_SECRET matches on both frontend and backend.

Video fails to render

Verify that ffmpeg is installed: ffmpeg -version.
Ensure chrome headless dependencies are present: npx remotion browser ensure.

OpenAI/Deepgram API errors

Check that your API keys are active and have sufficient balance limits.
Monitor service statuses: OpenAI Status, Deepgram Status.

Roadmap

Multi-voice narration tracks and character dialogue
Template marketplace for reusable scene packs
Cloud rendering worker pool (horizontal scaling of the render queue)
Streaming progress events via WebSockets

Acknowledgements

Nova Motion builds on the work of several open projects and APIs:

Remotion — React-based programmatic video rendering
WaveSpeed — Seedream, Seedance, and Lyria media models
OpenAI — GPT-4o-mini scripting and gpt-image-2 illustrations
ElevenLabs & Deepgram — neural TTS and speech-to-text
Luma — Ray 3.2 video generation
Pexels & Pixabay — stock media APIs
DigitalOcean & Cloudflare R2 — S3-compatible storage

Contributing

Contributions are welcome. Please read the Contributing Guide first - it covers setup, how to verify a mode end to end, and how to add a new one. Then open an issue to discuss what you would like to change before submitting a pull request.

Support

Star the repository if you find it useful.
Report bugs or request features via GitHub Issues.

Name		Name	Last commit message	Last commit date
.github		.github
public		public
render-server		render-server
shared		shared
src		src
.env.local.example		.env.local.example
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
components.json		components.json
eslint.config.mjs		eslint.config.mjs
next.config.ts		next.config.ts
package-lock.json		package-lock.json
package.json		package.json
postcss.config.mjs		postcss.config.mjs
remotion.config.ts		remotion.config.ts
tsconfig.json		tsconfig.json

Latest commit

History

Folders and files

Repository files navigation

Nova Motion

What is Nova Motion?

Highlights

Tech Stack

Current Outputs

Studio Preview

Project Structure

Video Generation Models (Compositions)

1. AI Storyboard Video (videoType: "AIStoryboardVideo")

2. Stock Video Short (videoType: "StockVideo")

3. Stock Image Short (videoType: "StockImage")

4. Typography/Layout Slide Videos (videoType: "SocialMedia" | "Explainer" | "General" | "TextAnimation")

5. Motion Graphics & Data Visualizations (videoType: "MotionGraphics")

6. AI Text-to-Video (videoType: "TextToVideo")

7. Micro Drama (videoType: "MicroDrama")

8. AI UGC Ad (videoType: "UGC")

9. Agentic Video Generator (videoType: "AgenticVideoGenerator")

10. Luma AI Ray 3.2 (videoType: "Luma")

11. Vox Collage Explainer (videoType: "VoxVideo")

12. Zack D Films 3D Short (videoType: "ZackDVideo")

13. AI Comic Drama Episode (videoType: "ComicDramaVideo")

14. Stickman Explainer Short (videoType: "StickmanExplainerVideo")

15. Whiteboard Animation Explainer (videoType: "WhiteboardVideo")

Quick Start (Local Development)

1. Install Dependencies

2. Set Up Environment Variables

3. Start the Render Server

4. Start the Next.js API Gateway

API Documentation

1. Generate Video (POST /api/videos)

Request Parameters

Endpoint A: AI Storyboard Video (videoType: "AIStoryboardVideo")

Endpoint B: Stock Video Short (videoType: "StockVideo")

Endpoint C: Stock Image Short (videoType: "StockImage")

Endpoint D: Typographic Slide Videos (videoType Options: "SocialMedia" | "Explainer" | "General" | "TextAnimation")

Endpoint E: Motion Graphics & Data Visualizations (videoType: "MotionGraphics")

Endpoint F: AI Text-to-Video (videoType: "TextToVideo")

Endpoint G: Micro Drama (videoType: "MicroDrama")

Endpoint H: AI UGC Ad (videoType: "UGC")

Exact JSON Request Payloads for All Video Modes

1. AI Storyboard Video (AIStoryboardVideo)

2. Stock Footage Video (StockVideo)

3. Stock Image Video (StockImage)

4. Social Media Typography Slide (SocialMedia)

5. Explainer Presentation Slide (Explainer)

6. General Slide Layout (General)

7. Text Animation / Kinetic Highlight (TextAnimation)

8. Motion Graphics Video (MotionGraphics)

9. AI Text-to-Video (TextToVideo)

10. Micro Drama (MicroDrama)

11. AI UGC Ad (UGC)

12. Luma Ray 3.2 Video (Luma)

Example A: Text-to-Video / Multi-Scene Ad (Scenario Picker)

Example B: Image-to-Video with Image Attachments (referenceImages)

Example C: Video Editing / Restyling Uploaded Video (sourceVideoUrl)

Example D: Video Reframing / Outpainting (sourceVideoUrl + aspectRatio)

13. Vox Collage Explainer (VoxVideo)

14. Zack D Films 3D Short (ZackDVideo)

15. AI Comic Drama Episode (ComicDramaVideo)

16. Stickman Explainer Short (StickmanExplainerVideo)

17. Whiteboard Animation Explainer (WhiteboardVideo)

API Modes: Generate vs. Direct Render

A. Generate Mode (AI Auto-styling)

B. Direct Render Mode (Full Custom Control)

C. Direct Render Mode for Motion Graphics (timeline)

2. Get Video Link / Job Status (GET /api/videos/:jobId)

Troubleshooting

Render server cannot connect

Video fails to render

OpenAI/Deepgram API errors

Roadmap

Acknowledgements

Contributing

Support

License

About

1. AI Storyboard Video (`videoType: "AIStoryboardVideo"`)

2. Stock Video Short (`videoType: "StockVideo"`)

3. Stock Image Short (`videoType: "StockImage"`)

4. Typography/Layout Slide Videos (`videoType: "SocialMedia" | "Explainer" | "General" | "TextAnimation"`)

5. Motion Graphics & Data Visualizations (`videoType: "MotionGraphics"`)

6. AI Text-to-Video (`videoType: "TextToVideo"`)

7. Micro Drama (`videoType: "MicroDrama"`)

8. AI UGC Ad (`videoType: "UGC"`)

9. Agentic Video Generator (`videoType: "AgenticVideoGenerator"`)

10. Luma AI Ray 3.2 (`videoType: "Luma"`)

11. Vox Collage Explainer (`videoType: "VoxVideo"`)

12. Zack D Films 3D Short (`videoType: "ZackDVideo"`)

13. AI Comic Drama Episode (`videoType: "ComicDramaVideo"`)

14. Stickman Explainer Short (`videoType: "StickmanExplainerVideo"`)

15. Whiteboard Animation Explainer (`videoType: "WhiteboardVideo"`)

1. Generate Video (`POST /api/videos`)

Endpoint A: AI Storyboard Video (`videoType: "AIStoryboardVideo"`)

Endpoint B: Stock Video Short (`videoType: "StockVideo"`)

Endpoint C: Stock Image Short (`videoType: "StockImage"`)

Endpoint D: Typographic Slide Videos (`videoType` Options: `"SocialMedia" | "Explainer" | "General" | "TextAnimation"`)

Endpoint E: Motion Graphics & Data Visualizations (`videoType: "MotionGraphics"`)

Endpoint F: AI Text-to-Video (`videoType: "TextToVideo"`)

Endpoint G: Micro Drama (`videoType: "MicroDrama"`)

Endpoint H: AI UGC Ad (`videoType: "UGC"`)

1. AI Storyboard Video (`AIStoryboardVideo`)

2. Stock Footage Video (`StockVideo`)

3. Stock Image Video (`StockImage`)

4. Social Media Typography Slide (`SocialMedia`)

5. Explainer Presentation Slide (`Explainer`)

6. General Slide Layout (`General`)

7. Text Animation / Kinetic Highlight (`TextAnimation`)

8. Motion Graphics Video (`MotionGraphics`)

9. AI Text-to-Video (`TextToVideo`)

10. Micro Drama (`MicroDrama`)

11. AI UGC Ad (`UGC`)

12. Luma Ray 3.2 Video (`Luma`)

Example B: Image-to-Video with Image Attachments (`referenceImages`)

Example C: Video Editing / Restyling Uploaded Video (`sourceVideoUrl`)

Example D: Video Reframing / Outpainting (`sourceVideoUrl` + `aspectRatio`)

13. Vox Collage Explainer (`VoxVideo`)

14. Zack D Films 3D Short (`ZackDVideo`)

15. AI Comic Drama Episode (`ComicDramaVideo`)

16. Stickman Explainer Short (`StickmanExplainerVideo`)

17. Whiteboard Animation Explainer (`WhiteboardVideo`)

C. Direct Render Mode for Motion Graphics (`timeline`)

2. Get Video Link / Job Status (`GET /api/videos/:jobId`)