Voiceover Cutter
I designed and built a script-driven media tool that aligns recordings to text, generates AI narration, and assembles finished demo voiceovers.
Overview
Built with sales teammates to automate a repeated editing task, saving approximately 20 minutes per demo video based on team feedback. A Next.js interface connects to a Python processing service, with visible job progress and reusable intermediate clips.
- My contribution
- Interface, workflow, and AI-assisted implementation
- Stack
- Next.js, TypeScript, Python, FastAPI, and FFmpeg
- Speech services
- ElevenLabs Scribe and text-to-speech, with optional Whisper transcription
- Deployment
- Vercel frontend and containerized API on Google Cloud Run
Separating UI from processing
The browser handles script input, segment previews, voice selection, and job status. FastAPI handles the expensive work: transcription, alignment, cutting, synthesis, and verification. This keeps long-running media work outside the frontend request lifecycle.
A typed API contract exposes the job ID, status, processing stage, progress, errors, and output availability. The React interface polls the active job once a second and stops at a terminal state. Keeping the ID in localStorage lets a page reload reconnect to a job while the backend still has it.
Matching speech to a script
The difficult part is choosing the right take from a recording with repeated phrases. Word-level transcription feeds a dynamic-programming aligner that matches script words in order. Fuzzy matching tolerates transcription differences, while a cost for skipped transcript words favors compact passages over matches scattered across unrelated takes.
Cross-segment deduplication prevents two script lines from claiming the same audio. Boundary refinement uses silence and retake checks before FFmpeg cuts the selected spans. These decisions connect a text edit to a specific, inspectable piece of media.
Implementation
A recoverable pipeline, from script segments to a verified media file.
- 01
Parse an ordered segment model
Split human narration by non-empty line and keep AI VOICE blocks intact. Speaker labels select an ElevenLabs voice. Both generation and recording-edit modes assemble their clips in the original script order.
- 02
Expose job state and cancellation
A backend worker advances through named processing stages. The UI shows progress and logs, tolerates temporary polling failures, and reconnects after a reload. Cancellation is cooperative and takes effect at the next processing checkpoint.
- 03
Reuse completed work
A segment manifest records the previous cut. When human segments still match, the next job reuses those clips and regenerates the AI voice blocks. Changes to human text take the re-cut path using the existing source media.
- 04
Verify the assembled output
Re-transcribe the final file and compare it with the script. The activity panel exposes missing words, extras, and matched spans so a completed export can be reviewed before use.
- 05
Keep the recovery boundary explicit
Jobs run in a single-worker executor with in-memory state and temporary workspace files. Reload recovery works while that backend job exists; durable recovery across a server restart would require persistent job and artifact storage.
Voiceover system
Production paths
Generate voices with ElevenLabs
Export M4A
Match recording to script
Remove mistakes and retakes
Insert generated agent voices
Export MP4 or M4A
Script parsing and voice assignment
Normal text↝Narrator clip
Agent 1:↝Agent 1 clip
Agent 2:↝Agent 2 clip
Ordered clips↝Final file
Media processing architecture
The interface submits jobs and polls status; the Python service owns the media pipeline.
Re-run and recovery decisions
Finished media↝Re-transcribe
Transcript↝Compare to script
Differences↝Missing / extra words
Activity panel↝Review the result
End-to-end production workflow
How a script moves from source setup through assembly, review, selective regeneration, and export.