AI video manipulation techniques
How AI-generated and manipulated video is created — and what forensic traces each technique leaves behind.
20 min read
The landscape of AI video manipulation
AI video manipulation techniques range from subtle face retouching to full synthetic video generation. Understanding these techniques is essential for detection — each method leaves distinct forensic signatures.
We categorize manipulation techniques by their scope and method:
Face swap
Face swap replaces one person's face with another while preserving the original head pose, expression, and lighting. Modern implementations use autoencoder architectures with shared encoder weights.
Common tools
DeepFaceLab, FaceSwap, Roop/InsightFace, SimSwap. Commercial offerings include various mobile apps with one-click face swap capabilities.
Forensic traces
- Face boundary artifacts — blending seams between swapped face and original skin
- Resolution mismatch — swapped face region may have different effective resolution
- Inconsistent noise patterns between face and background regions
- Temporal flickering at face boundaries across frames
- Eye reflection inconsistencies — different reflections in each eye
Face reenactment
Face reenactment transfers facial expressions from a driving video to a target face. The target person appears to make expressions and head movements they never made. This is distinct from face swap — the identity remains the same, but the expressions are puppeteered.
Forensic traces
- Unnatural micro-expressions — subtle expression dynamics that differ from natural behavior
- Head pose limitations — extreme angles may produce distortions
- Teeth and interior mouth artifacts — often poorly reconstructed
- Hair and accessory handling — earrings, glasses may warp unnaturally
Lip sync puppetry
Lip sync manipulation modifies the mouth region of a video to match new audio. The target person appears to say words they never said. Often combined with voice cloning for maximum deceptive effect.
Forensic traces
- Phoneme-viseme misalignment — subtle timing errors between speech sounds and lip shapes
- Lower face texture inconsistency — jaw and chin region may differ from upper face
- Jaw motion range — generated jaw movements may exceed natural range
Voice cloning & audio synthesis
Voice cloning creates synthetic speech that mimics a specific person's voice. Text-to-speech (TTS) generates speech from text input. Both are increasingly used in combination with video manipulation.
Forensic traces
- Unnatural prosody — rhythm, stress, and intonation patterns that differ from natural speech
- Breathing artifacts — absent or synthetic breathing patterns between phrases
- Spectral anomalies — unusual frequency distributions in the voice band
- Environmental mismatch — cloned voice may lack room acoustics present in the video
Text-to-video generation
The newest frontier: generating entire videos from text descriptions. Models like Sora, Kling, Runway Gen-3, and Pika produce increasingly realistic video from text prompts.
Forensic traces
- Physics violations — objects that defy gravity, impossible reflections, incorrect shadows
- Temporal morphing — objects that gradually change shape across frames
- Unusual noise distribution — AI-generated noise differs fundamentally from sensor noise
- Metadata absence — no camera metadata, unusual encoder signatures
- Hand and finger anomalies — extra digits, impossible poses, morphing appendages
These artifacts are becoming less pronounced with each generation of models. Detection must evolve continuously to keep pace.
Implications for detection
Each manipulation technique leaves different forensic traces, which is why multi-module detection is essential. A detector tuned only for face swaps will miss lip sync manipulation. A pixel-level analyzer may miss audio-only deepfakes.
ClipForensics addresses this by running 15 independent modules covering visual, temporal, audio, and container-level analysis — ensuring comprehensive coverage across manipulation types.