ClipForensics validation study
Internal validation of our forensic pipeline — methodology, results, and honest assessment of performance.
Technical report · 15 min read
Study overview
This document describes our internal validation methodology for the ClipForensics forensic analysis pipeline. We test against known authentic and manipulated videos to evaluate per-module accuracy, fusion performance, and overall system reliability.
Important note: This is an internal validation study, not an independent audit. We present these results transparently, but encourage independent researchers to conduct their own evaluations. We will publish independent audit results when available.
Methodology
Test dataset composition
Our test dataset includes videos from the following categories:
- Authentic camera footage: Videos captured on known devices with verified provenance.
- Face swap deepfakes: Generated using DeepFaceLab, FaceSwap, and Roop with varying quality levels.
- Lip sync manipulation: Created using Wav2Lip and similar tools.
- AI-generated videos: Outputs from Sora, Runway, Pika, and Stable Video Diffusion.
- Legitimate edits: Professionally edited authentic footage (color grading, cropping, transitions).
- Social media re-encoded: Authentic videos downloaded from YouTube, TikTok, and WhatsApp.
Results
Overall system performance
15
Modules Tested
6
Verdict Categories
87%
Weighted Accuracy
< 5%
False Positive Rate
Weighted accuracy (87%): The percentage of test videos receiving a correct verdict category, weighted by confidence level. Higher-confidence predictions are weighted more heavily.
False positive rate (< 5%): The percentage of authentic videos incorrectly flagged as manipulated. We prioritize low false positive rates because falsely accusing authentic content is more harmful than missing some manipulations.
Inconclusive rate (12%): The percentage of test videos receiving an "Inconclusive" verdict. We view this as a feature, not a failure — it means the system correctly identifies when evidence is insufficient.
Per-Module Performance
Module-level accuracy
Individual module performance varies significantly by manipulation type. Container-level modules (metadata, compression) perform well across all categories. Visual modules (face manipulation, artifacts) are strongest against face swaps. Temporal modules (optical flow, biological motion) are strongest against AI-generated video.
This variation is precisely why the multi-module approach is essential — no single module provides adequate coverage across all manipulation types.
Honest Assessment
Known failure modes
Our validation identified the following consistent failure modes:
- High-quality Sora outputs with minimal post-processing remain difficult to detect reliably.
- Videos with extreme compression (WhatsApp quality) lose most forensic signals.
- Legitimate professional edits (heavy color grading, VFX) sometimes trigger false positives.
- Audio-only deepfakes (voice clone with authentic video) have lower detection rates.
For a complete list of limitations, see our detection limitations page.