Reproducibility
All experiments are implemented as Jupyter notebooks. Follow the steps below to reproduce our results.
Environment Setup
Prerequisites
- Python 3.10+
- CUDA 11.8+ (GPU recommended)
- 16GB+ RAM
- Jupyter Notebook or Kaggle / Google Colab environment
Each notebook installs its own dependencies in the first cells. No separate requirements.txt is needed.
Pipeline Notebooks
The pipeline is split across several notebooks. Run them in order, or use the all-in-one notebook for a single-run experience.
Data Preprocessing
Extract video frames into a structured dataset using FFmpeg.
Main Pipeline — Diffusion Inpainting
End-to-end object replacement: YOLOv8-seg masking, RAFT optical flow, Stable Diffusion inpainting with ControlNet, and flow-matching latent search for temporal coherence.
SAM2 vs YOLOv8 Mask Comparison
Extends the main pipeline with SAM2 video segmentation and produces quantitative mask comparison metrics (IoU, Dice, SSIM).
Alternative Approaches
GAN-Based Replacement
Pix2pix-style GAN with a temporal discriminator for frame-to-frame consistency.
All-in-One Modular Package
Self-contained Python package with OpenCV baseline and optional Stable Diffusion upgrade path.
Kaggle-Ready Version
Fixed Errors of Original Review Thesis
Debugged version of the main pipeline adapted for Kaggle (GPU/Internet toggle, Pillow pinning, and corrected errors).
Evaluation Metrics
Metrics are computed within the notebooks (primarily in the SAM2 vs YOLOv8 experiments notebook). The following metrics are reported:
- IoU — Intersection over Union of SAM2 vs YOLO masks
- Dice — Dice coefficient (F1 score of mask overlap)
- SSIM — Structural Similarity Index between mask methods
Expected Results
SAM2 vs YOLO mask comparison across 60 frames yields the following ranges:
| Metric | Range | Mean |
|---|---|---|
| IoU | 0.88 - 0.93 | 0.91 |
| Dice | 0.94 - 0.96 | 0.95 |
| SSIM | 0.98 - 0.99 | 0.98 |
Troubleshooting
- CUDA out of memory — Reduce batch size or switch to a Kaggle/Colab GPU runtime
- SAM2 checkpoint not found — Download from the official repo
- Low-quality inpainting — Ensure masks cover the full object; dilate masks by 5-10 pixels
- Dependency conflicts — Run notebooks in a fresh virtual environment or use Kaggle/Colab where dependencies are pre-installed