USER
Deep Fake Detection using CNN
A Mini-Project Report Submitted in the
Partial Fulfillment of the Requirements for the Award of the Degree of
BACHELOR OF TECHNOLOGY
IN
COMPUTER SCIENCE AND ENGINEERING
Submitted by
A.Manisha 21881A05D5
S.Reshma Sri 21881A05J1
SUPERVISOR
Dr.V.Lokeshwari Vinya
Assistant Professor
Department of Computer Science and Engineering
June, 2024
Department of Computer Science and Engineering
CERTIFICATE
This is to certify that the project titled Deep Fake Detection using CNN is carried out by
A.Manisha 21881A05D5
S.Reshma Sri 21881A05J1
in partial fulfillment of the requirements for the award of the degree of Bachelor of Technology in Computer and Science Engineering during the year 2023-24.
Signature of the Supervisor Signature of the HOD
Dr.V.Lokeshwari Vinya Dr. Ramesh karnati
Assistant Professor Professor and Head, CSE
Kacharam (V), Shamshabad (M), Ranga Reddy (Dist.)–501218, Hyderabad, T.S.
Ph: 08413-253335, 253201, Fax: 08413-253482, www.vardhaman.org
Acknowledgement
The satisfaction that accompanies the successful completion of the task would be put incomplete without the mention of the people who made it possible, whose constant guidance and encouragement crown all the efforts with success.
We wish to express our deep sense of gratitude to Dr.V.Lokeshwari Vinya, Assistant Professor and Project Supervisor, Department of Computer and Science Engineering, Vardhaman College of Engineering, for his able guidance and useful suggestions, which helped us in completing the project in time.
We are particularly thankful to Dr. Ramesh karnati, the Head of the Department, Department of Computer and Science Engineering, his guidance, intense support and encouragement, which helped us to mould our project into a successful one.
We show gratitude to our honorable Principal Dr. J.V.R. Ravindra, for providing all facilities and support.
We avail this opportunity to express our deep sense of gratitude and heartful thanks to Dr. <PRESIDIO_ANONYMIZED_PERSON>, Chairman and Sri Teegala Upender Reddy, Secretary of VCE, for providing a congenial atmosphere to complete this project successfully.
We also thank all the staff members of Electronics and Communication Engineering department for their valuable support and generous advice. Finally thanks to all our friends and family members for their continuous support and enthusiastic help.
A.Manisha
S.Reshma Sri
Abstract
Deep fake technology has rapidly evolved, generating highly realistic but synthetic images and videos that pose significant threats to security, privacy, and information integrity. This project focuses on developing a robust deep fake detection system utilizing Convolutional Neural Networks (CNNs) to effectively identify manipulated media. The proposed method leverages the powerful feature extraction capabilities of CNNs to distinguish between authentic and fake visual content. The detection system is trained on a diverse dataset of real and deep fake images and videos, ensuring the model learns to recognize subtle inconsistencies and artifacts introduced during the creation of deep fakes. Advanced preprocessing techniques and data augmentation are employed to enhance the model’s generalization ability. Performance evaluation is conducted using standard metrics such as accuracy, precision, recall, and F1-score. Preliminary results demonstrate that the CNN-based approach achieves high accuracy in detecting deep fakes, significantly outperforming traditional detection methods. The model’s effectiveness in various real-world scenarios is assessed, highlighting its potential application in fields such as digital forensics, social media monitoring, and media authentication. This work contributes to the ongoing efforts to safeguard digital content from malicious manipulations and reinforces the importance of continuous advancements in deep fake detection technologies.
Keywords: Convolutional Neural Networks,Deep fake Detection, Image manipulation, video manipulation
Table of Contents
Title Page No.
Acknowledgement i
Abstract ii
List of Tables v
List of Figures vi
Abbreviations vi
CHAPTER 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Overview of Deep Fakes . . . . . . . . . . . . . . . . . . . . 1
1.1.2 Importance of Detecting Deep Fakes . . . . . . . . . . . . . 1
1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Goals of the Report . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2 Specific Objectives Related to CNN-Based Detection . . . 2
1.3 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3.1 What the Report Will Cover . . . . . . . . . . . . . . . . . 3
1.3.2 Limitations and Assumptions . . . . . . . . . . . . . . . . . 3
1.4 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4.1 Brief Overview of the Chapters . . . . . . . . . . . . . . . 3
CHAPTER 2 Literature Survey . . . . . . . . . . . . . . . . . . . . . 4
CHAPTER 3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1.1 Real Data Collection . . . . . . . . . . . . . . . . . . . . . . 10
3.1.2 Deep Fake Data Collection . . . . . . . . . . . . . . . . . . 10
3.1.3 Dataset Composition . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.1 Data Cleaning . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.2 Data Annotation . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.3 Data Augmentation . . . . . . . . . . . . . . . . . . . . . . 12
3.2.4 Data Normalization . . . . . . . . . . . . . . . . . . . . . . 12
3.2.5 Data Splitting . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3 Data Pipeline Implementation . . . . . . . . . . . . . . . . . . . . 13
3.4 Challenges and Considerations . . . . . . . . . . . . . . . . . . . . 13
3.4.1 Data Imbalance . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.4.2 Privacy and Ethics . . . . . . . . . . . . . . . . . . . . . . . 13
3.4.3 Computational Resources . . . . . . . . . . . . . . . . . . . 13 CHAPTER 4 Architecture and Model . . . . . . . . . . . . . . . . . 14
CHAPTER 5 Implementation . . . . . . . . . . . . . . . . . . . . . . 16
5.1 Setup and Installation . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.1.1 Installing Required Libraries . . . . . . . . . . . . . . . . . 16
5.2 Model Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2.1 Training the Model . . . . . . . . . . . . . . . . . . . . . . 16
5.3 Inference on Images . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.3.1 Defining the Inference Function . . . . . . . . . . . . . . . 18
5.4 Inference on Videos . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.4.1 Defining the Video Inference Function . . . . . . . . . . . . 19
CHAPTER 6 Conclusions and Future Scope . . . . . . . . . . . . . 22
6.1 Summary of Findings . . . . . . . . . . . . . . . . . . . . . . . . . 22
6.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
6.3 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . 23
6.4 Final Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
CHAPTER 7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
7.0.1 Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
7.0.2 Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
7.0.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
List of Tables
1.1 Summary of Notable Deepfake Creation Tools . . . . . . . . . . . 2
v
List of Figures
4.1 Architecture Diagram used for deep fake detection of images . . 14
4.2 flow chart diagram used for deep fake detection of videos . . . . 15
5.1 training the model . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2 output for giving real image as input to the model . . . . . . . . 19
5.3 output for giving fake video as input to the model . . . . . . . . 21
5.4 output for giving real video as input to the model . . . . . . . . 21
vi
Abbreviations
Abbreviation Description
CNN Convolutional Neural Network
RNN Recurrent Neural Network
LSTM Long Short Term Memory Network
GAN Generative Adversarial Network
ResNext Residual Neural Network with Cardinality
CHAPTER 1
Introduction
1.1 Background
1.1.1 Overview of Deep Fakes
Deep fakes are synthetic media in which a person in an existing image or video is replaced with someone else’s likeness. This is achieved through advanced techniques in artificial intelligence, particularly using deep learning models. The advent of deep fakes has raised significant concerns due to their potential misuse in spreading misinformation, committing fraud, and undermining public trust in media.
1.1.2 Importance of Detecting Deep Fakes
Detecting deep fakes is crucial for maintaining the integrity of digital content. As deep fake technology becomes more sophisticated, it becomes increasingly challenging to distinguish between genuine and manipulated media. Effective detection methods are essential for preventing the malicious use of deep fakes in areas such as politics, entertainment, and social media. Reliable detection mechanisms are vital for protecting individuals’ reputations and ensuring the authenticity of digital communications.
1
Tool Key Features Additional Information
Faceswap - Uses two encoder-decoder pairs with shared parameters - Incorporates adversarial loss and perceptual loss
- Utilizes pre-trained face recognition models
- Supports multiple face extraction modes
- Implements DSSIM loss function for face reconstruction
Faceswap-
GAN - Reconstructs 3D faces from low-resolution images - Supports few-shot face reenactment
Few-Shot
Face
Transla-
tion GAN - Generates face images of virtual people with independent latent variables - Embeds 3D priors into adver-
sarial learning
DeepFaceLa b- Creates portrait images with rig-like control over
StyleGAN via 3D mor-
phable face models - Self-supervised without manual annotations
DFaker - Performs high-fidelity face swapping - Can be applied to any face pairs without subject-specific training
Reface.ai - Offers advanced face swapping technology - Provides seamless integration with social media platforms
Remaker.ai - Enables easy creation of deepfake videos - Supports various editing features for enhancing videos
Table 1.1: Summary of Notable Deepfake Creation Tools
1.2 Objectives
1.2.1 Goals of the Report
The primary goal of this report is to explore the application of Convolutional Neural Networks (CNNs) in the detection of deep fakes in both images and videos. It aims to provide a comprehensive understanding of the current
state-of-the-art techniques and evaluate their effectiveness.
1.2.2 Specific Objectives Related to CNN-Based Detec-
tion
- To review the fundamental concepts of CNNs and their applicability in image and video analysis. - To investigate various CNN architectures that
Department of Computer Science and Engineering 2
have been utilized for deep fake detection. - To analyze the performance of CNN-based methods in terms of accuracy, efficiency, and robustness. - To identify the challenges and limitations associated with CNN-based deep fake
detection.
1.3 Scope
1.3.1 What the Report Will Cover
This report will cover the theoretical background of deep fakes, the principles of CNNs, and their implementation in detecting deep fakes. It will include a review of existing literature, a comparison of different CNN architectures, and a discussion of experimental results from recent studies.
1.3.2 Limitations and Assumptions
The scope of this report is limited to the detection of deep fakes in digital images and videos. It assumes a basic understanding of machine learning and deep learning concepts. The report will not delve into the legal or ethical implications of deep fakes, although their importance is acknowledged.
1.4 Structure
1.4.1 Brief Overview of the Chapters
-Chapter 2: Literature Review - This chapter provides a review of the existing literature on deep fake detection techniques and the role of CNNs.
- Chapter 3: Methodology - This chapter details the methodologies used in CNN-based deep fake detection, including data preprocessing, model training, and evaluation.
- Chapter 4: Results and Discussion - This chapter presents the experimental results and discusses the performance of different CNN models. Chapter 5: Conclusion - This chapter summarizes the findings, discusses the implications, and suggests future research directions.
CHAPTER 2
Literature Survey
Shraddha Suratkar, Sayali Bhiungade, Jui Pitale, Komal Soni, Tushar Badgujar, and Faruk Kazi (2023) [1]explored a novel approach to deepfake video detection by integrating convolutional neural networks (CNNs) with recurrent neural networks (RNNs). Their study, published in the Journal of
Control and Decision, leverages the spatial feature extraction capabilities of CNNs and the temporal sequence analysis strengths of RNNs. This hybrid model enhances the detection of temporal inconsistencies and spatial anomalies in video frames, leading to improved accuracy in identifying deepfake videos. The researchers emphasize the importance of combining spatial and temporal analysis to effectively combat the sophisticated nature of deepfake technology. Their findings demonstrate that such an integrated approach can significantly outperform traditional methods relying solely on either spatial or temporal
features.
Anuj Badale, Lionel Castelino, Chaitanya Darekar, and Joanne Gomes (2018)[2] conducted an early study on deepfake detection utilizing neural networks, presented at the 15th IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS). Their research focuses on the capability of neural networks to detect subtle inconsistencies in deepfake videos by analyzing pixel-level anomalies and unusual patterns. They highlighted the potential of neural networks in identifying deepfake content by training models on large datasets of real and manipulated videos. This foundational work paved the way for more advanced neural network-based detection techniques and underscored the need for robust models capable of learning complex features associated with deepfake manipulations.
4
Shruti Agarwal, Hany Farid, Tarek El-Gaaly, and Ser-Nam Lim (2020)[3] developed a dual approach to deepfake video detection by examining both appearance and behavioral inconsistencies. Their study, presented at the IEEE International Workshop on Information Forensics and Security (WIFS), combines visual analysis with behavioral cues such as eye movement patterns, facial expressions, and lip-syncing accuracy. This comprehensive approach aims to enhance detection accuracy by leveraging multiple sources of information that may indicate manipulation. The authors demonstrate that considering behavioral anomalies, in addition to visual artifacts, significantly improves the robustness of deepfake detection systems.
Saadaldeen Rashid Ahmed, Emrullah Sonu¸c, Mohammed Rashid Ahmed, and Adil Deniz Duru (2022) [4]conducted an extensive survey on deepfake detection and recognition methods using convolutional neural networks (CNNs).
Their study, presented at the International Congress on Human-Computer Interaction, Optimization, and Robotic Applications (HORA), reviews various CNN architectures and their effectiveness in detecting deepfake media. They provide a detailed analysis of different techniques, highlighting the strengths and weaknesses of each approach. The authors emphasize the need for continuous innovation in CNN-based models to keep pace with the evolving deepfake generation technologies. Their survey serves as a valuable resource for researchers looking to understand the current state of deepfake detection and identify potential areas for further investigation.
Mohammed Sahib Mahdi Altaei and colleagues (2022)[5] focused on detecting deepfake manipulations in face images using advanced deep learning techniques. Published in the Wasit Journal of Computer and Mathematics Science, their research explores the effectiveness of deep learning models in identifying subtle facial inconsistencies introduced by deepfake algorithms. They developed a model that utilizes convolutional neural networks (CNNs) to capture and analyze fine-grained features in facial images. Their findings indicate that deep learning approaches can achieve high detection accuracy, especially when trained on diverse datasets. The study underscores the importance of developing robust models that can generalize well across different
types of deepfake manipulations.
David Gu¨era and Edward J. Delp (2018)[6] proposed a method for detecting deepfake videos using recurrent neural networks (RNNs), presented at the
15th IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS). Their approach leverages the temporal analysis capabilities of RNNs to identify inconsistencies over video frames, focusing on the sequential nature of video data. By examining temporal dependencies and anomalies, their model effectively distinguishes between genuine and manipulated videos. This research highlights the potential of RNNs in capturing dynamic features that are often indicative of deepfake manipulations, contributing to the development of more sophisticated and reliable video analysis tools.
Dmitry Gura, Bo Dong, Duaa Mehiar, and Nidal Al Said (2024)[7] developed a customized convolutional neural network (CNN) specifically designed for the accurate detection of deepfake images in video collections. Published in Computers, Materials and Continua, their work focuses on optimizing CNN architectures to improve feature extraction and detection precision. They introduce innovative techniques to enhance the model’s ability to discriminate between real and fake images, demonstrating superior performance compared to traditional CNN models. Their research emphasizes the importance of tailoring CNN architectures to address the unique challenges posed by deepfake detection, contributing to the advancement of more effective and efficient detection methods.
Sumaiya Thaseen Ikram, Shourya Chambial, Dhruv Sood, et al. (2023) [8]developed a hybrid CNN deep learning model aimed at enhancing the performance of deepfake video detection. Their study, published in the International Journal of Electrical and Computer Engineering Systems, integrates multiple CNN layers to capture a wide range of features and improve detection accuracy. By combining different layers, the model can analyze both high-level and low-level features in video frames, making it more adept at identifying subtle manipulations. The researchers demonstrate that this hybrid approach significantly boosts the detection capabilities, providing a robust solution to the challenges posed by increasingly sophisticated deepfake technologies.
Asad Malik, Minoru Kuribayashi, Sani M. Abdullahi, and Ahmad Neyaz Khan (2022)[9] conducted a comprehensive survey on deepfake detection techniques for human face images and videos, published in IEEE Access. Their work reviews various state-of-the-art methods and technologies used to identify deepfake media. They provide an extensive analysis of different approaches, including CNNs, RNNs, and hybrid models, discussing their effectiveness, strengths, and limitations. The survey highlights the rapid advancements in deepfake generation and the corresponding need for improved detection techniques. The authors also identify gaps in current research and suggest potential areas for future investigation, emphasizing the importance of continuous innovation in this field.
Huy H. Nguyen, Junichi Yamagishi, and <PRESIDIO_ANONYMIZED_PERSON> (2019)[10] proposed using capsule networks to detect fake images and videos, as outlined in their preprint on arXiv. Capsule networks are designed to capture spatial hierarchies and relationships within images, making them particularly effective for identifying manipulations. The authors demonstrate that capsule networks can preserve spatial information better than traditional CNNs, which enhances their ability to detect deepfake artifacts. Their research shows that capsule networks offer a promising alternative to existing methods, providing a robust framework for detecting complex manipulations in both images and videos.
Samay Pashine, Sagar Mandiya, Praveen Gupta, and Rashid Sheikh (2021) [11]conducted a survey on various solutions for detecting facial manipulations, with a focus on deepfake detection. Their preprint on arXiv reviews different techniques and algorithms used to identify altered facial features and expressions in deepfake media. The authors discuss the effectiveness of each approach, including machine learning and deep learning models, and highlight their respective advantages and limitations. They provide a comprehensive overview of current detection technologies and suggest improvements for enhancing detection accuracy and robustness. This survey serves as a valuable resource for researchers and practitioners looking to understand the landscape of facial manipulation detection solutions.
Yogesh Patel, Sudeep Tanwar, Pronaya Bhattacharya, Rajesh Gupta, Turki Alsuwian, Innocent Ewean Davidson, and Thokozile F. Mazibuko (2023) [12]introduced an improved dense CNN architecture designed specifically for deepfake image detection. Published in IEEE Access, their model leverages densely connected CNN layers to enhance feature extraction and improve detection accuracy. The dense connections allow the model to capture more detailed and nuanced features, making it more effective at identifying deepfake images. The authors demonstrate that their architecture outperforms traditional CNN models, providing a significant advancement in the field of deepfake detection and contributing to the development of more reliable and efficient detection systems.
ST Suganthi, Mohamed Uvaze Ahamed Ayoobkhan, Nebojsa Bacanin, K Venkatachalam, Hub´alovsky´ Stˇep´an, Trojovsky´ Pavel, et al.ˇ (2022)[13] developed a deep learning model for deepfake face recognition and detection, published in PeerJ Computer Science. Their research explores the application of deep learning techniques to identify manipulated faces in both images and videos. The model is designed to analyze facial features and detect inconsistencies indicative of deepfake manipulations. The authors propose novel methods to enhance the model’s accuracy and robustness, including advanced preprocessing techniques and optimized network architectures. Their work contributes to the ongoing efforts to improve deepfake detection technology and offers practical solutions for real-world applications.
Hasin Shahed Shad, Md Mashfiq Rizvee, Nishat Tasnim Roza, SM Ahsanul
Hoq, Mohammad Monirujjaman Khan, Arjun Singh, Atef Zaguia, and Sami Bourouis (2021)[14] conducted a comparative analysis of different deepfake image detection methods using convolutional neural networks (CNNs). Published in Computational Intelligence and Neuroscience, their study evaluates the performance of various CNN-based approaches in identifying deepfake images. The authors compare multiple models, assessing their strengths and weaknesses, and provide insights into the most effective techniques for detecting manipulated images. Their findings highlight the importance of selecting appropriate CNN architectures and training strategies to achieve high detection accuracy, offering valuable guidance for researchers and practitioners in the field.
Deressa Wodajo and Solomon Atnafu (2021)[15] proposed the use of convolutional vision transformers for deepfake video detection, as described in their preprint on arXiv. Their approach combines the strengths of convolutional neural networks (CNNs) and transformers to capture both spatial and temporal features in video data. The convolutional vision transformer architecture leverages the transformer model’s ability to handle sequential data effectively while also benefiting from the CNN’s spatial feature extraction capabilities. The authors demonstrate that this hybrid model achieves superior detection performance, providing a robust solution for identifying deepfake videos. Their research contributes to the advancement of deepfake detection technologies by integrating the best aspects of CNNs and transformers.
CHAPTER 3
Methodology
The effectiveness of a deep fake detection system largely depends on the quality and diversity of the dataset. Collecting a comprehensive dataset that includes a wide variety of real and deep fake images and videos is essential for training a CNN that can generalize well to unseen data.
3.1 Data Collection
3.1.1 Real Data Collection
Real images and videos can be sourced from various platforms:
• Public Datasets: Leveraging existing public datasets like ImageNet, CelebA, and YouTube-8M can provide a rich source of authentic content.
• Web Scraping: Custom scripts can be used to scrape real images and videos from the web, ensuring compliance with legal and ethical
standards.
• Crowdsourcing: Platforms like Amazon Mechanical Turk can be used to collect real images and videos from diverse contributors.
3.1.2 Deep Fake Data Collection
Generating and collecting deep fake data involves:
• Public Deep Fake Datasets: Utilizing existing datasets like FaceForensics++, DeepFakeDetection, and DFDC (Deep Fake Detection Challenge) which contain a variety of manipulated content.
• Custom Generation: Using deep fake generation tools like DeepFaceLab, FaceSwap, and FSGAN to create custom deep fake videos. This ensures the inclusion of the latest manipulation techniques.
10
3.1.3 Dataset Composition
A balanced dataset should include:
• A mix of high and low-quality images and videos.
• A diverse set of subjects to prevent bias.
• Various types of deep fake manipulations (e.g., face swaps, lip-syncing).
3.2 Data Preprocessing
Once the data is collected, preprocessing is required to prepare it for training the CNN model. This includes cleaning, augmenting, and transforming the
data.
3.2.1 Data Cleaning
Cleaning involves:
• Removing Duplicates: Ensuring no repeated images or videos to avoid skewing the training process.
• Quality Check: Filtering out corrupted or low-quality files that might hinder the training process.
3.2.2 Data Annotation
Labeling the dataset accurately is crucial:
• Manual Annotation: Using human annotators to label real and fake
content.
• Automated Tools: Leveraging tools like Labelbox or custom scripts to streamline the annotation process.
3.2.3 Data Augmentation
Augmentation techniques are employed to increase the diversity of the dataset and improve the model’s robustness:
• Geometric Transformations: Including rotations, translations, scaling, and flips.
• Color Adjustments: Varying brightness, contrast, saturation, and hue.
• Noise Injection: Adding random noise to make the model robust to
noisy inputs.
• Synthetic Data: Generating synthetic data using Generative Adversarial Networks (GANs) to simulate rare scenarios.
3.2.4 Data Normalization
Normalizing the data helps in standardizing the input, making the training process more stable and faster:
• Pixel Scaling: Rescaling pixel values to a range of 0 to 1 or -1 to 1.
• Mean Subtraction: Subtracting the dataset mean from each image to center the data.
3.2.5 Data Splitting
Splitting the dataset into training, validation, and test sets ensures the model’s performance is evaluated correctly:
• Training Set: Typically 70-80% of the data used for training the model.
• Validation Set: Around 10-15% of the data used for hyperparameter tuning and validation during training.
• Test Set: Remaining 10-15% of the data used for final evaluation of the model’s performance.
Department of Computer Science and Engineering 12
3.3 Data Pipeline Implementation
A robust data pipeline ensures seamless data handling from collection to model training:
• Data Loading: Efficient data loaders using libraries like TensorFlow Data API or PyTorch DataLoader.
• Data Transformation: Applying preprocessing steps on-the-fly during training to save memory and improve performance.
• Batch Processing: Processing data in batches to optimize GPU utilization and speed up training.
3.4 Challenges and Considerations
3.4.1 Data Imbalance
Addressing data imbalance is crucial to prevent the model from being biased towards the majority class. Techniques like oversampling, undersampling, and synthetic minority over-sampling technique (SMOTE) can be employed.
3.4.2 Privacy and Ethics
Ensuring the collection and use of data comply with privacy laws and
ethical standards:
• Consent: Obtaining consent from individuals whose data is being col-
lected.
• Anonymization: Anonymizing sensitive information to protect privacy.
3.4.3 Computational Resources
Processing and training on large datasets require significant computational resources. Leveraging cloud services or high-performance computing clusters can mitigate this issue.
CHAPTER 4
Architecture and Model
A robust deep fake detection model for images can leverage a Convolutional Neural Network (CNN) architecture. Input preprocessing involves normalizing and resizing images. The model includes multiple convolutional layers to extract spatial features, followed by pooling layers to reduce dimensionality. Batch normalization and dropout layers improve generalization and prevent overfitting. Extracted features are fed into fully connected layers, culminating in a softmax layer for binary classification (real or fake). VGG, custom CNN, and DenseNet architectures can be employed, with training performed using a dataset of real and fake images, optimized with cross-entropy loss and Adam.
Figure 4.1: Architecture Diagram used for deep fake detection of images
14
A robust deep fake detection model for videos involves several key components arranged in a structured architecture. The input video frames are preprocessed with face cropping and alignment to ensure uniformity. Facial segmentation is then performed to isolate relevant facial features. A CNN architecture extracts spatial features from these segmented faces, followed by attention mechanisms to focus on crucial regions. Attention masks are applied to highlight and analyze these regions effectively. Forensic analysis techniques are employed within the network to determine the authenticity of the frames. The final classification layer combines these analyses to output whether the video is real or fake.
Figure 4.2: flow chart diagram used for deep fake detection of videos
CHAPTER 5
Implementation
5.1 Setup and Installation
5.1.1 Installing Required Libraries
To start with the implementation, we need to install the necessary libraries.
The following code installs TensorFlow, OpenCV, NumPy, Scikit-learn, PyTorch, Torchvision, Torchaudio, Matplotlib, Albumentations, and TQDM.
!pipinstalltensorflowopencv-pythonopencv-python-headlessnumpyscikit-learn
!pip install torch torchvision torchaudio
!pip install matplotlib albumentations tqdm
5.2 Model Training
5.2.1 Training the Model
We then train the model using the CIFAR-10 dataset, filtered to include only two classes for simplicity. This step involves setting up data augmentation, normalization, and a training loop.
from torchvision import datasets, transforms from torch.utils.data import DataLoader, Subset
transform = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]), ])
16
train_dataset=datasets.CIFAR10
(root=’./data’,train=True,download=True,transform=transform) classes_to_keep = [0, 1] indices=[i for i,label in enumerate(train_dataset.targets) if label in classes_to_keep] subset_dataset = Subset(train_dataset, indices) subset_dataset_modified =[(image, classes_to_keep.index(label)) for image, label in subset_dataset] train_loader = DataLoader(subset_dataset_modified, batch_size=16, shuffle=True, num_workers=2)
num_epochs = 10
for epoch in range(num_epochs):
model.train() running_loss = 0.0 for images, labels in tqdm(train_loader):
images = images.to(device) labels = labels.to(device)
optimizer.zero_grad() outputs = model(images) loss = criterion(outputs, labels) loss.backward() optimizer.step() running_loss += loss.item()
print(f’Epoch [{epoch+1}/{num_epochs}],
Loss: {running_loss/len(train_loader):.4f}’)
Output:
Figure 5.1: training the model
5.3 Inference on Images
5.3.1 Defining the Inference Function
We define a function to perform inference on a single image to determine if it is real or fake.
import torchvision.transforms as transforms from torchvision import models
device = torch.device(’cuda’ if torch.cuda.is_available() else ’cpu’) model = models.resnet18(pretrained=True) model.fc = torch.nn.Linear(model.fc.in_features, 2) model = model.to(device)
transform = transforms.Compose([ transforms.ToPILImage(), transforms.Resize((224, 224)), transforms.ToTensor(), transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]) ])
def detect_deepfake(model, image_path, transform):
model.eval() image = cv2.imread(image_path) if image is None: raiseValueError(f"Imagenotfoundorunabletoload:{image_path}")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) transformed = transform(image) transformed = transformed.unsqueeze(0).to(device)
with torch.no_grad():
output = model(transformed)
_, pred = torch.max(output, 1)
return ’Fake’ if pred.item() == 1 else ’Real’
image_path = ’/content/drive/MyDrive/fake/pic.png’ result = detect_deepfake(model, image_path, transform) print(f’The image is: {result}’) Output:
Figure 5.2: output for giving real image as input to the model
5.4 Inference on Videos
5.4.1 Defining the Video Inference Function
We create a function to perform inference on a video, analyzing each frame to determine if the video is mostly real or fake.
import cv2 import torch import torch.nn as nn import torchvision.models as models from albumentations.pytorch import ToTensorV2 import albumentations as A
transform = A.Compose([
A.Resize(224, 224),
A.Normalize(mean=(0.485,0.456,0.406),std=(0.229,0.224,0.225)),
ToTensorV2() ]) device = torch.device(’cuda’ if torch.cuda.is_available() else ’cpu’) model = models.resnet50(pretrained=True) num_features = model.fc.in_features model.fc = nn.Linear(num_features, 2) model = model.to(device) def detect_deepfake_in_video(model, video_path, transform, device):
model.eval() cap = cv2.VideoCapture(video_path) if not cap.isOpened(): print("Error: Could not open video.") return ’Unknown’, 0, 0 frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) print(f"Total number of frames: {frame_count}") predictions = [] processed_frames = 0
while cap.isOpened():
ret, frame = cap.read() if not ret:
break
image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) transformed = transform(image=image) image = transformed[’image’].unsqueeze(0).to(device)
with torch.no_grad(): output = model(image)
_, predicted = torch.max(output.data, 1) predictions.append(predicted.item())
processed_frames += 1 print(f"Processed frame: {processed_frames}/{frame_count}")
cap.release() fake_frames = sum(predictions) real_frames = len(predictions) - fake_frames if fake_frames > real_frames:
return ’Fake’, fake_frames, real_frames else:
return ’Real’, fake_frames, real_frames video_path = ’/content/drive/MyDrive/deepfake_video.mp4’ result, fake_frames, real_frames = detect_deepfake_in_video
(model, video_path, transform, device) print(f"The video is: {result} (Fake frames: {fake_frames},
Real frames: {real_frames})") Output:
Figure 5.3: output for giving fake video as input to the model
Figure 5.4: output for giving real video as input to the model
CHAPTER 6
Conclusions and Future Scope
6.1 Summary of Findings
The research on deep fake detection using Convolutional Neural Networks (CNNs) for images and videos has yielded several significant findings:
• Effectiveness of CNNs: CNNs have proven to be highly effective in
identifying deep fakes, leveraging their ability to capture intricate patterns and features in images and videos that are indicative of synthetic content.
• Data Preprocessing Importance: Proper data preprocessing, including normalization, resizing, and augmentation, is crucial for improving model accuracy and generalizability.
• Feature Extraction: The use of pre-trained CNN models for feature extraction and subsequent fine-tuning on deep fake datasets enhances detection performance.
• Temporal Analysis: For video data, incorporating temporal features using 3D CNNs or recurrent neural networks (RNNs) significantly improves the model’s ability to detect deep fakes over sequences of frames.
• Visualization Tools: Effective visualization tools, such as heatmaps, saliency maps, and ROC curves, provide valuable insights into model performance and areas of improvement.
6.2 Contributions
The key contributions of this research include:
• Development of a Robust CNN Model: Creation of a CNN-based
framework that efficiently detects deep fakes in both images and videos.
22
• Comprehensive Data Pipeline: Establishment of a detailed data
preprocessing and augmentation pipeline tailored for deep fake detection.
• Enhanced Feature Analysis: Implementation of advanced feature extraction and visualization techniques to better understand model behavior and improve interpretability.
• Performance Benchmarks: Setting benchmarks for detection accuracy, precision, recall, and F1 score, facilitating comparison with future research.
6.3 Future Research Directions
The field of deep fake detection is rapidly evolving, and several avenues for future research can be explored:
• Advanced Model Architectures: Investigate the use of more advanced neural network architectures, such as transformer models or hybrid CNNRNN models, to further improve detection accuracy.
• Real-time Detection: Develop techniques for real-time deep fake detection, particularly in video streams, to enhance applicability in live
scenarios.
• Robustness and Generalization: Focus on improving the robustness and generalization of models to various types of deep fakes, including those generated by new and emerging techniques.
• Explainability and Transparency: Enhance the explainability of deep fake detection models, making it easier for users to understand the decision-making process and trust the results.
• Ethical Considerations: Address ethical issues related to the use of deep fake detection technology, such as privacy concerns and the potential for misuse.
6.4 Final Remarks
Deep fake detection remains a critical area of research in the context of digital security and media integrity. The advancements made through the application of CNNs have shown promise in effectively combating the threats posed by synthetic media. However, the dynamic nature of deep fake generation techniques necessitates continuous innovation and adaptation in detection methodologies. By pursuing the outlined future research directions and maintaining a focus on ethical considerations, the field can continue to progress and contribute to safeguarding digital content authenticity.
CHAPTER 7
Results
7.0.1 Images
Accuracy: CNN models typically achieve high accuracy rates in detecting deep fakes in images. For instance, state-of-the-art models often report accuracy levels above 90% on benchmark datasets.
Precision and Recall: Precision and recall metrics are crucial in understanding the performance. CNN models usually show high precision (correctly identifying real vs. fake images) and recall (correctly detecting all actual fake images), often exceeding 85%.
Confusion Matrix: Analysis of the confusion matrix reveals that CNNs can effectively distinguish between real and fake images, with minimal false positives and false negatives.
ROC-AUC: The Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) scores for CNN models are generally high (often above 0.9), indicating strong discriminative power.
7.0.2 Videos
Frame-wise Detection: When applied to video frames, CNN models can achieve frame-wise accuracy rates similar to those for images, typically above 90%. However, per-frame accuracy might slightly decrease due to motion blur and varying lighting conditions.
Temporal Consistency: By analyzing the temporal consistency across frames, models can improve overall video classification accuracy. Techniques like Long Short-Term Memory (LSTM) networks or temporal attention mechanisms are often employed in conjunction with CNNs.
Overall Video Accuracy: Combined models incorporating both spatial (CNN) and temporal features can achieve high overall accuracy in detecting deep fake videos, often reported in the range of 85-95%.
25
Precision and Recall in Videos: Similar to images, precision and recall for video deep fake detection are crucial metrics. Models usually show precision and recall values around 80-90%, ensuring robust detection across different video contexts.
Real-time Detection: Advanced models can be optimized for real-time detection, maintaining high accuracy while processing frames efficiently.
7.0.3 Summary
CNN-based models demonstrate strong performance in detecting deep fakes in both images and videos. While image detection can achieve very high accuracy and reliability, video detection benefits from additional temporal analysis to maintain high performance. Combining CNNs with attention mechanisms, forensic analysis, and temporal consistency checks further enhances the detection capabilities, making these models effective tools in combating deep fakes.
REFERENCES
[1] Shraddha Suratkar, Sayali Bhiungade, Jui Pitale, Komal Soni, Tushar Badgujar, and Faruk Kazi. “Deep-fake video detection approaches using convolutional–recurrent neural networks”. In: Journal of Control and Decision 10.2 (2023), pp. 198–214.
[2] Anuj Badale, Lionel Castelino, Chaitanya Darekar, and Joanne Gomes. “Deep fake detection using neural networks”. In: 15th IEEE international conference on advanced video and signal based surveillance (AVSS). 2018.
[3] Shruti Agarwal, Hany Farid, Tarek El-Gaaly, and Ser-Nam Lim. “Detecting deep-fake videos from appearance and behavior”. In: 2020 IEEE international workshop on information forensics and security (WIFS). IEEE. 2020, pp. 1–6.
[4] Saadaldeen Rashid Ahmed, Emrullah Sonuc¸, Mohammed Rashid Ahmed, and Adil Deniz Duru. “Analysis survey on deepfake detection and recognition with convolutional neural networks”. In: 2022 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA). IEEE. 2022, pp. 1–7.
[5] Mohammed Sahib Mahdi Altaei et al. “Detection of Deep Fake in Face Images Using Deep Learning”. In: Wasit Journal of Computer and Mathematics Science 1.4 (2022).
[6] David Gu¨era and Edward J Delp. “Deepfake video detection using recurrent neural networks”. In: 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS). IEEE. 2018, pp. 1–6.
[7] Dmitry Gura, Bo Dong, Duaa Mehiar, and Nidal Al Said. “Customized Convolutional Neural Network for Accurate Detection of Deep Fake Images in Video Collections.” In: Computers, Materials & Continua 79.2 (2024).
[8] Sumaiya Thaseen Ikram, Shourya Chambial, Dhruv Sood, et al. “A performance enhancement of deepfake video detection through the use of a hybrid CNN Deep learning model”. In: International journal of electrical and computer engineering systems 14.2 (2023), pp. 169–178.
[9] Asad Malik, Minoru Kuribayashi, Sani M Abdullahi, and Ahmad Neyaz Khan. “DeepFake detection for human face images and videos: A survey”. In: Ieee Access 10 (2022), pp. 18757–18775.
[10] Huy H Nguyen, Junichi Yamagishi, and <PRESIDIO_ANONYMIZED_PERSON>. “Use of a capsule network to detect fake images and videos”. In: arXiv preprint arXiv:1910.12467 (2019).
[11] Samay Pashine, Sagar Mandiya, Praveen Gupta, and Rashid Sheikh. “Deep fake detection: Survey of facial manipulation detection solutions”. In: arXiv preprint arXiv:2106.12605 (2021).
27
[12] Yogesh Patel, Sudeep Tanwar, Pronaya Bhattacharya, Rajesh Gupta, Turki Alsuwian, Innocent Ewean Davidson, and Thokozile F Mazibuko. “An improved dense cnn architecture for deepfake image detection”. In: IEEE Access 11 (2023), pp. 22081–22095.
[13] ST Suganthi, Mohamed Uvaze Ahamed Ayoobkhan, Nebojsa Bacanin, K Venkatachalam, Hub´alovsky` Stˇep´an, Trojovsky` Pavel, et al. “Deepˇ learning model for deep fake face recognition and detection”. In: PeerJ Computer Science 8 (2022), e881.
[14] Hasin Shahed Shad, Md Mashfiq Rizvee, Nishat Tasnim Roza, SM Ahsanul Hoq, Mohammad Monirujjaman Khan, Arjun Singh, Atef Zaguia, and Sami Bourouis. “[Retracted] Comparative Analysis of Deepfake Image Detection Method Using Convolutional Neural Network”. In: Computational intelligence and neuroscience 2021.1 (2021), p. 3111676.
[15] Deressa Wodajo and Solomon Atnafu. “Deepfake video detection using convolutional vision transformer”. In: arXiv preprint arXiv:2102.11126 (2021).
this is the report of a project do similar report with all sections length being same but the context of a new project whose code is
" from tkinter import messagebox
from tkinter import *
from tkinter import simpledialog
import tkinter
from tkinter import filedialog
from tkinter.filedialog import askopenfilename
import cv2 as cv
import numpy as np
import os
from keras.preprocessing.image import load_img
from keras.utils import to_categorical
from keras.preprocessing.image import img_to_array
# from sklearn.model_selection import train_test_split
import pickle
from keras.models import load_model
from keras.applications import VGG16
from keras.layers import Flatten
from keras.layers import Dropout
from keras.layers import Dense
from keras.layers import Input
from keras.models import Model
from keras.optimizers import Adam
from keras.models import model_from_json
main = tkinter.Tk()
main.title("SSLA Based Traffic Sign and Lane Detection for Autonomous cars")
main.geometry("1300x1200")
global filename
global model
old = None
class_labels = ['Speed limit (20km/h)','Speed limit (30km/h)','Speed limit (50km/h)','Speed limit (60km/h)','Speed limit (70km/h)',
'Speed limit (80km/h)','End of speed limit (80km/h)','Speed limit (100km/h)','Speed limit (120km/h)','No passing','Stop','No Entry',
'General caution','Traffic signals']
def cannyDetection(img):
grayImg = cv.cvtColor(img, cv.COLOR_RGB2GRAY)
blurImg = cv.GaussianBlur(grayImg, (5, 5), 0)
cannyImg = cv.Canny(blurImg, 50, 150)
return cannyImg
def segmentDetection(img):
height = img.shape[0]
polygons = np.array([[(0, height), (800, height), (380, 290)]])
maskImg = np.zeros_like(img)
cv.fillPoly(maskImg, polygons, 255)
segmentImg = cv.bitwise_and(img, maskImg)
return segmentImg
def calculateLines(frame, lines):
left = []
right = []
for line in lines:
x1, y1, x2, y2 = line.reshape(4)
parameters = np.polyfit((x1, x2), (y1, y2), 1)
slope = parameters[0]
y_intercept = parameters[1]
if slope < 0:
left.append((slope, y_intercept))
else:
right.append((slope, y_intercept))
left_avg = np.average(left, axis = 0)
right_avg = np.average(right, axis = 0)
left_line = calculateCoordinates(frame, left_avg)
right_line = calculateCoordinates(frame, right_avg)
return np.array([left_line, right_line])
def calculateCoordinates(frame, parameters):
global old
#print(str(parameters)+" "+str(type(parameters))+" "+str(np.isnan(parameters)))
if old is None:
old = parameters
if np.isnan(parameters.any()) == False:
parameters = old
slope, intercept = parameters
y1 = frame.shape[0]
y2 = int(y1 - 150)
x1 = int((y1 - intercept) / slope)
x2 = int((y2 - intercept) / slope)
return np.array([x1, y1, x2, y2])
def visualizeLines(frame, lines):
lines_visualize = np.zeros_like(frame)
if lines is not None:
for x1, y1, x2, y2 in lines:
cv.line(lines_visualize, (x1, y1), (x2, y2), (0, 255, 0), 5)
return lines_visualize
def loadModel():
global model
model = load_model('model/model.h5')
pathlabel.config(text="Machine Learning Traffic Sign Detection Model Loaded")
text.delete('1.0', END)
text.insert(END,"Machine Learning Traffic Sign Detection Model Loaded\n\n");
def detectSignal():
global model
filename = filedialog.askopenfilename(initialdir="Videos")
pathlabel.config(text=filename)
text.delete('1.0', END)
text.insert(END,filename+" loaded\n\n")
text.update_idletasks()
cap = cv.VideoCapture(filename)
while (cap.isOpened()):
ret, frame = cap.read()
if frame is not None:
canny = cannyDetection(frame)
cv.imshow("cannyImage", canny)
segment = segmentDetection(canny)
hough = cv.HoughLinesP(segment, 2, np.pi / 180, 100, np.array([]), minLineLength = 100, maxLineGap = 50)
if hough is not None:
lines = calculateLines(frame, hough)
linesVisualize = visualizeLines(frame, lines)
cv.imshow("hough", linesVisualize)
output = cv.addWeighted(frame, 0.9, linesVisualize, 1, 1)
cv.imwrite("test.jpg",output)
temps = cv.imread("test.jpg")
h, w, c = temps.shape
image = load_img("test.jpg", target_size=(80, 80))
image = img_to_array(image) / 255.0
image = np.expand_dims(image, axis=0)
(boxPreds, labelPreds) = model.predict(image)
print(boxPreds)
boxPreds = boxPreds[0]
startX = int(boxPreds[0] * w)
startY = int(boxPreds[1] * h)
endX = int(boxPreds[2] * w)
endY = int(boxPreds[3] * h)
predict= np.argmax(labelPreds, axis=1)
predict = predict[0]
accuracy = np.amax(labelPreds, axis=1)
print(str(class_labels[predict])+" "+str(accuracy))
if accuracy > 0.97:
cv.putText(output, "Recognized As "+str(class_labels[predict]), (startX, startY), cv.FONT_HERSHEY_SIMPLEX, 0.7, (0, 255, 0), 2)
cv.rectangle(output, (startX, startY), (endX, endY), (0, 255, 0), 2)
cv.imshow("output", output)
if cv.waitKey(10) & 0xFF == ord('q'):
break
cap.release()
cv.destroyAllWindows()
def close():
main.destroy()
font = ('times', 16, 'bold')
title = Label(main, text='SSLA Based Traffic Sign and Lane Detection for Autonomous cars',anchor=W, justify=CENTER)
title.config(bg='yellow4', fg='white')
title.config(font=font)
title.config(height=3, width=120)
title.place(x=0,y=5)
font1 = ('times', 14, 'bold')
upload = Button(main, text="Generate & Load Machine Learning Model", command=loadModel)
upload.place(x=50,y=100)
upload.config(font=font1)
pathlabel = Label(main)
pathlabel.config(bg='yellow4', fg='white')
pathlabel.config(font=font1)
pathlabel.place(x=50,y=150)
markovButton = Button(main, text="Upload Video & Detect Hough Lane, Signal", command=detectSignal)
markovButton.place(x=50,y=200)
markovButton.config(font=font1)
predictButton = Button(main, text="Exit", command=close)
predictButton.place(x=50,y=250)
predictButton.config(font=font1)
font1 = ('times', 12, 'bold')
text=Text(main,height=15,width=78)
scroll=Scrollbar(text)
text.configure(yscrollcommand=scroll.set)
text.place(x=450,y=100)
text.config(font=font1)
main.config(bg='magenta3')
main.mainloop()"