Souha-BH/BERT_Resnet50

Model Card for Multimodal Risk Behavior Detection Model

Model Overview

The Multimodal Risk Behavior Detection Model is designed to detect risky health behaviors in TikTok videos. By leveraging both visual and textual features from TikTok video content, the model can classify whether a video portrays risky health behaviors, such as smoking, alcohol consumption, or unhealthy eating habits. The model integrates two pre-trained architectures: BERT for text feature extraction and ResNet50 for video frame analysis, combining their outputs to make predictions.

Training Data

The model was trained on the "Detecting Risky Health Behaviors in TikTok Videos" dataset: https://huggingface.co/datasets/Souha-BH/DetectingRiskyHealthBehaviorsInTikTokVideos. This dataset includes video metadata, captions, and video clips, which are labeled as either risky or non-risky. The data was collected using the Apify TikTok Hashtag Scraper and annotated for risky health behaviors. The model uses the text column (captions) and the corresponding video files from the dataset to extract text and visual features.

• Training-Validation-Test Split: The dataset was split into training, validation, and test sets using the "Split" column.
• Training set: Used to train the model's parameters.
• Validation set: Used to tune hyperparameters and avoid overfitting.
• Test set: Used to evaluate the final performance of the model.

Model Architecture

The Multimodal Risk Behavior Detection Model follows a multimodal approach that integrates both textual and visual modalities.

• Textual Features: Extracted using BERT (bert-base-uncased), with tokenized video captions passed through BERT's transformer layers.
• Visual Features: Extracted using ResNet50, where frames from each TikTok video are resized and processed to generate high-level visual embeddings.
• Feature Fusion: The embeddings from BERT and ResNet50 are concatenated and passed through a series of fully connected layers with ReLU activations and dropout regularization to prevent overfitting.
• Classification Layer: The final layer is a single-unit sigmoid layer that outputs a probability between 0 and 1, with 0.5 as the threshold for classification.

Training Procedure

• Loss Function: Binary Cross-Entropy Loss (BCE) was used to compute the error between predicted probabilities and true labels.
• Optimizer: Adam optimizer with a learning rate of 2e-5.
• Batch Size: 4 video samples per batch.
• Epochs: The model was trained for 5 epochs.
• Video Frame Limit: Each video was sampled for 10 frames to reduce computational overhead.
• Augmentation and Normalization: Frames were resized to 224x224 and normalized using ImageNet's mean and standard deviation.

Evaluation Metrics

The model was evaluated on the test set using the following metrics:

• Accuracy: Measures the proportion of correct predictions.
• Precision: Measures how many of the predicted "risky" videos were actually risky.
• Recall: Measures how many of the actual risky videos were correctly identified.
• F1 Score: The harmonic mean of precision and recall, balancing both metrics.
• ROC-AUC: Measures the area under the ROC curve, showing the model's ability to distinguish between risky and non-risky videos.

Model Performance

After training for 5 epochs, the model's performance on the test set was as follows:

• Accuracy: 63.33%
• Precision: 55.00%
• Recall: 84.62%
• F1 Score: 66.67%
• ROC-AUC: 65.84%

Usage

• Input: A TikTok video and its corresponding caption.
• Output: A probability score indicating the likelihood that the video depicts a risky health behavior.

Limitations

• Data Balance: If the dataset is imbalanced (more non-risky videos than risky ones), the model may struggle with precision.
• Contextual Understanding: The model relies heavily on textual captions. If a caption does not explicitly describe risky behavior, the model may underperform.
• Limited Frame Sampling: Only 10 frames per video are processed, which may miss important content, especially for longer videos.