---
title: Multi Modal Emotion Recognition
emoji: 📈
colorFrom: gray
colorTo: blue
sdk: gradio
sdk_version: "4.44.0"
app_file: app.py
pinned: false
license: mit
---

# Multi Modal Emotion Recognition 📈

This application allows users to analyze emotions from videos using state-of-the-art models for both audio and visual content. You can upload videos (maximum length of 2 minutes) to extract emotions from both speech and facial expressions in real-time.

## Features:
- **Audio Emotion Detection:** Uses OpenAI's Whisper model for transcription and Cardiff NLP's RoBERTa model for emotion recognition in text.
- **Visual Emotion Analysis:** Leverages Salesforce's BLIP model for image captioning and J-Hartmann's DistilRoBERTa for visual emotion recognition.

## Instructions:
1. Upload a video file (maximum length: **2 minutes**).
2. The app will analyze both the audio and visual components of the video to extract and display emotions in real-time.

## Models Used:
The models have been handpicked after numerous trials and are optimized for this task. Below are the models and the corresponding research papers:

1. **Cardiff NLP RoBERTa for Emotion Recognition from Text:**
   - [Model: cardiffnlp/twitter-roberta-base-emotion](https://huggingface.co/cardiffnlp/twitter-roberta-base-emotion)
   - [Paper: RoBERTa Sentiment & Emotion Analysis](https://arxiv.org/pdf/2010.12421)

2. **Salesforce BLIP for Image Captioning and Visual Emotion Analysis:**
   - [Model: Salesforce/blip-image-captioning-base](https://huggingface.co/Salesforce/blip-image-captioning-base)
   - [Paper: BLIP - Bootstrapping Language-Image Pre-training](https://arxiv.org/abs/2201.12086)

3. **J-Hartmann DistilRoBERTa for Emotion Recognition from Images:**
   - [Model: j-hartmann/emotion-english-distilroberta-base](https://huggingface.co/j-hartmann/emotion-english-distilroberta-base)

4. **OpenAI Whisper for Speech-to-Text Transcription:**
   - [Model: openai/whisper-base](https://huggingface.co/openai/whisper-base)
   - [Paper: Whisper - Speech Recognition](https://arxiv.org/abs/2212.04356)

These models were selected based on extensive trials to ensure the best performance for this multimodal emotion recognition task.

## Access the App:
You can try the app [here](https://huggingface.co/spaces/Pradheep1647/multi-modal-emotion-recognition).

## License:
This project is licensed under the MIT License.