README.md

---
title: BlinkBlur
emoji: 👀
sdk: gradio
sdk_version: 4.10.0
python_version: 3.9
app_file: app.py
---

# Eye Blink / Blurriness detectors

### Try [BlinkBlur on HuggingFace 🤗](https://huggingface.co/spaces/ArnoBen/BlinkBlur)

## Packages
```requirements.txt
gradio==4.10.0
matplotlib==3.8.2
mediapipe==0.10.9
numpy==1.26.2
opencv-contrib-python==4.8.1.78
opencv-python==4.8.1.78
pandas==2.1.4
pytorch-lightning==2.1.2
scikit-learn==1.3.2
seaborn==0.13.0
torch==2.1.2
torchvision==0.16.2
tqdm==4.66.1
```

## How to use
```commandline
usage: main.py [-h] [-w] [-p PATH] [-e] [-b]

optional arguments:
  -h, --help            show this help message and exit
  -w, --webcam          Runs eye blink detection on the user's webcam.
                        Overrides other options.
  -p PATH, --path PATH  path of image to process
  -e, --eyes            Detects eye blink on the input image
  -b, --blur            Estimates image blur
```

You can try with images available in `examples/`.

Command example:
```commandline
python main.py -w
python main.py --path examples/close.jpg --eyes
python main.py -p examples/open_far.jpg -e
python main.py -p examples/blurry_1.jpg --blur
python main.py -p examples/sharp_1.jpg -b
```

## Metrics

### Closed-eyes detector
Here are some evaluation metrics on the model, calculated with the code in `closed_eyes_detection/training/test.py`.

![Metrics](assets/closed_eyes_metrics.png)

The confusion matrix has been calculated on a classification threshold of 0.49,
this corresponds to the red cross on the curve.

My assumption is that we want to make sure we don't keep a picture where the user is blinking
so we need to be more strict on it to maximize recall. 

### Blurriness estimator

The blurriness estimator uses the variance of the laplacian to assess the blurriness.

Here are some metrics and statistics on the blurriness detection task, 
calculated with the code in `blurriness_estimation/calculate_metrics.py`.

<img src="assets/distribution.png" width="400"/>

The red vertical line indicates the chosen threshold, given by the following metrics:

![Metrics](assets/blurriness_metrics.png)

## Assumptions
- Input images are in RGB format, in a format readable by OpenCV/PIL
- For live webcam detection, the user has at least 1 camera
- The person's face is not rotated
- The person's face is not too close to the sides of the frame (this can lead to a handled error during cropping)
 
## Decisions
- If several persons appear, only the first one detected will be processed
- I use MediaPipe's Face Detection to get eyes coordinates, then crop the eyes region and feed them into a custom model.
- The blurriness estimation is based on an edge detector (variance of laplacian). 
The decision is done with a threshold of 250, selected based on metrics detailed in the Metrics section.

## Datasets

- I trained the closed eye detection model on [this kaggle dataset](https://www.kaggle.com/datasets/tauilabdelilah/mrl-eye-dataset).
- The blurriness estimator was tested on [this kaggle dataset](https://www.kaggle.com/datasets/kwentar/blur-dataset)