README.md

---
library_name: py-feat
pipeline_tag: image-feature-extraction
license: apache-2.0
language:
- en
---


# MP_Blendshapes

## Model Description
MP_Blendshapes has been ported to pytorch from Google's [mediapipe](https://github.com/google-ai-edge/mediapipe) library using Liam Schoneveld's github [repository](https://github.com/nlml/deconstruct-mediapipe). The inputs are 146/473 of mediapipe's FaceMeshV2 high density landmark model. The 52 blendshapes are similar to [ARKit Face Blendshapes](https://arkit-face-blendshapes.com/) and loosely correspond to Facial Action Unit Coding System.

See the mediapipe [model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf) for more details.

## Model Details
- **Model Type**: MLP-Mixer (Keras)
- **Framework**: pytorch

## Model Sources
- **Repository**: [GitHub Repository](https://github.com/cosanlab/py-feat)
- **Model Card**: [Mediapipe blendshape model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf)
- **Paper**: [Attention mesh: High-fidelity face mesh prediction in real-time](https://arxiv.org/abs/2006.10962)

## Citation
If you use the mp_blendshapes model in your research or application, please cite the following paper:

Grishchenko, I., Ablavatski, A., Kartynnik, Y., Raveendran, K., & Grundmann, M. (2020). Attention mesh: High-fidelity face mesh prediction in real-time. arXiv preprint arXiv:2006.10962.

```
@article{grishchenko2020attention,
  title={Attention mesh: High-fidelity face mesh prediction in real-time},
  author={Grishchenko, Ivan and Ablavatski, Artsiom and Kartynnik, Yury and Raveendran, Karthik and Grundmann, Matthias},
  journal={arXiv preprint arXiv:2006.10962},
  year={2020}
}
```

## Example Useage

```python
import torch
import pandas as pd
from huggingface_hub import hf_hub_download
from feat.au_detectors.MP_Blendshapes.MP_Blendshapes_test import MediaPipeBlendshapesMLPMixer     
from feat.utils import MP_BLENDSHAPE_MODEL_LANDMARKS_SUBSET, MP_BLENDSHAPE_NAMES

device = 'cpu'

# Load model and weights
blendshape_detector = MediaPipeBlendshapesMLPMixer()
model_path = hf_hub_download(repo_id="py-feat/mp_blendshapes", filename="face_blendshapes.pth")
blendshape_model_file = hf_hub_download(repo_id='py-feat/resmasknet', filename="ResMaskNet_Z_resmasking_dropout1_rot30.pth")
blendshape_checkpoint = torch.load(blendshape_model_file, map_location=device)["net"]
blendshape_detector.load_state_dict(blendshape_checkpoint)
blendshape_detector.eval()
blendshape_detector.to(device)


# Test model
face_image = "path/to/your/test_image.jpg"  # Replace with your extracted face image that is [224, 224]

# Extract Landmarks
landmark_detector = torch.load('/Users/lukechang/Dropbox/py-feat/mediapipe/model/face_landmarks_detector_Nx3x256x256_onnx.pth', weights_only=False)
landmark_detector.eval()
landmark_detector.to(device)
landmark_results = landmark_detector(torch.tensor(face_image).to(device))

# Blendshape Classification
landmarks = landmark_results[0].reshape(1,478,3)[:,:,:2]
img_size = torch.tensor((face_image_width, face_image_height)).unsqueeze(0).unsqueeze(0)
landmarks = landmarks * img_size
blendshapes = blendshape_detector(landmarks)
blendshape_results = pd.Series(blendshape_results.squeeze().detach().numpy(), index=BLENDSHAPE_NAMES)
        
```