NeuroSync Open Source Audio2Face Blendshape Transformer Model

02/04/2025 Update to model.pth + utils > audio > processing > audio_processing.py

Increased accuracy (timing and overall face shows more natural movement overall, brows, squint, cheeks + mouth shapes)
More smoothness during playback.
Works better with more voices and styles of speaking.
Fixed end of generation wobble from reflect padding, changed to replicating the end frame as it was sometimes predicting extra mouth based on the reflection if having to reflect a lot for the final generation of a sentence or paragraph.

This preview of the new model is a modest increase in capability that requires both model.pth and model.py to be replaced with the new versions.

Model Overview

The NeuroSync audio-to-face blendshape transformer seq2seq model converts sequences of audio features into corresponding facial blendshape coefficients, enabling real-time character animation. It integrates seamlessly with Unreal Engine via LiveLink.

Features

Audio-to-Face Transformation: Converts raw audio features into facial blendshape coefficients.
Transformer Seq2Seq Architecture: Utilizes encoder-decoder layers to capture complex dependencies between audio and facial expressions.
Unreal Engine Integration (LiveLink): Stream facial blendshapes in real time with the NeuroSync Player.

Usage

Local API

Set up your local API using the NeuroSync Local API repository to process audio files and stream generated blendshapes.

Model Architecture

Encoder: Processes audio features with a transformer encoder using positional encodings.
Decoder: Uses cross-attention in a transformer decoder to generate blendshape coefficients.
Output: Produces 68 blendshape coefficients, 61 relate to ARKit and 7 "emotions".

Blendshape Coefficients

Included: Eye movements (e.g., EyeBlinkLeft, EyeSquintRight), jaw movements (e.g., JawOpen, JawRight), mouth movements (e.g., MouthSmileLeft, MouthPucker), brow movements (e.g., BrowInnerUp, BrowDownLeft), and cheek/nose movements (e.g., CheekPuff, NoseSneerRight).
Note: Coefficients 62–68 (related to emotional states) should be ignored or used for additive sliders since they are not streamed into LiveLink.

Community & Resources

Live Demo

Twitch: Talk to a NeuroSync prototype live on Twitch

YouTube Channel

For tutorials, updates, and more, visit our YouTube channel.

NeuroSync License

This software uses a dual-license model:

1. Free License (MIT License)

For individuals and businesses earning under $1M per year:

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

2. Commercial License (For Businesses Earning $1M+ Per Year)

Businesses or organizations with annual revenue of $1,000,000 or more must obtain a commercial license to use this software.

To acquire a commercial license, please contact us.

Compliance

By using this software, you agree to these licensing terms. If your business exceeds the revenue threshold, you must transition to a commercial license or cease using the software.

References

For any questions or further support, please feel free to contribute to the repository or raise an issue.

AnimaVR
/

NEUROSYNC_Audio_To_Face_Blendshape

You need to agree to share your contact information to access this model