README.md

---
library_name: transformers
tags:
- speech
- text-to-speech
datasets:
- openslr/librispeech_asr
- MushanW/GLOBE
- MikhailT/hifi-tts
---

# Model Card for Model ID
Voicera is a AR text-to-speech model trained on ~1000hrs of speech data.
speech is converted to discrete tokens using "Multi-Scale Neural Audio Codec (SNAC)" model
**NB: This is not a SOTA model, and not accuarate enough for production usecase**



## Model Details

### Model Description

"Voicera" is a text-to-speech (TTS) model designed for generating speech from written text. 
It uses a GPT-2 type architecture, which helps in creating natural and expressive speech. 
The model converts audio into tokens using the "Multi-Scale Neural Audio Codec (SNAC)" model, allowing it to understand and produce speech sounds. 
Voicera aims to provide clear and understandable speech, focusing on natural pronunciation and intonation. 
It's a project to explore TTS technology and improve audio output quality. 



- **Developed by:** Lwasinam Dilli
- **Funded by :** Lwasinam Dilli
- **Model type:** GPT2-Transformer architecture
- **License:** Free and Open to use I guess :)


### Model Sources 

<!-- Provide the basic links for the model. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo :** [Demos](https://lwasinam.github.io/)



## How to Get Started with the Model

There are three models, We have the base model and two other finetuned on jenny and expresso datasets
The best of all currently is the Jenny finetune
Here are colab link to all 3 respectively
1. [Base Model](https://colab.research.google.com/drive/10nPKliRs1C3ofv2J16_HGDlmzfd-yBtj#scrollTo=r17orAuZ45Q2)
2. [Jenny-Finetune](https://colab.research.google.com/drive/1MSzGGqIhGYVCn76alsX9oBzwC4EtOQSR#scrollTo=Oz0DG-MtovBw)
3. [Expresso-Finetune](https://colab.research.google.com/drive/1wzwSOtpT1CpEMvbcjvvgEKQZoQa5bX2p#scrollTo=YrBUwCNYmmUW&uniqifier=1)


## Training Details

### Training Data
Training data consist of clean subset of Hifi, Libri-Speech, Libri-TTs and Globe datasets

### Training Procedure
During training, audio tokens are generated from snac model and concatenated with text tokens, They are all trained in an autoregressive manner
but since we're interested in just audio tokens, text token loss is reduced by 0.1.

#### Preprocessing 

Hugging Face had pretty much all the datasets I needed. I just had to filter out audio more than 10secs due to compute restraints 


#### Training Hyperparameters
 - Weight decay 0.1
 - batch_size 1 with grad_accumulation of 32
 - Scheduler : CosineAnnealingWarmRestart with minimum learning rate of 1e-7 and Num of steps for Warm Restart being 500


## Evaluation

I should probably work on this, the loss went down and the output got better :)

### Results
 Check out the demo page her -> [Demo](https://lwasinam.github.io/)

#### Summary



- **Hardware Type:** Tesla P100
- **Hours used:** 300+hrs
- **Cloud Provider:** Kaggle :)




## Citation [optional]

**BibTeX:**

```
@software{Betker_TorToiSe_text-to-speech_2022,
author = {Betker, James},
month = apr,
title = {{TorToiSe text-to-speech}},
url = {https://github.com/neonbjb/tortoise-tts},
version = {2.0},
year = {2022}
}

@software{Siuzdak_SNAC_Multi-Scale_Neural_2024,
author = {Siuzdak, Hubert},
month = feb,
title = {{SNAC: Multi-Scale Neural Audio Codec}},
url = {https://github.com/hubertsiuzdak/snac},
year = {2024}
}

``` 


## Model Card Authors [optional]

Lwasinam Dilli