README.md

---
title: VALL-E
emoji: 🐻‍❄️
colorFrom: green
colorTo: purple
sdk: gradio
sdk_version: 3.41.2
app_file: app.py
pinned: false
license: agpl-3.0
---

Recommended settings for stable speech:
* `NAR levels`: 7; less NAR levels reduces the quality of the final waveform (this may also be strictly because, when EnCodec is fed a sequence with less RVQ bin levels than what it was initialized with, it'll sound worse).
* `Temperature (AR)`: [0.85, 1.1]; It's ***really*** tough to find a one-size-fits-all value.
* `Temperature (NAR)`: [0.15, 0.85]; This is even harder to nail decent values. Too high and you'll hear artifacts from the NAR, too low and you might not have acoustic detail recreated.
* `Dynamic Temperature`: checked; Dynamic temperature seems to definitely help resolve issues with a model that is not strongly trained. Pairable with every other sampling technique.
* `Top P`: [0.85, 0.95] || 1; I feel this is cope.
* `Top K`: [768, 1024] || 0; I also feel this is cope.
* `Beam Width`: 0 || 16; beam searching helps find potential best candidates, but I'm not sure how well it helps in the realm of audio. Incompatible with mirostat.
* `Repetition Penalty`: 1.35; this and the length decay miraculously are what helps stabilize output; I have my theories.
* `Repetition Penalty Length Decay`: 0.2; this helps not severly dampen the model's output when applying rep. pen.
* `Length Penalty`: 0; this only be messed with if you're consistently having either too short output, or too long output. The AR is trained decently enough to know when to emit a STOP token.
* `Mirostat (Tau)`: [2.0, 8.0]; The "surprise value" when performing mirostat sampling, which seems to be much more favorable in comparison to typical top-k/top-p or beam search sampling. The "best" values are still unknown.
* `Mirostat (Eta)`: [0.05, 0.3]; The "learning rate" (decay value?) applied each step for mirostat sampling.

This Space:
* houses experimental models and the necessary inferencing code for my [VALL-E](https://git.ecker.tech/mrq/vall-e) implementation. I hope to gain some critical feedback with the outputs.
* utilizes a T4 with a narcoleptic 5-minute sleep timer, as I do not have another system to (easily) host this myself with a 6800XT (or two) while I'm training off my 4070Ti and 7900XTX.

The model is:
* utilizing an RetNet for faster training/inferencing with conforming dimensionality (1024 dim, 4096 ffn dim, 16 heads, 12 layers) targetting the full eight RVQ-bins (albeit the model was originally trained at two then four).
* trained on ~12.5K hour dataset composed of LibriTTS-R, LibriLight (`small`+`medium`+`duplicated`), generously donated audiobooks, and vidya voice clip rips (including some Japanese kusoge gacha clips).
* a "monolothic" approach to sharing the retention-based transformer weights between AR and NAR tasks for no immediately discernable penalties (besides retraining).
* utilizing DeepSpeed to inference using its int8 quantized inferencing (allegedly), and Vocos for better output quality.
    - I do need to add a toggle between different dtypes to gauge any perceptable quality/throughput gains/losses.
* currently still being trained, and any updates to it will be pushed back to this repo.

I am also currently training an experimental model with double the layers (24 layers instead) to gauge its performance. Depending on how well it performs, I may pivot to that too, but for now, I'm starting to doubt the time investment in training it.