metadata
title: VALL-E
emoji: 🐻❄️
colorFrom: green
colorTo: purple
sdk: gradio
sdk_version: 3.41.2
app_file: app.py
pinned: false
license: agpl-3.0
Recommended settings for stable speech:
NAR levels: 7; less NAR levels reduces the quality of the final waveform (this may also be strictly because, when EnCodec is fed a sequence with less RVQ bin levels than what it was initialized with, it'll sound worse).Temperature (AR): [0.85, 1.1]; It's really tough to find a one-size-fits-all value.Temperature (NAR): [0.15, 0.85]; This is even harder to nail decent values. Too high and you'll hear artifacts from the NAR, too low and you might not have acoustic detail recreated.Dynamic Temperature: checked; Dynamic temperature seems to definitely help resolve issues with a model that is not strongly trained. Pairable with every other sampling technique.Top P: [0.85, 0.95] || 1; I feel this is cope.Top K: [768, 1024] || 0; I also feel this is cope.Beam Width: 0 || 16; beam searching helps find potential best candidates, but I'm not sure how well it helps in the realm of audio. Incompatible with mirostat.Repetition Penalty: 1.35; this and the length decay miraculously are what helps stabilize output; I have my theories.Repetition Penalty Length Decay: 0.2; this helps not severly dampen the model's output when applying rep. pen.Length Penalty: 0; this only be messed with if you're consistently having either too short output, or too long output. The AR is trained decently enough to know when to emit a STOP token.Mirostat (Tau): [2.0, 8.0]; The "surprise value" when performing mirostat sampling, which seems to be much more favorable in comparison to typical top-k/top-p or beam search sampling. The "best" values are still unknown.Mirostat (Eta): [0.05, 0.3]; The "learning rate" (decay value?) applied each step for mirostat sampling.
This Space:
- houses experimental models and the necessary inferencing code for my VALL-E implementation. I hope to gain some critical feedback with the outputs.
- utilizes a T4 with a narcoleptic 5-minute sleep timer, as I do not have another system to (easily) host this myself with a 6800XT (or two) while I'm training off my 4070Ti and 7900XTX.
The model is:
- utilizing an RetNet for faster training/inferencing with conforming dimensionality (1024 dim, 4096 ffn dim, 16 heads, 12 layers) targetting the full eight RVQ-bins (albeit the model was originally trained at two then four).
- trained on ~12.5K hour dataset composed of LibriTTS-R, LibriLight (
small+medium+duplicated), generously donated audiobooks, and vidya voice clip rips (including some Japanese kusoge gacha clips). - a "monolothic" approach to sharing the retention-based transformer weights between AR and NAR tasks for no immediately discernable penalties (besides retraining).
- utilizing DeepSpeed to inference using its int8 quantized inferencing (allegedly), and Vocos for better output quality.
- I do need to add a toggle between different dtypes to gauge any perceptable quality/throughput gains/losses.
- currently still being trained, and any updates to it will be pushed back to this repo.
I am also currently training an experimental model with double the layers (24 layers instead) to gauge its performance. Depending on how well it performs, I may pivot to that too, but for now, I'm starting to doubt the time investment in training it.