license: cc-by-nc-nd-4.0

AudioLDM 2

AudioLDM 2 is a latent text-to-audio diffusion model capable of generating realistic audio samples given any text input. It is available in the 🧨 Diffusers library from v0.21.0 onwards.

Model Details

AudioLDM 2 was proposed in the paper AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining by Haohe Liu et al.

AudioLDM takes a text prompt as input and predicts the corresponding audio. It can generate text-conditional sound effects, human speech and music.

Checkpoint Details

This is the original, base version of the AudioLDM 2 model, also referred to as audioldm2-full.

There are three official AudioLDM 2 checkpoints. Two of these checkpoints are applicable to the general task of text-to-audio generation. The third checkpoint is trained exclusively on text-to-music generation. All checkpoints share the same model size for the text encoders and VAE. They differ in the size and depth of the UNet. See table below for details on the three official checkpoints:

Checkpoint	Task	UNet Model Size	Total Model Size	Training Data / h
audioldm2	Text-to-audio	350M	1.1B	1150k
audioldm2-large	Text-to-audio	750M	1.5B	1150k
audioldm2-music	Text-to-music	350M	1.1B	665k
audioldm2-gigaspeech	Text-to-speech	350M	1.1B	10k
audioldm2-ljspeech	Text-to-speech	350M	1.1B

Model Sources

Usage

First, install the required packages:

pip install --upgrade diffusers transformers accelerate

Text-to-Speech

For text-to-speech generation, the AudioLDM2Pipeline can be used to load pre-trained weights and generate text-conditional audio outputs:

import scipy
import torch
from diffusers import AudioLDM2Pipeline

repo_id = "anhnct/audioldm2_gigaspeech"
pipe = AudioLDM2Pipeline.from_pretrained(repo_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

# define the prompts
prompt = "An female actor say with angry voice"
transcript = "wish you have a good day, i hope you never forget me"
negative_prompt = "low quality"

# set the seed for generator
generator = torch.Generator("cuda").manual_seed(1)

# run the generation
audio = pipe(
    prompt,
    negative_prompt=negative_prompt,
    transcription=transcript_1,
    num_inference_steps=200,
    audio_length_in_s=8.0,
    num_waveforms_per_prompt=1,
    generator=generator,
    max_new_tokens=512
).audios

# save the best audio sample (index 0) as a .wav file
scipy.io.wavfile.write("techno_2.wav", rate=16000, data=audio[0])

Citation

BibTeX:

@article{liu2023audioldm2,
  title={"AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining"},
  author={Haohe Liu and Qiao Tian and Yi Yuan and Xubo Liu and Xinhao Mei and Qiuqiang Kong and Yuping Wang and Wenwu Wang and Yuxuan Wang and Mark D. Plumbley},
  journal={arXiv preprint arXiv:2308.05734},
  year={2023}
}