demo_part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b6ee1ede",
   "metadata": {},
   "source": [
    "## Cross-Lingual Voice Clone Demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7f043ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "import se_extractor\n",
    "from api import ToneColorConverter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15116b59",
   "metadata": {},
   "source": [
    "### Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aacad912",
   "metadata": {},
   "outputs": [],
   "source": [
    "ckpt_converter = 'checkpoints/converter'\n",
    "device = 'cuda:0'\n",
    "output_dir = 'outputs'\n",
    "\n",
    "tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)\n",
    "tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')\n",
    "\n",
    "os.makedirs(output_dir, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db80fcf",
   "metadata": {},
   "source": [
    "In this demo, we will use OpenAI TTS as the base speaker to produce multi-lingual speech audio. The users can flexibly change the base speaker according to their own needs. Please create a file named `.env` and place OpenAI key as `OPENAI_API_KEY=xxx`. We have also provided a Chinese base speaker model (see `demo_part1.ipynb`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b245ca3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# Please create a file named .env and place your\n",
    "# OpenAI key as OPENAI_API_KEY=xxx\n",
    "load_dotenv() \n",
    "\n",
    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))\n",
    "\n",
    "response = client.audio.speech.create(\n",
    "    model=\"tts-1\",\n",
    "    voice=\"nova\",\n",
    "    input=\"This audio will be used to extract the base speaker tone color embedding. \" + \\\n",
    "        \"Typically a very short audio should be sufficient, but increasing the audio \" + \\\n",
    "        \"length will also improve the output audio quality.\"\n",
    ")\n",
    "\n",
    "response.stream_to_file(f\"{output_dir}/openai_source_output.mp3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f67740c",
   "metadata": {},
   "source": [
    "### Obtain Tone Color Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8add279",
   "metadata": {},
   "source": [
    "The `source_se` is the tone color embedding of the base speaker. \n",
    "It is an average for multiple sentences with multiple emotions\n",
    "of the base speaker. We directly provide the result here but\n",
    "the readers feel free to extract `source_se` by themselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63ff6273",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_speaker = f\"{output_dir}/openai_source_output.mp3\"\n",
    "source_se, audio_name = se_extractor.get_se(base_speaker, tone_color_converter, vad=True)\n",
    "\n",
    "reference_speaker = 'resources/example_reference.mp3'\n",
    "target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter, vad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a40284aa",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73dc1259",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the base speaker tts\n",
    "text = [\n",
    "    \"MyShell is a decentralized and comprehensive platform for discovering, creating, and staking AI-native apps.\",\n",
    "    \"MyShell es una plataforma descentralizada y completa para descubrir, crear y apostar por aplicaciones nativas de IA.\",\n",
    "    \"MyShell est une plateforme décentralisée et complète pour découvrir, créer et miser sur des applications natives d'IA.\",\n",
    "    \"MyShell ist eine dezentralisierte und umfassende Plattform zum Entdecken, Erstellen und Staken von KI-nativen Apps.\",\n",
    "    \"MyShell è una piattaforma decentralizzata e completa per scoprire, creare e scommettere su app native di intelligenza artificiale.\",\n",
    "    \"MyShellは、AIネイティブアプリの発見、作成、およびステーキングのための分散型かつ包括的なプラットフォームです。\",\n",
    "    \"MyShell — это децентрализованная и всеобъемлющая платформа для обнаружения, создания и стейкинга AI-ориентированных приложений.\",\n",
    "    \"MyShell هي منصة لامركزية وشاملة لاكتشاف وإنشاء ورهان تطبيقات الذكاء الاصطناعي الأصلية.\",\n",
    "    \"MyShell是一个去中心化且全面的平台，用于发现、创建和投资AI原生应用程序。\",\n",
    "    \"MyShell एक विकेंद्रीकृत और व्यापक मंच है, जो AI-मूल ऐप्स की खोज, सृजन और स्टेकिंग के लिए है।\",\n",
    "    \"MyShell é uma plataforma descentralizada e abrangente para descobrir, criar e apostar em aplicativos nativos de IA.\"\n",
    "]\n",
    "src_path = f'{output_dir}/tmp.wav'\n",
    "\n",
    "for i, t in enumerate(text):\n",
    "\n",
    "    response = client.audio.speech.create(\n",
    "        model=\"tts-1\",\n",
    "        voice=\"alloy\",\n",
    "        input=t,\n",
    "    )\n",
    "\n",
    "    response.stream_to_file(src_path)\n",
    "\n",
    "    save_path = f'{output_dir}/output_crosslingual_{i}.wav'\n",
    "\n",
    "    # Run the tone color converter\n",
    "    encode_message = \"@MyShell\"\n",
    "    tone_color_converter.convert(\n",
    "        audio_src_path=src_path, \n",
    "        src_se=source_se, \n",
    "        tgt_se=target_se, \n",
    "        output_path=save_path,\n",
    "        message=encode_message)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9d70c38e1c0b038dbdffdaa4f8bfa1f6767c43760905c87a9fbe7800d18c6c35"
  },
  "kernelspec": {
   "display_name": "Python 3.9.18 ('openvoice')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}