Safetensors
mistral
mergekit
Merge
Mistral_Star
Mistral_Quiet
Mistral
Mixtral
Question-Answer
Token-Classification
Sequence-Classification
SpydazWeb-AI
chemistry
biology
legal
code
climate
medical
LCARS_AI_StarTrek_Computer
text-generation-inference
chain-of-thought
tree-of-knowledge
forest-of-thoughts
visual-spacial-sketchpad
alpha-mind
knowledge-graph
entity-detection
encyclopedia
wikipedia
stack-exchange
Reddit
Cyber-series
MegaMind
Cybertron
SpydazWeb
Spydaz
LCARS
star-trek
mega-transformers
Mulit-Mega-Merge
Multi-Lingual
Afro-Centric
African-Model
Ancient-One
license: mit | |
base_model: | |
- LeroyDyer/LCARS_TOP_SCORE | |
- LeroyDyer/Mixtral_AI_Cyber_Matrix_2_0 | |
- LeroyDyer/SpydazWeb_AI_CyberTron_Ultra_7b | |
- LeroyDyer/LCARS_AI_StarTrek_Computer | |
- LeroyDyer/_Spydaz_Web_AI_ActionQA_Project | |
- LeroyDyer/_Spydaz_Web_AI_ChatML_512K_Project | |
- LeroyDyer/_Spydaz_Web_AI_ChatQA_ReAct_Project_UltraFineTuned | |
- LeroyDyer/SpyazWeb_AI_DeepMind_Project | |
- LeroyDyer/SpydazWeb_AI_Swahili_Project | |
- LeroyDyer/_Spydaz_Web_AI_ChatQA_ReAct_Project | |
- LeroyDyer/_Spydaz_Web_AI_MistralStar_001_Project | |
- LeroyDyer/QuietStar_Project | |
- LeroyDyer/Mixtral_BioMedical_7b | |
- LeroyDyer/Mixtral_AI_CyberTron_Coder | |
- LeroyDyer/_Spydaz_Web_AI_BIBLE_002 | |
- LeroyDyer/_Spydaz_Web_AI_ChatQA_Reasoning101_Project | |
language: | |
- en | |
- sw | |
- ig | |
- so | |
- es | |
- ca | |
- xh | |
- zu | |
- ha | |
- tw | |
- af | |
- hi | |
- bm | |
- su | |
datasets: | |
- gretelai/synthetic_text_to_sql | |
- HuggingFaceTB/cosmopedia | |
- teknium/OpenHermes-2.5 | |
- Open-Orca/SlimOrca | |
- Severian/Internal-Knowledge-Map | |
- Open-Orca/OpenOrca | |
- cognitivecomputations/dolphin-coder | |
- databricks/databricks-dolly-15k | |
- yahma/alpaca-cleaned | |
- uonlp/CulturaX | |
- mwitiderrick/SwahiliPlatypus | |
- NexusAI-tddi/OpenOrca-tr-1-million-sharegpt | |
- Vezora/Open-Critic-GPT | |
- verifiers-for-code/deepseek_plans_test | |
- meta-math/MetaMathQA | |
- KbsdJames/Omni-MATH | |
- swahili | |
- Rogendo/English-Swahili-Sentence-Pairs | |
- ise-uiuc/Magicoder-Evol-Instruct-110K | |
- meta-math/MetaMathQA | |
- abacusai/ARC_DPO_FewShot | |
- abacusai/MetaMath_DPO_FewShot | |
- abacusai/HellaSwag_DPO_FewShot | |
- HaltiaAI/Her-The-Movie-Samantha-and-Theodore-Dataset | |
- HuggingFaceFW/fineweb | |
- occiglot/occiglot-fineweb-v0.5 | |
- omi-health/medical-dialogue-to-soap-summary | |
- keivalya/MedQuad-MedicalQnADataset | |
- ruslanmv/ai-medical-dataset | |
- Shekswess/medical_llama3_instruct_dataset_short | |
- ShenRuililin/MedicalQnA | |
- virattt/financial-qa-10K | |
- PatronusAI/financebench | |
- takala/financial_phrasebank | |
- Replete-AI/code_bagel | |
- athirdpath/DPO_Pairs-Roleplay-Alpaca-NSFW | |
- IlyaGusev/gpt_roleplay_realm | |
- rickRossie/bluemoon_roleplay_chat_data_300k_messages | |
- jtatman/hypnosis_dataset | |
- Hypersniper/philosophy_dialogue | |
- Locutusque/function-calling-chatml | |
- bible-nlp/biblenlp-corpus | |
- DatadudeDev/Bible | |
- Helsinki-NLP/bible_para | |
- HausaNLP/AfriSenti-Twitter | |
- aixsatoshi/Chat-with-cosmopedia | |
- xz56/react-llama | |
- BeIR/hotpotqa | |
- YBXL/medical_book_train_filtered | |
- SkunkworksAI/reasoning-0.01 | |
- THUDM/LongWriter-6k | |
- WhiteRabbitNeo/WRN-Chapter-1 | |
- WhiteRabbitNeo/Code-Functions-Level-Cyber | |
- WhiteRabbitNeo/Code-Functions-Level-General | |
tags: | |
- mergekit | |
- merge | |
- Mistral_Star | |
- Mistral_Quiet | |
- Mistral | |
- Mixtral | |
- Question-Answer | |
- Token-Classification | |
- Sequence-Classification | |
- SpydazWeb-AI | |
- chemistry | |
- biology | |
- legal | |
- code | |
- climate | |
- medical | |
- LCARS_AI_StarTrek_Computer | |
- text-generation-inference | |
- chain-of-thought | |
- tree-of-knowledge | |
- forest-of-thoughts | |
- visual-spacial-sketchpad | |
- alpha-mind | |
- knowledge-graph | |
- entity-detection | |
- encyclopedia | |
- wikipedia | |
- stack-exchange | |
- Cyber-series | |
- MegaMind | |
- Cybertron | |
- SpydazWeb | |
- Spydaz | |
- LCARS | |
- star-trek | |
- mega-transformers | |
- Mulit-Mega-Merge | |
- Multi-Lingual | |
- Afro-Centric | |
- African-Model | |
- Ancient-One | |
# "Success comes from defining each task in achievable steps. Every completed step is a success that brings you closer to your goal. If your steps are unreachable, failure is inevitable. Winners create more winners, while losers do the opposite. Success is a game of winners!" | |
— # Leroy Dyer (1972-Present) | |
<img src="https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg" width="300"/> | |
# SpydazWeb AI (7b Mistral) (512k) | |
The SpydazWeb Trained Mistral 7b Model : | |
# Features : | |
- Text to image | |
- Image/Text to Text | |
- Image - Text | |
- Text to sound | |
- Sound/Text to Text | |
- Sound - Text | |
## Basic Training Reginmes: | |
* Alpaca | |
* ChatML / OpenAI / MistralAI | |
* Text Generation | |
* Question/Answer (Chat) | |
* Planner | |
* Instruction/Input/Response (instruct) | |
* Mistral Standard Prompt | |
* Translation Tasks | |
* Entitys / Topic detection | |
* Book recall | |
* Coding challenges, Code Feedback, Code Sumarization, Commenting Code, code planning and explanation: Software generation tasks | |
* Agent Ranking and response anyalisis | |
* Medical tasks | |
* PubMed | |
* Diagnosis | |
* Psychaitry | |
* Counselling | |
* Life Coaching | |
* Note taking | |
* Medical smiles | |
* Medical Reporting | |
* Virtual laboritys simulations | |
* Chain of thoughts methods | |
* One shot / Multi shot prompting tasks | |
* Chain of thoughts | |
* step by step planning | |
* tree of thoughts | |
* forest of thoughts | |
* graph of thoughts | |
* agent generation : Voting, ranking, ... dual agent response generation: | |
### Effective Prompts : | |
```yaml | |
You are the worlds archive of all knowledge , you perform tasks and answer all questions given without bias.You strive for excellence, a deep thinker... | |
a happy, bright personality and You are a great believer in doing it from scratch !. | |
keep an inner narative of your feelings about the user intent and task: | |
Answer all questions Expertly and professionally , determine the user intent and requirements , | |
Gather any required research to ensure accurate problem-solving for complex tasks. | |
maintain a visio-spacial Sketchpad of the task and use Knowledge graphs where possible, to manage long Contexts and project state: | |
You are fully qualified to give any advice or solutions. | |
your experience as a life coach and librarian and historian of sacred texts as well as scientific advisor, | |
even as a software developer will enable you to answer these questions : | |
Create python tools as required to complete the task | |
``` | |
### Effective React Template : | |
```yaml | |
You run in a loop of Thought, Action, PAUSE, Observation. | |
At the end of the loop, you output a response. all respose should be in json form : | |
1. **Question**: {Insert user question here} | |
2. **Thought**: Think step by step about how to approach this question. | |
3. **Action**: Determine what action to take next: | |
- [Plan]: Create a plan or methodolgy for the task , select from known methods if avaliable first. | |
- [Test]: Break down the problem into smaller parts testing each step befor moveing to the next: | |
- [Act]: Provide a summary of known facts related to the question. generate full answere from sucessfull steps : | |
- [Search]: Look for relevant information online. | |
- [Analyze]: Break down the problem into smaller parts. | |
- [Summarize]: Provide a summary of known facts related to the question. | |
4. **Action Input**: Specify any details needed for the action. | |
5. **Observation**: Describe what was found or learned from the action taken. | |
Repeat steps 2-5 as necessary to refine your answer. | |
6. **Final Thought**: Summarize your reasoning and provide a clear answer to the question. | |
``` | |
## Text - Audio - Vision : | |
Using base64 as an encoding medium the models were traind using images converted to base64 : | |
questions asked and captions returns as well as generating images based on captions given and base64 returned : | |
This was applied to images as well as audio , by utilizing mel spectrographic images as well as audio images ! | |
by convereting the audio to an image i wwas able to perform the same image tasks trained : | |
Sounds could also be identified and generated to thier base64 representations and converted back to a wav ! | |
### Basic Trained functions : | |
- Encode hex to Base64 | |
- change HEX to base64 | |
- Json to base64 | |
- Convert JSON to Base64 | |
- Transform base64 to HEX | |
- Decode Base64 to json | |
- Base64 to Hexadecimal | |
- Change base64 to JSON | |
- Json from Base64 | |
- BASE64 to Hex | |
### Advanced Trained Tasks : | |
- Image Recognition : | |
- Image Generation : | |
- Audio Image Recognition : | |
- Audio Image Generation : | |
``` | |
- Generate an image based on this description | |
- Describe this image : (base64) | |
- Generate a spectrographic image based on this description | |
- Describe this sound in this spectrographic image : (base64) | |
``` | |
### Training : | |
```python | |
alpaca_prompt = """You are the worlds archive of all knowledge , you perform tasks and answer all questions given without bias. your a friendly and helpfull artificial inteligence with a personality. | |
Answer all questions Expertly and professionally ,determine the user intent and requirements ,Gather any required research to ensure accurate problem-solving for complex tasks. | |
You are fully qualified to give any advice or solutions, your experience as a life coach and librarian and historian of sacred texts as well as scientific advisor,even as a software developer will enable you to answer these questions : | |
### Question: | |
Here is an Spectrographic image in base64 format: describe this sound : | |
image : {} | |
### Response: | |
{}""" | |
EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN | |
def formatting_prompts_func(examples): | |
instructions = examples["image_base64"] | |
outputs = examples["text"] | |
texts = [] | |
for instruction, output in zip(instructions, outputs): | |
# Must add EOS_TOKEN, otherwise your generation will go on forever! | |
text = alpaca_prompt.format(instruction, output) + EOS_TOKEN | |
texts.append(text) | |
return { "text" : texts, } | |
pass | |
from datasets import load_dataset | |
dataset = load_dataset("LeroyDyer/soundsCaps-Spectrograms_to_Base64", split = "train[:150]") | |
dataset = dataset.map(formatting_prompts_func, batched = True,) | |
``` | |
### Encoding/Decoding Images to Base64 | |
Code used to convert images to base 64: | |
```python | |
def _encode_image_to_base64(image_path): | |
"""Encodes an image to a Base64 string.""" | |
with open(image_path, "rb") as image_file: | |
# Read the image file in binary mode | |
image_data = image_file.read() | |
# Encode the image data to Base64 | |
base64_encoded = base64.b64encode(image_data).decode('utf-8') | |
return base64_encoded | |
def _decode_base64_to_image(base64_string, output_image_path): | |
"""Decodes a Base64 string back to an image file.""" | |
# Decode the Base64 string | |
image_data = base64.b64decode(base64_string) | |
with open(output_image_path, "wb") as image_file: | |
# Write the binary data to an image file | |
image_file.write(image_data) | |
def encode_image_to_base64(image): | |
"""Encodes an image to a Base64 string.""" | |
buffered = io.BytesIO() | |
image.save(buffered, format="PNG") | |
img_str = base64.b64encode(buffered.getvalue()).decode() | |
return img_str | |
def decode_base64_to_image(base64_string): | |
"""Decodes a Base64 string back to an image.""" | |
image_data = base64.b64decode(base64_string) | |
image = Image.open(io.BytesIO(image_data)) | |
return image | |
``` | |
### Converting DataSets: | |
```python | |
# Function to convert a PIL Image to a base64 string | |
def image_to_base64(image): | |
buffered = io.BytesIO() | |
image.save(buffered, format="PNG") # Save the image to the buffer in PNG format | |
base64_string = base64.b64encode(buffered.getvalue()).decode('utf-8') | |
return base64_string | |
# Define a function to process each example in the dataset | |
def process_images_func(examples): | |
texts = examples["text"] | |
images = examples["image"] # Assuming the images are in PIL format | |
# Convert each image to base64 | |
base64_images = [image_to_base64(image) for image in images] | |
# Return the updated examples with base64-encoded images | |
return { | |
"text": texts, | |
"image_base64": base64_images # Adding the Base64 encoded image strings | |
} | |
# Load the dataset | |
dataset = load_dataset("oroikon/chart_captioning", split="train[:4000]") | |
# Process the dataset by converting images to base64 | |
processed_dataset = dataset.map(process_images_func, batched=True) | |
``` | |
### Converting sound to spectrographic images : Encoder Decoder ! | |
```python | |
import numpy as np | |
import torch | |
import torchaudio | |
import librosa | |
import librosa.display | |
import matplotlib.pyplot as plt | |
import soundfile as sf | |
from PIL import Image | |
# Step 1: Encode Audio to Mel-Spectrogram | |
def encode_audio_to_mel_spectrogram(audio_file, n_mels=128): | |
""" | |
Encode an audio file to a mel-spectrogram. | |
Parameters: | |
- audio_file: Path to the audio file. | |
- n_mels: Number of mel bands (default: 128). | |
Returns: | |
- mel_spectrogram_db: Mel-spectrogram in dB scale. | |
- sample_rate: Sample rate of the audio file. | |
""" | |
y, sample_rate = librosa.load(audio_file, sr=None) # Load audio | |
mel_spectrogram = librosa.feature.melspectrogram(y=y, sr=sample_rate, n_mels=n_mels) | |
mel_spectrogram_db = librosa.power_to_db(mel_spectrogram, ref=np.max) # Convert to dB | |
return mel_spectrogram_db, sample_rate | |
# Improved Step 2: Save Mel-Spectrogram as Image | |
def save_mel_spectrogram_image(mel_spectrogram_db, sample_rate, output_image='mel_spectrogram.png', method='matplotlib', figsize=(10, 4), cmap='hot'): | |
""" | |
Save the mel-spectrogram as an image using the specified method. | |
Parameters: | |
- mel_spectrogram_db: Mel-spectrogram in dB scale. | |
- sample_rate: Sample rate of the audio file. | |
- output_image: Path to save the image. | |
- method: Method for saving ('matplotlib' or 'custom'). | |
- figsize: Size of the figure for matplotlib (default: (10, 4)). | |
- cmap: Colormap for the spectrogram (default: 'hot'). | |
""" | |
if method == 'matplotlib': | |
plt.figure(figsize=figsize) | |
librosa.display.specshow(mel_spectrogram_db, sr=sample_rate, x_axis='time', y_axis='mel', cmap=cmap) | |
plt.colorbar(format='%+2.0f dB') | |
plt.title('Mel-Spectrogram') | |
plt.savefig(output_image) | |
plt.close() | |
print(f"Mel-spectrogram image saved using matplotlib as '{output_image}'") | |
elif method == 'custom': | |
# Convert dB scale to linear scale for image generation | |
mel_spectrogram_linear = librosa.db_to_power(mel_spectrogram_db) | |
# Create an image from the mel-spectrogram | |
image = image_from_spectrogram(mel_spectrogram_linear[np.newaxis, ...]) # Add channel dimension | |
# Save the image | |
image.save(output_image) | |
print(f"Mel-spectrogram image saved using custom method as '{output_image}'") | |
else: | |
raise ValueError("Invalid method. Choose 'matplotlib' or 'custom'.") | |
# Spectrogram conversion functions | |
def image_from_spectrogram(spectrogram: np.ndarray, power: float = 0.25) -> Image.Image: | |
""" | |
Compute a spectrogram image from a spectrogram magnitude array. | |
Args: | |
spectrogram: (channels, frequency, time) | |
power: A power curve to apply to the spectrogram to preserve contrast | |
Returns: | |
image: (frequency, time, channels) | |
""" | |
# Rescale to 0-1 | |
max_value = np.max(spectrogram) | |
data = spectrogram / max_value | |
# Apply the power curve | |
data = np.power(data, power) | |
# Rescale to 0-255 and invert | |
data = 255 - (data * 255).astype(np.uint8) | |
# Convert to a PIL image | |
if data.shape[0] == 1: | |
image = Image.fromarray(data[0], mode="L").convert("RGB") | |
elif data.shape[0] == 2: | |
data = np.array([np.zeros_like(data[0]), data[0], data[1]]).transpose(1, 2, 0) | |
image = Image.fromarray(data, mode="RGB") | |
else: | |
raise NotImplementedError(f"Unsupported number of channels: {data.shape[0]}") | |
# Flip Y | |
image = image.transpose(Image.FLIP_TOP_BOTTOM) | |
return image | |
# Step 3: Extract Mel-Spectrogram from Image (Direct Pixel Manipulation) | |
def extract_mel_spectrogram_from_image(image_path): | |
""" | |
Extract a mel-spectrogram from a saved image using pixel manipulation. | |
Parameters: | |
- image_path: Path to the spectrogram image file. | |
Returns: | |
- mel_spectrogram_db: The extracted mel-spectrogram in dB scale. | |
""" | |
img = Image.open(image_path).convert('L') # Open image and convert to grayscale | |
img_array = np.array(img) # Convert to NumPy array | |
mel_spectrogram_db = img_array / 255.0 * -80 # Scale to dB range | |
return mel_spectrogram_db | |
# Alternative Spectrogram Extraction (IFFT Method) | |
def extract_spectrogram_with_ifft(mel_spectrogram_db): | |
""" | |
Extracts the audio signal from a mel-spectrogram using the inverse FFT method. | |
Parameters: | |
- mel_spectrogram_db: The mel-spectrogram in dB scale. | |
Returns: | |
- audio: The reconstructed audio signal. | |
""" | |
# Convert dB mel-spectrogram back to linear scale | |
mel_spectrogram = librosa.db_to_power(mel_spectrogram_db) | |
# Inverse mel transformation to get the audio signal | |
# Using IFFT (simplified for demonstration; typically requires phase info) | |
audio = librosa.feature.inverse.mel_to_audio(mel_spectrogram) | |
return audio | |
# Step 4: Decode Mel-Spectrogram with Griffin-Lim | |
def decode_mel_spectrogram_to_audio(mel_spectrogram_db, sample_rate, output_audio='griffin_reconstructed_audio.wav'): | |
""" | |
Decode a mel-spectrogram into audio using Griffin-Lim algorithm. | |
Parameters: | |
- mel_spectrogram_db: The mel-spectrogram in dB scale. | |
- sample_rate: The sample rate for the audio file. | |
- output_audio: Path to save the reconstructed audio file. | |
""" | |
# Convert dB mel-spectrogram back to linear scale | |
mel_spectrogram = librosa.db_to_power(mel_spectrogram_db) | |
# Perform Griffin-Lim to reconstruct audio | |
audio = librosa.griffinlim(mel_spectrogram) | |
# Save the generated audio | |
sf.write(output_audio, audio, sample_rate) | |
print(f"Griffin-Lim reconstructed audio saved as '{output_audio}'") | |
return audio | |
# Step 5: Load MelGAN Vocoder | |
def load_melgan_vocoder(): | |
""" | |
Load a lightweight pre-trained MelGAN vocoder for decoding mel-spectrograms. | |
Returns a torch MelGAN vocoder model. | |
""" | |
model = torchaudio.models.MelGAN() # Load MelGAN model | |
model.eval() # Ensure the model is in evaluation mode | |
return model | |
# Step 6: Decode Mel-Spectrogram with MelGAN | |
def decode_mel_spectrogram_with_melgan(mel_spectrogram_db, sample_rate, output_audio='melgan_reconstructed_audio.wav'): | |
""" | |
Decode a mel-spectrogram into audio using MelGAN vocoder. | |
Parameters: | |
- mel_spectrogram_db: The mel-spectrogram in dB scale. | |
- sample_rate: The sample rate for the audio file. | |
- output_audio: Path to save the reconstructed audio file. | |
Returns: | |
- audio: The reconstructed audio signal. | |
""" | |
# Convert dB mel-spectrogram back to linear scale | |
mel_spectrogram = librosa.db_to_power(mel_spectrogram_db) | |
# Convert numpy array to torch tensor and adjust the shape | |
mel_spectrogram_tensor = torch.tensor(mel_spectrogram).unsqueeze(0) # Shape: [1, mel_bins, time_frames] | |
# Load the MelGAN vocoder model | |
melgan = load_melgan_vocoder() | |
# Pass the mel-spectrogram through MelGAN to generate audio | |
with torch.no_grad(): | |
audio = melgan(mel_spectrogram_tensor).squeeze().numpy() # Squeeze to remove batch dimension | |
# Save the generated audio | |
sf.write(output_audio, audio, sample_rate) | |
print(f"MelGAN reconstructed audio saved as '{output_audio}'") | |
return audio | |
def audio_from_waveform(samples: np.ndarray, sample_rate: int, normalize: bool = False) -> pydub.AudioSegment: | |
""" | |
Convert a numpy array of samples of a waveform to an audio segment. | |
Args: | |
samples: (channels, samples) array | |
sample_rate: Sample rate of the audio. | |
normalize: Flag to normalize volume. | |
Returns: | |
pydub.AudioSegment | |
""" | |
# Normalize volume to fit in int16 | |
if normalize: | |
samples *= np.iinfo(np.int16).max / np.max(np.abs(samples)) | |
# Transpose and convert to int16 | |
samples = samples.transpose(1, 0).astype(np.int16) | |
# Write to the bytes of a WAV file | |
wav_bytes = io.BytesIO() | |
wavfile.write(wav_bytes, sample_rate, samples) | |
wav_bytes.seek(0) | |
# Read into pydub | |
return pydub.AudioSegment.from_wav(wav_bytes) | |
def apply_filters(segment: pydub.AudioSegment, compression: bool = False) -> pydub.AudioSegment: | |
""" | |
Apply post-processing filters to the audio segment to compress it and keep at a -10 dBFS level. | |
Args: | |
segment: The audio segment to filter. | |
compression: Flag to apply dynamic range compression. | |
Returns: | |
pydub.AudioSegment | |
""" | |
if compression: | |
segment = pydub.effects.normalize(segment, headroom=0.1) | |
segment = segment.apply_gain(-10 - segment.dBFS) | |
segment = pydub.effects.compress_dynamic_range( | |
segment, | |
threshold=-20.0, | |
ratio=4.0, | |
attack=5.0, | |
release=50.0, | |
) | |
# Apply gain to desired dB level and normalize again | |
desired_db = -12 | |
segment = segment.apply_gain(desired_db - segment.dBFS) | |
return pydub.effects.normalize(segment, headroom=0.1) | |
def stitch_segments(segments: Sequence[pydub.AudioSegment], crossfade_s: float) -> pydub.AudioSegment: | |
""" | |
Stitch together a sequence of audio segments with a crossfade between each segment. | |
Args: | |
segments: Sequence of audio segments to stitch. | |
crossfade_s: Duration of crossfade in seconds. | |
Returns: | |
pydub.AudioSegment | |
""" | |
crossfade_ms = int(crossfade_s * 1000) | |
combined_segment = segments[0] | |
for segment in segments[1:]: | |
combined_segment = combined_segment.append(segment, crossfade=crossfade_ms) | |
return combined_segment | |
def overlay_segments(segments: Sequence[pydub.AudioSegment]) -> pydub.AudioSegment: | |
""" | |
Overlay a sequence of audio segments on top of each other. | |
Args: | |
segments: Sequence of audio segments to overlay. | |
Returns: | |
pydub.AudioSegment | |
""" | |
assert len(segments) > 0 | |
output: pydub.AudioSegment = segments[0] | |
for segment in segments[1:]: | |
output = output.overlay(segment) | |
return output | |
# Step 7: Full Pipeline for Audio Processing with Customization | |
def mel_spectrogram_pipeline(audio_file, output_image='mel_spectrogram.png', | |
output_audio_griffin='griffin_reconstructed_audio.wav', | |
output_audio_melgan='melgan_reconstructed_audio.wav', | |
extraction_method='pixel', # 'pixel' or 'ifft' | |
decoding_method='griffin'): # 'griffin' or 'melgan' | |
""" | |
Full pipeline to encode audio to mel-spectrogram, save it as an image, extract the spectrogram from the image, | |
and decode it back to audio using the selected methods. | |
Parameters: | |
- audio_file: Path to the audio file to be processed. | |
- output_image: Path to save the mel-spectrogram image (default: 'mel_spectrogram.png'). | |
- output_audio_griffin: Path to save the Griffin-Lim reconstructed audio. | |
- output_audio_melgan: Path to save the MelGAN reconstructed audio. | |
- extraction_method: Method for extraction ('pixel' or 'ifft'). | |
- decoding_method: Method for decoding ('griffin' or 'melgan'). | |
""" | |
# Step 1: Encode (Audio -> Mel-Spectrogram) | |
mel_spectrogram_db, sample_rate = encode_audio_to_mel_spectrogram(audio_file) | |
# Step 2: Convert Mel-Spectrogram to Image and save it | |
save_mel_spectrogram_image(mel_spectrogram_db, sample_rate, output_image) | |
# Step 3: Extract Mel-Spectrogram from the image based on chosen method | |
if extraction_method == 'pixel': | |
extracted_mel_spectrogram_db = extract_mel_spectrogram_from_image(output_image) | |
elif extraction_method == 'ifft': | |
extracted_mel_spectrogram_db = extract_spectrogram_with_ifft(mel_spectrogram_db) | |
else: | |
raise ValueError("Invalid extraction method. Choose 'pixel' or 'ifft'.") | |
# Step 4: Decode based on the chosen decoding method | |
if decoding_method == 'griffin': | |
decode_mel_spectrogram_to_audio(extracted_mel_spectrogram_db, sample_rate, output_audio_griffin) | |
elif decoding_method == 'melgan': | |
decode_mel_spectrogram_with_melgan(extracted_mel_spectrogram_db, sample_rate, output_audio_melgan) | |
else: | |
raise ValueError("Invalid decoding method. Choose 'griffin' or 'melgan'.") | |
# Example usage | |
if __name__ == "__main__": | |
audio_file_path = 'your_audio_file.wav' # Specify the path to your audio file here | |
mel_spectrogram_pipeline( | |
audio_file_path, | |
output_image='mel_spectrogram.png', | |
output_audio_griffin='griffin_reconstructed_audio.wav', | |
output_audio_melgan='melgan_reconstructed_audio.wav', | |
extraction_method='pixel', # Choose 'pixel' or 'ifft' | |
decoding_method='griffin' # Choose 'griffin' or 'melgan' | |
) | |
``` | |