--- license: apache-2.0 base_model: facebook/wav2vec2-base tags: - generated_from_trainer metrics: - wer model-index: - name: wav2vec2-jv-base-openslr results: [] datasets: - openslr/openslr language: - jv pipeline_tag: automatic-speech-recognition --- # wav2vec2-jv-base-openslr This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on the [OpenSLR41](https://openslr.org/41/) datasets. It achieves the following results on the evaluation set: - Loss: 0.2843 - Wer: 0.1502 ## Model description The model is a fine-tuned version of wav2vec2, specifically adapted using the OpenSLR 41 dataset, which is focused on the Javanese language domain. This adaptation enables the model to effectively recognize and process spoken Javanese, leveraging the robust capabilities of the wav2vec2 architecture combined with domain-specific training data. ## Intended uses & limitations This model is intended for transcribing spoken Javanese language from audio recordings. It achieves a Word Error Rate (WER) of 15%, indicating that while the model performs reasonably well, it still produces significant transcription errors. Users should be aware that the accuracy may vary, particularly in cases with challenging audio conditions or less common dialects. Additionally, this model requires input audio at a sample rate of 16kHz, which may limit its applicability for recordings at different sample rates or lower quality audio files. ## Training and evaluation data The model use OpenSLR41 datasets, and split into 2 section (training and testing), then the model is trained using 1xA100 GPU with a training duration of 4-5 hours. ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0001 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 1000 - num_epochs: 65 - mixed_precision_training: Native AMP ### Log Data | Training results | Training Loss | Epoch | Step | Validation Loss | Wer | |:-------------:|:-------:|:-----:|:---------------:|:------:| | 0.5361 | 2.8329 | 2000 | 0.4626 | 0.4238 | | 0.332 | 5.6657 | 4000 | 0.3857 | 0.3749 | | 0.242 | 8.4986 | 6000 | 0.3456 | 0.3060 | | 0.1893 | 11.3314 | 8000 | 0.3250 | 0.2846 | | 0.1566 | 14.1643 | 10000 | 0.3260 | 0.2640 | | 0.1433 | 16.9972 | 12000 | 0.2891 | 0.2516 | | 0.124 | 19.8300 | 14000 | 0.3172 | 0.2433 | | 0.1103 | 22.6629 | 16000 | 0.3099 | 0.2453 | | 0.1015 | 25.4958 | 18000 | 0.3087 | 0.2295 | | 0.088 | 28.3286 | 20000 | 0.3250 | 0.2054 | | 0.0831 | 31.1615 | 22000 | 0.3127 | 0.2143 | | 0.0748 | 33.9943 | 24000 | 0.2973 | 0.1923 | | 0.0696 | 36.8272 | 26000 | 0.3103 | 0.2026 | | 0.0622 | 39.6601 | 28000 | 0.3292 | 0.2068 | | 0.0564 | 42.4929 | 30000 | 0.2965 | 0.1916 | | 0.0507 | 45.3258 | 32000 | 0.3061 | 0.1819 | | 0.0475 | 48.1586 | 34000 | 0.2784 | 0.1881 | | 0.0448 | 50.9915 | 36000 | 0.2872 | 0.1764 | | 0.0413 | 53.8244 | 38000 | 0.2854 | 0.1716 | | 0.0357 | 56.6572 | 40000 | 0.2862 | 0.1723 | | 0.0328 | 59.4901 | 42000 | 0.2887 | 0.1654 | | 0.0324 | 62.3229 | 44000 | 0.2843 | 0.1502 | ### How to run (Gradio Web) ```python import torch import torchaudio import gradio as gr import numpy as np from transformers import pipeline, AutoProcessor, AutoModelForCTC device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # Load the model and processor MODEL_NAME = "" processor = AutoProcessor.from_pretrained(MODEL_NAME) model = AutoModelForCTC.from_pretrained(MODEL_NAME) # Move model to GPU model.to(device) # Create the pipeline with the model and processor transcriber = pipeline("automatic-speech-recognition", model=model, tokenizer=processor.tokenizer, feature_extractor=processor.feature_extractor, device=device) def transcribe(audio): sr, y = audio y = y.astype(np.float32) y /= np.max(np.abs(y)) return transcriber({"sampling_rate": sr, "raw": y})["text"] demo = gr.Interface( transcribe, gr.Audio(sources=["upload"]), "text", ) demo.launch(share=True) ``` ### How to run ```python import torch import torchaudio import gradio as gr import numpy as np from transformers import pipeline, AutoProcessor, AutoModelForCTC device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # Load the model and processor MODEL_NAME = "" processor = AutoProcessor.from_pretrained(MODEL_NAME) model = AutoModelForCTC.from_pretrained(MODEL_NAME) # Move model to GPU model.to(device) # Load audio file AUDIO_PATH = "" audio_input, sample_rate = torchaudio.load(AUDIO_PATH) # Ensure the audio is mono (1 channel) if audio_input.shape[0] > 1: audio_input = torch.mean(audio_input, dim=0, keepdim=True) # Resample audio if necessary if sample_rate != 16000: resampler = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=16000) audio_input = resampler(audio_input) # Process the audio input input_values = processor(audio_input.squeeze(), sampling_rate=16000, return_tensors="pt").input_values # Move input values to GPU input_values = input_values.to(device) # Perform inference with torch.no_grad(): logits = model(input_values).logits # Decode the logits to text predicted_ids = torch.argmax(logits, dim=-1) transcription = processor.batch_decode(predicted_ids)[0] print("Transcription:", transcription) ``` ### Framework versions - Transformers 4.44.0 - Pytorch 2.2.1+cu118 - Datasets 2.20.0 - Tokenizers 0.19.1