|
---
|
|
language:
|
|
- en
|
|
license: apache-2.0
|
|
pipeline_tag: automatic-speech-recognition
|
|
tags:
|
|
- audio
|
|
- automatic-speech-recognition
|
|
- hf-asr-leaderboard
|
|
widget:
|
|
- example_title: Librispeech sample 1
|
|
src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
|
|
- example_title: Librispeech sample 2
|
|
src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
|
|
model-index:
|
|
- name: whisper-medium
|
|
results:
|
|
- task:
|
|
type: automatic-speech-recognition
|
|
name: Automatic Speech Recognition
|
|
dataset:
|
|
name: Afrispeech-200
|
|
type: intronhealth/afrispeech-200
|
|
config: clean
|
|
split: test
|
|
args:
|
|
language: en
|
|
metrics:
|
|
- type: wer
|
|
value: 0
|
|
name: Test WER
|
|
---
|
|
|
|
# Afrispeech-Whisper-Medium-All
|
|
|
|
This model builds upon the capabilities of Whisper Medium (a pre-trained model for speech recognition and translation trained on a massive 680k hour dataset). While Whisper demonstrates impressive generalization abilities, this model takes it a step further to be very specific for African accents.
|
|
|
|
**Fine-tuned on the AfriSpeech-200 dataset**, specifically designed for African accents, this model offers enhanced performance for speech recognition tasks on African languages.
|
|
- Dataset: https://huggingface.co/datasets/intronhealth/afrispeech-200
|
|
- Paper: https://arxiv.org/abs/2310.00274
|
|
|
|
## Transcription
|
|
|
|
In this example, the context tokens are 'unforced', meaning the model automatically predicts the output language
|
|
(English) and task (transcribe).
|
|
|
|
```python
|
|
>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
|
|
>>> from datasets import load_dataset
|
|
|
|
>>> # load model and processor
|
|
>>> processor = WhisperProcessor.from_pretrained("intronhealth/afrispeech-whisper-medium-all")
|
|
>>> model = WhisperForConditionalGeneration.from_pretrained("intronhealth/afrispeech-whisper-medium-all")
|
|
>>> model.config.forced_decoder_ids = None
|
|
|
|
>>> # load dummy dataset and read audio files
|
|
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
|
|
>>> sample = ds[0]["audio"]
|
|
>>> input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features
|
|
|
|
>>> # generate token ids
|
|
>>> predicted_ids = model.generate(input_features)
|
|
>>> # decode token ids to text
|
|
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)
|
|
['<|startoftranscript|><|en|><|transcribe|><|notimestamps|> Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.<|endoftext|>']
|
|
|
|
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
|
|
[' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.']
|
|
```
|
|
The context tokens can be removed from the start of the transcription by setting `skip_special_tokens=True`.
|
|
|
|
|
|
|
|
## Long-Form Transcription
|
|
|
|
The Whisper model is intrinsically designed to work on audio samples of up to 30s in duration. However, by using a chunking
|
|
algorithm, it can be used to transcribe audio samples of up to arbitrary length. This is possible through Transformers
|
|
[`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline)
|
|
method. Chunking is enabled by setting `chunk_length_s=30` when instantiating the pipeline. With chunking enabled, the pipeline
|
|
can be run with batched inference. It can also be extended to predict sequence level timestamps by passing `return_timestamps=True`:
|
|
|
|
```python
|
|
>>> import torch
|
|
>>> from transformers import pipeline
|
|
>>> from datasets import load_dataset
|
|
|
|
>>> device = "cuda:0" if torch.cuda.is_available() else "cpu"
|
|
|
|
>>> pipe = pipeline(
|
|
>>> "automatic-speech-recognition",
|
|
>>> model="intronhealth/afrispeech-whisper-medium-all",
|
|
>>> chunk_length_s=30,
|
|
>>> device=device,
|
|
>>> )
|
|
|
|
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
|
|
>>> sample = ds[0]["audio"]
|
|
|
|
>>> prediction = pipe(sample.copy(), batch_size=8)["text"]
|
|
" Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel."
|
|
|
|
>>> # we can also return timestamps for the predictions
|
|
>>> prediction = pipe(sample.copy(), batch_size=8, return_timestamps=True)["chunks"]
|
|
[{'text': ' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.',
|
|
'timestamp': (0.0, 5.44)}]
|
|
```
|
|
|
|
Refer to the blog post [ASR Chunking](https://huggingface.co/blog/asr-chunking) for more details on the chunking algorithm.
|
|
|
|
|
