Whisper-small-ru-pruned-ft

Model info

This is a finetuned version of pruned whisper-small model (waveletdeboshir/whisper-small-ru-pruned) for Russian.

Model was finetuned on russian part of mozilla-foundation/common_voice_15_0 with Specaugment, Colored Noise augmentation and Noise from file augmentation.

Metrics

metric	dataset	waveletdeboshir/whisper-small-ru-pruned	waveletdeboshir/whisper-small-ru-pruned-ft
WER (without punctuation)	common_voice_15_0_test	0.1748	0.1092
WER	common_voice_15_0_test	0.2492	0.1571

Limitations

Because texts in Common Voice don't contain digits and other characters except letters and punctuation signs, model lost an ability to predict numbers and special characters.

Size

Only 10% tokens was left including special whisper tokens (no language tokens except <|ru|> and <|en|>, no timestamp tokens), 200 most popular tokens from tokenizer and 4000 most popular Russian tokens computed by tokenization of russian text corpus.

Model size is 15% less then original whisper-small:

	openai/whisper-small	waveletdeboshir/whisper-small-ru-pruned
n of parameters	242 M	205 M
n of parameters (with proj_out layer)	281 M	208 M
model file size	967 Mb	821 Mb
vocab_size	51865	4207

Usage

Model can be used as an original whisper:

>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
>>> import torchaudio

>>> # load audio
>>> wav, sr = torchaudio.load("audio.wav")

>>> # load model and processor
>>> processor = WhisperProcessor.from_pretrained("waveletdeboshir/whisper-small-ru-pruned-ft")
>>> model = WhisperForConditionalGeneration.from_pretrained("waveletdeboshir/whisper-small-ru-pruned-ft")

>>> input_features = processor(wav[0], sampling_rate=sr, return_tensors="pt").input_features 

>>> # generate token ids
>>> predicted_ids = model.generate(input_features)
>>> # decode token ids to text
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)
['<|startoftranscript|><|ru|><|transcribe|><|notimestamps|> Начинаем работу.<|endoftext|>']

The context tokens can be removed from the start of the transcription by setting skip_special_tokens=True.

Other pruned whisper models

• waveletdeboshir/whisper-tiny-ru-pruned
• waveletdeboshir/whisper-base-ru-pruned