microsoft

/

wavlm-base-plus-sv

like 29

Follow

Microsoft 8.19k

When I run the embedding extraction snippet from model card I get

Some weights of the model checkpoint at microsoft/wavlm-base-plus-sv were not used when initializing WavLMForXVector: ['wavlm.encoder.pos_conv_embed.conv.weight_g', 'wavlm.encoder.pos_conv_embed.conv.weight_v']
- This IS expected if you are initializing WavLMForXVector from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing WavLMForXVector from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Some weights of WavLMForXVector were not initialized from the model checkpoint at microsoft/wavlm-base-plus-sv and are newly initialized: ['wavlm.encoder.pos_conv_embed.conv.parametrizations.weight.original0', 'wavlm.encoder.pos_conv_embed.conv.parametrizations.weight.original1']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

I am little confused, it looks like some of the weights are not loaded from the pretrained model and then initialized. This is not expected in this situation, right?

Same problem. Was able to use the model properly with transformers version <=4.29.0

Yes, downgrading transformers version to 4.29.0 helped

@stachu86 , the model runs with memory leak, the memory keeps increasing even when inferencing on the same audio. Have you encountered such problem? If not, what versions of torch and cuda do you use?

@sourcesur I have torch 2.2.1 and CUDA 12.3, I did not notice any memory leake, however, I ended up with code based on WavLMForXVector documentation, so maybe with torch.no_grad() will help?

@stachu86 , I was using the model card example, the torch.no_grad() helped, thanks!