File size: 3,386 Bytes
f07425a 844ccbe f07425a 844ccbe f07425a 844ccbe f07425a 6534ede f07425a 6534ede f07425a 6534ede f07425a 844ccbe f07425a 844ccbe 6534ede 844ccbe c85e695 12b17c8 c85e695 b525c98 f07425a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 |
---
license: apache-2.0
tags:
- contrastive learning
- CLAP
- audio classification
- zero-shot classification
---
# tinyCLAP: Distilling Contrastive Language-Audio Pretrained models
[](https://arxiv.org/abs/2311.14517)
This repository contains the official implementation of [tinyCLAP](https://arxiv.org/abs/2311.14517).
To access the project website, using [this link](https://francescopaissan.it/tinyclapweb/).
## Requirements
To clone the repo and install requirements:
```setup
git clone https://github.com/fpaissan/tinyCLAP & cd tinyCLAP
pip install -r extra_requirements.txt
```
## Training
To train the model(s) in the paper, run this command:
```bash
MODEL_NAME=phinet_alpha_1.50_beta_0.75_t0_6_N_7
./run_tinyCLAP.sh $MODEL_NAME
```
Note that `MODEL_NAME` is formatted such that the script will automatically parse the configuration for the student model.
You can change parameters by changing the model name.
Please note:
- To use the original CLAP encoder in the distillation setting, replace the model name with `Cnn14`;
- To reproduce the variants of PhiNet from the manuscript, refer to the hyperparameters listed in Table 1.
## Evaluation
The command to evaluate the model on each dataset varies slightly among datasets.
Below are listed all the necessary commands.
### ESC50
```bash
python train_clap.py hparams/distill_clap.yaml --experiment_name tinyCLAP_$MODEL_NAME --zs_eval True --esc_folder $PATH_TO_ESC
```
### UrbanSound8K
```bash
python train_clap.py hparams/distill_clap.yaml --experiment_name tinyCLAP_$MODEL_NAME --zs_eval True --us8k_folder $PATH_TO_US8K
```
### TUT17
```bash
python train_clap.py hparams/distill_clap.yaml --experiment_name tinyCLAP_$MODEL_NAME --zs_eval True --tut17_folder $PATH_TO_TUT17
```
## Pre-trained Models
You can download pretrained models from the [tinyCLAP HF](https://huggingface.co/fpaissan/tinyCLAP).
_Note_: The checkpoints on HF contain the entire CLAP module (complete of text encoder and teacher encoder).
To run inference using the pretrained models, please use:
```bash
python train_clap.py hparams/distill_clap.yaml --pretrained_clap fpaissan/tinyCLAP/$MODEL_NAME.ckpt --zs_eval True --tut17_folder $PATH_TO_TUT17
```
This command will automatically download the checkpoint, if present in the zoo of pretrained models. Make sure to change the dataset configuration file based on the evaluation.
A list of available models with their computational cost is described in the follwing table:
| audioenc_name_student | Params [M] | ESC-50 | UrbanSound8K | TUT17 |
|:-----:|:----------:|:------:|:------------:|:-----:|
| MSFT CLAP | 82.8 | 80.7% | 72.1% | 25.2% |
| Cnn14 | 82.8 | 81.3% | 72.3% | 23.7% |
| phinet_alpha_1.50_beta_0.75_t0_6_N_7 | 4.4 | 77.3% | 69.7% | 21.9% |
The original paper's checkpoints are available through [this link](https://www.dropbox.com/scl/fi/e3aj76vxwlb4w6hs3mclv/tinyCLAP_results.zip?rlkey=7fl426tz1vf686oyosvja8i9s&dl=0).
## Citing tinyCLAP
```
@inproceedings{paissan2024tinyclap,
title={tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models},
author={Paissan, Francesco and Farella, Elisabetta},
journal={Interspeech 2024},
year={2024}
}
``` |