README.md

---
license: apache-2.0
language:
- en
---

# [Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation](https://yannqi.github.io/AVS-COMBO/)

[Qi Yang](https://yannqi.github.io/), Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan and [Shiming Xiang](https://people.ucas.ac.cn/~xiangshiming)

This repository provides the pretrained checkpoints for the paper "Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation" accepted by CVPR 2024.

## 🔥What's New

- (2024. 3.14) Our checkpoints are available to the public!
- (2024. 3.12) Our code is available to the public🌲!
- (2024. 2.27) Our paper(COMBO) is accepted by CVPR 2024!
- (2023.11.17) We completed the implemention of COMBO and push the code.

<!-- ## 🪵 TODO List -->
 

## 🛠️ Getting Started

### 1. Environments

- Linux or macOS with Python ≥ 3.6

```shell
# recommended
pip install -r requirements.txt
pip install soundfile
# build MSDeformAttention
cd model/modeling/pixel_decoder/ops
sh make.sh
```

- Preprocessing for detectron2

 For using Siam-Encoder Module (SEM), we refine 1-line code of the detectron2.

 The refined file that requires attention is located at:

 `conda_envs/xxx/lib/python3.xx/site-packages/detectron2/checkpoint/c2_model_loading.py`
 (refine the `xxx` to your own environment)

 Commenting out the following code in [L287](https://github.com/facebookresearch/detectron2/blob/cc9266c2396d5545315e3601027ba4bc28e8c95b/detectron2/checkpoint/c2_model_loading.py#L287) will allow the code to run without errors:

```python
# raise ValueError("Cannot match one checkpoint key to multiple keys in the model.") 
```

- Install Semantic-SAM (Optional)

```shell
# Semantic-SAM
pip install git+https://github.com/cocodataset/panopticapi.git
git clone https://github.com/UX-Decoder/Semantic-SAM
cd Semantic-SAM
python -m pip install -r requirements.txt
```

Find out more at [Semantic-SAM](https://github.com/UX-Decoder/Semantic-SAM)

### 2. Datasets

Please refer to the link [AVSBenchmark](https://github.com/OpenNLPLab/AVSBench) to download the datasets. You can put the data under `data` folder or rename your own folder. Remember to modify the path in config files. The `data` directory is as bellow:

```
|--AVS_dataset
 |--AVSBench_semantic/
 |--AVSBench_object/Multi-sources/
 |--AVSBench_object/Single-source/
```

### 3. Download Pre-Trained Models

- The pretrained backbone is available from benchmark AVSBench pretrained backbones[TODO].

```
|--pretrained
 |--detectron2/R-50.pkl
 |--detectron2/d2_pvt_v2_b5.pkl
 |--vggish-10086976.pth
 |--vggish_pca_params-970ea276.pth
```

### 4. Maskiges pregeneration

- Generate class-agnostic masks (Optional)

```shell
sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh train # or ms3, avss
sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh val 
sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh test
```

- Generate Maskiges (Optional)

```shell
python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split train # or ms3, avss
python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split val
python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split test
```

- Move Maskiges to the following folder
 Note: For convenience, we provide pre-generated Maskiges for S4\MS3\AVSS subset on the TODO hugging face link.

```
|--AVS_dataset
 |--AVSBench_semantic/pre_SAM_mask/
 |--AVSBench_object/Multi-sources/ms3_data/pre_SAM_mask/
 |--AVSBench_object/Single-source/s4_data/pre_SAM_mask/
```

### 5. Train

```shell
# ResNet-50
sh scripts/res_train_avs4.sh # or ms3, avss
```

```shell
# PVTv2
sh scripts/pvt_train_avs4.sh # or ms3, avss
```

### 6. Test

```shell
# ResNet-50
sh scripts/res_test_avs4.sh # or ms3, avss
```

```shell
# PVTv2
sh scripts/pvt_test_avs4.sh # or ms3, avss
```

## 🤝 Citing COMBO

```
@misc{yang2023cooperation,
 title={Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation},
 author={Qi Yang and Xing Nie and Tong Li and Pengfei Gao and Ying Guo and Cheng Zhen and Pengfei Yan and Shiming Xiang},
 year={2023},
 eprint={2312.06462},
 archivePrefix={arXiv},
 primaryClass={cs.CV}
}
```