Update README.md

README.md

@@ -102,129 +102,158 @@ pipeline_tag: feature-extraction
 tags:
 - music
 ---
-# CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages
+# **CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages**
 
 <p align="center">
   <img src="overview.png" alt="CLaMP 3 Overview" width="50%">
 </p>
 
+## **Overview**
+CLaMP 3 is a **multimodal and multilingual** framework for **music information retrieval (MIR)**. By using **contrastive learning**, it aligns **sheet music, audio, performance signals, and multilingual text** into a **shared representation space**, enabling retrieval across unaligned musical modalities.  
 
-## Overview
-CLaMP 3 is a unified framework for cross-modal and cross-lingual music information retrieval (MIR). By using contrastive learning, it aligns sheet music, audio, performance signals, and multilingual text into a shared representation space, enabling retrieval across unaligned musical modalities. Key features include:
+### **Key Features**  
+- **Multimodal Support:**  
+   - **Sheet Music:** Uses **Interleaved ABC notation**, with a context size of **512 bars**.  
+   - **Performance Signals:** Processes **MIDI Text Format (MTF)** data, with a context size of **512 MIDI messages**.  
+   - **Audio Recordings:** Works with features extracted by **[MERT](https://arxiv.org/abs/2306.00107)**, with a context size of **640 seconds of audio**.
 
-- **Multimodal Support:**
-   1. **Sheet Music:** Uses Interleaved ABC notation.
-   2. **Performance Signals:** Processes MIDI Text Format (MTF) data.
-   3. **Audio Recordings:** Works with audio features extracted by [MERT](https://arxiv.org/abs/2306.00107).
+- **Multilingual Capabilities:**  
+   - Trained on **27 languages** and generalizes to all **100 languages** supported by **[XLM-R](https://arxiv.org/abs/1911.02116)**.  
 
-- **Multilingual Capabilities:** Trained on **27 languages** and generalizes to all **100 languages** supported by [XLM-R](https://arxiv.org/abs/1911.02116).
+- **Datasets & Benchmarking:**  
+   - **[M4-RAG](https://huggingface.co/datasets/sander-wood/m4-rag):** A **large-scale** dataset of **2.31M high-quality music-text pairs** across 27 languages and 194 countries.  
+   - **[WikiMT-X](https://huggingface.co/datasets/sander-wood/wikimt-x):** A MIR benchmark containing **1,000 triplets** of sheet music, audio, and diverse text annotations.  
 
-- **Dataset and Benchmark:**
-  - Trained on **M4-RAG**, a large-scale dataset of 2.31M high-quality music-text pairs across 27 languages and 194 countries.
-  - Introduces **WikiMT-X**, a benchmark containing 1,000 triplets of sheet music, audio, and text.
+### **Applications**  
+CLaMP 3 supports a **wide range of music research tasks**, including but not limited to:  
+- **Semantic Retrieval:** Find music based on **descriptions** or retrieve textual metadata for **audio or symbolic** inputs.  
+- **Zero-Shot Classification:** Categorize **music by genre, region, or other attributes** without labeled training data.  
+- **Music Quality Assessment:** Compute the **semantic distance** between reference and generated music features, similar to **Fréchet Inception Distance (FID)**.  
+- **Cross-Modal Generative Model Evaluation:** Assess **text-to-music generation, music captioning**, and **symbolic-to-audio synthesis** models.  
+- **Computational Musicology:** By visualizing the distribution of data within the **shared representation space**, researchers can explore regional music patterns, stylistic similarities, and cross-cultural influences. 
 
-CLaMP 3 supports a wide range of applications in MIR and music research, including but not limited to:  
-- **Semantic retrieval:** Searching for music based on descriptive text or retrieving textual metadata based on audio or symbolic representations.  
-- **Zero-shot classification:** Categorizing music by genre, region, or other attributes without labeled training data.  
-- **Music quality assessment:** Measuring the **semantic distance** between reference ground truth and generated music using metrics analogous to **Fréchet Inception Distance (FID)**, providing an objective alternative to human evaluation.  
-- **Evaluation of generative models:** Assessing **text-to-music generation**, **music captioning**, and **symbolic-to-audio synthesis** models by quantifying their alignment across different modalities.  
-- **Computational musicology:** Enabling studies in **geographical musicology**, analyzing regional distributions and cross-cultural influences through large-scale multimodal datasets.  
+Importantly, these applications are **not restricted to any specific music modality or language**, making CLaMP 3 a powerful tool for **diverse music AI research**.
 
-Importantly, these applications are **not restricted to any specific musical modality or language**. CLaMP 3's multimodal and multilingual design allows it to generalize across diverse datasets, making it a powerful tool for a wide range of music-related AI research.
+## **Links**
+- **CLaMP 3 Demo Page** *(Coming Soon...)*
+- **CLaMP 3 Paper** *(Coming Soon...)*
+- **[CLaMP 3 Code](https://github.com/sanderwood/clamp3)**
+- **[CLaMP 3 Model Weights](https://huggingface.co/sander-wood/clamp3/tree/main)**
+- **[M4-RAG Pre-training Dataset](https://huggingface.co/datasets/sander-wood/m4-rag)**
+- **[WikiMT-X Evaluation Benchmark](https://huggingface.co/datasets/sander-wood/wikimt-x)**
 
-### Links
-- CLaMP 3 Demo Page (Coming Soon...)
-- CLaMP 3 Paper (Coming Soon...)
-- [CLaMP 3 Code](https://github.com/sanderwood/clamp3)
-- [CLaMP 3 Model Weights](https://huggingface.co/sander-wood/clamp3/tree/main)
-- [M4-RAG Pre-training Dataset](https://huggingface.co/datasets/sander-wood/m4-rag)
-- [WikiMT-X Evaluation Benchmark](https://huggingface.co/datasets/sander-wood/wikimt-x)
+> **Note:** Ensure the model weights are placed in the `code/` folder, and verify the **configuration hyperparameters** before use.
 
-> **Note** Ensure the model weights for CLaMP 3 are placed under the `code/` folder for proper loading. Also, verify that the configuration hyperparameters are correctly set.
+## **Repository Structure**
+- **[code/](https://github.com/sanderwood/clamp3/tree/main/code)** → Training & feature extraction scripts.
+- **[classification/](https://github.com/sanderwood/clamp3/tree/main/classification)** → Linear classification training and prediction.  
+- **[preprocessing/](https://github.com/sanderwood/clamp3/tree/main/preprocessing)** → Convert data into **Interleaved ABC, MTF, or MERT-extracted features**.  
+- **[retrieval/](https://github.com/sanderwood/clamp3/tree/main/retrieval)** → Semantic search, retrieval evaluation, and similarity calculations.  
 
-## Repository Structure
-- [code/](https://github.com/sanderwood/clamp3/tree/main/code): Contains scripts for training CLaMP 3 and extracting features from music and text data. You can modify hyperparameters and file paths in the configuration files for custom training.
-  
-- [classification/](https://github.com/sanderwood/clamp3/tree/main/classification): Includes scripts for classification tasks using extracted features, such as training linear classification models and making predictions.
+## **Getting Started**
+### **Environment Setup**
+To set up the environment for CLaMP 3, run:  
+```bash
+conda env create -f environment.yml
+conda activate clamp3
+```
 
-- [preprocessing/](https://github.com/sanderwood/clamp3/tree/main/preprocessing): Scripts for converting your data into compatible formats (interleaved ABC notation, MTF, or MERT-extracted audio features). These are required for CLaMP 3 to work with the data.
+### **Data Preparation**
+#### **1. Convert Music Data to Compatible Formats**
+Before using CLaMP 3, preprocess **MusicXML files** into **Interleaved ABC**, **MIDI files** into **MTF**, and **audio files** into **MERT-extracted features**.
 
-- [retrieval/](https://github.com/sanderwood/clamp3/tree/main/retrieval): Provides scripts for evaluating model performance, conducting semantic searches, and calculating similarity metrics based on extracted feature vectors.
+> **Note:** Each script requires a manual edit of the `input_dir` variable at the top of the file before running, **except for the MERT extraction script (`extract_mert.py`), which takes command-line arguments for input and output paths.**
 
-> **Note** For detailed instructions on how to use the scripts in each folder, please refer to the individual README files within those directories. This main README provides only a high-level overview of the repository.
+##### **1.1 Convert MusicXML to Interleaved ABC Notation**
 
-## Getting Started
+CLaMP 3 requires **Interleaved ABC notation** for sheet music. To achieve this, first, convert **MusicXML** (`.mxl`, `.xml`, `.musicxml`) to **standard ABC** using [`batch_xml2abc.py`](https://github.com/sanderwood/clamp3/blob/main/preprocessing/abc/batch_xml2abc.py):
 
-### Environment Setup
-To set up the environment for CLaMP 3, run the following commands:
+```bash
+python batch_xml2abc.py
+```
+- **Input:** `.mxl`, `.xml`, `.musicxml`  
+- **Output:** `.abc` (Standard ABC)
+ 
+Next, process the standard ABC files into **Interleaved ABC notation** using [`batch_interleaved_abc.py`](https://github.com/sanderwood/clamp3/blob/main/preprocessing/abc/batch_interleaved_abc.py):
 
 ```bash
-conda env create -f environment.yml
-conda activate clamp3
+python batch_interleaved_abc.py
+```
+- **Input:** `.abc` (Standard ABC)  
+- **Output:** `.abc` *(Interleaved ABC for CLaMP 3)*  
+
+##### **1.2 Convert MIDI to MTF Format**
+CLaMP 3 processes **performance signals** in **MIDI Text Format (MTF)**. Convert **MIDI files** (`.mid`, `.midi`) into **MTF format** using [`batch_midi2mtf.py`](https://github.com/sanderwood/clamp3/blob/main/preprocessing/midi/batch_midi2mtf.py):
+
+```bash
+python batch_midi2mtf.py
+```
+- **Input:** `.mid`, `.midi`  
+- **Output:** `.mtf` *(MTF for CLaMP 3)*  
+
+##### **1.3 Extract Audio Features using MERT**
+For audio processing, CLaMP 3 uses **MERT-extracted features** instead of raw waveforms. Extract **MERT-based features** from raw audio (`.mp3`, `.wav`) using [`extract_mert.py`](https://github.com/sanderwood/clamp3/blob/main/preprocessing/audio/extract_mert.py):
+
+```bash
+python extract_mert.py --input_path <input_path> --output_path <output_path> --model_path musichubert_hf/MERT-v1-95M --mean_features
+```
+- **Input:** `.mp3`, `.wav`  
+- **Output:** `.npy` *(Processed audio features for CLaMP 3)*  
+
+### **Training and Feature Extraction**
+#### **1. Training Models**
+Modify **[config.py](https://github.com/sanderwood/clamp3/blob/main/code/config.py)** to adjust **hyperparameters and data paths**.
+
+To train CLaMP 3 on **symbolic music**, use **[train_clamp3_symbolic.py](https://github.com/sanderwood/clamp3/blob/main/code/train_clamp3_symbolic.py)**:
+
+```bash
+python -m torch.distributed.launch --nproc_per_node=<GPUs> --use_env train_clamp3_symbolic.py
+```
+
+For **audio data**, use **[train_clamp3_audio.py](https://github.com/sanderwood/clamp3/blob/main/code/train_clamp3_audio.py)**:
+
+```bash
+python -m torch.distributed.launch --nproc_per_node=<GPUs> --use_env train_clamp3_audio.py
+```
+
+Alternatively, you can use **pre-trained weights**:
+- **[CLaMP 3 SAAS (Optimal for Audio)](https://huggingface.co/sander-wood/clamp3/blob/main/weights_clamp3_saas.pth)**
+- **[CLaMP 3 C2 (Optimal for Symbolic Music)](https://huggingface.co/sander-wood/clamp3/blob/main/weights_clamp3_c2.pth)**  
+
+By default, CLaMP 3 is configured for the **SAAS version**, which provides **optimal performance on audio data**. If working primarily with **symbolic music**, download the **C2 variant** and modify **line 66 in `config.py`** from **saas** to **c2**.
+
+#### **2. Feature Extraction**
+After training (or using pre-trained weights), extract features using [`extract_clamp3.py`](https://github.com/sanderwood/clamp3/blob/main/code/extract_clamp3.py):
+
+```bash
+accelerate launch extract_clamp3.py --epoch <epoch> <input_dir> <output_dir> [--get_global]
+```
+- **`--epoch <epoch>`:** (Optional) Specify the checkpoint epoch.
+- **`<input_dir>`:** Directory containing the input files.
+- **`<output_dir>`:** Destination folder for the output `.npy` features.
+- **`--get_global`**: (Optional) Flag to extract a global semantic vector for each input.
+
+All extracted features are stored as `.npy` files.
+
+> **Note**: In this project, we use the global semantic vectors (via average pooling and a linear layer) for both classification and retrieval tasks.
+
+### **Retrieval and Classification**
+#### **1. Semantic Search**
+Retrieve **similar music features** using **[`semantic_search.py`](https://github.com/sanderwood/clamp3/tree/main/retrieval/semantic_search.py)**:  
+```bash
+python semantic_search.py <query_file> <reference_folder> [--top_k TOP_K]
+```
+> **Note:** Zero-shot classification is essentially **semantic search**, where the query feature is compared against class prototypes.  
+
+#### **2. Classification**
+Train a linear classifier using **[`train_cls.py`](https://github.com/sanderwood/clamp3/tree/main/classification/train_cls.py)**:  
+```bash
+python train_cls.py --train_folder <path> --eval_folder <path> [--num_epochs <int>] [--learning_rate <float>] [--balanced_training]
+```
+Run inference with **[`inference_cls.py`](https://github.com/sanderwood/clamp3/tree/main/classification/inference_cls.py)**:  
+```bash
+python inference_cls.py <weights_path> <feature_folder> <output_file>
 ```
 
-### Data Preparation
-1. **Convert Files**: Navigate to the `preprocessing/` folder and convert your music files into a compatible format (interleaved ABC notation, MTF, or MERT-extracted audio features) suitable for use with CLaMP 3. Whether you are training or performing inference, **you must use these preprocessing scripts to ensure the data is in the correct format**.
-   1. **Interleaved ABC Notation**:
-      - Convert MusicXML files to ABC using [batch_xml2abc.py](https://github.com/sanderwood/clamp3/blob/main/preprocessing/abc/batch_xml2abc.py).
-      - Process the ABC files into interleaved notation using [batch_interleaved_abc.py](https://github.com/sanderwood/clamp3/blob/main/preprocessing/abc/batch_interleaved_abc.py).
-   2. **MTF**:
-      - Convert MIDI files to MTF format using [batch_midi2mtf.py](https://github.com/sanderwood/clamp3/blob/main/preprocessing/midi/batch_midi2mtf.py).
-   3. **MERT-extracted Audio Features**:
-      - Extract audio features using MERT by running the scripts [extract_mert.py](https://github.com/sanderwood/clamp3/tree/main/preprocessing/audio/extract_mert.py). These features will be saved as `.npy` files and are ready for use in CLaMP 3.
-
-2. **Prepare Text Metadata (Optional)**: If you plan to train the model, you will need to prepare corresponding metadata for each music file. The metadata should be in JSON format, containing details like title, artist, region, language, and description.
-
-   Example:
-   ```json
-   {
-       "filepaths": ["audio/--/---aL9TdeI4.npy"],
-       "id": "---aL9TdeI4",
-       "title": "Mairi's Wedding",
-       "artists": ["Noel McLoughlin"],
-       "region": "United Kingdom of Great Britain and Northern Ireland",
-       "language": "English",
-       "genres": ["Folk", "Traditional"],
-       "tags": ["Scottish", "Wedding", "Traditional", "Folk", "Celtic"],
-       "background": "Mairi's Wedding is a Scottish folk song...",
-       "analysis": "The song has a lively and upbeat Scottish folk rhythm...",
-       "description": "A traditional folk song with a joyful celebration...",
-       "scene": "The setting is a picturesque Scottish village on a sunny morning...",
-       "translations": { "language": "Vietnamese", "background": "Bài hát \"Đám Cưới Mairi\"..." }
-   }
-   ```
-
-   Once your JSON files are ready, merge them into a single `.jsonl` file and structure the directories as shown:
-
-   ```
-   /your-data-folder/
-   ├── abc/
-   ├── audio/
-   ├── mtf/
-   ├── merged_output.jsonl
-   ```
-
-### Training and Feature Extraction
-2. **Training Models**: If you want to train CLaMP 3, check the training scripts in the [code/](https://github.com/sanderwood/clamp3/tree/main/code) folder and modify the [config.py](https://github.com/sanderwood/clamp3/blob/main/code/config.py) file to set your hyperparameters and data paths.
-
-3. **Extracting Features**: After training (or if you have pre-trained weights), extract features from **preprocessed** data using [extract_clamp3.py](https://github.com/sanderwood/clamp3/blob/main/code/extract_clamp3.py). The script automatically detects the modality based on the file extension (e.g., `.txt`, `.abc`, `.mtf`, `.npy`). Make sure your data has already been converted into CLaMP 3–compatible formats by following the scripts in the [preprocessing/](https://github.com/sanderwood/clamp3/tree/main/preprocessing)` folder.
-
-### Classification and Retrieval
-4. **Classification**: To perform classification on the extracted features, navigate to the [classification/](https://github.com/sanderwood/clamp3/tree/main/classification) directory. You’ll find scripts for training and making predictions using linear classification models.
-
-5. **Semantic Search**: To conduct semantic searches using the extracted features, refer to the scripts in the [retrieval/](https://github.com/sanderwood/clamp3/tree/main/retrieval) folder.
-
-## Citation
-Coming Soon...
-<!-- If you use CLaMP 3, M4-RAG, or WikiMT-X in your research, please cite the following paper:
-
-bibtex
-@misc{wu2024clamp2multimodalmusic,
-      title={CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models}, 
-      author={Shangda Wu and Yashan Wang and Ruibin Yuan and Zhancheng Guo and Xu Tan and Ge Zhang and Monan Zhou and Jing Chen and Xuefeng Mu and Yuejie Gao and Yuanliang Dong and Jiafeng Liu and Xiaobing Li and Feng Yu and Maosong Sun},
-      year={2024},
-      eprint={2410.13267},
-      archivePrefix={arXiv},
-      primaryClass={cs.SD},
-      url={https://arxiv.org/abs/2410.13267}, 
-} -->
+## **Citation**
+*Coming Soon...*