File size: 3,485 Bytes
814186b 4ec1180 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 |
---
license: apache-2.0
---
<div align="center">
<h1>Chimera: Improving Generalist Model with<br>Domain-Specific Experts</h1>
[[ Paper ]](https://huggingface.co/papers/2412.05983) [[ Website ]](https://unimodal4reasoning.github.io/chimera_page/) [[ Dataset🤗 ]]() [[ Github ]](https://github.com/UniModal4Reasoning/Chimera)
</div>
## 🛠️ Installation
- Clone this repository:
```bash
git clone https://github.com/UniModal4Reasoning/Chimera.git
```
- Create a conda virtual environment and activate it:
```bash
conda create -n chimera python=3.9 -y
conda activate chimera
```
- Install dependencies using `requirements.txt`:
```bash
pip install -r requirements.txt
```
- Install other requirements:
```bash
cd chimera/
pip install --upgrade pip # enable PEP 660 support
pip install -e .
```
### Additional Instructions
- Install `flash-attn==2.3.4`:
```bash
pip install flash-attn==2.3.4 --no-build-isolation
```
Alternatively you can compile from source:
```bash
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout v2.3.4
python setup.py install
```
## Quick Start
### Multi-modal reasoning
```python
from chimera.chimera_infer import Chimera4easyuse
import torch
from PIL import Image
# prepare model
# model_path = "U4R/Chimera-Reasoner-2B"
# model_path = "U4R/Chimera-Reasoner-4B"
model_path = "U4R/Chimera-Reasoner-8B"
generation_config = dict(max_new_tokens=256, do_sample=False)
model = Chimera4easyuse(model_path, dtype = torch.bfloat16, generation_config= generation_config)
# prepare input
image_path = "path/to/image"
user_prompt = "<image>\nuser prompt"
input_image = Image.open(image_path).convert('RGB')
response = model.get_response(user_prompt, [input_image])
print(response)
```
### Visual content extraction
```python
from chimera.chimera_infer import Chimera4easyuse
import torch
from PIL import Image
# prepare model
model_path = "U4R/Chimera-Extractor-1B"
generation_config = dict(max_new_tokens=4096, do_sample=False, no_repeat_ngram_size = 20)
model = Chimera4easyuse(model_path, dtype = torch.float16, generation_config= generation_config)
# prepare input
image_path = "path/to/document"
user_prompt = "<image>\nAs a smart PDF to Markdown conversion tool, please convert the content of the provided PDF into Markdown format."
input_image = Image.open(image_path).convert('RGB')
response = model.get_response(user_prompt, [input_image])
print(response)
```
## License
Chimera is released under the [Apache License 2.0](LICENSE)
## Citation
If you find our models / code / papers useful in your research, please consider giving ⭐ and citations 📝, thx :)
```bibtex
@misc{peng2024chimeraimprovinggeneralistmodel,
title={Chimera: Improving Generalist Model with Domain-Specific Experts},
author={Tianshuo Peng and Mingsheng Li and Hongbin Zhou and Renqiu Xia and Renrui Zhang and Lei Bai and Song Mao and Bin Wang and Conghui He and Aojun Zhou and Botian Shi and Tao Chen and Bo Zhang and Xiangyu Yue},
year={2024},
eprint={2412.05983},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.05983},
}
```
## Contact Us
If you encounter any issues or have questions, please feel free to contact us via [email protected] or [email protected].
|