README.md

---
license: apache-2.0
language:
- en
base_model:
- Qwen/Qwen2.5-1.5B-Instruct
- laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg-soup
pipeline_tag: question-answering
metrics:
- accuracy
library_name: transformers
---

[Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions](https://arxiv.org/abs/2412.08737)

# Model Card for Euclid-convnext-xxlarge (Version on 12/05/2024)

A multimodal large language models specifically trained for strong low-level geometric perception. 

## Model Details

### Model Description

Euclid is trained on 1.6M synthetic geometry images with high-fidelity question-answer pairs using a curriculum learning approach.

It combines a ConvNeXt visual encoder with a Qwen-2.5 language model, connected through a 2-layer MLP multimodal connector.


### Model Sources

- **Repository:** https://github.com/euclid-multimodal/Euclid
- **Paper:** https://arxiv.org/abs/2412.08737
- **Demo:** https://euclid-multimodal.github.io/

## Uses

The model is trained for precise low-level geometric perception tasks which is able to perform
- Point-on-line detection
- Point-on-circle detection
- Angle classification
- Length comparison
- Geometric annotation understanding

Please refer to our [repo](https://github.com/euclid-multimodal/Euclid) for full input format.

### Limitations and Applications

Our model is not designed to handle:
- Comprehensive image understanding tasks
- Advanced cognitive reasoning beyond geometric analysis

However, the model demonstrates strength in low-level visual perception. 

This capability makes it potentially valuable for serving as a base model for specialized downstream fintuning including:

- Robotic vision and automation systems
- Medical imaging and diagnostic support
- Industrial quality assurance and inspection
- Geometric education and visualization tools

### Example Usage

Clone our Euclid [repo](https://github.com/euclid-multimodal/Euclid) first, set up the environment, then run:

```
pip install -U "huggingface_hub[cli]"
huggingface-cli download --cache-dir $MODEL_PATH EuclidAI/Euclid-convnext-xxlarge
python euclid/eval/run_euclid_geo.py --model_path $MODEL_PATH --device cuda
```



## Evaluation Results

Performance on Geoperception benchmark tasks:

| Model | POL | POC | ALC | LHC | PEP | PRA | EQL | Overall |
|-------|-----|-----|-----|-----|-----|-----|-----|----------|
| Random Baseline | 0.43 | 2.63 | 59.92 | 51.36 | 0.25 | 0.00 | 0.02 | 16.37 |
| Pixtral-12B | 22.85 | 53.21 | 47.33 | 51.43 | 22.53 | 37.11 | **58.45** | 41.84 |
| Gemini-1.5-Pro | 24.42 | **69.80** | 57.96 | 79.05 | **39.60** | **77.59** | 52.27 | 57.24 |
| EUCLID-ConvNeXt-Large | 80.54 | 57.76 | 86.37 | 88.24 | 42.23 | 64.94 | 34.45 | 64.93 |
| EUCLID-ConvNeXt-XXLarge | **82.98** | 61.45 | **90.56** | **90.82** | **46.96** | 70.52 | 31.94 | **67.89** |


## Citation

If you find Euclid useful for your research and applications, please cite using this BibTeX:
```bibtex
@article{zhang2024euclid,
  title={Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions},
  author={Zhang, Jiarui and Liu, Ollie and Yu, Tianyu and Hu, Jinyi and Neiswanger, Willie},
  journal={arXiv preprint arXiv:2412.08737},
  year={2024}
}