---
license: cc-by-nc-4.0
base_model: lmms-lab/llava-onevision-qwen2-7b-mid-stage-a4
model-index:
- name: llama3-siglip-taco-8b
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 🌮 TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action

<h3 align="left"> <a href="https://taco-project.github.io/">🌐 Website</a> | <a href="https://arxiv.org/pdf/2412.05479">📑 Arxiv</a> | <a href="">🤗 Model Weights</a> | <a href="">💻 Demo</a></h3>
    
<h5 align="left"> If you like our project or are interested in its updates, please star us :) Thank you! ⭐ </h2>

## Model description
We introduce TACO as a family of multi-modal large action models designed to improve performance on such complex, multi-step and multi-modal tasks. During inference, TACO produces chains-of-thought-and–action (CoTA), executes intermediate steps by invoking external tools such as OCR, depth estimation and calculator, then integrates both the thoughts and action outputs to produce coherent responses. Our TACO models outperform the instruction-tuned baseline across 8 benchmarks, achieving a 3.6% improvement on average, with gains up to 15% in MMVet tasks involving OCR, mathematical reasoning and spatial reasoning. 

<p align="center">
  <img src="teaser.png" width="1000" style="margin-bottom: 0.2;"/>
  <p align="center">Figure 1. TACO vs. other multi-modal models</p>
<p>

## Usage

See our [Github repository](https://github.com/SalesforceAIResearch/TACO). 

## Intended uses & limitations

This model is intended to be used on complex, multi-step and multi-modal question answering tasks. It is trained to answer visual questions with some of the following 15 actions：```OCR```, ```LocalizeObjects```, ```GetObjects```, 
```EstimateRegionDepth```, ```EstimateObjectDepth```, ```Crop```, ```ZoomIn```, ```QueryLanguageModel```, ```GetImageToImagesSimilarity```, ```GetImageToTextsSimilarity```, 
```GetTextToImagesSimilarity```, ```DetectFaces```, ```QueryKnowledgeBase```, ```Calculate```, and ```SolveMathEquation```. Additionally, the ```Terminate``` action
is also supported for the model to provide a final answer. 

For other types of tasks that don't benefit from the actions above, you might need to train a new model or further finetune it with other actions.

## Training and evaluation data

See our [paper](https://arxiv.org/pdf/2412.05479) for details.

## Training procedure and hyperparameters 

See our [paper](https://arxiv.org/pdf/2412.05479) for details.

## Training results

See our [paper](https://arxiv.org/pdf/2412.05479) for details.

### License information
This release is for research purposes only in support of an academic paper. This repository is licensed under the noncommercial license [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). 

### Citation
Please cite us if you find our repository helpful. Thank you!
```
@misc{ma2024tacolearningmultimodalaction,
      title={TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action}, 
      author={Zixian Ma and Jianguo Zhang and Zhiwei Liu and Jieyu Zhang and Juntao Tan and Manli Shu and Juan Carlos Niebles and Shelby Heinecke and Huan Wang and Caiming Xiong and Ranjay Krishna and Silvio Savarese},
      year={2024},
      eprint={2412.05479},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.05479}, 
}
```