Update README.md
Browse files
README.md
CHANGED
|
@@ -14,33 +14,45 @@ The Aquila-VL-2B model is a vision-language model (VLM) trained based on the [LL
|
|
| 14 |
|
| 15 |
The model was trained on our self-built Infinity-MM dataset, which contains approximately 40 million image-text pairs. This dataset is a combination of open-source data collected from the internet and synthetic instruction data generated using open-source VLM models.
|
| 16 |
|
| 17 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
# Evaluation
|
| 20 |
|
| 21 |
-
We evaluated the model using the [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) tool. Whenever possible, we prioritized using the
|
| 22 |
-
|
| 23 |
-
|
|
| 24 |
-
|
| 25 |
-
|
|
| 26 |
-
|
|
| 27 |
-
|
|
| 28 |
-
|
|
| 29 |
-
|
|
| 30 |
-
|
|
| 31 |
-
|
|
| 32 |
-
|
|
| 33 |
-
|
|
| 34 |
-
|
|
| 35 |
-
|
|
| 36 |
-
|
|
| 37 |
-
|
|
| 38 |
-
|
|
| 39 |
-
|
|
| 40 |
-
|
|
| 41 |
-
| MathVision
|
| 42 |
-
|
|
| 43 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
|
| 46 |
|
|
@@ -50,8 +62,9 @@ For comparison models, evaluations were conducted in a local environment, so the
|
|
| 50 |
|
| 51 |
* We plan to train models of various sizes.
|
| 52 |
* Future training will incorporate multi-image and video data.
|
| 53 |
-
*
|
| 54 |
-
*
|
|
|
|
| 55 |
|
| 56 |
# Disclaimer
|
| 57 |
-
The resources, including code, data, and model weights, associated with this project are restricted for academic research purposes only and cannot be used for commercial purposes. The content produced the model is influenced by uncontrollable variables such as randomness, and therefore, the accuracy of the output cannot be guaranteed by this project. This project does not accept any legal liability for the content of the model output, nor does it assume responsibility for any losses incurred due to the use of associated resources and output results.
|
|
|
|
| 14 |
|
| 15 |
The model was trained on our self-built Infinity-MM dataset, which contains approximately 40 million image-text pairs. This dataset is a combination of open-source data collected from the internet and synthetic instruction data generated using open-source VLM models.
|
| 16 |
|
| 17 |
+
|
| 18 |
+
We have open-sourced Infinity-MM dataset and related resources. We hope you enjoy using them!
|
| 19 |
+
|
| 20 |
+
## News
|
| 21 |
+
- `2024/10/25`: The **Aquila-VL-2B** model and Infinity-MM dataset are now available. We have also released the technical report simultaneously.
|
| 22 |
+
|
| 23 |
+
<!-- We plan to open-source the Infinity-MM dataset, training scripts, and related resources in the near future. For more technical details, stay tuned for our upcoming technical report. -->
|
| 24 |
|
| 25 |
# Evaluation
|
| 26 |
|
| 27 |
+
We evaluated the model using the [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) tool. Whenever possible, we prioritized using the OpenAI API for test sets that support API-based evaluation.
|
| 28 |
+
|
| 29 |
+
| Benchmark | MiniCPM-V-2 | InternVL2-2B | XinYuan-VL-2B | Qwen2-VL-2B-Instruct | Aquila-VL-2B |
|
| 30 |
+
| :--------------------------- | :---------: | :----------: | :-----------: | :------------------: | :----------: |
|
| 31 |
+
| MMBench-EN<sub>test</sub> | 69.4 | 73.4 | **78.9** | 74.9 | 78.8 |
|
| 32 |
+
| MMBench-CN<sub>test</sub> | 65.9 | 70.9 | 76.1 | 73.9 | **76.4** |
|
| 33 |
+
| MMBench_V1.1<sub>test</sub> | 65.2 | 69.7 | **75.4** | 72.7 | 75.2 |
|
| 34 |
+
| MMT-Bench<sub>test</sub> | 54.5 | 53.3 | 57.2 | 54.8 | **58.2** |
|
| 35 |
+
| RealWorldQA | 55.4 | 57.3 | 63.9 | 62.6 | **63.9** |
|
| 36 |
+
| HallusionBench | 36.8 | 38.1 | 36.0 | 41.5 | **43.0** |
|
| 37 |
+
| SEEDBench2<sub>plus</sub> | 51.8 | 60.0 | 63.0 | 62.4 | **63.0** |
|
| 38 |
+
| LLaVABench | 66.1 | 64.8 | 42.4 | 52.5 | **68.4** |
|
| 39 |
+
| MMStar | 41.6 | 50.2 | 51.9 | 47.8 | **54.9** |
|
| 40 |
+
| POPE | 86.6 | 85.3 | **89.4** | 88.0 | 83.6 |
|
| 41 |
+
| MMVet | 44.0 | 41.1 | 42.7 | **50.7** | 44.3 |
|
| 42 |
+
| MMMU<sub>val</sub> | 39.6 | 34.9 | 43.6 | 41.7 | **47.4** |
|
| 43 |
+
| ScienceQA<sub>test</sub> | 80.4 | 94.1 | 86.6 | 78.1 | **95.2** |
|
| 44 |
+
| AI2D<sub>test</sub> | 64.8 | 74.4 | 74.2 | 74.6 | **75.0** |
|
| 45 |
+
| MathVista<sub>testmini</sub> | 39.0 | 45.0 | 47.1 | 47.9 | **59.0** |
|
| 46 |
+
| MathVerse<sub>testmini</sub> | 19.8 | 24.7 | 22.2 | 21.0 | **26.2** |
|
| 47 |
+
| MathVision | 15.4 | 12.6 | 16.3 | 17.5 | **18.4** |
|
| 48 |
+
| DocVQA<sub>test</sub> | 71.0 | 86.9 | 87.6 | **89.9** | 85.0 |
|
| 49 |
+
| InfoVQA<sub>test</sub> | 40.0 | 59.5 | 59.1 | **65.4** | 58.3 |
|
| 50 |
+
| ChartQA<sub>test</sub> | 59.6 | 71.4 | 57.1 | 73.5 | **76.5** |
|
| 51 |
+
| TextVQA<sub>val</sub> | 74.3 | 73.5 | 77.6 | **79.9** | 76.4 |
|
| 52 |
+
| OCRVQA<sub>testcore</sub> | 54.4 | 40.2 | 67.6 | **68.7** | 64.0 |
|
| 53 |
+
| VCR<sub>en easy</sub> | 27.6 | 51.6 | 67.7 | 68.3 | **70.0** |
|
| 54 |
+
| OCRBench | 613 | 784 | 782 | **810** | 772 |
|
| 55 |
+
| Average | 53.5 | 58.8 | 60.9 | 62.1 | **64.1** |
|
| 56 |
|
| 57 |
|
| 58 |
|
|
|
|
| 62 |
|
| 63 |
* We plan to train models of various sizes.
|
| 64 |
* Future training will incorporate multi-image and video data.
|
| 65 |
+
<!-- * A comprehensive technical report will be released. -->
|
| 66 |
+
<!-- * We will open-source the Infinity-MM dataset and training code. -->
|
| 67 |
+
|
| 68 |
|
| 69 |
# Disclaimer
|
| 70 |
+
The resources, including code, data, and model weights, associated with this project are restricted for academic research purposes only and cannot be used for commercial purposes. The content produced the model is influenced by uncontrollable variables such as randomness, and therefore, the accuracy of the output cannot be guaranteed by this project. This project does not accept any legal liability for the content of the model output, nor does it assume responsibility for any losses incurred due to the use of associated resources and output results.[]()
|