File size: 4,891 Bytes
aba1922 c6c887e aba1922 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb
- YanweiLi/MGM-Instruction
language:
- zh
tags:
- vlm
- MLLM
- Multi-Modal LLM
---
<div align="center">
<img src="assets/a.png" />
<h1>Bumblebee</h1>
<h3>Surpassed Close-Source Models: A Breakthrough in Open-Source Advancements</h3>
<p align="center">
weights <a href="https://huggingface.co/MonolithFoundation/Bumblebee">🤗</a> | <a href="https://github.com/MonolithFoundation/Bumblebee/issues/1"> wechat group </a> | <a href="https://github.com/MonolithFoundation/Bumblebee"> code </a>
</p>
</div>
> 👷We are stressly working on the code cleaning and documentation. Stay tuned! (weights are available)
**Bumblebee** is a simple open-source MLLM that has surpassed QwenVL Max on MMBench-Test-CN with a score of **75.8**, using only **open-source data** and **14B LLM parameters**.
Thanks to the utilization of the Qwen2 14B model, Bumblebee guarantees superior Chinese performance and outperforms any other open-source model in terms of understanding.
We have released the weights solely for research purposes, with the hope of accelerating the democratization of the multi-modal open-source community and making AGI with vision more accessible to everyone.
- **Bumblebee has a Simple Design**: It is a 14B LLM-driven multi-modal model, similar to the LLaVA architecture design, incorporating MGM's dual tower visual information input, resulting in enhanced performance in vision tasks.
- **Bumblebee is Fast:** With a simple **576 tokens/per** image input, it surpasses QwenVL Max in Chinese without the need for any image slicing techniques. With this limited token length, Bumblebee can be easily adapted for video understanding.
- **Bumblebee is Evolving:** We are currently working on even more extensive and refined designs for multi-modal models. The next generation is expected to be even more powerful.
## News
- _2024.04.25_: First version **Bumblebee** droped;
## Results
Here are the metrics we have evaluated, although due to limited compute resources and time, we were unable to cover all leaderboards. However, we have focused on the metrics that are more convincing and where cheating is not possible.
| Model | MMB-CN-Test | MMB-EN-Test | MMStar(A) | MMStar(C) | MMStar(F) |
| ---------- | ----------- | ----------- | --------- | --------- | --------- |
| Bumblebee | **75.8** | 76.8 | **43.8** | **63.2** | **41.2** |
| QwenVL Max | 75.7 | 77.6 | 36.5 | 46 | 27.2 |
| Yi-VL-34B | 70.7 | 72.4 | 40.5 | 59.2 | 34 |
| Omini12B | 62 | 71.7 | 39.6 | 63.6 | 32.4 |
| GPT4V-1106 | 77 | 74.4 | 50.4 | 65.2 | 38.8 |
> It is worth noting that Bumblebee is an open-source model and possibly the smallest among these closed-source models.
> Regarding MMStar, as it is a brand new leaderboard, we believe it provides a more fair assessment of a model's true ability, rather than relying solely on potentially cheating leaderboards. Therefore, we give more precise recognition to MMStar here.
The results clearly indicate that Bumblebee has achieved remarkable progress compared to other state-of-the-art open-source models. It has even demonstrated compatibility with certain closed-source models. Notably, on MMbench-CN-Test and MMStar, where cheating is not possible, Bumblebee has surpassed QwenVL Max using only 14B parameters.
## Install
1. Download the weights
2. Get the codes
One can first clone this repo:
```
git clone https://github.com/MonolithFoundation/Bumblebee.git
cd Bumblebee
```
Using MGM like inference code to inference, make sure loading correct vision encoder pretrained weights. (to be added)
## Demo
Impressive Chinese results showed by Bumblebee:
As the result suggested, **Bumblebee** might be one of the most capable Chinese MLLM opensourced as for now.
It shows good abilities at:
- OCR understanding;
- Reasoning;
- Table and Graphic comprehension;
- Math calculate;
If want try it, just download our weights and inference via MGM!
## Future Plan
**Bumblebee** is just the first model release aimed at catching up closed-source models. While it is capable of performing many tasks, it has not yet surpassed closed models by a significant margin.
In our pursuit of advancement, we are currently developing even more advanced technologies that will surpass the current state-of-the-art multimodal models. The path we are paving includes:
- Video and long sequences support, which is our ultimate goal;
- The development of a specialized model that works exclusively with documents, incorporating some innovative techniques.
- Hopefully we can make a new **Optimus Prime** model in the near future.
If you were interested in the new progress, please **star and watch** this repo! |