| <br> | |
| <p align="center"> | |
| <img src="assets/logo.jpg" width="400"/> | |
| <p> | |
| <br> | |
| <p align="center"> | |
| Qwen-VL <a href="https://modelscope.cn/models/qwen/Qwen-VL/summary">🤖 <a> | <a href="https://huggingface.co/Qwen/Qwen-VL">🤗</a>  ｜ Qwen-VL-Chat <a href="https://modelscope.cn/models/qwen/Qwen-VL-Chat/summary">🤖 <a>| <a href="https://huggingface.co/Qwen/Qwen-VL-Chat">🤗</a>  ｜  <a href="https://modelscope.cn/studios/qwen/Qwen-VL-Chat-Demo/summary">Demo</a>  ｜  <a href="https://github.com/QwenLM/Qwen-VL/blob/main/visual_memo.md">Report</a>   |   <a href="https://discord.gg/9bjvspyu">Discord</a> | |
| </p> | |
| <br> | |
| **Qwen-VL** (Qwen Large Vision Language Model) is the visual multimodal version of the large model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-VL accepts image, text, and bounding box as inputs, outputs text and bounding box. The features of Qwen-VL include: | |
| - **Strong performance**: It significantly surpasses existing open-source Large Vision Language Models (LVLM) under similar scale settings on multiple English evaluation benchmarks (including Zero-shot caption, VQA, DocVQA, and Grounding). | |
| - **Multi-lingual LVLM support text recognization**: Qwen-VL naturally supports multi-lingual conversation, and it promotes end-to-end recognition of Chinese and English bi-lingual text in images. | |
| - **Multi-image interleaved conversations**: This feature allows for the input and comparison of multiple images, as well as the ability to specify questions related to the images and engage in multi-image storytelling. | |
| - **First generalist model support grounding in Chinese**: Detecting bounding boxes through open-domain language expression in both Chinese and English. | |
| - **Fine-grained recognization and understanding**: Compared to the 224 resolution currently used by other open-source LVLM, the 448 resolution promotes fine-grained text recognition, document QA, and bounding box annotation. | |
| We release two models of the Qwen-VL series: | |
| - Qwen-VL: The pre-trained LVLM model uses Qwen-7B as the initialization of the LLM, and [Openclip ViT-bigG](https://github.com/mlfoundations/open_clip) as the initialization of the visual encoder. And connects them with a randomly initialized cross-attention layer. Qwen-VL was trained on about 1.5B image-text paired data. The final image input resolution is 448. | |
| - Qwen-VL-Chat: A multimodal LLM-based AI assistant, which is trained with alignment techniques. | |
| For more details about Qwen-VL, please refer to our [technical memo](visual_memo.md). | |
| ## Evaluation | |
| We evaluated the model's ability from two perspectives: | |
| 1. **Standard Benchmarks**: We evaluate the model's basic task capabilities on four major categories of multimodal tasks: | |
| - Zero-shot Caption: Evaluate model's zero-shot image captioning ability on unseen datasets; | |
| - General VQA: Evaluate the general question-answering ability of pictures, such as the judgment, color, number, category, etc; | |
| - Text-based VQA: Evaluate the model's ability to recognize text in pictures, such as document QA, chart QA, etc; | |
| - Referring Expression Comprehension: Evaluate the ability to localize a target object in an image described by a referring expression. | |
| 2. **TouchStone**: To evaluate the overall text-image dialogue capability and alignment level with humans, we have constructed a benchmark called TouchStone, which is based on scoring with GPT4 to evaluate the LVLM model. | |
| - The TouchStone benchmark covers a total of 300+ images, 800+ questions, and 27 categories. Such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc; | |
| - In order to break the current limitation of GPT4 in terms of direct image input, TouchStone provides fine-grained image annotations by human labeling. These detailed annotations, along with the questions and the model's output, are then presented to GPT4 for scoring. | |
| - The benchmark includes both English and Chinese versions. | |
| The results of the evaluation are as follows: | |
| Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has a more comprehensive coverage in terms of capability range. | |
| <p align="center"> | |
| <img src="assets/radar.png" width="600"/> | |
| <p> | |
| ### Zero-shot Caption & General VQA | |
| <table> | |
| <thead> | |
| <tr> | |
| <th rowspan="2">Model type</th> | |
| <th rowspan="2">Model</th> | |
| <th colspan="2">Zero-shot Caption</th> | |
| <th colspan="5">General VQA</th> | |
| </tr> | |
| <tr> | |
| <th>NoCaps</th> | |
| <th>Flickr30K</th> | |
| <th>VQAv2<sup>dev</sup></th> | |
| <th>OK-VQA</th> | |
| <th>GQA</th> | |
| <th>SciQA-Img<br>(0-shot)</th> | |
| <th>VizWiz<br>(0-shot)</th> | |
| </tr> | |
| </thead> | |
| <tbody align="center"> | |
| <tr> | |
| <td rowspan="12">Generalist<br>Models</td> | |
| <td>Flamingo-9B</td> | |
| <td>-</td> | |
| <td>61.5</td> | |
| <td>51.8</td> | |
| <td>44.7</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>28.8</td> | |
| </tr> | |
| <tr> | |
| <td>Flamingo-80B</td> | |
| <td>-</td> | |
| <td>67.2</td> | |
| <td>56.3</td> | |
| <td>50.6</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>31.6</td> | |
| </tr> | |
| <tr> | |
| <td>Unified-IO-XL</td> | |
| <td>100.0</td> | |
| <td>-</td> | |
| <td>77.9</td> | |
| <td>54.0</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td>Kosmos-1</td> | |
| <td>-</td> | |
| <td>67.1</td> | |
| <td>51.0</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>29.2</td> | |
| </tr> | |
| <tr> | |
| <td>Kosmos-2</td> | |
| <td>-</td> | |
| <td>66.7</td> | |
| <td>45.6</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td>BLIP-2 (Vicuna-13B)</td> | |
| <td>103.9</td> | |
| <td>71.6</td> | |
| <td>65.0</td> | |
| <td>45.9</td> | |
| <td>32.3</td> | |
| <td>61.0</td> | |
| <td>19.6</td> | |
| </tr> | |
| <tr> | |
| <td>InstructBLIP (Vicuna-13B)</td> | |
| <td><strong>121.9</strong></td> | |
| <td>82.8</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>49.5</td> | |
| <td>63.1</td> | |
| <td>33.4</td> | |
| </tr> | |
| <tr> | |
| <td>Shikra (Vicuna-13B)</td> | |
| <td>-</td> | |
| <td>73.9</td> | |
| <td>77.36</td> | |
| <td>47.16</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td><strong>Qwen-VL (Qwen-7B)</strong></td> | |
| <td>121.4</td> | |
| <td><b>85.8</b></td> | |
| <td><b>78.8</b></td> | |
| <td><b>58.6</b></td> | |
| <td><b>59.3</b></td> | |
| <td><b>67.1</b></td> | |
| <td><b>34.3</b></td> | |
| </tr> | |
| <tr> | |
| <td>Qwen-VL (4-shot)</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>63.6</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>39.1</td> | |
| </tr> | |
| <tr> | |
| <td>Qwen-VL-Chat</td> | |
| <td>-</td> | |
| <td>81.5</td> | |
| <td>-</td> | |
| <td>56.69</td> | |
| <td>-</td> | |
| <td>68.22</td> | |
| <td>37.05</td> | |
| </tr> | |
| <tr> | |
| <td>Qwen-VL-Chat (4-shot)</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>60.6</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>45.5</td> | |
| </tr> | |
| <tr> | |
| <td>Previous SOTA<br>(Per Task Fine-tuning)</td> | |
| <td>-</td> | |
| <td>127.0<br>(PALI-17B)</td> | |
| <td>84.5<br>(InstructBLIP<br>-FlanT5-XL)</td> | |
| <td>86.1<br>(PALI-X<br>-55B)</td> | |
| <td>66.1<br>(PALI-X<br>-55B)</td> | |
| <td>72.1<br>(CFR)</td> | |
| <td>92.53<br>(LLaVa+<br>GPT-4)</td> | |
| <td>70.9<br>(PALI-X<br>-55B)</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| - For zero-shot image captioning, Qwen-VL achieves the **SOTA** on Flickr30K and competitive results on Nocaps with InstructBlip. | |
| - For general VQA, Qwen-VL achieves the **SOTA** under the same generalist LVLM scale settings. | |
| ### Text-based VQA (focuse on text understanding capabilities in images) | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Model type</th> | |
| <th>Model</th> | |
| <th>TextVQA</th> | |
| <th>DocVQA</th> | |
| <th>ChartQA</th> | |
| <th>AI2D</th> | |
| <th>OCR-VQA</th> | |
| </tr> | |
| </thead> | |
| <tbody align="center"> | |
| <tr> | |
| <td rowspan="5">Generalist Models</td> | |
| <td>BLIP-2 (Vicuna-13B)</td> | |
| <td>42.4</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td>InstructBLIP (Vicuna-13B)</td> | |
| <td>50.7</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td>mPLUG-DocOwl (LLaMA-7B)</td> | |
| <td>52.6</td> | |
| <td>62.2</td> | |
| <td>57.4</td> | |
| <td>-</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td>Pic2Struct-Large (1.3B)</td> | |
| <td>-</td> | |
| <td><b>76.6</b></td> | |
| <td>58.6</td> | |
| <td>42.1</td> | |
| <td>71.3</td> | |
| </tr> | |
| <tr> | |
| <td>Qwen-VL (Qwen-7B)</td> | |
| <td><b>63.8</b></td> | |
| <td>65.1</td> | |
| <td><b>65.7</b></td> | |
| <td><b>62.3</b></td> | |
| <td><b>75.7</b></td> | |
| </tr> | |
| <tr> | |
| <td>Specialist SOTAs<br>(Specialist/Finetuned)</td> | |
| <td>PALI-X-55B (Single-task FT)<br>(Without OCR Pipeline)</td> | |
| <td>71.44</td> | |
| <td>80.0</td> | |
| <td>70.0</td> | |
| <td>81.2</td> | |
| <td>75.0</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| - In text-related recognition/QA evaluation, Qwen-VL achieves the SOTA under the generalist LVLM scale settings. | |
| - Resolution is important for several above evaluations. While most open-source LVLM models with 224 resolution are incapable of these evaluations or can only solve these by cutting images, Qwen-VL scales the resolution to 448 so that it can be evaluated end-to-end. Qwen-VL even outperforms Pic2Struct-Large models of 1024 resolution on some tasks. | |
| ### Referring Expression Comprehension | |
| <table> | |
| <thead> | |
| <tr> | |
| <th rowspan="2">Model type</th> | |
| <th rowspan="2">Model</th> | |
| <th colspan="3">RefCOCO</th> | |
| <th colspan="3">RefCOCO+</th> | |
| <th colspan="2">RefCOCOg</th> | |
| <th>GRIT</th> | |
| </tr> | |
| <tr> | |
| <th>val</th> | |
| <th>test-A</th> | |
| <th>test-B</th> | |
| <th>val</th> | |
| <th>test-A</th> | |
| <th>test-B</th> | |
| <th>val-u</th> | |
| <th>test-u</th> | |
| <th>refexp</th> | |
| </tr> | |
| </thead> | |
| <tbody align="center"> | |
| <tr> | |
| <td rowspan="8">Generalist Models</td> | |
| <td>GPV-2</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>51.50</td> | |
| </tr> | |
| <tr> | |
| <td>OFA-L*</td> | |
| <td>79.96</td> | |
| <td>83.67</td> | |
| <td>76.39</td> | |
| <td>68.29</td> | |
| <td>76.00</td> | |
| <td>61.75</td> | |
| <td>67.57</td> | |
| <td>67.58</td> | |
| <td>61.70</td> | |
| </tr> | |
| <tr> | |
| <td>Unified-IO</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td><b>78.61</b></td> | |
| </tr> | |
| <tr> | |
| <td>VisionLLM-H</td> | |
| <td></td> | |
| <td>86.70</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td>Shikra-7B</td> | |
| <td>87.01</td> | |
| <td>90.61</td> | |
| <td>80.24 </td> | |
| <td>81.60</td> | |
| <td>87.36</td> | |
| <td>72.12</td> | |
| <td>82.27</td> | |
| <td>82.19</td> | |
| <td>69.34</td> | |
| </tr> | |
| <tr> | |
| <td>Shikra-13B</td> | |
| <td>87.83 </td> | |
| <td>91.11</td> | |
| <td>81.81</td> | |
| <td>82.89</td> | |
| <td>87.79</td> | |
| <td>74.41</td> | |
| <td>82.64</td> | |
| <td>83.16</td> | |
| <td>69.03</td> | |
| </tr> | |
| <tr> | |
| <td>Qwen-VL-7B</td> | |
| <td><b>89.36</b></td> | |
| <td>92.26</td> | |
| <td><b>85.34</b></td> | |
| <td><b>83.12</b></td> | |
| <td>88.25</td> | |
| <td><b>77.21</b></td> | |
| <td><b>85.58</b></td> | |
| <td><b>85.48</b></td> | |
| <td>78.22</td> | |
| </tr> | |
| <tr> | |
| <td>Qwen-VL-7B-Chat</td> | |
| <td><b>88.55</b></td> | |
| <td><b>92.27</b></td> | |
| <td>84.51</td> | |
| <td>82.82</td> | |
| <td><b>88.59</b></td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td rowspan="3">Specialist SOTAs<br>(Specialist/Finetuned)</td> | |
| <td>G-DINO-L</td> | |
| <td>90.56 </td> | |
| <td>93.19</td> | |
| <td>88.24</td> | |
| <td>82.75</td> | |
| <td>88.95</td> | |
| <td>75.92</td> | |
| <td>86.13</td> | |
| <td>87.02</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td>UNINEXT-H</td> | |
| <td>92.64 </td> | |
| <td>94.33</td> | |
| <td>91.46</td> | |
| <td>85.24</td> | |
| <td>89.63</td> | |
| <td>79.79</td> | |
| <td>88.73</td> | |
| <td>89.37</td> | |
| <td>-</td> | |
| </tr> | |
| <tr> | |
| <td>ONE-PEACE</td> | |
| <td>92.58 </td> | |
| <td>94.18</td> | |
| <td>89.26</td> | |
| <td>88.77</td> | |
| <td>92.21</td> | |
| <td>83.23</td> | |
| <td>89.22</td> | |
| <td>89.27</td> | |
| <td>-</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| - Qwen-VL achieves the **SOTA** in all above referring expression comprehension benchmarks. | |
| - Qwen-VL has not been trained on any Chinese grounding data, but it can still generalize to the Chinese Grounding tasks in a zero-shot way by training Chinese Caption data and English Grounding data. | |
| We provide all of the above evaluation scripts for reproducing our experimental results. Please read [eval/EVALUATION.md](eval/EVALUATION.md) for more information. | |
| ### Chat evaluation | |
| TouchStone is a benchmark based on scoring with GPT4 to evaluate the abilities of the LVLM model on text-image dialogue and alignment levels with humans. It covers a total of 300+ images, 800+ questions, and 27 categories, such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc. Please read [touchstone/README_CN.md](touchstone/README.md) for more information. | |
| #### English evaluation | |
| | Model | Score | | |
| |---------------|-------| | |
| | PandaGPT | 488.5 | | |
| | MiniGPT4 | 531.7 | | |
| | InstructBLIP | 552.4 | | |
| | LLaMA-AdapterV2 | 590.1 | | |
| | mPLUG-Owl | 605.4 | | |
| | LLaVA | 602.7 | | |
| | Qwen-VL-Chat | 645.2 | | |
| #### Chinese evaluation | |
| | Model | Score | | |
| |---------------|-------| | |
| | VisualGLM | 247.1 | | |
| | Qwen-VL-Chat | 401.2 | | |
| Qwen-VL-Chat has achieved the best results in both Chinese and English alignment evaluation. | |
| ## Requirements | |
| * python 3.8 and above | |
| * pytorch 1.12 and above, 2.0 and above are recommended | |
| * CUDA 11.4 and above are recommended (this is for GPU users) | |
| ## Quickstart | |
| Below, we provide simple examples to show how to use Qwen-VL and Qwen-VL-Chat with 🤖 ModelScope and 🤗 Transformers. | |
| Before running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries. | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| Now you can start with ModelScope or Transformers. More usage aboue vision encoder, please refer to [FAQ](FAQ.md). | |
| #### 🤗 Transformers | |
| To use Qwen-VL-Chat for the inference, all you need to do is to input a few lines of codes as demonstrated below. However, **please make sure that you are using the latest code.** | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| from transformers.generation import GenerationConfig | |
| import torch | |
| torch.manual_seed(1234) | |
| # Note: The default behavior now has injection attack prevention off. | |
| tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-VL-Chat", trust_remote_code=True) | |
| # use bf16 | |
| # model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat", device_map="auto", trust_remote_code=True, bf16=True).eval() | |
| # use fp16 | |
| # model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval() | |
| # use cpu only | |
| # model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat", device_map="cpu", trust_remote_code=True).eval() | |
| # use cuda device | |
| model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat", device_map="cuda", trust_remote_code=True).eval() | |
| # Specify hyperparameters for generation | |
| model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-VL-Chat", trust_remote_code=True) | |
| # 1st dialogue turn | |
| query = tokenizer.from_list_format([ | |
| {'image': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'}, | |
| {'text': '这是什么'}, | |
| ]) | |
| response, history = model.chat(tokenizer, query=query, history=None) | |
| print(response) | |
| # 图中是一名年轻女子在沙滩上和她的狗玩耍，狗的品种可能是拉布拉多。她们坐在沙滩上，狗的前腿抬起来，似乎在和人类击掌。两人之间充满了信任和爱。 | |
| # 2st dialogue turn | |
| response, history = model.chat(tokenizer, '输出"击掌"的检测框', history=history) | |
| print(response) | |
| # <ref>击掌</ref><box>(517,508),(589,611)</box> | |
| image = tokenizer.draw_bbox_on_latest_picture(response, history) | |
| if image: | |
| image.save('1.jpg') | |
| else: | |
| print("no box") | |
| ``` | |
| <p align="center"> | |
| <img src="assets/demo_highfive.jpeg" width="500"/> | |
| <p> | |
| ## FAQ | |
| If you meet problems, please refer to [FAQ](FAQ.md) and the issues first to search a solution before you launch a new issue. | |
| ## License Agreement | |
| Researchers and developers are free to use the codes and model weights of both Qwen-7B and Qwen-7B-Chat. We also allow their commercial use. Check our license at [LICENSE](LICENSE) for more details. | |
| ## Contact Us | |
| If you are interested to leave a message to either our research team or product team, feel free to send an email to qianwen[email protected]. | |