Improve model card and add metadata

README.md

@@ -1,20 +1,20 @@
 ---
-license: cc
+base_model:
+- Qwen/Qwen2.5-1.5B-Instruct
+- google/siglip-so400m-patch14-384
 datasets:
 - weizhiwang/Open-Qwen2VL-Data
 - MAmmoTH-VL/MAmmoTH-VL-Instruct-12M
 language:
 - en
-base_model:
-- Qwen/Qwen2.5-1.5B-Instruct
-- google/siglip-so400m-patch14-384
+license: cc
 pipeline_tag: image-text-to-text
+library_name: transformers
 ---
 
 # Model Card for Open-Qwen2VL
 
-<!-- Provide a quick summary of what the model is/does. -->
-
+Open-Qwen2VL is a multimodal model that takes images and text as input and produces text as output.  This model is described in the paper [Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources](https://huggingface.co/papers/2504.00595).  The code is available at [https://github.com/Victorwz/Open-Qwen2VL](https://github.com/Victorwz/Open-Qwen2VL).
 
 <!-- Please follow my reproduced implementation [LLaVA-Unified](https://github.com/Victorwz/LLaVA-Unified) for more details on fine-tuning LLaVA model with Llama-3 as the foundatiaon LLM. -->
 
@@ -48,7 +48,8 @@ vlm.to(device, dtype=torch.bfloat16)
 image_url = "https://huggingface.co/adept/fuyu-8b/resolve/main/bus.png"
 # image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
 image = [vlm.vision_backbone.image_transform(Image.open(requests.get(image_url, stream=True).raw).convert("RGB")).unsqueeze(0)]
-user_prompt = '<image>' + '\n' + "Describe the image."
+user_prompt = '<image>' + '
+' + "Describe the image."
 
 # Generate!
 generated_text = vlm.generate_batch(
@@ -62,36 +63,14 @@ print(generated_text[0])
 ```
 The image caption results look like:
 ```
-The image depicts a blue and orange bus parked on the side of a street. The bus is a modern, single-decker model, and it is prominently displayed in the foreground. The bus has a digital display at the front, showing the route number "616" and the destination "Morton Best." The bus is labeled with the logo of "Stagecoach," a well-known bus operator in the UK. The bus also features a slogan on its side that reads "Stronger for you," indicating a focus on providing reliable and efficient public transportation.
-
-The bus is parked on a grassy area adjacent to a sidewalk. The grass is well-maintained, and there are a few trees planted along the sidewalk, providing some shade. The sidewalk itself is made of concrete and appears to be clean and well-kept. 
-
-In the background, there are residential buildings, which are typical of a suburban area. These buildings have pitched roofs and are constructed with a mix of brick and plaster. The architecture suggests a typical British suburban neighborhood. The sky above is partly cloudy, with patches of blue sky visible, indicating fair weather.
-
-The bus is parked in a designated bus stop area, as indicated by the presence of a bus stop sign and a bus stop shelter, although the shelter is not visible in the image. The bus's license plate is visible and reads "Y600 HJX." The overall scene suggests a typical day in a suburban area where public transportation is readily available for residents.
-
-The image does not contain any people, vehicles other than the bus, or any other notable objects. The focus is primarily on the bus and its surroundings. The bus appears to be in good condition, indicating that it is well-maintained and likely in regular use. The presence of the bus stop and the well-kept environment suggest that the area is well-organized and that public transportation is an integral part of the community's daily life.
+The image depicts a blue and orange bus parked on the side of a street. ...
 ```
 
-<!-- # Fine-Tune LLaVA-Llama-3 on Your Visual Instruction Data
-Please refer to our [LLaVA-Unified](https://github.com/Victorwz/LLaVA-Unified) git repo for fine-tuning data preparation and scripts. The data loading function and fastchat conversation template are changed due to a different tokenizer.
-
-## Benchmark Results
-
-
-| Model                 |      MMMU Val     |
-| :-------------------- | :---------------: |
-| LLaVA-v1.5-7B         |       35.3        |
-| LLaVA-Llama-3-8B      |       36.7        | 
-
-Please refer to `eval_outputs/LLaVA-Llama-3-8B_mmmu_val.json` for reproduce the benchmark performance on MMMU validation set. -->
+<!-- # Fine-Tune LLaVA-Llama-3 on Your Visual Instruction Data ... -->
 
 ## Citation
 <!-- 
 ```bibtex
 @misc{wang2024llavallama3,
-  title={LLaVA-Llama-3-8B: A reproduction towards LLaVA-v1.5 based on Llama-3-8B LLM backbone},
-  author={Wang, Weizhi},
-  year={2024}
-}
-``` -->
+...
+``` -->