--- license: apache-2.0 language: - en - zh base_model: - Qwen/Qwen2.5-7B-Instruct --- ## Introduction ![lidar_map](statics/sail.png) SAIL-VL is a state-of-the-art vision-language model (VLM) developed by the Bytedance Douyin Content Team. The goal of SAIL-VL is to develope a high-performance vision language model that facilitates deployment on mobile devices and ensures accessibility and affordability for a broad audience. Through careful tuning of data and training recipes, SAIL-VL demonstrates that even a small VLM can benefit significantly from data scaling. Our model outperforms Qwen2-VL, InternVL2.5-MPO and even recent SoTA models of comparable sizes. In a word, SAIL-VL is a foundational VLM for vision-language applications. Welcome to explore its capabilities and feel free to contact us for any questions or opportunities. ## News🚀🚀🚀 - 2024-2-19: 📖 We released our 8B model, check out at [🤗SAIL-VL-8B](https://huggingface.co/BytedanceDouyinContent/SAIL-VL-8B) ~ - 2024-1-10: 📖 We released our paper on Arxiv: [Scalable Vision Language Model Training via High Quality Data Curation ](https://arxiv.org/abs/2501.05952) - 2024-12-25: 🚀 We ranked the 1st in [OpenCompass Multi-modal Leaderboard](https://rank.opencompass.org.cn/leaderboard-multimodal/?m=REALTIME) among models of 2B parameters. ## Model Card ### Model Architecture: | Architecture | ViT | LLM | Adapter | Token Merge | Resolution | | --- | --- | --- | --- | --- | --- | | [🤗SAIL-VL-2B](https://huggingface.co/BytedanceDouyinContent/SAIL-VL-2B) | [🤗InternViT-300M](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [🤗Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) | 2-layer MLP | 2x2 | 448x448xN | | [🤗SAIL-VL-8B](https://huggingface.co/BytedanceDouyinContent/SAIL-VL-8B) | [🤗InternViT-300M](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [🤗Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | 2-layer MLP | 2x2 | 448x448xN | ### Training Recipes Overview: Sail-VL benefits from high-quality data and carefully curated training recipes. We find the data quality, quantity and the design of curriculum training pipeline are crucial for model performance. With the proper design and data, the model's capacity scales effectively with data expansion at all stages, leading to enhanced performance. ![](statics/paper_page.png) ## Evaluation SAIL-VL is competitive compared with Qwen2-VL, DeepSeekVL-2 and recently released InternVL2.5-MPO, please see the following table for details. ### Detail Evaluations: | Benchmark | **SAIL-VL-8B** | Qwen2-VL-8B | InternVL2.5-MPO-8B | DeepSeekVL-2-Small | | --- | --- | --- | --- | --- | | **Overall Performance** | *74.5* | *73.0* | *74.3* | *72.7* | | **General VQA** | *68.3* | *68.5* | *71.2* | *66.8* | | **OCR VQA** | *79.8* | *79.6* | *76.3* | *79.0* | | **Math&Knowledge** | *83.3* | *71.0* | *83.2* | *79.0* | | **Hallucination** | *68.7* | *67.5* | *69.7* | *65.3* | | **General VQA** | | | | | | MMStar | 64.2 | 58.3 | 65.3 | 57.7 | | MMBench_DEV | 79.5 | 79.5 | 83.3 | 78.1 | | MMMU_VAL | 48.2 | 50.9 | 52.8 | 47.6 | | MME | 2244 | 2321 | 2321 | 2149 | | SEEDBench_IMG | 75.5 | 75.3 | 76.9 | 76.8 | | RealWorldQA | 71.9 | 69.7 | 70.2 | 70.2 | | MMVET | 58.3 | 62.6 | 66.8 | 60.3 | | **OCR VQA** | | | | | | AI2D_TEST | 83.7 | 82.9 | 84.1 | 82.0 | | DocVQA_Val | 92.2 | 93.7 | 92.1 | 92.3 | | InfoVQA_Val | 75.2 | 75.9 | 76.2 | 72.5 | | ChartQA_Test | 84.6 | 81.6 | 77.6 | 84.6 | | TextVQA_Val | 77.7 | 83.8 | 79.2 | 83.3 | | OCRVQA_Test | 61.4 | 56.2 | 36.7 | 54.5 | | OCRBench | 835 | 833 | 880 | 834 | | **Math&Knowledge** | | | | | | MathVistaMini | 68.4 | 57.3 | 68.5 | 61.8 | | ScienceQA_Val | 98.2 | 84.6 | 97.9 | 96.2 | | **Hallucination** | | | | | | HallucinationBench | 52.2 | 48.5 | 50.3 | 41.2 | | POPE | 85.2 | 86.5 | 89.1 | 89.4 | ## Demo Cases We visualize some examples to show the capabilities of our model. Our model is able to give detail and complex answer for a variety of questions. | Image | Question | Answer | | --- | --- | --- | | ![](statics/8b-1.jpg) | Please describe the image shortly.| The image prominently features the Golden Gate Bridge, an iconic suspension bridge in San Francisco, California. It spans across a body of water, presumably the San Francisco Bay. The photograph captures the bridge in a clear day with minimal cloud cover. Visible in the image's lower part is the adjacent cityscape and landscape, including parts of a building on the left side which looks like a low-lying brick structure, likely part of nearby infrastructure or a historical site in San Francisco. | | ![](statics/8b-2.jpeg) | Why is this meme funny? | This meme is funny because it cleverly plays on the contrast between the human's focused strategy and the dog's random play. The image juxtaposes a child intensely trying to predict the dog's next move with the dog itself, which shows a checkers board and the caption "Me playing it on random." It humorously suggests that the dog, despite its seemingly haphazard approach, is the true master of the game. This creates a comical situation where a human child is perplexed by the dog's unpredictable moves, while the dog seems completely at ease with its own casual, random play style. | | ![](statics/8b-3.jpg) | Do you recognize this place ? | Yes, I recognize the place. This is the Temple of Heaven, located in Beijing, China. It's a famous historical site and architectural gem known for its distinctive appearance and cultural significance. The Temple of Heaven is renowned for its circular shape and beautiful architecture, combining elements of East Asian design with unique features that set it apart from other Chinese temple structures. It was originally built in the early 15th century during the Ming Dynasty and has since become an iconic representation of Chinese architecture and imperial history. | | ![](statics/8b-3.jpg) | Help me write a travel plan for this place. | Travel Plan for Temple of Heaven:\n 1. Morning: Depart early to avoid crowds.\n 2. Arrive at Temple of Heaven, explore the main structure, learn about its significance.\n 3. Take a guided tour of the surrounding gardens and historical sites.\n 4. Enjoy an authentic lunch in the vicinity. \n 5. Continue exploration, visit the Prayer Hall. \n 6. Visit during sunset for a serene experience. \n 7. End with evening meals showcasing local cuisine. \n 8. Take time to reflect on the experience as you leave. | | ![](statics/8b-4.jpeg) | Introduce me this painting. | This iconic artwork is "Starry Night," painted by Vincent van Gogh in 1889. The canvas captures a swirling night sky filled with vibrant blue and yellow hues, giving it a mesmerizing, dream-like quality. Van Gogh's bold, expressive brushstrokes bring the night scene to life, creating a sense of movement and energy that captivates viewers. Below the ethereal sky, a small village and church anchor the composition, adding a touch of human presence to the vast, celestial expanse. This masterpiece exemplifies Van Gogh's unique style, showcasing his deep emotional connection to nature and the universe. | ## How to Use The basic usage and dynamic crop strategy of SAIL-VL follows InternVL2, you can easily switch Intern-VL series of models to our model. Here is a simple example of using our model: ### Requirements: ``` pip3 install einops transformers timm ``` ### Code: ```Python import numpy as np import torch import torchvision.transforms as T from PIL import Image from torchvision.transforms.functional import InterpolationMode from transformers import AutoModel, AutoTokenizer IMAGENET_MEAN = (0.485, 0.456, 0.406) IMAGENET_STD = (0.229, 0.224, 0.225) def build_transform(input_size): MEAN, STD = IMAGENET_MEAN, IMAGENET_STD transform = T.Compose([ T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img), T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC), T.ToTensor(), T.Normalize(mean=MEAN, std=STD) ]) return transform def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size): best_ratio_diff = float('inf') best_ratio = (1, 1) area = width * height for ratio in target_ratios: target_aspect_ratio = ratio[0] / ratio[1] ratio_diff = abs(aspect_ratio - target_aspect_ratio) if ratio_diff < best_ratio_diff: best_ratio_diff = ratio_diff best_ratio = ratio elif ratio_diff == best_ratio_diff: if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]: best_ratio = ratio return best_ratio def dynamic_preprocess(image, min_num=1, max_num=10, image_size=448, use_thumbnail=False): orig_width, orig_height = image.size aspect_ratio = orig_width / orig_height # calculate the existing image aspect ratio target_ratios = set( (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if i * j <= max_num and i * j >= min_num) target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1]) # find the closest aspect ratio to the target target_aspect_ratio = find_closest_aspect_ratio( aspect_ratio, target_ratios, orig_width, orig_height, image_size) # calculate the target width and height target_width = image_size * target_aspect_ratio[0] target_height = image_size * target_aspect_ratio[1] blocks = target_aspect_ratio[0] * target_aspect_ratio[1] # resize the image resized_img = image.resize((target_width, target_height)) processed_images = [] for i in range(blocks): box = ( (i % (target_width // image_size)) * image_size, (i // (target_width // image_size)) * image_size, ((i % (target_width // image_size)) + 1) * image_size, ((i // (target_width // image_size)) + 1) * image_size ) # split the image split_img = resized_img.crop(box) processed_images.append(split_img) assert len(processed_images) == blocks if use_thumbnail and len(processed_images) != 1: thumbnail_img = image.resize((image_size, image_size)) processed_images.append(thumbnail_img) return processed_images def load_image(image_file, input_size=448, max_num=10): image = Image.open(image_file).convert('RGB') transform = build_transform(input_size=input_size) images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num) pixel_values = [transform(image) for image in images] pixel_values = torch.stack(pixel_values) return pixel_values path = "BytedanceDouyinContent/SAIL-VL-8B" model = AutoModel.from_pretrained( path, torch_dtype=torch.bfloat16, trust_remote_code=True).eval().cuda() tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False) # set the max number of tiles in `max_num` pixel_values = load_image('./test.png', max_num=10).to(torch.bfloat16).cuda() generation_config = dict(max_new_tokens=1024, do_sample=True) # pure-text conversation question = 'Hello, who are you?' response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True) print(f'User: {question} Assistant: {response}') question = 'Can you tell me a story?' response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True) print(f'User: {question} Assistant: {response}') # single-image single-round conversation question = ' Please describe the image shortly.' response = model.chat(tokenizer, pixel_values, question, generation_config) print(f'User: {question} Assistant: {response}') # single-image multi-round conversation question = ' Please describe the image in detail.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True) print(f'User: {question} Assistant: {response}') question = 'Please write a poem according to the image.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True) print(f'User: {question} Assistant: {response}') ```
## Acknowledge Our model is built upon numerous outstanding open-source projects, and we are grateful for their contributions. We extend special thanks to the InternVL team and Qwen team for their great base models, and to the BAAI team (Infinity-MM) for their generous release of data. ## Citation ``` @article{dong2025scalable, title={Scalable vision language model training via high quality data curation}, author={Dong, Hongyuan and Kang, Zijian and Yin, Weijie and Liang, Xiao and Feng, Chao and Ran, Jiao}, journal={arXiv preprint arXiv:2501.05952}, year={2025} } ``` ## Contributions This work is conducted by Bytedance Douyin Content Team, authored by: ``` {Hongyuan Dong, Zijian Kang, Weijie Yin}, Xiao Liang, Chao Feng, Jiao Ran {*} Equal Contributions. ``` We also appreciate the support from the model evaluation team: ``` Zirui Guo, Yan Qiu, Yaling Mou, Ming Jiang ``` And from AI platform team: ``` Huiyu Yu, Lin Dong, Yong Zhang ``` ## License This project is licensed under [Apache License 2.0](LICENSE). ## Contact If you have any question, please feel free to contact us: BytedanceDouyinContent@bytedance.com