taochenxin commited on
Commit
4e78b36
·
verified ·
1 Parent(s): d4495bc

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +211 -214
README.md CHANGED
@@ -1,215 +1,212 @@
1
- ---
2
- license: mit
3
- ---
4
- ---
5
- license: mit
6
- pipeline_tag: image-text-to-text
7
- library_name: transformers
8
- base_model:
9
- - internlm/internlm2-chat-1_8b
10
- base_model_relation: merge
11
- language:
12
- - multilingual
13
- tags:
14
- - internvl
15
- - vision-language model
16
- - monolithic
17
- ---
18
- # HoVLE
19
-
20
- [\[📜 HoVLE Paper\]]() [\[🚀 Quick Start\]](#quick-start)
21
-
22
- <a id="radar"></a>
23
-
24
-
25
- ## Introduction
26
-
27
- <p align="middle">
28
- <img src="assets/intro.png" width="95%" />
29
- </p>
30
-
31
- We introduce **HoVLE**, a novel monolithic vision-language model (VLM) that processes images and texts in a unified manner. HoVLE introduces a holistic embedding module that projects image and text inputs into a shared embedding space, allowing the Large Language Model (LLM) to interpret images in the same way as texts.
32
-
33
- HoVLE significantly surpasses previous monolithic VLMs and demonstrates competitive performance with compositional VLMs. This work narrows the gap between monolithic and compositional VLMs, providing a promising direction for the development of monolithic VLMs.
34
-
35
- This repository releases the HoVLE model with 2.6B parameters. It is built upon [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b). Please refer to [HoVLE (HD)](https://huggingface.co/taochenxin/HoVLE-HD) for the high-definition version. For more details, please refer to our [paper]().
36
-
37
-
38
- ## Model Details
39
- <p align="middle">
40
- <img src="assets/overview.png" width="90%" />
41
- </p>
42
-
43
- | | Details |
44
- | :---------------------------: | :---------- |
45
- | Architecture | The whole model consists of a holistic embedding module and an LLM. The holistic embedding module consists of the same causal Transformer layers as the LLM. It accepts both images and texts as input, and projects them into a unified embedding space. These embeddings are then forwarded into the LLM, constituting a monolithic VLM. |
46
- | Stage I (Distillation) | The first stage trains the holistic embedding module to distill the image feature from a pre-trained visual encoder and the text embeddings from an LLM, providing general encoding abilities. Only the holistic embedding module is trainable. |
47
- | Stage II (Alignment) | The second stage combines the holistic embedding module with the LLM to perform auto-regressive training, aligning different modalities to a shared embedding space. Only the holistic embedding module is trainable. |
48
- | Stage III (Instruction Tuning) | A visual instruction tuning stage is incorporated to further strengthen the whole VLM to follow instructions. The whole model is trainable. |
49
-
50
-
51
-
52
- ## Performance
53
- <p align="middle">
54
- <img src="assets/performance1.png" width="90%" />
55
- </p>
56
- <p align="middle">
57
- <img src="assets/performance2.png" width="90%" />
58
- </p>
59
-
60
- - Sources of the results include the original papers, our evaluation with [VLMEvalKit](https://github.com/open-compass/VLMEvalKit), and [OpenCompass](https://rank.opencompass.org.cn/leaderboard-multimodal/?m=REALTIME).
61
- - Please note that evaluating the same model using different testing toolkits can result in slight differences, which is normal. Updates to code versions and variations in environment and hardware can also cause minor discrepancies in results.
62
-
63
-
64
-
65
- Limitations: Although we have made efforts to ensure the safety of the model during the training process and to encourage the model to generate text that complies with ethical and legal requirements, the model may still produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Please do not propagate such content. We are not responsible for any consequences resulting from the dissemination of harmful information.
66
-
67
-
68
-
69
- ## Quick Start
70
-
71
- We provide an example code to run HoVLE inference using `transformers`.
72
-
73
- > Please use transformers==4.37.2 to ensure the model works normally.
74
-
75
-
76
- ### Inference with Transformers
77
-
78
- ```python
79
- import numpy as np
80
- import torch
81
- import torchvision.transforms as T
82
- from decord import VideoReader, cpu
83
- from PIL import Image
84
- from torchvision.transforms.functional import InterpolationMode
85
- from transformers import AutoModel, AutoTokenizer
86
-
87
- IMAGENET_MEAN = (0.485, 0.456, 0.406)
88
- IMAGENET_STD = (0.229, 0.224, 0.225)
89
-
90
- def build_transform(input_size):
91
- MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
92
- transform = T.Compose([
93
- T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
94
- T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
95
- T.ToTensor(),
96
- T.Normalize(mean=MEAN, std=STD)
97
- ])
98
- return transform
99
-
100
- def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
101
- best_ratio_diff = float('inf')
102
- best_ratio = (1, 1)
103
- area = width * height
104
- for ratio in target_ratios:
105
- target_aspect_ratio = ratio[0] / ratio[1]
106
- ratio_diff = abs(aspect_ratio - target_aspect_ratio)
107
- if ratio_diff < best_ratio_diff:
108
- best_ratio_diff = ratio_diff
109
- best_ratio = ratio
110
- elif ratio_diff == best_ratio_diff:
111
- if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
112
- best_ratio = ratio
113
- return best_ratio
114
-
115
- def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
116
- orig_width, orig_height = image.size
117
- aspect_ratio = orig_width / orig_height
118
-
119
- # calculate the existing image aspect ratio
120
- target_ratios = set(
121
- (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
122
- i * j <= max_num and i * j >= min_num)
123
- target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
124
-
125
- # find the closest aspect ratio to the target
126
- target_aspect_ratio = find_closest_aspect_ratio(
127
- aspect_ratio, target_ratios, orig_width, orig_height, image_size)
128
-
129
- # calculate the target width and height
130
- target_width = image_size * target_aspect_ratio[0]
131
- target_height = image_size * target_aspect_ratio[1]
132
- blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
133
-
134
- # resize the image
135
- resized_img = image.resize((target_width, target_height))
136
- processed_images = []
137
- for i in range(blocks):
138
- box = (
139
- (i % (target_width // image_size)) * image_size,
140
- (i // (target_width // image_size)) * image_size,
141
- ((i % (target_width // image_size)) + 1) * image_size,
142
- ((i // (target_width // image_size)) + 1) * image_size
143
- )
144
- # split the image
145
- split_img = resized_img.crop(box)
146
- processed_images.append(split_img)
147
- assert len(processed_images) == blocks
148
- if use_thumbnail and len(processed_images) != 1:
149
- thumbnail_img = image.resize((image_size, image_size))
150
- processed_images.append(thumbnail_img)
151
- return processed_images
152
-
153
- def load_image(image_file, input_size=448, max_num=12):
154
- image = Image.open(image_file).convert('RGB')
155
- transform = build_transform(input_size=input_size)
156
- images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
157
- pixel_values = [transform(image) for image in images]
158
- pixel_values = torch.stack(pixel_values)
159
- return pixel_values
160
-
161
-
162
- path = 'taochenxin/HoVLE/'
163
- model = AutoModel.from_pretrained(
164
- path,
165
- torch_dtype=torch.bfloat16,
166
- low_cpu_mem_usage=True,
167
- trust_remote_code=True).eval().cuda()
168
- tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
169
-
170
- # set the max number of tiles in `max_num`
171
- pixel_values = load_image('./examples_image.jpg', max_num=12).to(torch.bfloat16).cuda()
172
- generation_config = dict(max_new_tokens=1024, do_sample=True)
173
-
174
- # pure-text conversation (纯文本对话)
175
- question = 'Hello, who are you?'
176
- response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
177
- print(f'User: {question}\nAssistant: {response}')
178
-
179
- question = 'Can you tell me a story?'
180
- response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)
181
- print(f'User: {question}\nAssistant: {response}')
182
-
183
- # single-image single-round conversation (单图单轮对话)
184
- question = '<image>\nPlease describe the image shortly.'
185
- response = model.chat(tokenizer, pixel_values, question, generation_config)
186
- print(f'User: {question}\nAssistant: {response}')
187
-
188
- # single-image multi-round conversation (单图多轮对话)
189
- question = '<image>\nPlease describe the image in detail.'
190
- response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
191
- print(f'User: {question}\nAssistant: {response}')
192
-
193
- question = 'Please write a poem according to the image.'
194
- response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
195
- print(f'User: {question}\nAssistant: {response}')
196
-
197
- ```
198
-
199
-
200
- ## License
201
-
202
- This project is released under the MIT license, while InternLM2 is licensed under the Apache-2.0 license.
203
-
204
- ## Citation
205
-
206
- If you find this project useful in your research, please consider citing:
207
-
208
- ```BibTeX
209
- @article{tao2024hovle,
210
- title={HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding},
211
- author={Tao, Chenxin and Su, Shiqian and Zhu, Xizhou and Zhang, Chenyu and Chen, Zhe and Liu, Jiawen and Wang, Wenhai and Lu, Lewei and Huang, Gao and Qiao, Yu and Dai, Jifeng},
212
- journal={},
213
- year={2024}
214
- }
215
  ```
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: image-text-to-text
4
+ library_name: transformers
5
+ base_model:
6
+ - internlm/internlm2-chat-1_8b
7
+ base_model_relation: merge
8
+ language:
9
+ - multilingual
10
+ tags:
11
+ - internvl
12
+ - vision-language model
13
+ - monolithic
14
+ ---
15
+ # HoVLE
16
+
17
+ [\[📜 HoVLE Paper\]]() [\[🚀 Quick Start\]](#quick-start)
18
+
19
+ <a id="radar"></a>
20
+
21
+
22
+ ## Introduction
23
+
24
+ <p align="middle">
25
+ <img src="assets/intro.png" width="95%" />
26
+ </p>
27
+
28
+ We introduce **HoVLE**, a novel monolithic vision-language model (VLM) that processes images and texts in a unified manner. HoVLE introduces a holistic embedding module that projects image and text inputs into a shared embedding space, allowing the Large Language Model (LLM) to interpret images in the same way as texts.
29
+
30
+ HoVLE significantly surpasses previous monolithic VLMs and demonstrates competitive performance with compositional VLMs. This work narrows the gap between monolithic and compositional VLMs, providing a promising direction for the development of monolithic VLMs.
31
+
32
+ This repository releases the HoVLE model with 2.6B parameters. It is built upon [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b). Please refer to [HoVLE (HD)](https://huggingface.co/taochenxin/HoVLE-HD) for the high-definition version. For more details, please refer to our [paper]().
33
+
34
+
35
+ ## Model Details
36
+ <p align="middle">
37
+ <img src="assets/overview.png" width="90%" />
38
+ </p>
39
+
40
+ | | Details |
41
+ | :---------------------------: | :---------- |
42
+ | Architecture | The whole model consists of a holistic embedding module and an LLM. The holistic embedding module consists of the same causal Transformer layers as the LLM. It accepts both images and texts as input, and projects them into a unified embedding space. These embeddings are then forwarded into the LLM, constituting a monolithic VLM. |
43
+ | Stage I (Distillation) | The first stage trains the holistic embedding module to distill the image feature from a pre-trained visual encoder and the text embeddings from an LLM, providing general encoding abilities. Only the holistic embedding module is trainable. |
44
+ | Stage II (Alignment) | The second stage combines the holistic embedding module with the LLM to perform auto-regressive training, aligning different modalities to a shared embedding space. Only the holistic embedding module is trainable. |
45
+ | Stage III (Instruction Tuning) | A visual instruction tuning stage is incorporated to further strengthen the whole VLM to follow instructions. The whole model is trainable. |
46
+
47
+
48
+
49
+ ## Performance
50
+ <p align="middle">
51
+ <img src="assets/performance1.png" width="90%" />
52
+ </p>
53
+ <p align="middle">
54
+ <img src="assets/performance2.png" width="90%" />
55
+ </p>
56
+
57
+ - Sources of the results include the original papers, our evaluation with [VLMEvalKit](https://github.com/open-compass/VLMEvalKit), and [OpenCompass](https://rank.opencompass.org.cn/leaderboard-multimodal/?m=REALTIME).
58
+ - Please note that evaluating the same model using different testing toolkits can result in slight differences, which is normal. Updates to code versions and variations in environment and hardware can also cause minor discrepancies in results.
59
+
60
+
61
+
62
+ Limitations: Although we have made efforts to ensure the safety of the model during the training process and to encourage the model to generate text that complies with ethical and legal requirements, the model may still produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Please do not propagate such content. We are not responsible for any consequences resulting from the dissemination of harmful information.
63
+
64
+
65
+
66
+ ## Quick Start
67
+
68
+ We provide an example code to run HoVLE inference using `transformers`.
69
+
70
+ > Please use transformers==4.37.2 to ensure the model works normally.
71
+
72
+
73
+ ### Inference with Transformers
74
+
75
+ ```python
76
+ import numpy as np
77
+ import torch
78
+ import torchvision.transforms as T
79
+ from decord import VideoReader, cpu
80
+ from PIL import Image
81
+ from torchvision.transforms.functional import InterpolationMode
82
+ from transformers import AutoModel, AutoTokenizer
83
+
84
+ IMAGENET_MEAN = (0.485, 0.456, 0.406)
85
+ IMAGENET_STD = (0.229, 0.224, 0.225)
86
+
87
+ def build_transform(input_size):
88
+ MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
89
+ transform = T.Compose([
90
+ T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
91
+ T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
92
+ T.ToTensor(),
93
+ T.Normalize(mean=MEAN, std=STD)
94
+ ])
95
+ return transform
96
+
97
+ def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
98
+ best_ratio_diff = float('inf')
99
+ best_ratio = (1, 1)
100
+ area = width * height
101
+ for ratio in target_ratios:
102
+ target_aspect_ratio = ratio[0] / ratio[1]
103
+ ratio_diff = abs(aspect_ratio - target_aspect_ratio)
104
+ if ratio_diff < best_ratio_diff:
105
+ best_ratio_diff = ratio_diff
106
+ best_ratio = ratio
107
+ elif ratio_diff == best_ratio_diff:
108
+ if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
109
+ best_ratio = ratio
110
+ return best_ratio
111
+
112
+ def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
113
+ orig_width, orig_height = image.size
114
+ aspect_ratio = orig_width / orig_height
115
+
116
+ # calculate the existing image aspect ratio
117
+ target_ratios = set(
118
+ (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
119
+ i * j <= max_num and i * j >= min_num)
120
+ target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
121
+
122
+ # find the closest aspect ratio to the target
123
+ target_aspect_ratio = find_closest_aspect_ratio(
124
+ aspect_ratio, target_ratios, orig_width, orig_height, image_size)
125
+
126
+ # calculate the target width and height
127
+ target_width = image_size * target_aspect_ratio[0]
128
+ target_height = image_size * target_aspect_ratio[1]
129
+ blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
130
+
131
+ # resize the image
132
+ resized_img = image.resize((target_width, target_height))
133
+ processed_images = []
134
+ for i in range(blocks):
135
+ box = (
136
+ (i % (target_width // image_size)) * image_size,
137
+ (i // (target_width // image_size)) * image_size,
138
+ ((i % (target_width // image_size)) + 1) * image_size,
139
+ ((i // (target_width // image_size)) + 1) * image_size
140
+ )
141
+ # split the image
142
+ split_img = resized_img.crop(box)
143
+ processed_images.append(split_img)
144
+ assert len(processed_images) == blocks
145
+ if use_thumbnail and len(processed_images) != 1:
146
+ thumbnail_img = image.resize((image_size, image_size))
147
+ processed_images.append(thumbnail_img)
148
+ return processed_images
149
+
150
+ def load_image(image_file, input_size=448, max_num=12):
151
+ image = Image.open(image_file).convert('RGB')
152
+ transform = build_transform(input_size=input_size)
153
+ images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
154
+ pixel_values = [transform(image) for image in images]
155
+ pixel_values = torch.stack(pixel_values)
156
+ return pixel_values
157
+
158
+
159
+ path = 'taochenxin/HoVLE/'
160
+ model = AutoModel.from_pretrained(
161
+ path,
162
+ torch_dtype=torch.bfloat16,
163
+ low_cpu_mem_usage=True,
164
+ trust_remote_code=True).eval().cuda()
165
+ tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
166
+
167
+ # set the max number of tiles in `max_num`
168
+ pixel_values = load_image('./examples_image.jpg', max_num=12).to(torch.bfloat16).cuda()
169
+ generation_config = dict(max_new_tokens=1024, do_sample=True)
170
+
171
+ # pure-text conversation (纯文本对话)
172
+ question = 'Hello, who are you?'
173
+ response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
174
+ print(f'User: {question}\nAssistant: {response}')
175
+
176
+ question = 'Can you tell me a story?'
177
+ response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)
178
+ print(f'User: {question}\nAssistant: {response}')
179
+
180
+ # single-image single-round conversation (单图单轮对话)
181
+ question = '<image>\nPlease describe the image shortly.'
182
+ response = model.chat(tokenizer, pixel_values, question, generation_config)
183
+ print(f'User: {question}\nAssistant: {response}')
184
+
185
+ # single-image multi-round conversation (单图多轮对话)
186
+ question = '<image>\nPlease describe the image in detail.'
187
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
188
+ print(f'User: {question}\nAssistant: {response}')
189
+
190
+ question = 'Please write a poem according to the image.'
191
+ response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
192
+ print(f'User: {question}\nAssistant: {response}')
193
+
194
+ ```
195
+
196
+
197
+ ## License
198
+
199
+ This project is released under the MIT license, while InternLM2 is licensed under the Apache-2.0 license.
200
+
201
+ ## Citation
202
+
203
+ If you find this project useful in your research, please consider citing:
204
+
205
+ ```BibTeX
206
+ @article{tao2024hovle,
207
+ title={HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding},
208
+ author={Tao, Chenxin and Su, Shiqian and Zhu, Xizhou and Zhang, Chenyu and Chen, Zhe and Liu, Jiawen and Wang, Wenhai and Lu, Lewei and Huang, Gao and Qiao, Yu and Dai, Jifeng},
209
+ journal={},
210
+ year={2024}
211
+ }
 
 
 
212
  ```