RaushanTurganbay HF staff commited on
Commit
061e2d4
β€’
1 Parent(s): 4d835bc

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +235 -3
README.md CHANGED
@@ -1,3 +1,235 @@
1
- ---
2
- license: llama2
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: llama2
5
+ pipeline_tag: image-text-to-text
6
+ ---
7
+
8
+ # LLaVA-NeXT-Video Model Card
9
+
10
+ Check out also the Google Colab demo to run Llava on a free-tier Google Colab instance: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CZggLHrjxMReG-FNOmqSOdi4z7NPq6SO?usp=sharing)
11
+
12
+ Disclaimer: The team releasing LLaVa-NeXT-Video did not write a model card for this model so this model card has been written by the Hugging Face team.
13
+
14
+ ## πŸ“„ Model details
15
+
16
+ **Model type:**
17
+ LLaVA-Next-Video is an open-source chatbot trained by fine-tuning LLM on multimodal instruction-following data. The model is buit on top of LLaVa-NeXT by tuning on a mix of video and image data to achieves better video understanding capabilities. The videos were sampled uniformly to be 32 frames per clip.
18
+ The model is a current SOTA among open-source models on [VideoMME bench](https://arxiv.org/abs/2405.21075).
19
+ Base LLM: [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-13b-v1.5)
20
+
21
+ <img src="http://drive.google.com/uc?export=view&id=1fVg-r5MU3NoHlTpD7_lYPEBWH9R8na_4">
22
+
23
+
24
+ **Model date:**
25
+ LLaVA-Next-Video-7B was trained in April 2024.
26
+
27
+ **Paper or resources for more information:** https://github.com/LLaVA-VL/LLaVA-NeXT
28
+
29
+
30
+ ## πŸ“š Training dataset
31
+
32
+ ### Image
33
+ - 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP.
34
+ - 158K GPT-generated multimodal instruction-following data.
35
+ - 500K academic-task-oriented VQA data mixture.
36
+ - 50K GPT-4V data mixture.
37
+ - 40K ShareGPT data.
38
+
39
+ ### Video
40
+ - 100K VideoChatGPT-Instruct.
41
+
42
+ ## πŸ“Š Evaluation dataset
43
+ A collection of 4 benchmarks, including 3 academic VQA benchmarks and 1 captioning benchmark.
44
+
45
+
46
+
47
+ ## πŸš€ How to use the model
48
+
49
+ First, make sure to have `transformers >= 4.42.0`.
50
+ The model supports multi-visual and multi-prompt generation. Meaning that you can pass multiple images/videos in your prompt. Make sure also to follow the correct prompt template (`USER: xxx\nASSISTANT:`) and add the token `<image>` or `<video>` to the location where you want to query images/videos:
51
+
52
+ Below is an example script to run generation in `float16` precision on a GPU device:
53
+
54
+ ```python
55
+ import av
56
+ import torch
57
+ from transformers import LlavaNextVideoProcessor, LlavaNextVideoForConditionalGeneration
58
+
59
+ model_id = "llava-hf/LLaVA-NeXT-Video-7B-32K-hf"
60
+
61
+ model = LlavaNextVideoForConditionalGeneration.from_pretrained(
62
+ model_id,
63
+ torch_dtype=torch.float16,
64
+ low_cpu_mem_usage=True,
65
+ ).to(0)
66
+
67
+ processor = LlavaNextVideoProcessor.from_pretrained(model_id)
68
+
69
+ def read_video_pyav(container, indices):
70
+ '''
71
+ Decode the video with PyAV decoder.
72
+ Args:
73
+ container (`av.container.input.InputContainer`): PyAV container.
74
+ indices (`List[int]`): List of frame indices to decode.
75
+ Returns:
76
+ result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
77
+ '''
78
+ frames = []
79
+ container.seek(0)
80
+ start_index = indices[0]
81
+ end_index = indices[-1]
82
+ for i, frame in enumerate(container.decode(video=0)):
83
+ if i > end_index:
84
+ break
85
+ if i >= start_index and i in indices:
86
+ frames.append(frame)
87
+ return np.stack([x.to_ndarray(format="rgb24") for x in frames])
88
+
89
+
90
+ # define a chat histiry and use `apply_chat_template` to get correctly formatted prompt
91
+ # Each value in "content" has to be a list of dicts with types ("text", "image", "video")
92
+ conversation = [
93
+ {
94
+
95
+ "role": "user",
96
+ "content": [
97
+ {"type": "text", "text": "Why is this video funny?"},
98
+ {"type": "video"},
99
+ ],
100
+ },
101
+ ]
102
+
103
+ prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
104
+
105
+ video_path = hf_hub_download(repo_id="raushan-testing-hf/videos-test", filename="sample_demo_1.mp4", repo_type="dataset")
106
+ container = av.open(video_path)
107
+
108
+ # sample uniformly 8 frames from the video, can sample more for longer videos
109
+ total_frames = container.streams.video[0].frames
110
+ indices = np.arange(0, total_frames, total_frames / 8).astype(int)
111
+ clip = read_video_pyav(container, indices)
112
+ inputs_video = processor(text=prompt, videos=clip, padding=True, return_tensors="pt").to(model.device)
113
+
114
+ output = model.generate(**inputs_video, max_new_tokens=100, do_sample=False)
115
+ print(processor.decode(output[0][2:], skip_special_tokens=True))
116
+ ```
117
+
118
+ ### Inference with images as inputs
119
+
120
+ To generate from images use the below code after loading the model as shown above:
121
+
122
+ ```python
123
+ import requests
124
+ from PIL import Image
125
+
126
+ conversation = [
127
+ {
128
+ "role": "user",
129
+ "content": [
130
+ {"type": "text", "text": "What are these?"},
131
+ {"type": "image"},
132
+ ],
133
+ },
134
+ ]
135
+ prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
136
+
137
+ image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
138
+ raw_image = Image.open(requests.get(image_file, stream=True).raw)
139
+ inputs_image = processor(prompt, images=raw_image, return_tensors='pt').to(0, torch.float16)
140
+
141
+ output = model.generate(**inputs_video, max_new_tokens=100, do_sample=False)
142
+ print(processor.decode(output[0][2:], skip_special_tokens=True))
143
+ ```
144
+
145
+ ### Inference with images and videos as inputs
146
+
147
+ To generate from images and videos in one generate use the below code after loading the model as shown above:
148
+
149
+ ```python
150
+ conversation_1 = [
151
+ {
152
+ "role": "user",
153
+ "content": [
154
+ {"type": "text", "text": "What's the content of the image>"},
155
+ {"type": "image"},
156
+ ],
157
+ }
158
+ ]
159
+ conversation_2 = [
160
+ {
161
+ "role": "user",
162
+ "content": [
163
+ {"type": "text", "text": "Why is this video funny?"},
164
+ {"type": "video"},
165
+ ],
166
+ },
167
+ ]
168
+ prompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)
169
+ prompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)
170
+
171
+ s = processor(text=[prompt_1, prompt_2], images=image, videos=clip, padding=True, return_tensors="pt").to(model.device)
172
+
173
+ # Generate
174
+ generate_ids = model.generate(**inputs, max_new_tokens=100)
175
+ out = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
176
+ print(out)
177
+ ```
178
+
179
+ ### Model optimization
180
+
181
+ #### 4-bit quantization through `bitsandbytes` library
182
+
183
+ First make sure to install `bitsandbytes`, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:
184
+
185
+ ```diff
186
+ model = LlavaNextVideoForConditionalGeneration.from_pretrained(
187
+ model_id,
188
+ torch_dtype=torch.float16,
189
+ low_cpu_mem_usage=True,
190
+ + load_in_4bit=True
191
+ )
192
+ ```
193
+
194
+ #### Use Flash-Attention 2 to further speed-up generation
195
+
196
+ First make sure to install `flash-attn`. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:
197
+
198
+ ```diff
199
+ model = LlavaNextVideoForConditionalGeneration.from_pretrained(
200
+ model_id,
201
+ torch_dtype=torch.float16,
202
+ low_cpu_mem_usage=True,
203
+ + use_flash_attention_2=True
204
+ ).to(0)
205
+ ```
206
+
207
+
208
+ ## πŸ”’ License
209
+ Llama 2 is licensed under the LLAMA 2 Community License,
210
+ Copyright (c) Meta Platforms, Inc. All Rights Reserved.
211
+
212
+
213
+ ## ✏️ Citation
214
+ If you find our paper and code useful in your research:
215
+
216
+ ```BibTeX
217
+ @misc{zhang2024llavanextvideo,
218
+ title={LLaVA-NeXT: A Strong Zero-shot Video Understanding Model},
219
+ url={https://llava-vl.github.io/blog/2024-04-30-llava-next-video/},
220
+ author={Zhang, Yuanhan and Li, Bo and Liu, haotian and Lee, Yong jae and Gui, Liangke and Fu, Di and Feng, Jiashi and Liu, Ziwei and Li, Chunyuan},
221
+ month={April},
222
+ year={2024}
223
+ }
224
+ ```
225
+
226
+ ```BibTeX
227
+ @misc{liu2024llavanext,
228
+ title={LLaVA-NeXT: Improved reasoning, OCR, and world knowledge},
229
+ url={https://llava-vl.github.io/blog/2024-01-30-llava-next/},
230
+ author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Li, Bo and Zhang, Yuanhan and Shen, Sheng and Lee, Yong Jae},
231
+ month={January},
232
+ year={2024}
233
+ }
234
+ ```
235
+