Hugging Face's logo

llava-hf
/
llava-v1.6-34b-hf

Image-Text-to-Text
Transformers
Safetensors
English
llava_next
vision
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
20
llava-v1.6-34b-hf / README.md
merve's picture
merve HF staff
Update README.md
e69afc2 verified
preview code
|
raw
history blame
3.16 kB
metadata
tags:
  - vision
  - image-text-to-text

LLaVa-Next, leveraging NousResearch/Nous-Hermes-2-Yi-34B as LLM

The LLaVA-NeXT model was proposed in LLaVA-NeXT: Improved reasoning, OCR, and world knowledge by Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee. LLaVa-NeXT (also called LLaVa-1.6) improves upon LLaVa by increasing the input image resolution and training on an improved visual instruction tuning dataset to improve OCR and common sense reasoning.

Disclaimer: The team releasing LLaVa-NeXT did not write a model card for this model so this model card has been written by the Hugging Face team.

Model description

LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases. LLaVA 1.6 improves on LLaVA 1.5 BY:

  • Using Mistral-7B and Nous-Hermes-2-Yi-34B (for this checkpoint) which has better commercial licenses, and bilingual support
  • More diverse and high quality data mixture
  • Dynamic high resolution

image/png

Intended uses & limitations

You can use the raw model for tasks like image captioning, visual question answering, multimodal chatbot use cases. See the model hub to look for other versions on a task that interests you.

How to use

Here's the prompt template for this model:

"[INST] <image>\nWhat is shown in this image? [/INST]"

You can load and use the model like following:

from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image
import requests

processor = LlavaNextProcessor.from_pretrained("NousResearch/Nous-Hermes-2-Yi-34B")

model = LlavaNextForConditionalGeneration.from_pretrained("NousResearch/Nous-Hermes-2-Yi-34B", torch_dtype=torch.float16, low_cpu_mem_usage=True) 
model.to("cuda:0")

# prepare image and text prompt, using the appropriate prompt template
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)
prompt = ""<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n<image>\nWhat is shown in this image?<|im_end|><|im_start|>assistant\n""

inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")

# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)

print(processor.decode(output[0], skip_special_tokens=True))

BibTeX entry and citation info 

@misc{liu2023improved,
      title={Improved Baselines with Visual Instruction Tuning}, 
      author={Haotian Liu and Chunyuan Li and Yuheng Li and Yong Jae Lee},
      year={2023},
      eprint={2310.03744},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}