HuanjinYao/Mulberry_llava_8b
Image-Text-to-Text
•
Updated
•
956
•
2
Huanjin Yao
HuanjinYao
AI & ML interests
None yet
Recent Activity
updated
a model
about 1 month ago
HuanjinYao/Mulberry_llava_8b
updated
a collection
about 1 month ago
Mulberry
updated
a model
about 1 month ago
HuanjinYao/Mulberry_qwen2vl_7b
Organizations
HuanjinYao's activity
updated
a
model
about 1 month ago
updated
a
collection
about 1 month ago
updated
a
model
about 1 month ago
published
a
model
about 1 month ago
updated
a
dataset
about 2 months ago
updated
a
collection
about 2 months ago
updated
a
model
about 2 months ago
published
a
dataset
about 2 months ago
updated
a
collection
about 2 months ago
published
a
model
about 2 months ago
Add metadata
#1 opened about 2 months ago
by
nielsr
authored
a
paper
2 months ago
upvoted
a
paper
3 months ago
liked
a
Space
9 months ago
updated
a
model
9 months ago
About video instruction data
2
#15 opened 9 months ago
by
HuanjinYao
reacted to
merve's
post with 👀
10 months ago
Post
2061
Do we fully leverage ViT encoders in vision language models?
A new paper (by @HuanjinYao et al) built a dense connector that does it better! HuanjinYao/DenseConnector-v1.5-8B
HuanjinYao/denseconnector-66500e173fc8c9f05dc98dea
VLMs consist of an image encoder block, a projection layer that projects image embeddings to text embedding space and then a text decoder sequentially connected 📖
This paper explores using intermediate states of image encoder and not a single output 🤩
The authors explore three different ways of instantiating dense connector: sparse token integration, sparse channel integration and dense channel integration. (see paper on how they do it Dense Connector for MLLMs (2405.13800))
They explore all three of them integrated to LLaVA 1.5 and found out each of the new models are superior to the original LLaVA 1.5 🥹 I tried the model and it seems to work very well. As part of the release, the authors have released various ckpts based on different decoders (Vicuna 7/13B and Llama 3-8B) that you can find in the collection 🤗
A new paper (by @HuanjinYao et al) built a dense connector that does it better! HuanjinYao/DenseConnector-v1.5-8B
HuanjinYao/denseconnector-66500e173fc8c9f05dc98dea
VLMs consist of an image encoder block, a projection layer that projects image embeddings to text embedding space and then a text decoder sequentially connected 📖
This paper explores using intermediate states of image encoder and not a single output 🤩
The authors explore three different ways of instantiating dense connector: sparse token integration, sparse channel integration and dense channel integration. (see paper on how they do it Dense Connector for MLLMs (2405.13800))
They explore all three of them integrated to LLaVA 1.5 and found out each of the new models are superior to the original LLaVA 1.5 🥹 I tried the model and it seems to work very well. As part of the release, the authors have released various ckpts based on different decoders (Vicuna 7/13B and Llama 3-8B) that you can find in the collection 🤗
updated
a
Space
10 months ago
updated
a
collection
10 months ago
updated
a
model
10 months ago