Update README.md
Browse files
README.md
CHANGED
@@ -11,6 +11,12 @@ license: mit
|
|
11 |
This repository features LLaVA v1.5 trained with the Phi-3-mini-3.8B LLM. This integration aims to leverage the strengths of both models to offer advanced vision-language understanding.
|
12 |
|
13 |
|
|
|
|
|
|
|
|
|
|
|
|
|
14 |
## Key Components
|
15 |
|
16 |
- **Base Large Language Model (LLM):** [Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct)
|
|
|
11 |
This repository features LLaVA v1.5 trained with the Phi-3-mini-3.8B LLM. This integration aims to leverage the strengths of both models to offer advanced vision-language understanding.
|
12 |
|
13 |
|
14 |
+
## Training Strategy
|
15 |
+
- **Pretraining:** Only Vision-to-Language projector is trained. The rest of the model is frozen.
|
16 |
+
- **Fine-tuning:** LLM is LoRA fine-tuned. Only the vision-backbone (CLIP) is kept frozen.
|
17 |
+
- **Note:** The repository contains merged weights.
|
18 |
+
|
19 |
+
|
20 |
## Key Components
|
21 |
|
22 |
- **Base Large Language Model (LLM):** [Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct)
|