diff --git a/.gitattributes b/.gitattributes index c7d9f3332a950355d5a77d85000f05e6f45435ea..165eec68232dd8ed07006b22a5100c6b052321ff 100644 --- a/.gitattributes +++ b/.gitattributes @@ -32,3 +32,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +figs/online_demo.png filter=lfs diff=lfs merge=lfs -text +figs/overview.png filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..c7c3c5249bb6aa37a65921894aa6e7f65e165576 --- /dev/null +++ b/README.md @@ -0,0 +1,159 @@ +# MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models +[Deyao Zhu](https://tsutikgiau.github.io/)* (On Job Market!), [Jun Chen](https://junchen14.github.io/)* (On Job Market!), [Xiaoqian Shen](https://xiaoqian-shen.github.io), [Xiang Li](https://xiangli.ac.cn), and [Mohamed Elhoseiny](https://www.mohamed-elhoseiny.com/). *Equal Contribution + +**King Abdullah University of Science and Technology** + + + + +## Online Demo + +Click the image to chat with MiniGPT-4 around your images +[![demo](figs/online_demo.png)](https://minigpt-4.github.io) + + +## Examples + | | | +:-------------------------:|:-------------------------: +![find wild](figs/examples/wop_2.png) | ![write story](figs/examples/ad_2.png) +![solve problem](figs/examples/fix_1.png) | ![write Poem](figs/examples/rhyme_1.png) + +More examples can be found in the [project page](https://minigpt-4.github.io). + + + +## Introduction +- MiniGPT-4 aligns a frozen visual encoder from BLIP-2 with a frozen LLM, Vicuna, using just one projection layer. +- We train MiniGPT-4 with two stages. The first traditional pretraining stage is trained using roughly 5 million aligned image-text pairs in 10 hours using 4 A100s. After the first stage, Vicuna is able to understand the image. But the generation ability of Vicuna is heavilly impacted. +- To address this issue and improve usability, we propose a novel way to create high-quality image-text pairs by the model itself and ChatGPT together. Based on this, we then create a small (3500 pairs in total) yet high-quality dataset. +- The second finetuning stage is trained on this dataset in a conversation template to significantly improve its generation reliability and overall usability. To our surprise, this stage is computationally efficient and takes only around 7 minutes with a single A100. +- MiniGPT-4 yields many emerging vision-language capabilities similar to those demonstrated in GPT-4. + + +![overview](figs/overview.png) + + +## Getting Started +### Installation + +**1. Prepare the code and the environment** + +Git clone our repository, creating a python environment and ativate it via the following command + +```bash +git clone https://github.com/Vision-CAIR/MiniGPT-4.git +cd MiniGPT-4 +conda env create -f environment.yml +conda activate minigpt4 +``` + + +**2. Prepare the pretrained Vicuna weights** + +The current version of MiniGPT-4 is built on the v0 versoin of Vicuna-13B. +Please refer to our instruction [here](PrepareVicuna.md) +to prepare the Vicuna weights. +The final weights would be in a single folder with the following structure: + +``` +vicuna_weights +├── config.json +├── generation_config.json +├── pytorch_model.bin.index.json +├── pytorch_model-00001-of-00003.bin +... +``` + +Then, set the path to the vicuna weight in the model config file +[here](minigpt4/configs/models/minigpt4.yaml#L16) at Line 16. + +**3. Prepare the pretrained MiniGPT-4 checkpoint** + +To play with our pretrained model, download the pretrained checkpoint +[here](https://drive.google.com/file/d/1a4zLvaiDBr-36pasffmgpvH5P7CKmpze/view?usp=share_link). +Then, set the path to the pretrained checkpoint in the evaluation config file +in [eval_configs/minigpt4_eval.yaml](eval_configs/minigpt4_eval.yaml#L10) at Line 11. + + + +### Launching Demo Locally + +Try out our demo [demo.py](demo.py) on your local machine by running + +``` +python demo.py --cfg-path eval_configs/minigpt4_eval.yaml --gpu-id 0 +``` + +Here, we load Vicuna as 8 bit by default to save some GPU memory usage. +Besides, the default beam search width is 1. +Under this setting, the demo cost about 23G GPU memory. +If you have a more powerful GPU with larger GPU memory, you can run the model +in 16 bit by setting low_resource to False in the config file +[minigpt4_eval.yaml](eval_configs/minigpt4_eval.yaml) and use a larger beam search width. + + +### Training +The training of MiniGPT-4 contains two alignment stages. + +**1. First pretraining stage** + +In the first pretrained stage, the model is trained using image-text pairs from Laion and CC datasets +to align the vision and language model. To download and prepare the datasets, please check +our [first stage dataset preparation instruction](dataset/README_1_STAGE.md). +After the first stage, the visual features are mapped and can be understood by the language +model. +To launch the first stage training, run the following command. In our experiments, we use 4 A100. +You can change the save path in the config file +[train_configs/minigpt4_stage1_pretrain.yaml](train_configs/minigpt4_stage1_pretrain.yaml) + +```bash +torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/minigpt4_stage1_pretrain.yaml +``` + +A MiniGPT-4 checkpoint with only stage one training can be downloaded +[here](https://drive.google.com/file/d/1u9FRRBB3VovP1HxCAlpD9Lw4t4P6-Yq8/view?usp=share_link). +Compared to the model after stage two, this checkpoint generate incomplete and repeated sentences frequently. + + +**2. Second finetuning stage** + +In the second stage, we use a small high quality image-text pair dataset created by ourselves +and convert it to a conversation format to further align MiniGPT-4. +To download and prepare our second stage dataset, please check our +[second stage dataset preparation instruction](dataset/README_2_STAGE.md). +To launch the second stage alignment, +first specify the path to the checkpoint file trained in stage 1 in +[train_configs/minigpt4_stage1_pretrain.yaml](train_configs/minigpt4_stage2_finetune.yaml). +You can also specify the output path there. +Then, run the following command. In our experiments, we use 1 A100. + +```bash +torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/minigpt4_stage2_finetune.yaml +``` + +After the second stage alignment, MiniGPT-4 is able to talk about the image coherently and user-friendly. + + + + +## Acknowledgement + ++ [BLIP2](https://huggingface.co/docs/transformers/main/model_doc/blip-2) The model architecture of MiniGPT-4 follows BLIP-2. Don't forget to check this great open-source work if you don't know it before! ++ [Lavis](https://github.com/salesforce/LAVIS) This repository is built upon Lavis! ++ [Vicuna](https://github.com/lm-sys/FastChat) The fantastic language ability of Vicuna with only 13B parameters is just amazing. And it is open-source! + + +If you're using MiniGPT-4 in your research or applications, please cite using this BibTeX: +```bibtex +@misc{zhu2022minigpt4, + title={MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models}, + author={Deyao Zhu and Jun Chen and Xiaoqian Shen and xiang Li and Mohamed Elhoseiny}, + year={2023}, +} +``` + + +## License +This repository is under [BSD 3-Clause License](LICENSE.md). +Many codes are based on [Lavis](https://github.com/salesforce/LAVIS) with +BSD 3-Clause License [here](LICENSE_Lavis.md). diff --git a/examples/ad_1.png b/examples/ad_1.png new file mode 100644 index 0000000000000000000000000000000000000000..d0378e43e9a0e797b2ab32f4d8f6261fa2224408 Binary files /dev/null and b/examples/ad_1.png differ diff --git a/examples/ad_2.png b/examples/ad_2.png new file mode 100644 index 0000000000000000000000000000000000000000..674248b723bee885c43a55d85d83ea1c0fa41477 Binary files /dev/null and b/examples/ad_2.png differ diff --git a/examples/cook_1.png b/examples/cook_1.png new file mode 100644 index 0000000000000000000000000000000000000000..d8cdb45c98492afd4f975b8626bb590b580616a5 Binary files /dev/null and b/examples/cook_1.png differ diff --git a/examples/cook_2.png b/examples/cook_2.png new file mode 100644 index 0000000000000000000000000000000000000000..d08272b3733dda976bfa78733d9ca4eb544fee52 Binary files /dev/null and b/examples/cook_2.png differ diff --git a/examples/describe_1.png b/examples/describe_1.png new file mode 100644 index 0000000000000000000000000000000000000000..02f3c92f54749fa354a5f8c617f24301728555b2 Binary files /dev/null and b/examples/describe_1.png differ diff --git a/examples/describe_2.png b/examples/describe_2.png new file mode 100644 index 0000000000000000000000000000000000000000..20bf8c7cd86c03f9ed77f95d912057438997277d Binary files /dev/null and b/examples/describe_2.png differ diff --git a/examples/fact_1.png b/examples/fact_1.png new file mode 100644 index 0000000000000000000000000000000000000000..1f7522871916c3d1bd113cfb88c45386d8abda7a Binary files /dev/null and b/examples/fact_1.png differ diff --git a/examples/fact_2.png b/examples/fact_2.png new file mode 100644 index 0000000000000000000000000000000000000000..de6ef53ef7afd72894711a3a5288a24d62c39182 Binary files /dev/null and b/examples/fact_2.png differ diff --git a/examples/fix_1.png b/examples/fix_1.png new file mode 100644 index 0000000000000000000000000000000000000000..023cfe6610747868805c70001e14f2b408f3cebb Binary files /dev/null and b/examples/fix_1.png differ diff --git a/examples/fix_2.png b/examples/fix_2.png new file mode 100644 index 0000000000000000000000000000000000000000..f60da5ff9bdef7018e98a92b19c0e59d31acd059 Binary files /dev/null and b/examples/fix_2.png differ diff --git a/examples/fun_1.png b/examples/fun_1.png new file mode 100644 index 0000000000000000000000000000000000000000..f720ea603f88019e24dbcb569328c3083c832baf Binary files /dev/null and b/examples/fun_1.png differ diff --git a/examples/fun_2.png b/examples/fun_2.png new file mode 100644 index 0000000000000000000000000000000000000000..1d37a8068feda3f7ecc2d0b22893d26071e68b64 Binary files /dev/null and b/examples/fun_2.png differ diff --git a/examples/logo_1.png b/examples/logo_1.png new file mode 100644 index 0000000000000000000000000000000000000000..8bbe438bdc05ce023251045575c6b7e7b04f210f Binary files /dev/null and b/examples/logo_1.png differ diff --git a/examples/op_1.png b/examples/op_1.png new file mode 100644 index 0000000000000000000000000000000000000000..3dbb2ff51ca08f62171f48167bbb97ad604cc4d0 Binary files /dev/null and b/examples/op_1.png differ diff --git a/examples/op_2.png b/examples/op_2.png new file mode 100644 index 0000000000000000000000000000000000000000..2cd3e1f8b0326dea14d45bf866b244deb38ef409 Binary files /dev/null and b/examples/op_2.png differ diff --git a/examples/people_1.png b/examples/people_1.png new file mode 100644 index 0000000000000000000000000000000000000000..7e95c42c710aef5efe94a52da280fd7451f185d7 Binary files /dev/null and b/examples/people_1.png differ diff --git a/examples/people_2.png b/examples/people_2.png new file mode 100644 index 0000000000000000000000000000000000000000..aec6c83b217c96af91668dbc566e05a14238b2a8 Binary files /dev/null and b/examples/people_2.png differ diff --git a/examples/rhyme_1.png b/examples/rhyme_1.png new file mode 100644 index 0000000000000000000000000000000000000000..7d133878d8b534867253c7be7b2805faffbd6ad7 Binary files /dev/null and b/examples/rhyme_1.png differ diff --git a/examples/rhyme_2.png b/examples/rhyme_2.png new file mode 100644 index 0000000000000000000000000000000000000000..6cf9bf8958302e461dec8b58dd7cbbe2224a8e5c Binary files /dev/null and b/examples/rhyme_2.png differ diff --git a/examples/story_1.png b/examples/story_1.png new file mode 100644 index 0000000000000000000000000000000000000000..3eb6ccb93fb5c866eeb758ba962904e5c3d57875 Binary files /dev/null and b/examples/story_1.png differ diff --git a/examples/story_2.png b/examples/story_2.png new file mode 100644 index 0000000000000000000000000000000000000000..9d37142a9ae32f20d47ebd10cf1c395f10b363f7 Binary files /dev/null and b/examples/story_2.png differ diff --git a/examples/web_1.png b/examples/web_1.png new file mode 100644 index 0000000000000000000000000000000000000000..8943842c08609713b78d95ab3b5c418995569505 Binary files /dev/null and b/examples/web_1.png differ diff --git a/examples/wop_1.png b/examples/wop_1.png new file mode 100644 index 0000000000000000000000000000000000000000..88f37d672bb2dd3dac34caced2ed4bebcfe15412 Binary files /dev/null and b/examples/wop_1.png differ diff --git a/examples/wop_2.png b/examples/wop_2.png new file mode 100644 index 0000000000000000000000000000000000000000..8255974176014db0b388617821630bdb438b5e6b Binary files /dev/null and b/examples/wop_2.png differ diff --git a/figs/examples/ad_1.png b/figs/examples/ad_1.png new file mode 100644 index 0000000000000000000000000000000000000000..d0378e43e9a0e797b2ab32f4d8f6261fa2224408 Binary files /dev/null and b/figs/examples/ad_1.png differ diff --git a/figs/examples/ad_2.png b/figs/examples/ad_2.png new file mode 100644 index 0000000000000000000000000000000000000000..674248b723bee885c43a55d85d83ea1c0fa41477 Binary files /dev/null and b/figs/examples/ad_2.png differ diff --git a/figs/examples/cook_1.png b/figs/examples/cook_1.png new file mode 100644 index 0000000000000000000000000000000000000000..d8cdb45c98492afd4f975b8626bb590b580616a5 Binary files /dev/null and b/figs/examples/cook_1.png differ diff --git a/figs/examples/cook_2.png b/figs/examples/cook_2.png new file mode 100644 index 0000000000000000000000000000000000000000..d08272b3733dda976bfa78733d9ca4eb544fee52 Binary files /dev/null and b/figs/examples/cook_2.png differ diff --git a/figs/examples/describe_1.png b/figs/examples/describe_1.png new file mode 100644 index 0000000000000000000000000000000000000000..02f3c92f54749fa354a5f8c617f24301728555b2 Binary files /dev/null and b/figs/examples/describe_1.png differ diff --git a/figs/examples/describe_2.png b/figs/examples/describe_2.png new file mode 100644 index 0000000000000000000000000000000000000000..20bf8c7cd86c03f9ed77f95d912057438997277d Binary files /dev/null and b/figs/examples/describe_2.png differ diff --git a/figs/examples/fact_1.png b/figs/examples/fact_1.png new file mode 100644 index 0000000000000000000000000000000000000000..1f7522871916c3d1bd113cfb88c45386d8abda7a Binary files /dev/null and b/figs/examples/fact_1.png differ diff --git a/figs/examples/fact_2.png b/figs/examples/fact_2.png new file mode 100644 index 0000000000000000000000000000000000000000..de6ef53ef7afd72894711a3a5288a24d62c39182 Binary files /dev/null and b/figs/examples/fact_2.png differ diff --git a/figs/examples/fix_1.png b/figs/examples/fix_1.png new file mode 100644 index 0000000000000000000000000000000000000000..023cfe6610747868805c70001e14f2b408f3cebb Binary files /dev/null and b/figs/examples/fix_1.png differ diff --git a/figs/examples/fix_2.png b/figs/examples/fix_2.png new file mode 100644 index 0000000000000000000000000000000000000000..f60da5ff9bdef7018e98a92b19c0e59d31acd059 Binary files /dev/null and b/figs/examples/fix_2.png differ diff --git a/figs/examples/fun_1.png b/figs/examples/fun_1.png new file mode 100644 index 0000000000000000000000000000000000000000..f720ea603f88019e24dbcb569328c3083c832baf Binary files /dev/null and b/figs/examples/fun_1.png differ diff --git a/figs/examples/fun_2.png b/figs/examples/fun_2.png new file mode 100644 index 0000000000000000000000000000000000000000..1d37a8068feda3f7ecc2d0b22893d26071e68b64 Binary files /dev/null and b/figs/examples/fun_2.png differ diff --git a/figs/examples/logo_1.png b/figs/examples/logo_1.png new file mode 100644 index 0000000000000000000000000000000000000000..8bbe438bdc05ce023251045575c6b7e7b04f210f Binary files /dev/null and b/figs/examples/logo_1.png differ diff --git a/figs/examples/op_1.png b/figs/examples/op_1.png new file mode 100644 index 0000000000000000000000000000000000000000..3dbb2ff51ca08f62171f48167bbb97ad604cc4d0 Binary files /dev/null and b/figs/examples/op_1.png differ diff --git a/figs/examples/op_2.png b/figs/examples/op_2.png new file mode 100644 index 0000000000000000000000000000000000000000..2cd3e1f8b0326dea14d45bf866b244deb38ef409 Binary files /dev/null and b/figs/examples/op_2.png differ diff --git a/figs/examples/people_1.png b/figs/examples/people_1.png new file mode 100644 index 0000000000000000000000000000000000000000..7e95c42c710aef5efe94a52da280fd7451f185d7 Binary files /dev/null and b/figs/examples/people_1.png differ diff --git a/figs/examples/people_2.png b/figs/examples/people_2.png new file mode 100644 index 0000000000000000000000000000000000000000..aec6c83b217c96af91668dbc566e05a14238b2a8 Binary files /dev/null and b/figs/examples/people_2.png differ diff --git a/figs/examples/rhyme_1.png b/figs/examples/rhyme_1.png new file mode 100644 index 0000000000000000000000000000000000000000..7d133878d8b534867253c7be7b2805faffbd6ad7 Binary files /dev/null and b/figs/examples/rhyme_1.png differ diff --git a/figs/examples/rhyme_2.png b/figs/examples/rhyme_2.png new file mode 100644 index 0000000000000000000000000000000000000000..6cf9bf8958302e461dec8b58dd7cbbe2224a8e5c Binary files /dev/null and b/figs/examples/rhyme_2.png differ diff --git a/figs/examples/story_1.png b/figs/examples/story_1.png new file mode 100644 index 0000000000000000000000000000000000000000..3eb6ccb93fb5c866eeb758ba962904e5c3d57875 Binary files /dev/null and b/figs/examples/story_1.png differ diff --git a/figs/examples/story_2.png b/figs/examples/story_2.png new file mode 100644 index 0000000000000000000000000000000000000000..9d37142a9ae32f20d47ebd10cf1c395f10b363f7 Binary files /dev/null and b/figs/examples/story_2.png differ diff --git a/figs/examples/web_1.png b/figs/examples/web_1.png new file mode 100644 index 0000000000000000000000000000000000000000..8943842c08609713b78d95ab3b5c418995569505 Binary files /dev/null and b/figs/examples/web_1.png differ diff --git a/figs/examples/wop_1.png b/figs/examples/wop_1.png new file mode 100644 index 0000000000000000000000000000000000000000..88f37d672bb2dd3dac34caced2ed4bebcfe15412 Binary files /dev/null and b/figs/examples/wop_1.png differ diff --git a/figs/examples/wop_2.png b/figs/examples/wop_2.png new file mode 100644 index 0000000000000000000000000000000000000000..8255974176014db0b388617821630bdb438b5e6b Binary files /dev/null and b/figs/examples/wop_2.png differ diff --git a/figs/online_demo.png b/figs/online_demo.png new file mode 100644 index 0000000000000000000000000000000000000000..cc3dde7a4ac01cf1bcb8096de63c0ec583070846 --- /dev/null +++ b/figs/online_demo.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:236107e23897574578111e0a4bbe6c0475e794b5eca13943f692755b1c71c8df +size 1259978 diff --git a/figs/overview.png b/figs/overview.png new file mode 100644 index 0000000000000000000000000000000000000000..83ac7cc1657d98262d34459268421713074b4a96 --- /dev/null +++ b/figs/overview.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:378307bed387d530087773826ac3b4686e106cf2d1e4a24daed27573f8cc14cb +size 2532475