Wanfq commited on
Commit
fb0b3b2
1 Parent(s): 857cb02

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +276 -9
README.md CHANGED
@@ -15,7 +15,8 @@ library_name: transformers
15
 
16
  <div id="top" align="center">
17
 
18
- _**Knowledge Fusion of Large Language Models**_
 
19
 
20
  <h4> |<a href="https://arxiv.org/abs/2401.10491"> 📑 Paper </a> |
21
  <a href="https://huggingface.co/Wanfq/FuseLLM-7B"> 🤗 Model </a> |
@@ -37,12 +38,26 @@ _<sup>†</sup> Sun Yat-sen University,
37
 
38
 
39
  ## News
 
40
  - **Jan 22, 2024:** 🔥 We're excited to announce that the FuseLLM-7B, which is the fusion of [Llama-2-7B](https://huggingface.co/meta-llama/Llama-2-7b-hf), [OpenLLaMA-7B](https://huggingface.co/openlm-research/open_llama_7b_v2), and [MPT-7B](https://huggingface.co/mosaicml/mpt-7b), is now available on 🤗 [Huggingface Models](https://huggingface.co/Wanfq/FuseLLM-7B). Happy exploring!
41
 
 
 
 
 
 
 
 
 
 
42
  ## Contents
43
 
44
  - [Overview](#overview)
45
  - [Model Release](#model-release)
 
 
 
 
46
  - [Citation](#citation)
47
  - [Acknowledgements](#acknowledgments)
48
 
@@ -50,7 +65,7 @@ _<sup>†</sup> Sun Yat-sen University,
50
 
51
  In this study, we explore the realm of knowledge fusion for LLMs to create a unified model that combines the capabilities and distinctive strengths of multiple structurally diverse LLMs. To achieve this, we introduce FuseLLM, which first leverages the generative distributions of these source LLMs to externalize both their collective knowledge and individual strengths, and subsequently transfer them to the target LLM through lightweight continual training.
52
 
53
- Compared with model ensemble which requires the parallel deployment of multiple LLMs or weight merging which is generally limited to LLMs with identical architectures, FuseLLM supports the fusion of multiple LLMs with **diverse architectures** by explicitly transferring their knowledge and capabilities to a **single** target LLM.
54
 
55
  <p align="center">
56
  <img src="./assets/fig_1.png" width="95%"> <br>
@@ -59,26 +74,278 @@ Compared with model ensemble which requires the parallel deployment of multiple
59
 
60
  ## Model Release
61
 
62
- We release the FuseLLM-7B on [Huggingface Models](https://huggingface.co/models?sort=trending&search=FuseLLM), which is the fusion of three popular open-source LLMs that possess distinct architectures and functionalities: [Llama-2-7B](https://huggingface.co/meta-llama/Llama-2-7b-hf), [OpenLLaMA-7B](https://huggingface.co/openlm-research/open_llama_7b_v2), and [MPT-7B](https://huggingface.co/mosaicml/mpt-7b).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
 
64
- Evaluations across three benchmarks, which consist of a total of **42** tasks spanning reasoning, commonsense, and code generation, confirm that the target model trained by our method outperforms each source LLM and the casual language model baseline in most tasks.
65
 
66
  <p align="center">
67
- <img src="./assets/fig_2.png" width="95%"> <br>
68
  </p>
69
 
70
- To further illustrate the effectiveness of FuseLLM, we incorporate additional generative benchmarks related to knowledge-based question-answering, reading comprehension, content analysis, machine translation, and theorem application. The results highlight FuseLLM’s superiority over all source LLMs and the baseline.
 
 
71
 
72
  <p align="center">
73
- <img src="./assets/fig_3.png" width="95%"> <br>
74
  </p>
75
 
76
- Since FuseLLM is also applicable to instruction-tuning models, we assess the instruction-following performance on the Vicuna Benchmark using GPT-4 as an evaluator. The results demonstrate that FuseLLM surpasses each individual source instruction-tuning LLM and the baseline , achieving the best performance with GPT-4 judgment.
 
 
 
77
 
78
  <p align="center">
79
- <img src="./assets/fig_4.png" width="50%"> <br>
80
  </p>
81
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
 
83
  ## Citation
84
 
 
15
 
16
  <div id="top" align="center">
17
 
18
+ **Knowledge Fusion of Large Language Models**
19
+
20
 
21
  <h4> |<a href="https://arxiv.org/abs/2401.10491"> 📑 Paper </a> |
22
  <a href="https://huggingface.co/Wanfq/FuseLLM-7B"> 🤗 Model </a> |
 
38
 
39
 
40
  ## News
41
+ - **Jan 23, 2024:** 🔥🔥 We release the code for FuseLLM, including the data construction and model training process!
42
  - **Jan 22, 2024:** 🔥 We're excited to announce that the FuseLLM-7B, which is the fusion of [Llama-2-7B](https://huggingface.co/meta-llama/Llama-2-7b-hf), [OpenLLaMA-7B](https://huggingface.co/openlm-research/open_llama_7b_v2), and [MPT-7B](https://huggingface.co/mosaicml/mpt-7b), is now available on 🤗 [Huggingface Models](https://huggingface.co/Wanfq/FuseLLM-7B). Happy exploring!
43
 
44
+
45
+ ## WIP
46
+
47
+ | Source LLMs | Target LLM |
48
+ |------------------------------------------------------|-------------------|
49
+ | Mixtral-8x7B-v0.1, SOLAR-10.7B-v1.0, Mistral-7B-v0.1 | Mistral-7B-v0.1 |
50
+ | Mixtral-8x7B-v0.1, SOLAR-10.7B-v1.0, Mistral-7B-v0.1 | SOLAR-10.7B-v1.0 |
51
+ | Mixtral-8x7B-v0.1, SOLAR-10.7B-v1.0, Mistral-7B-v0.1 | Mixtral-8x7B-v0.1 |
52
+
53
  ## Contents
54
 
55
  - [Overview](#overview)
56
  - [Model Release](#model-release)
57
+ - [Quick Start](#quick-start)
58
+ - [Data Construction](#data-construction)
59
+ - [Training](#training)
60
+ - [Evaluation](#evaluation)
61
  - [Citation](#citation)
62
  - [Acknowledgements](#acknowledgments)
63
 
 
65
 
66
  In this study, we explore the realm of knowledge fusion for LLMs to create a unified model that combines the capabilities and distinctive strengths of multiple structurally diverse LLMs. To achieve this, we introduce FuseLLM, which first leverages the generative distributions of these source LLMs to externalize both their collective knowledge and individual strengths, and subsequently transfer them to the target LLM through lightweight continual training.
67
 
68
+ Unlike model ensemble approaches that require the **parallel deployment of multiple LLMs**, or weight merging techniques that are typically **limited to LLMs with identical architectures**, FuseLLM is designed to support **the fusion of multiple LLMs with diverse architectures into a more potent LLM**. By explicitly transferring their knowledge and capabilities to a single target LLM, FuseLLM offers a powerful and flexible solution for the knowledge fusion of LLMs.
69
 
70
  <p align="center">
71
  <img src="./assets/fig_1.png" width="95%"> <br>
 
74
 
75
  ## Model Release
76
 
77
+ We release the FuseLLM-7B on 🤗 [Huggingface Models](https://huggingface.co/models?sort=trending&search=FuseLLM), which is the fusion of three popular open-source LLMs that possess distinct architectures and functionalities: [Llama-2-7B](https://huggingface.co/meta-llama/Llama-2-7b-hf), [OpenLLaMA-7B](https://huggingface.co/openlm-research/open_llama_7b_v2), and [MPT-7B](https://huggingface.co/mosaicml/mpt-7b).
78
+
79
+ Here are the evaluation results of FuseLLM.
80
+
81
+ ### General Reasoning & Commonsense Reasoning
82
+
83
+ We first show the performance of FuseLLM on Big-Bench Hard and CommonSense benchmarks, which evaluate the general reasoning and commonsense reasoning abilities respectively.
84
+
85
+ <p align="center">
86
+ <img src="./assets/fig_4.png" width="95%"> <br>
87
+ </p>
88
+
89
+ ### Code Generation & Text Generation
90
+
91
+ We then evaluate FuseLLM on MultiPL-E, which is a multilingual programming benchmark to assess the code generation performance. We also conduct experiments on several text generation benchmarks, including TrivialQA (question-answering), DROP (reading comprehension), LAMBADA (content analysis), IWSLT2017 (machine translation), and SCIBench (theorem application).
92
+
93
+ <p align="center">
94
+ <img src="./assets/fig_5.png" width="95%"> <br>
95
+ </p>
96
+
97
+ ### Instruction Following
98
 
99
+ FuseLLM is also applicable to the fusion of instruction-tuned LLMs. We further evaluate the Vicuna Benchmark, which assesses the instruction following ability.
100
 
101
  <p align="center">
102
+ <img src="./assets/fig_6.png" width="50%"> <br>
103
  </p>
104
 
105
+ ### FuseLLM vs. Knowledge Distillation
106
+
107
+ As knowledge distillation is also a method for enhancing the performance of LLMs by utilizing representations, we compare FuseLLM with Llama-2 KD, which is distilled from Llama-2 13B.
108
 
109
  <p align="center">
110
+ <img src="./assets/fig_7.png" width="50%"> <br>
111
  </p>
112
 
113
+ ### FuseLLM vs. Model Ensemble & Weight Merging
114
+
115
+ To compare FuseLLM with existing fusion methods (such as model ensemble and weight merging), we simulate scenarios to ensure model fusion with an identical structure where multiple source LLMs are derived from the same base model but are continually trained on different corpus. We then test the perplexity of these fusion methods on different benchmarks.
116
+
117
 
118
  <p align="center">
119
+ <img src="./assets/fig_8.png" width="50%"> <br>
120
  </p>
121
 
122
+ ## Quick Start
123
+
124
+ ### Setup
125
+
126
+ We use `python 3.9` in this project.
127
+
128
+ Then, we have to install all the libraries listed in `requirements.txt`.
129
+
130
+ ```bash
131
+ pip install -r requirements.txt
132
+ ```
133
+
134
+ ### Usage
135
+
136
+ ```python
137
+ from transformers import AutoTokenizer, AutoModelForCausalLM
138
+ tokenizer = AutoTokenizer.from_pretrained("Wanfq/FuseLLM-7B", use_fast=False)
139
+ model = AutoModelForCausalLM.from_pretrained("Wanfq/FuseLLM-7B", torch_dtype="auto")
140
+ model.cuda()
141
+ inputs = tokenizer("<your text here>", return_tensors="pt").to(model.device)
142
+ tokens = model.generate(
143
+ **inputs,
144
+ max_new_tokens=512,
145
+ temperature=0.6,
146
+ top_p=0.9,
147
+ do_sample=True,
148
+ )
149
+ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
150
+ ```
151
+
152
+ We also find `Exllama v2 Quantizations` version on [FuseLLM-7B-exl2](https://huggingface.co/bartowski/FuseLLM-7B-exl2), it uses [ExLlamaV2 v0.0.11](https://github.com/turboderp/exllamav2/releases/tag/v0.0.11) for quantization.
153
+
154
+ ## Data Construction
155
+
156
+ We use the [MiniPile](https://huggingface.co/datasets/JeanKaddour/minipile) dataset for continual training.
157
+
158
+ Here we show the scripts to obtain representations from multiple LLMs for model fusion.
159
+
160
+ 1. Split long text
161
+
162
+ ```bash
163
+ python ./src/utils/split_long_text.py \
164
+ --base_model_name_or_path "<path_to_llama_2_7b>" \
165
+ --blending_model_name_or_path "<path_to_open_llama_7b_v2>" \
166
+ --another_blending_model_name_or_path "<path_to_mpt_7b>" \
167
+ --dataset "<path_to_minipile>" \
168
+ --dataset_save_dir "<path_to_minipile_split>" \
169
+ --cache_dir "<path_to_cache_dir>" \
170
+ --block_size 2048 \
171
+ --preprocessing_num_workers 80
172
+ ```
173
+
174
+ 2. Get representations for each LLM
175
+
176
+ ```bash
177
+ # We split the dataset into 8 splits, then process each split on a GPU.
178
+ # Please run this script for llama_2_7b, open_llama_7b_v2, and mpt_7b.
179
+ for i in {0..7}; do
180
+ export CUDA_VISIBLE_DEVICES=${i}
181
+ python ./src/utils/forward_for_logits.py \
182
+ --model_name_or_path "<path_to_each_model>" \
183
+ --dataset "<path_to_minipile_split>" \
184
+ --dataset_save_dir "${i}_8_<path_to_minipile_split_each_model_representation>" \
185
+ --dataset_split_num 8 \
186
+ --dataset_index ${i} \
187
+ --cache_dir "<path_to_cache_dir>" \
188
+ --model_max_length 2048 \
189
+ --training_mode full \
190
+ --load_in_half bf16 \
191
+ --batch_size 8 \
192
+ --preprocessing_num_workers 80 \
193
+ --top_k_logits 10 \
194
+ --save_per_token_metric 2>&1 > "${i}_8_<path_to_log_file>" 2>&1 &
195
+ unset CUDA_VISIBLE_DEVICES
196
+ sleep 30
197
+ done
198
+
199
+ wait
200
+ ```
201
+
202
+ 3. Align representations from different LLMs
203
+
204
+ ```bash
205
+ # Get vocab mapping from different LLMs.
206
+
207
+ # llama_2_7b <-> open_llama_7b_v2
208
+ python ./src/utils/vocab_mapping.py \
209
+ --base_model_name_or_path "<path_to_llama_2_7b>" \
210
+ --blending_model_name_or_path "<path_to_open_llama_7b_v2>" \
211
+ --dataset_dir "<path_to_minipile_split>" \
212
+ --vocab_mapping_save_dir "<path_to_llama_2_7b_open_llama_7b_v2_vocab_mapping>" \
213
+ --cache_dir "<path_to_cache_dir>" \
214
+ --model_max_length 2048 \
215
+ --vocab_mapping_type "default" \
216
+ --num_process 1
217
+
218
+ # llama_2_7b <-> mpt_7b
219
+ python ./src/utils/vocab_mapping.py \
220
+ --base_model_name_or_path "<path_to_llama_2_7b>" \
221
+ --blending_model_name_or_path "<path_to_mpt_7b>" \
222
+ --dataset_dir "<path_to_minipile_split>" \
223
+ --vocab_mapping_save_dir "<path_to_llama_2_7b_mpt_7b_vocab_mapping>" \
224
+ --cache_dir "<path_to_cache_dir>" \
225
+ --model_max_length 2048 \
226
+ --vocab_mapping_type "default" \
227
+ --num_process 1
228
+ ```
229
+
230
+ ```bash
231
+ # Align representations from different LLMs.
232
+
233
+ # llama_2_7b <-> open_llama_7b_v2
234
+ for i in {0..7}; do
235
+ python ./src/utils/token_alignment.py \
236
+ --base_model_name_or_path "<path_to_llama_2_7b>" \
237
+ --blending_model_name_or_path "<path_to_open_llama_7b_v2>" \
238
+ --base_dataset_dir "${i}_8_<path_to_minipile_split_llama_2_7b_representation>" \
239
+ --blending_dataset_dir "${i}_8_<path_to_minipile_split_open_llama_7b_v2_representation>" \
240
+ --dataset_save_dir "${i}_8_<path_to_minipile_split_llama_2_7b_open_llama_7b_v2_aligned_representation>" \
241
+ --cache_dir "<path_to_cache_dir>" \
242
+ --model_max_length 2048 \
243
+ --preprocessing_num_workers 80 \
244
+ --batch_size 100 \
245
+ --blending_model_index 0 \
246
+ --vocab_align_type "soft" \
247
+ --vocab_mapping_save_dir "<path_to_llama_2_7b_open_llama_7b_v2_vocab_mapping>" \
248
+ --metric_level "sequence"
249
+ done
250
+
251
+ # llama_2_7b <-> mpt_7b
252
+ for i in {0..7}; do
253
+ python ./src/utils/token_alignment.py \
254
+ --base_model_name_or_path "<path_to_llama_2_7b>" \
255
+ --blending_model_name_or_path "<path_to_mpt_7b>" \
256
+ --base_dataset_dir "${i}_8_<path_to_minipile_split_llama_2_7b_open_llama_7b_v2_aligned_representation>" \
257
+ --blending_dataset_dir "${i}_8_<path_to_minipile_split_mpt_7b_representation>" \
258
+ --dataset_save_dir "${i}_8_<path_to_minipile_split_llama_2_7b_open_llama_7b_v2_mpt_7b_aligned_representation>" \
259
+ --cache_dir "<path_to_cache_dir>" \
260
+ --model_max_length 2048 \
261
+ --preprocessing_num_workers 80 \
262
+ --batch_size 100 \
263
+ --blending_model_index 1 \
264
+ --vocab_align_type "soft" \
265
+ --vocab_mapping_save_dir "<path_to_llama_2_7b_mpt_7b_vocab_mapping>" \
266
+ --metric_level "sequence"
267
+ done
268
+ ```
269
+
270
+ 4. Packing all features to speed up training.
271
+
272
+ ```bash
273
+ for i in {0..7}; do
274
+ python3 ./src/utils/packing.py \
275
+ --dataset_dir "${i}_8_<path_to_minipile_split_llama_2_7b_open_llama_7b_v2_mpt_7b_aligned_representation>" \
276
+ --dataset_save_dir "${i}_8_<path_to_miniplie_fusellm_processed>" \
277
+ --cache_dir "<path_to_cache_dir>" \
278
+ --model_max_length 2048 \
279
+ --preprocessing_num_workers 80 \
280
+ --batch_size 1000 \
281
+ --metric_level "sequence"
282
+ ```
283
+
284
+ The final processed data is at `${i}_8_<path_to_miniplie_fusellm_processed>`, where `i in {0..7}`.
285
+
286
+ ## Training
287
+
288
+ Here, we show the script for FuseLLM training.
289
+
290
+ ```bash
291
+ export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
292
+
293
+ deepspeed --master_port=20001 ./src/train.py \
294
+ --training_mode full \
295
+ --deepspeed ./config/zero_stage2_config.json \
296
+ --model_name_or_path "<path_to_llama_2_7b>" \
297
+ --output_dir "<path_to_save_fusellm_7b>" \
298
+ --model_max_length 2048 \
299
+ --logging_steps 1 \
300
+ --save_strategy steps \
301
+ --save_steps 500 \
302
+ --save_total_limit 1 \
303
+ --evaluation_strategy steps \
304
+ --per_device_eval_batch_size 1 \
305
+ --logging_strategy steps \
306
+ --do_train \
307
+ --do_eval \
308
+ --bf16 True \
309
+ --tf32 True \
310
+ --warmup_ratio 0.008 \
311
+ --lr_scheduler_type cosine \
312
+ --dataset_name "0_8_<path_to_miniplie_fusellm_processed>,1_8_<path_to_miniplie_fusellm_processed>,2_8_<path_to_miniplie_fusellm_processed>,3_8_<path_to_miniplie_fusellm_processed>,4_8_<path_to_miniplie_fusellm_processed>,5_8_<path_to_miniplie_fusellm_processed>,6_8_<path_to_miniplie_fusellm_processed>,7_8_<path_to_miniplie_fusellm_processed>" \
313
+ --per_device_train_batch_size 1 \
314
+ --gradient_accumulation_steps 16 \
315
+ --num_train_epochs 1 \
316
+ --eval_steps 500 \
317
+ --optim adamw_torch \
318
+ --adam_beta1 0.9 \
319
+ --adam_beta2 0.95 \
320
+ --learning_rate 1e-5 \
321
+ --weight_decay 0.1 \
322
+ --max_grad_norm 1.0 \
323
+ --seed 42 \
324
+ --gradient_checkpointing True \
325
+ --use_flash_attn True \
326
+ --report_to tensorboard \
327
+ --do_distill \
328
+ --distill_with_ref_model True \
329
+ --distill_with_aligned_model_0 True \
330
+ --distill_with_aligned_model_1 True \
331
+ --distill_loss_type "ce" \
332
+ --distill_teacher_temperature 1.0 \
333
+ --lm_loss_weight 0.9 \
334
+ --distill_greater_as_gt True \
335
+ --distill_greater_as_gt_type "hard" \
336
+ --dataloader_num_workers 10 \
337
+ --remove_unused_columns False 2>&1 | tee "<path_to_log_file>"
338
+ ```
339
+
340
+ ## Evaluation
341
+
342
+ The evaluation code we used in our evaluation are list as follows:
343
+
344
+ - [Big-Bench Hard](https://github.com/allenai/open-instruct/tree/main/eval)
345
+ - [CommonSense: ARC-easy, ARC-challenge, BoolQ, HellaSwag, OpenBookQA](https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.3.0)
346
+ - [MultiPL-E](https://github.com/bigcode-project/bigcode-evaluation-harness)
347
+ - [Text Generation: TrivialQA, DROP, LAMBADA, IWSLT2017, SciBench](https://github.com/open-compass/opencompass)
348
+ - [Vicuna Bench](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge)
349
 
350
  ## Citation
351