README.md

---
base_model: FuseAI/FuseChat-Llama-3.2-3B-Instruct
tags:
- llama-cpp
- gguf-my-repo
license: llama3.2
---

# Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q5_K_M-GGUF
This model was converted to GGUF format from [`FuseAI/FuseChat-Llama-3.2-3B-Instruct`](https://huggingface.co/FuseAI/FuseChat-Llama-3.2-3B-Instruct) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/FuseAI/FuseChat-Llama-3.2-3B-Instruct) for more details on the model.

---
Model details:
-
We present FuseChat-3.0, a series of models crafted to enhance 
performance by integrating the strengths of multiple source LLMs into 
more compact target LLMs. To achieve this fusion, we utilized four 
powerful source LLMs: Gemma-2-27B-It, Mistral-Large-Instruct-2407, 
Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For the target LLMs, 
we employed three widely-used smaller models—Llama-3.1-8B-Instruct, 
Gemma-2-9B-It, and Qwen-2.5-7B-Instruct—along with two even more compact
 models—Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. The implicit 
model fusion process involves a two-stage training pipeline comprising 
Supervised Fine-Tuning (SFT) to mitigate distribution discrepancies 
between target and source LLMs, and Direct Preference Optimization (DPO)
 for learning preferences from multiple source LLMs. The resulting 
FuseChat-3.0 models demonstrated substantial improvements in tasks 
related to general conversation, instruction following, mathematics, and
 coding. Notably, when Llama-3.1-8B-Instruct served as the target LLM, 
our fusion approach achieved an average improvement of 6.8 points across
 14 benchmarks. Moreover, it showed significant improvements of 37.1 and
 30.1 points on instruction-following test sets AlpacaEval-2 and 
Arena-Hard respectively. We have released the FuseChat-3.0 models on Huggingface, stay tuned for the forthcoming dataset and code.



	
		
	

		Overview
	



Combining the strengths of multiple large language models (LLMs) 
represents a promising approach to enhance individual model 
capabilities. Model fusion is a technique that integrates the strengths 
of robust source LLMs into a target LLM.


Previous iterations of the FuseChat
 series employed probabilistic distribution matrices generated by source
 models to transfer knowledge to target models. We refer to this method 
as explicit model fusion (EMF) because it involves a 
well-defined knowledge transfer process. While applicable to models with
 varying architectures and sizes, and without increasing memory overhead
 during inference, this approach presents notable challenges such as 
vocabulary alignment and the merging of distribution matrices from 
different LLMs. These issues complicate model fusion, reduce its 
efficiency, and may introduce noise and errors and affect the fusion 
results.


FuseChat-3.0, however, takes a different approach by enhancing a 
single LLM through implicit learning from robust open-source LLMs, a 
process we term implicit model fusion (IMF). The 
concept of IMF has been widely utilized to improve the performance of 
weaker models. For instance, a weak model can be boosted through 
fine-tuning with outputs from stronger LLMs. Moreover, a reward model 
can be trained using outputs from various LLMs, enabling it to learn and
 capture the differences in capabilities between the LLMs. Zephyr 
further collects responses from multiple LLMs and ranks them with GPT-4 
to obtain preference data for training the policy. Inspired by recent 
alignment techniques, we propose an IMF method to transfer the 
capabilities of source LLMs to a target LLM through preference 
optimization.


Our IMF method follows a three-stage process aimed at effectively 
transferring capabilities from source LLMs to a target LLM. First, 
during dataset construction, we sample N responses from
 each of the source LLMs and annotate these responses using an external 
reward model. Second, in the supervised fine-tuning (SFT)
 stage, we fine-tune the target model using the best responses, which 
not only enhances the target model's capabilities but also helps 
mitigate the distributional gap between the source and target models. 
Finally, in the direct preference optimization (DPO) 
stage, we optimize the target model by using the best and worst 
responses from the source models as preference pairs, further enhancing 
the target model's performance. The complete pipeline will be detailed 
in the following paragraph.



	
		
	

		Dataset
	




	
		
	

		Prompt Selection
	



Our datasets were designed to enhance model's instruction following, 
general conversation, mathematics, coding, and Chinese-language 
capabilities. We selected data from open-source community datasets, 
applying targeted filtering and preprocessing. Key datasets and 
filtering criteria included:


Instruction Following & General Conversation: Sourced from UltraFeedback, Magpie-Pro-DPO-100K-v0.1, and HelpSteer2, excluding code and math data.
Mathematics: Selected from OpenMathInstruct-2, with nearly 60,000 unique samples.
Coding: Curated from leetcode and self-oss-instruct-sc2-exec-filter-50k, retaining prompts with test cases.
Chinese Language: Integrated alpaca_gpt4_zh and Magpie-Qwen2-Pro-200K-Chinese, filtering out code and math prompts to retain approximately 10,000 high-quality samples.



	
		
	

		Response Sampling
	



For each dataset's prompts, we synthesized responses mainly from four different series of source models, specifically Gemma-2-27b-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.


Instruction Following & General Conversation: We sampled each prompt five times from all the source models.
Mathematics: We retained the responses generated by
 Llama-3.1-405B-Instruct from the original dataset (OpenMathInstruct-2) 
and additionally sampled responses using Qwen-2.5-Math-72B-Instruct.
Coding: We sampled each prompt eight times for all source models.
Chinese Language: We included single response sampled exclusively from Qwen-2.5-72B-Instruct.

---
## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)

```bash
brew install llama.cpp

```
Invoke the llama.cpp server or the CLI.

### CLI:
```bash
llama-cli --hf-repo Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q5_K_M-GGUF --hf-file fusechat-llama-3.2-3b-instruct-q5_k_m.gguf -p "The meaning to life and the universe is"
```

### Server:
```bash
llama-server --hf-repo Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q5_K_M-GGUF --hf-file fusechat-llama-3.2-3b-instruct-q5_k_m.gguf -c 2048
```

Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```

Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```

Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q5_K_M-GGUF --hf-file fusechat-llama-3.2-3b-instruct-q5_k_m.gguf -p "The meaning to life and the universe is"
```
or 
```
./llama-server --hf-repo Triangle104/FuseChat-Llama-3.2-3B-Instruct-Q5_K_M-GGUF --hf-file fusechat-llama-3.2-3b-instruct-q5_k_m.gguf -c 2048
```