README.md

---
license: other
base_model:
- beomi/Llama-3-Open-Ko-8B-Instruct-preview
- cognitivecomputations/dolphin-2.9-llama3-8b
- NousResearch/Meta-Llama-3-8B-Instruct
- NousResearch/Meta-Llama-3-8B
- abacusai/Llama-3-Smaug-8B
- Locutusque/Llama-3-Orca-1.0-8B
library_name: transformers
tags:
- mergekit
- merge
- llama

---
# 🇰🇷 SmartLlama-3-Ko-8B
<a href="https://ibb.co/C8Tcw1F"><img src="https://i.ibb.co/QQ1gJbG/smartllama3.png" alt="smartllama3" border="0"></a><br />

SmartLlama-3-Ko-8B is a sophisticated AI model that integrates the capabilities of several advanced language models. This merged model is designed to excel in a variety of tasks ranging from technical problem-solving to multilingual communication.

## 📕 Merge Details

### Component Models and Contributions

### 1. NousResearch/Meta-Llama-3-8B and Meta-Llama-3-8B-Instruct
- **General Language Understanding and Instruction-Following**: These base models provide a robust foundation in general language understanding. The instruct version is optimized to follow detailed user instructions, enhancing the model's utility in task-oriented dialogues.

### 2. cognitivecomputations/dolphin-2.9-llama3-8b
- **Complex Problem-Solving and Depth of Understanding**: Enhances the model's capabilities in technical and scientific domains, improving its performance in complex problem-solving and areas requiring intricate understanding.

### 3. abacusai/Llama-3-Smaug-8B
- **Multi-Turn Conversational Abilities**: Improves performance in real-world multi-turn conversations, crucial for applications in customer service and interactive learning.A multi-turn conversation refers to a dialogue that consists of several back-and-forth exchanges between participants. Unlike a single-turn interaction, where the conversation might end after one question and one response, multi-turn conversations require ongoing engagement from both sides. In such conversations, the context from previous messages is often crucial in shaping the response of each participant, making it necessary for them to remember or keep track of what was said earlier.For AI systems like chatbots or virtual assistants, the ability to handle multi-turn conversations is crucial. It allows the AI to engage more naturally and effectively with users, simulating human-like interactions. This capability is particularly important in customer service, where understanding the history of a customer’s issue can lead to more accurate and helpful responses, or in scenarios like therapy or tutoring, where the depth of the conversation can significantly impact the effectiveness of the interaction.

### 4. Locutusque/Llama-3-Orca-1.0-8B
- **Specialization in Math, Coding, and Writing**: Enhances the model's ability to handle mathematical equations, generate computer code, and produce high-quality written content.

### 5. beomi/Llama-3-Open-Ko-8B-Instruct-preview
- **Enhanced Korean Language Capabilities**: Specifically trained to understand and generate Korean, valuable for bilingual or multilingual applications targeting Korean-speaking audiences.

### Merging Technique: DARE TIES
- **Balanced Integration**: The DARE TIES method ensures that each component model contributes its strengths in a balanced manner, maintaining a high level of performance across all integrated capabilities.

### Overall Capabilities
SmartLlama-3-Ko-8B is highly capable and versatile, suitable for:

- **Technical and Academic Applications**: Enhanced capabilities in math, coding, and technical writing.
- **Customer Service and Interactive Applications**: Advanced conversational skills and sustained interaction handling.
- **Multilingual Communication**: Specialized training in Korean enhances its utility in global or region-specific settings.

This comprehensive capability makes SmartLlama-3-Ko-8B not only a powerful tool for general-purpose AI tasks but also a specialized resource for industries and applications demanding high levels of technical and linguistic precision.

## 🖋️ Merge Method

This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using [NousResearch/Meta-Llama-3-8B](https://huggingface.co/NousResearch/Meta-Llama-3-8B) as a base.

## 🎭 Models Merged

The following models were included in the merge:
* [beomi/Llama-3-Open-Ko-8B-Instruct-preview](https://huggingface.co/beomi/Llama-3-Open-Ko-8B-Instruct-preview)
* [cognitivecomputations/dolphin-2.9-llama3-8b](https://huggingface.co/cognitivecomputations/dolphin-2.9-llama3-8b)
* [NousResearch/Meta-Llama-3-8B-Instruct](https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct)
* [abacusai/Llama-3-Smaug-8B](https://huggingface.co/abacusai/Llama-3-Smaug-8B)
* [Locutusque/Llama-3-Orca-1.0-8B](https://huggingface.co/Locutusque/Llama-3-Orca-1.0-8B)

## 🗞️ Configuration

The following YAML configuration was used to produce this model:

```yaml
models:
  - model: NousResearch/Meta-Llama-3-8B
    # Base model providing a general foundation without specific parameters
  - model: NousResearch/Meta-Llama-3-8B-Instruct
    parameters:
      density: 0.58
      weight: 0.25  
  - model: cognitivecomputations/dolphin-2.9-llama3-8b
    parameters:
      density: 0.52
      weight: 0.15  
  - model: Locutusque/Llama-3-Orca-1.0-8B
    parameters:
      density: 0.52
      weight: 0.15  
  - model: abacusai/Llama-3-Smaug-8B
    parameters:
      density: 0.52
      weight: 0.15  
  - model: beomi/Llama-3-Open-Ko-8B-Instruct-preview
    parameters:
      density: 0.53
      weight: 0.2   
merge_method: dare_ties
base_model: NousResearch/Meta-Llama-3-8B
parameters:
  int8_mask: true
dtype: bfloat16


```

### 🎊 Test Result

**Korean Multi Turn Conversation**
<a href="https://ibb.co/TKPGx9G"><img src="https://i.ibb.co/0BYLRHL/Screenshot-2024-04-30-at-2-42-18-PM.png" alt="Screenshot-2024-04-30-at-2-42-18-PM" border="0"></a>
<a href="https://ibb.co/v40tkNj"><img src="https://i.ibb.co/hF3qVGm/Screenshot-2024-04-30-at-8-26-57-AM.png" alt="Screenshot-2024-04-30-at-8-26-57-AM" border="0"></a>

**Programming**
<a href="https://ibb.co/6tZLqwx"><img src="https://i.ibb.co/n10tKmv/Screenshot-2024-04-30-at-8-30-35-AM.png" alt="Screenshot-2024-04-30-at-8-30-35-AM" border="0"></a>

**Physics & Math**
<a href="https://ibb.co/jDhVNk0"><img src="https://i.ibb.co/jDhVNk0/Screenshot-2024-04-30-at-1-06-16-PM.png" alt="Screenshot-2024-04-30-at-1-06-16-PM" border="0"></a>
<a href="https://ibb.co/KKgN4j5"><img src="https://i.ibb.co/KKgN4j5/Screenshot-2024-04-30-at-1-06-31-PM.png" alt="Screenshot-2024-04-30-at-1-06-31-PM" border="0"></a>
<a href="https://ibb.co/ZzKHP5j"><img src="https://i.ibb.co/ZzKHP5j/Screenshot-2024-04-30-at-1-06-47-PM.png" alt="Screenshot-2024-04-30-at-1-06-47-PM" border="0"></a>