Hugging Face's logo

AIJapanese
/
Moriyasu_Qwen2_JP_7B

Text Generation
Transformers
Safetensors
Japanese
qwen2
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
Moriyasu_Qwen2_JP_7B
File size: 7,001 Bytes 
1c9a1ee
 
 
 
 
 
73b751c
 
fd60b6b
15c7188
0696ded
 
02210a7
fd60b6b
 
 
 
 
 
 
 
8f71e0d
3501a5e
 
00022e8
36b32c3
 
ac81f2e
36b32c3
1f0fe1b
fd60b6b
bf25490
 
 
 
 
 
 
 
 
 
 
00022e8
bf25490
 
 
 
 
 
 
 
 
 
6dc4fa5
bf25490
 
1781c0a
bf25490
1781c0a
 
 
 
 
 
00022e8
3f8b407
 
ac81f2e
 
 
 
 
 
 
00022e8
ac81f2e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
efea65e
ac81f2e
 
 
 
 
 
 
 
 
c6c7d16
 
 
 
 
 
 
 
 
 
 
9a6fd46
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
 
---
license: apache-2.0
language:
- ja
pipeline_tag: text-generation
library_name: transformers
---

# Moriyasu_Qwen2_JP_7B

### Model Description

**Moriyasu_Qwen2_JP_7B** is a large language model trained by Moriyasu. Based on [Qwen/Qwen2-7B](https://huggingface.co/Qwen/Qwen2-7B), it has been enhanced for Japanese usage through additional pre-training and instruction tuning.  

# Model Performance

### JGLUE tasks
We used the [lm-evaluation-harness](https://github.com/Stability-AI/lm-evaluation-harness/tree/jp-stable) repo to evaluate across 8 tasks, and the results are as follows:


|Model|JCommonsenseQA|JNLI|JMARC|JSQuAD|JAQKET-V2|XL-SUM|XWINOGRAD|MGSM|JA AVG|
|---|---|---|---|---|---|---|---|---|---|
|   |3-shot|3-shot|0-shot|2-shot|1-shot|1-shot|0-shot|5-shot|   |
|   |Acc.|Balanced Acc.|Balanced Acc.|Char-F1|Char-F1|ROUGE-2|Acc.|Acc.|   |
| Moriyasu_Qwen2_JP_7B (ours) | **0.9491** | **0.9111** | 0.9550 | 0.8748 | 0.8924 | 0.1966 | **0.8238** | 0.5560 | **0.7699** | 
| Qwen2-7B-Instruct | 0.9080 | 0.7807 | 0.9329 | 0.9290 | 0.8334 | 0.1905 | 0.7216 | **0.6120** | 0.7385 | 
| SakanaAI/EvoLLM-JP-v1-7B | 0.8919 | 0.6602 | 0.9555 | 0.9210 | 0.8641 | **0.2331** | 0.8165 | 0.4760 | 0.7273 |
| Llama-3-ELYZA-JP-8B |0.9240 | 0.6485 | **0.9567** | 0.9204 | 0.8743 | 0.2135 | 0.7821 | 0.4920 | 0.7264 |
| Llama-3-Swallow-8B-Instruct-v0.1 | 0.9249 | 0.6212 | 0.9427 | **0.9373** | **0.9083** | 0.1961 | 0.7404 | 0.5000 | 0.7214 | 
| Tanuki-8B-dpo-v1.0| 0.7918 | 0.4305 | 0.9226 | 0.8229 | 0.7799 | 0.1168 | 0.7039 | 0.4360 | 0.6256 | 


### Japanese tasks

For this evaluation, we used [swallow-evaluation](https://github.com/swallow-llm/swallow-evaluation) repo to evaluate our model.
The results of other models are taken from the report
[Llama-3.1-Swallow-8B-Instruct-v0.2](https://huggingface.co/tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2) .

|Model|JCom.|JEMHopQA|NIILC|JSQuAD|XL-Sum|MGSM|WMT20-en-ja|WMT20-ja-en|JMMLU|JHumanEval|Ja Avg|
|---|---|---|---|---|---|---|---|---|---|---|---|
|   |4-shot|4-shot|4-shot|4-shot|1-shot|4-shot|4-shot|4-shot|5-shot|0-shot|   |
|   |EM acc|Char-F1|Char-F1|Char-F1|ROUGE-2|EM acc|BLEU|BLEU|EM acc|pass@1|   |
| Moriyasu_Qwen2_JP_7B (ours)| **0.9321** | 0.4823 | **0.6046** | **0.9201** | 0.1382 | 0.5560 | 0.2636 | 0.1892 | 0.5273 | 0.2976 | 0.4911 |
| RakutenAI-7B-chat | 0.9035 | 0.2600 | 0.4619 | 0.8647 | 0.1339 | 0.2120 | 0.2667 | 0.1966 | 0.4504 | 0.2299 | 0.3980 |
| Qwen2-7B-Instruct | 0.8856 | 0.3902 | 0.3859 | 0.8967 | 0.1277 | 0.5720 | 0.2041 | 0.1909 | 0.5713 | **0.5683** | 0.4793 |
| Qwen2.5-7B-Instruct | 0.9151 | 0.4293 | 0.3910 | 0.8908 | 0.1676 | **0.6240** | 0.2108 | 0.1916 | **0.6252** | 0.5305 | 0.4976 |
| Tanuki-8B-dpo-v1.0 | 0.2770 | 0.2937 | 0.3710 | 0.6669 | 0.1016 | 0.4280 | 0.2385 | 0.1820 | 0.3078 | 0.2555 | 0.3122 |
| Llama 3 8B Instruct | 0.8785 | 0.3812 | 0.3936 | 0.8955 | 0.1273 | 0.4160 | 0.2143 | 0.2035 | 0.4719 | 0.2872 | 0.4269 |
| Llama 3.1 8B Instruct | 0.8829 | 0.4272 | 0.4112 | 0.8856 | 0.1481 | 0.5280 | 0.2174 | 0.1990 | 0.5086 | 0.4976 | 0.4706 |
| Llama 3 Youko 8B Instruct | 0.9196 | 0.4850 | 0.5178 | 0.9001 | 0.2085 | 0.4680 | 0.2559 | 0.1906 | 0.4691 | 0.2695 | 0.4684 |
| Llama-3-ELYZA-JP-8B | 0.9017 | 0.5124 | 0.5016 | 0.9113 | 0.1677 | 0.4600 | 0.2509 | 0.1846 | 0.4829 | 0.3811 | 0.4754 |
| Llama 3 heron brain 8B v0.3 | 0.9231 | 0.4933 | 0.5694 | 0.9056 | **0.2178** | 0.4560 | 0.2771 | 0.2168 | 0.4993 | 0.3177 | 0.4876 |
| Llama 3 Swallow 8B Instruct | 0.9178 | 0.4963 | 0.5168 | 0.9088 | 0.1296 | 0.4880 | 0.2522 | 0.2254 | 0.4835 | 0.3927 | 0.4811 |
| Llama 3.1 Swallow 8B Instruct v0.1| 0.9240 | **0.5874** | 0.5736 | 0.9170 | 0.1380 | 0.5080 | 0.2820 | **0.2282** | 0.5301 | 0.3665 | 0.5055 |
| Llama 3.1 Swallow 8B Instruct v0.2| 0.9294 | 0.5601 | 0.5988 | 0.9148 | 0.1372 | 0.5280 | **0.2878** | 0.2270 | 0.5504 | 0.4079 | **0.5141** |

### Japanese MTBench

For this evaluation, we use [FastChat](https://github.com/Stability-AI/FastChat/tree/jp-stable) and **gpt-4o-2024-08-06** for judgement and reference answer. 

Due to limited computational resources, we conducted evaluations on only a select number of models.

|Model|coding|extraction|humanities|math|reasoning|roleplay|stem|writing|JMTAvg|
|---|---|---|---|---|---|---|---|---|---|
| Moriyasu_Qwen2_JP_7B (ours)       | **0.515** | 0.710 | **0.845** | **0.685** | **0.585** | **0.815** | **0.710** | **0.765** | **0.704** |
| Llama-3-ELYZA-JP-8B               | 0.365 | **0.72** | 0.730 | 0.400 | 0.555 | 0.670 | 0.580 | 0.785 | 0.601 |
| Llama 3.1 Swallow 8B Instruct v0.1| 0.480 | 0.680 | 0.705 | 0.475 | 0.425 | 0.710 | 0.620 | 0.645 | 0.592 |

### Elyza task 100:

For this benchmark, we use  [Elyza task 100](https://huggingface.co/datasets/elyza/ELYZA-tasks-100) dataset and gpt4o scoring prompt of Elyza. Link prompt from [this blog](https://zenn.dev/elyza/articles/7ece3e73ff35f4)

|Model|Score|
|---|---|
| Moriyasu_Qwen2_JP_7B (ours)       | 3.37 |
| Llama-3-ELYZA-JP-8B               | **3.66** | 
| Llama 3.1 Swallow 8B Instruct v0.1| 3.32 |

### Nejumi leaderboard 3
We will contact Nejumi soon to evaluate on this benchmark


# Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
path = 'AIJapanese/Moriyasu_Qwen2_JP_7B'
model = AutoModelForCausalLM.from_pretrained(
    path,
    torch_dtype=torch.bfloat16, 
    device_map="auto",
    use_cache=True
)
tokenizer = AutoTokenizer.from_pretrained(path)

system_prompt = "あなたは誠実で優秀な日本人アシスタントです。常に可能な限り最も役立つ回答を提供するように努めてください。"
prompt = "日本で一番高い山は何ですか "
conversation = [{"role": "system", "content": system_prompt }]
conversation.append({"role": "user", "content": prompt})
text = tokenizer.apply_chat_template(
    conversation,
    tokenize=False,
    add_generation_prompt=True)

model_inputs = tokenizer(text,return_tensors="pt").to(model.device)
generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=2048,
    temperature = 0.2,
    #top_p=0.95,
    #top_k=40,
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
```

# Training Datasets

### Pre-training dataset

The model is continually pre-trained on Japanese data from the Qwen2-7b model while maintaining the model's English ability (80% Japanese, 20% English). We use about 120 billion tokens sampled from, Japanese and English Wikipedia articles, Japanese CC-100 Japanese C4, Japanese OSCAR ,The Pile, Webfined, Japanese websites, book data, mathematics and code,...

### Instruction Tuning
We generated about 1 million Instruction data from various methods such as generated data, translated data, and data manually tagged by humans.

# Contact:
If you have any questions, please contact me at: [email protected]