Aratako
/

Swallow-MoE-4x7B-lisa

@@ -1,199 +1,86 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+base_model:
+- tokyotech-llm/Swallow-7b-hf
+- tokyotech-llm/Swallow-7b-instruct-hf
+- nitky/Superswallow-7b-v0.1
+- nitky/Superswallow-7b-v0.2
+- nitky/Superswallow-7b-v0.3
 library_name: transformers
+tags:
+- merge
+- moe
+- lisa
+license: cc-by-nc-sa-4.0
+datasets:
+- kunishou/amenokaku-code-instruct
+- llm-jp/oasst1-21k-en
+- hieunguyenminh/roleplay
+- meta-math/MetaMathQA
+- kunishou/jp-effective-instructions
+language:
+- ja
 ---
+# Swallow-MoE-4x7B-lisa
+## 概要
+[tokyotech-llm/Swallow-7b-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-hf)をベースに、以下の4モデルをgate_mode=randomでMoEし、その後[LISA](https://arxiv.org/abs/2403.17919)という手法でインストラクションチューニングを施したモデルです。
+- [tokyotech-llm/Swallow-7b-instruct-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-instruct-hf)
+- [nitky/Superswallow-7b-v0.1](https://huggingface.co/nitky/Superswallow-7b-v0.1)
+- [nitky/Superswallow-7b-v0.2](https://huggingface.co/nitky/Superswallow-7b-v0.2)
+- [nitky/Superswallow-7b-v0.3](https://huggingface.co/nitky/Superswallow-7b-v0.3)
+お試しで作ってみたものなので、性能にはあまり期待しないでください。以下にベンチマーク結果も記載しております。
+**なお、この学習で使ったLISAの実装には[不具合がある可能性](https://github.com/OptimalScale/LMFlow/issues/726)が指摘されており、正常に学習できていない可能性があります。**
+## データセット
+以下の合計14327件のデータを学習に利用しました。プロンプトフォーマットはAlpacaを利用しています。
+- [kunishou/amenokaku-code-instruct](https://huggingface.co/datasets/kunishou/amenokaku-code-instruct)の各sourceから最大100件、計1475件
+- [kunishou/jp-effective-instructions](https://huggingface.co/datasets/kunishou/jp-effective-instructions)のinstructionとoutputがともに11文字以上のデータ、計5050件
+- [llm-jp/oasst1-21k-en](https://huggingface.co/datasets/llm-jp/oasst1-21k-en)よりランダムな1000件（英語）
+- [hieunguyenminh/roleplay](https://huggingface.co/datasets/hieunguyenminh/roleplay)よりランダムな1000件（英語）
+- [meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA)よりランダムな1000件（英語）
+- [ichikara-instruction](https://liat-aip.sakura.ne.jp/wp/llm%E3%81%AE%E3%81%9F%E3%82%81%E3%81%AE%E6%97%A5%E6%9C%AC%E8%AA%9E%E3%82%A4%E3%83%B3%E3%82%B9%E3%83%88%E3%83%A9%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3%E3%83%87%E3%83%BC%E3%82%BF%E4%BD%9C%E6%88%90/llm%E3%81%AE%E3%81%9F%E3%82%81%E3%81%AE%E6%97%A5%E6%9C%AC%E8%AA%9E%E3%82%A4%E3%83%B3%E3%82%B9%E3%83%88%E3%83%A9%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3%E3%83%87%E3%83%BC%E3%82%BF-%E5%85%AC%E9%96%8B/)より、4802件
+なお、ichikara-instructionの利用によりCC-BY-NC-SAを継承します。
+## 学習の設定
+主な学習パラメータは以下の通りです。なお、学習途中でのエラーのため2epochs程度しか学習できておりません。
+- lisa_activated_layers: 8
+- lisa_interval_steps: 13
+- learning_rate: 5e-5
+- num_train_epochs: 約2epochs
+- batch_size: 64
+- max_seq_length: 2048
+## 評価
+マージに利用したモデル群と本モデルの[japanese-mt-bench](https://github.com/Stability-AI/FastChat/tree/jp-stable/fastchat/llm_judge)の結果は以下の通りです。（シングルターン）
+Swallow-instructよりはスコアが高く、Superswallowよりは低いという何とも言えない結果になっております。
+とはいえ、少量のデータセット・たった2epochsの学習でSwallow-instructを超えられているのは一定の成果とも言えるかもしれません。
+|Model|Size|Coding|Extraction|Humanities|Math|Reasoning|Roleplay|STEM|Writing|avg_score|
+|---|---|---|---|---|---|---|---|---|---|---|
+| Swallow-7b-instruct-hf | 7B | 2.0 | 4.6 | 5.4 | 1.7 | 2.8 | 5.0 | 5.9 | 6.9 | 4.2875 |
+| Superswallow-7b-v0.1 | 7B | 2.0 | 5.1 | 7.8 | 2.1 | 3.6 | 6.2 | 7.3 | 7.5 | 5.2000 |
+| Superswallow-7b-v0.2 | 7B | 2.2 | 5.8 | 6.7 | 2.5 | 4.3 | 5.5 | 6.6 | 5.8 | 4.9250 |
+| Superswallow-7b-v0.3 | 7B | 2.1 | 4.6 | 8.3 | 2.1 | 5.0 | 6.3 | 7.7 | 8.9 | 5.6250 |
+| **This model** | **4x7B** | **2.0** | **3.4** | **7.5** | **1.9** | **2.6** | **5.5** | **6.3** | **7.5** | **4.5875** |
+![レーダーチャート](./japanese_mt_bench.png)
+同様に、jsquad(jsquad-1.1-0.3, 2-shots)、jcommonsenseqa(jcommonsenseqa-1.1-0.3, 3-shots)、jnli(jnli-1.3-0.3, 3-shots)、marc_ja(marc_ja-1.1-0.3, 3-shots)結果は以下の通りです。（jsquadは100で割り、それぞれ小数点以下第4位を四捨五入）
+ここでもSwallow-instructよりはスコアが高く、Superswallowよりは低い結果になっています。なお、こちらは参考として本モデルのインストラクションチューニング前（MoEのみ）のモデルのスコアも載せてあります。
+|Model|Size|jsquad(exact_match)|jcommonsenseqa(acc)|jnli(acc)|marc_ja(acc)|average|
+|---|---|---|---|---|---|---|
+| Swallow-7b-instruct-hf | 7B | 0.757 | 0.831 | 0.212 | 0.945 | 0.686 |
+| Superswallow-7b-v0.1 | 7B | 0.441 | 0.846 | 0.374 | 0.966 | 0.657 |
+| Superswallow-7b-v0.2 | 7B | 0.722 | 0.846 | 0.381 | 0.964 | 0.728 |
+| Superswallow-7b-v0.3 | 7B | 0.721 | 0.850 | 0.362 | 0.964 | 0.724 |
+| **This model without fine-tuning** | **4x7B** | **0.674** | **0.809** | **0.333** | **0.952** | **0.692** |
+| **This model** | **4x7B** | **0.741** | **0.806** | **0.385** | **0.948** | **0.719** |