Update README.md
Browse files
README.md
CHANGED
@@ -114,19 +114,35 @@ model-index:
|
|
114 |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TeeZee/NEBULA-23B-v1.0
|
115 |
name: Open LLM Leaderboard
|
116 |
---
|
117 |
-
|
118 |
|
119 |
![image/png](https://huggingface.co/TeeZee/NEBULA-23B-v1.0/resolve/main/NEBULA-23B-v1.0.jpg)
|
120 |
|
121 |
- 108 layers,DUS procedure, mistral(32)->SOLAR(48)->GALAXY(72)->NEBULA(108)
|
|
|
122 |
- model created as a extension of depth upscaling procedure used for SOLAR by upstage
|
123 |
|
124 |
-
|
125 |
- model can and will produce NSFW content
|
126 |
-
-
|
|
|
|
|
|
|
|
|
127 |
|
128 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
129 |
- Alpaca
|
|
|
|
|
|
|
|
|
|
|
130 |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
131 |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_TeeZee__NEBULA-23B-v1.0)
|
132 |
|
|
|
114 |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TeeZee/NEBULA-23B-v1.0
|
115 |
name: Open LLM Leaderboard
|
116 |
---
|
117 |
+
# NEBULA-23.8B-v1.0
|
118 |
|
119 |
![image/png](https://huggingface.co/TeeZee/NEBULA-23B-v1.0/resolve/main/NEBULA-23B-v1.0.jpg)
|
120 |
|
121 |
- 108 layers,DUS procedure, mistral(32)->SOLAR(48)->GALAXY(72)->NEBULA(108)
|
122 |
+
- 23.8B parameters
|
123 |
- model created as a extension of depth upscaling procedure used for SOLAR by upstage
|
124 |
|
125 |
+
## Results
|
126 |
- model can and will produce NSFW content
|
127 |
+
- GSM8k evaluation seems to be often broken, HellaSwag, Winograde and TQA show that its a smart model
|
128 |
+
- RP and ERP work suprizingly good and I didn't encounter any GPTismsm yet
|
129 |
+
- lower memory footprint than 20B and 23B models
|
130 |
+
- follows character card very well
|
131 |
+
- NSFW output feels fresh comparing to existing models
|
132 |
|
133 |
+
## Finetuning for RP
|
134 |
+
- SFT using MinervaAI/Aesir-Preview dataset, 10 epochs
|
135 |
+
- DPO using athirdpath/DPO_Pairs-Roleplay-Alpaca-NSFW dataset, 1 epoch
|
136 |
+
- SFT using 1xAda6000, DPO 1x3090, around 24h for each
|
137 |
+
- jupyter notebooks or mergekit configs for anyone wanting to reproduce/reuse scripts - just drop me a message
|
138 |
+
|
139 |
+
## Prompt template
|
140 |
- Alpaca
|
141 |
+
- chat template is embedded in tokenizer config, should load automatically
|
142 |
+
|
143 |
+
All comments are greatly appreciated, download, test and if you appreciate my work, consider buying me my fuel:
|
144 |
+
<a href="https://www.buymeacoffee.com/TeeZee" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>
|
145 |
+
|
146 |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
147 |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_TeeZee__NEBULA-23B-v1.0)
|
148 |
|