ehartford commited on
Commit
096d4a1
1 Parent(s): 0bbc170

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +237 -0
README.md ADDED
@@ -0,0 +1,237 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ base_model: meta-llama/Meta-Llama-3-8B
4
+ tags:
5
+ - generated_from_trainer
6
+ - axolotl
7
+ model-index:
8
+ - name: out
9
+ results: []
10
+ datasets:
11
+ - cognitivecomputations/Dolphin-2.9
12
+ - teknium/OpenHermes-2.5
13
+ - m-a-p/CodeFeedback-Filtered-Instruction
14
+ - cognitivecomputations/dolphin-coder
15
+ - cognitivecomputations/samantha-data
16
+ - HuggingFaceH4/ultrachat_200k
17
+ - microsoft/orca-math-word-problems-200k
18
+ - abacusai/SystemChat-1.1
19
+ - Locutusque/function-calling-chatml
20
+ - internlm/Agent-FLAN
21
+ ---
22
+
23
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
24
+ should probably proofread and complete it, then remove this comment. -->
25
+
26
+ # Dolphin 2.9 Llama 3 8b 🐬
27
+
28
+ Curated and trained by Eric Hartford, Lucas Atkins, and Fernando Fernandes, and Cognitive Computations
29
+
30
+ Discord: https://discord.gg/8fbBeC7ZGx
31
+
32
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/63111b2d88942700629f5771/ldkN1J0WIDQwU4vutGYiD.png" width="600" />
33
+
34
+ A bug has been found in the Dolphin 2.9 dataset in SystemConversations that causes the model to overly talk about the "SYSTEM MESSAGE". To counter this, we recommend you add a statement in the system message directing the model not to mention the system message. An example system message is "The assistant is named Dolphin. A helpful and friendly AI assistant, Dolphin avoids discussing the system message unless directly asked about it."
35
+
36
+ My appreciation for the sponsors of Dolphin 2.9:
37
+ - [Crusoe Cloud](https://crusoe.ai/) - provided excellent on-demand 10xL40S node
38
+
39
+ This model is based on Llama-3-8b, and is governed by [META LLAMA 3 COMMUNITY LICENSE AGREEMENT](LICENSE)
40
+
41
+ The base model has 8k context, and the full-weight fine-tuning was with 4k sequence length.
42
+
43
+ It took 2.5 days on 8x L40S provided by Crusoe Cloud
44
+
45
+ This model was trained FFT on all parameters, using ChatML prompt template format.
46
+
47
+ example:
48
+
49
+ ```
50
+ <|im_start|>system
51
+ You are Dolphin, a helpful AI assistant.<|im_end|>
52
+ <|im_start|>user
53
+ {prompt}<|im_end|>
54
+ <|im_start|>assistant
55
+
56
+ ```
57
+
58
+ Dolphin-2.9 has a variety of instruction, conversational, and coding skills. It also has initial agentic abilities and supports function calling.
59
+
60
+ Dolphin is uncensored. I have filtered the dataset to remove alignment and bias. This makes the model more compliant. You are advised to implement your own alignment layer before exposing the model as a service. It will be highly compliant with any requests, even unethical ones. Please read my blog post about uncensored models. https://erichartford.com/uncensored-models You are responsible for any content you create using this model. Enjoy responsibly.
61
+
62
+ Dolphin is licensed according to Meta's Llama license. I grant permission for any use, including commercial, that falls within accordance with Meta's Llama-3 license. Dolphin was trained on data generated from GPT4, among other models.
63
+
64
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
65
+ <details><summary>See axolotl config</summary>
66
+
67
+ axolotl version: `0.4.0`
68
+ ```yaml
69
+ base_model: meta-llama/Meta-Llama-3-8B
70
+ model_type: AutoModelForCausalLM
71
+ tokenizer_type: AutoTokenizer
72
+ tokenizer_use_fast: false
73
+
74
+
75
+ load_in_8bit: false
76
+ load_in_4bit: false
77
+ strict: false
78
+ model_config:
79
+
80
+ datasets:
81
+ - path: /workspace/datasets/dolphin-2.9/dolphin201-sharegpt2.jsonl
82
+ type: sharegpt
83
+ conversation: chatml
84
+ - path: /workspace/datasets/dolphin-2.9/Ultrachat200kunfiltered.jsonl
85
+ type: sharegpt
86
+ conversation: chatml
87
+ - path: /workspace/datasets/dolphin-2.9/dolphin-coder-translate-sharegpt2.jsonl
88
+ type: sharegpt
89
+ conversation: chatml
90
+ - path: /workspace/datasets/dolphin-2.9/dolphin-coder-codegen-sharegpt2.jsonl
91
+ type: sharegpt
92
+ conversation: chatml
93
+ - path: /workspace/datasets/dolphin-2.9/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
94
+ type: sharegpt
95
+ conversation: chatml
96
+ - path: /workspace/datasets/dolphin-2.9/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
97
+ type: sharegpt
98
+ conversation: chatml
99
+ - path: /workspace/datasets/dolphin-2.9/not_samantha_norefusals.jsonl
100
+ type: sharegpt
101
+ conversation: chatml
102
+ - path: /workspace/datasets/dolphin-2.9/Orca-Math-resort-unfiltered.jsonl
103
+ type: sharegpt
104
+ conversation: chatml
105
+ - path: /workspace/datasets/dolphin-2.9/agent_instruct_react_unfiltered.jsonl
106
+ type: sharegpt
107
+ conversation: chatml
108
+ - path: /workspace/datasets/dolphin-2.9/toolbench_instruct_j1s1_3k_unfiltered.jsonl
109
+ type: sharegpt
110
+ conversation: chatml
111
+ - path: /workspace/datasets/dolphin-2.9/toolbench_negative_unfiltered.jsonl
112
+ type: sharegpt
113
+ conversation: chatml
114
+ - path: /workspace/datasets/dolphin-2.9/toolbench_react_10p_unfiltered.jsonl
115
+ type: sharegpt
116
+ conversation: chatml
117
+ - path: /workspace/datasets/dolphin-2.9/toolbench_tflan_cot_30p_unfiltered.jsonl
118
+ type: sharegpt
119
+ conversation: chatml
120
+ - path: /workspace/datasets/dolphin-2.9/openhermes200k_unfiltered.jsonl
121
+ type: sharegpt
122
+ conversation: chatml
123
+ - path: /workspace/datasets/dolphin-2.9/SystemConversations.jsonl
124
+ type: sharegpt
125
+ conversation: chatml
126
+
127
+ chat_template: chatml
128
+
129
+
130
+ dataset_prepared_path: /workspace/datasets/dolphin-2.9/thingy
131
+ val_set_size: 0.0002
132
+ output_dir: ./out
133
+
134
+ sequence_len: 4096
135
+ sample_packing: true
136
+ pad_to_sequence_len: true
137
+
138
+ gradient_accumulation_steps: 4
139
+ micro_batch_size: 3
140
+ num_epochs: 3
141
+ logging_steps: 1
142
+ optimizer: adamw_8bit
143
+ lr_scheduler: cosine
144
+ learning_rate: 2e-5
145
+
146
+ wandb_project: dolphin-2.9-mixtral-8x22b
147
+ wandb_watch:
148
+ wandb_run_id:
149
+ wandb_log_model:
150
+
151
+ train_on_inputs: false
152
+ group_by_length: false
153
+ bf16: auto
154
+ fp16:
155
+ tf32: false
156
+
157
+ gradient_checkpointing: true
158
+ gradient_checkpointing_kwargs:
159
+ use_reentrant: false
160
+ early_stopping_patience:
161
+ resume_from_checkpoint:
162
+ local_rank:
163
+ logging_steps: 1
164
+ xformers_attention:
165
+ flash_attention: true
166
+ saves_per_epoch: 4
167
+ save_total_limit: 2
168
+ save_steps:
169
+ evals_per_epoch: 4
170
+ eval_sample_packing: false
171
+ debug:
172
+ deepspeed: deepspeed_configs/zero3_bf16.json
173
+ weight_decay: 0.05
174
+ fsdp:
175
+ fsdp_config:
176
+ special_tokens:
177
+ eos_token: "<|im_end|>"
178
+ pad_token: "<|end_of_text|>"
179
+ tokens:
180
+ - "<|im_start|>"
181
+ - "<|im_end|>"
182
+
183
+ ```
184
+
185
+ </details><br>
186
+
187
+ ## Quants
188
+
189
+ GGUF : https://huggingface.co/QuantFactory/dolphin-2.9-llama3-8b-GGUF
190
+
191
+ GGUF with imatrix: https://huggingface.co/bartowski/dolphin-2.9-llama3-8b-GGUF
192
+
193
+ Exllamav2: https://huggingface.co/bartowski/dolphin-2.9-llama3-8b-exl2
194
+
195
+ ## Training procedure
196
+
197
+ ### Training hyperparameters
198
+
199
+ The following hyperparameters were used during training:
200
+ - learning_rate: 2e-05
201
+ - train_batch_size: 3
202
+ - eval_batch_size: 3
203
+ - seed: 42
204
+ - distributed_type: multi-GPU
205
+ - num_devices: 8
206
+ - gradient_accumulation_steps: 4
207
+ - total_train_batch_size: 96
208
+ - total_eval_batch_size: 24
209
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
210
+ - lr_scheduler_type: cosine
211
+ - lr_scheduler_warmup_steps: 7
212
+ - num_epochs: 3
213
+
214
+ ### Training results
215
+
216
+ | Training Loss | Epoch | Step | Validation Loss |
217
+ |:-------------:|:------:|:----:|:---------------:|
218
+ | 1.146 | 0.0005 | 1 | 1.1064 |
219
+ | 0.6962 | 0.2501 | 555 | 0.6636 |
220
+ | 0.6857 | 0.5001 | 1110 | 0.6503 |
221
+ | 0.6592 | 0.7502 | 1665 | 0.6419 |
222
+ | 0.6465 | 1.0002 | 2220 | 0.6317 |
223
+ | 0.5295 | 1.2395 | 2775 | 0.6408 |
224
+ | 0.5302 | 1.4895 | 3330 | 0.6351 |
225
+ | 0.5188 | 1.7396 | 3885 | 0.6227 |
226
+ | 0.521 | 1.9896 | 4440 | 0.6168 |
227
+ | 0.3968 | 2.2289 | 4995 | 0.6646 |
228
+ | 0.3776 | 2.4789 | 5550 | 0.6619 |
229
+ | 0.3983 | 2.7290 | 6105 | 0.6602 |
230
+
231
+
232
+ ### Framework versions
233
+
234
+ - Transformers 4.40.0
235
+ - Pytorch 2.2.2+cu121
236
+ - Datasets 2.18.0
237
+ - Tokenizers 0.19.1