Update README.md
Browse files
README.md
CHANGED
@@ -193,52 +193,3 @@ The training was done for 1 epochs We used 4 x [RTX 3090s](https://www.nvidia.c
|
|
193 |

|
194 |
|
195 |
|
196 |
-
optimizer: paged_adamw_8bit
|
197 |
-
lr_scheduler: constant_with_warmup
|
198 |
-
learning_rate: 1e-6
|
199 |
-
max_grad_norm: 0.01
|
200 |
-
|
201 |
-
train_on_inputs: false
|
202 |
-
group_by_length: false
|
203 |
-
bf16: auto
|
204 |
-
fp16:
|
205 |
-
tf32: true
|
206 |
-
|
207 |
-
gradient_checkpointing: unsloth
|
208 |
-
early_stopping_patience:
|
209 |
-
resume_from_checkpoint:
|
210 |
-
local_rank:
|
211 |
-
logging_steps: 1
|
212 |
-
xformers_attention:
|
213 |
-
flash_attention: true
|
214 |
-
|
215 |
-
warmup_steps: 25
|
216 |
-
evals_per_epoch: 4
|
217 |
-
eval_table_size:
|
218 |
-
eval_max_new_tokens: 128
|
219 |
-
saves_per_epoch: 1
|
220 |
-
debug:
|
221 |
-
deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16_cpuoffload_params.json
|
222 |
-
weight_decay: 0.0
|
223 |
-
fsdp:
|
224 |
-
fsdp_config:
|
225 |
-
|
226 |
-
|
227 |
-
```
|
228 |
-
|
229 |
-
</details><br>
|
230 |
-
|
231 |
-
## Credits
|
232 |
-
|
233 |
-
Thank you to [Lucy Knada](https://huggingface.co/lucyknada), [Intervitens](https://huggingface.co/intervitens),[Cgato](https://huggingface.co/cgato), [Kubernetes Bad](https://huggingface.co/kubernetes-bad) and the rest of [Anthracite](https://huggingface.co/anthracite-org)
|
234 |
-
|
235 |
-
|
236 |
-
## Training
|
237 |
-
The training was done for 1 epochs We used 4 x [RTX 3090s](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/) GPUs graciously provided by [Intervitens](https://huggingface.co/intervitens) for the fine-tuning of the model.
|
238 |
-
|
239 |
-
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
240 |
-
|
241 |
-
## Safety
|
242 |
-
|
243 |
-

|
244 |
-
|
|
|
193 |

|
194 |
|
195 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|