Qurtana commited on
Commit
0997768
·
verified ·
1 Parent(s): 4b742c5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -23,4 +23,8 @@ datasets:
23
 
24
  This model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
25
 
 
 
 
 
26
  [<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
 
23
 
24
  This model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
25
 
26
+ Trained using rank-stablized QLoRA with r = 64 and alpha = 5 for one epoch using the "ChatML" data prep.
27
+
28
+ The following heads were targeted: "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", and "lm_head".
29
+
30
  [<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)