File size: 1,364 Bytes
9c3a0b8 738d8c2 9c3a0b8 48b6e9f 738d8c2 9c3a0b8 8624ed0 738d8c2 845252b 738d8c2 5024ed7 9c3a0b8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
base_model: LumiOpen/Viking-7B
language:
- en
- fi
- sv
- 'no'
- da
- is
- nn
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
datasets:
- Gryphe/Sonnet3.5-SlimOrcaDedupCleaned
- mpasila/Sonnet3.5-SlimOrcaDedupCleaned-4k-context
---
This is a test model because the previous attempt failed. So it turns out I also trained this incorrectly due to the dataset using wrongly formatted ShareGPT so after I trained it again "correctly" for another full epoch I noticed that so.. I will have to train it yet again.. but at least now it should be fixed. But this model is using the wrongly formatted dataset. The next model should be better.
Prompt format is: ChatML
LoRA: [mpasila/Viking-SlimSonnet-v0.2-LoRA-7B](https://huggingface.co/mpasila/Viking-SlimSonnet-v0.2-LoRA-7B)
Trained with regular LoRA (not quantized/QLoRA) and LoRA rank was 128 and Alpha set to 32. Trained for 5000 steps (0.11 epoch).
# Uploaded model
- **Developed by:** mpasila
- **License:** apache-2.0
- **Finetuned from model :** LumiOpen/Viking-7B
This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|