File size: 1,645 Bytes
03bf099
348797f
03bf099
111adf5
 
21b44fc
111adf5
348797f
111adf5
 
 
348797f
 
 
 
 
 
 
 
 
 
a3c29b2
 
 
 
 
 
 
 
 
 
 
 
348797f
 
a3c29b2
348797f
a3c29b2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
---
library_name: peft
---
## Introduction
Krakowiak-7B is a finetuned version of Meta's [Llama2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf). It was trained on the modified and updated dataset originally created by [Chris Ociepa](https://huggingface.co/datasets/szymonindy/ociepa-raw-self-generated-instructions-pl)
containing ~ 50K instructions. Making it one of the best and biggest available LLM's.
Name [krakowiak](https://www.youtube.com/watch?v=OeQ6jYzt6cM) refers to one of the most popular and characteristic Polish folk dances, with its very lively, even wild, tempo, and long, easy strides, demonstrating spirited abandon and elegance at the same time.

## How to test it?
The model can be ran using the Huggingface library or in the browser using this [Google Colab](https://colab.research.google.com/drive/1IM7j57g9ZHj-Pw2EXGyacNuKHjvK3pIc?usp=sharing)
## Training procedure

The following `bitsandbytes` quantization config was used during training:
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: bfloat16

The following `bitsandbytes` quantization config was used during training:
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: bfloat16
### Framework versions

- PEFT 0.4.0

- PEFT 0.4.0