Llama-2-ko-OpenOrca-gugugo-13B

This model was trained for PoC purposes. This is part of an experiment to check whether model performance improves when fine-tuned with large data of about 1 million samples.

[Note] There are still many people/customers who have the wrong idea that 'Always the more data, the better,' so I showed it directly with experimental data. In fine-tuning, data quality is much more important than simply preparing a lot of data, and keyword distribution within the dataset is also important!

For example, when searching for process and comparison keywords in the kkullm dataset, each is about 1% of the entire dataset.

Model Details

• Base Model: beomi/llama-2-koen-13b

Datasets

Trained on 1 million samples from the dataset. The training infrastructure used AWS g5.12xlarge x 2ea (total of NVIDIA A10G 8 GPUs).

• OpenOrca-gugugo-ko

Hyperparameters

The hyperparameters are simply heuristic values. For reference only:

learning_rate = 3e-5
lr_scheduler = "constant_with_warmup"
batch_size = 1
gradient_accumulation_steps = 8
lora_alpha = 16
lora_r = 16
lora_dropout = 0.1
lora_target_modules = "[gate_proj, down_proj, up_proj, q_proj, k_proj, o_proj, v_proj]"
use_flash_attention_2 = True

License

• Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License, under LLAMA 2 COMMUNITY LICENSE AGREEMENT

This model was created as a personal experiment, unrelated to the organization I work for.