See axolotl config

axolotl version: 0.4.0

base_model: meta-llama/Meta-Llama-3-8B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
#  - path: taozi555/bagel
#    type: sharegpt
  - path: MinervaAI/Aesir-Preview
    type: sharegpt
  - path: KaraKaraWitch/PIPPA-ShareGPT-formatted
    type: sharegpt
chat_template: chatml

dataset_prepared_path: last_run_prepared
val_set_size: 0.001
output_dir: /workspace/llama3-8b-pippa
adapter: qlora
lora_model_dir:

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
lora_modules_to_save:
  - embed_tokens
  - lm_head

wandb_project: waifu
wandb_entity:
wandb_watch:
wandb_run_id:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
adam_beta2: 0.95
adam_epsilon: 0.00001
max_grad_norm: 1.0
lr_scheduler: cosine
learning_rate: 0.0002
optimizer: paged_adamw_32bit

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
#bfloat16: true

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10

eval_steps: 100
eval_table_size:
eval_table_max_new_tokens:
eval_sample_packing: false
saves_per_epoch: 
save_steps: 100
save_total_limit: 2
debug:
#deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16_cpuoffload_all.json
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
  eos_token: "<|im_end|>"
  pad_token: "<|im_end|>"
tokens:
  - "<|im_start|>"

workspace/llama3-8b-pippa

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.5946

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
4.6425	0.0	1	4.4372
1.9054	0.21	100	1.6499
1.6536	0.41	200	1.6101
1.7332	0.62	300	1.5973
1.7975	0.82	400	1.6079
1.669	1.01	500	1.5992
1.5612	1.21	600	1.5926
1.6936	1.42	700	1.5868
1.6197	1.62	800	1.5707
1.6831	1.83	900	1.5690
1.4055	2.02	1000	1.5902
1.4736	2.22	1100	1.5987
1.4137	2.43	1200	1.5899
1.4527	2.63	1300	1.5854
1.507	2.84	1400	1.5814
1.4538	3.03	1500	1.5900
1.4501	3.24	1600	1.5938
1.3612	3.44	1700	1.5928
1.4801	3.65	1800	1.5922
1.3502	3.85	1900	1.5946

Framework versions

• PEFT 0.10.0
• Transformers 4.40.0.dev0
• Pytorch 2.2.0+cu121
• Datasets 2.15.0
• Tokenizers 0.15.0