See axolotl config

axolotl version: 0.6.0

base_model: meta-llama/Llama-3.2-1B-Instruct
hub_model_id: kweinmeister/Llama-3.2-1B-Instruct-gsm8k

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: openai/gsm8k
    type: alpaca_chat.load_qa
    name: "main"
    train_on_split: "train"

val_set_size: 0.1
output_dir: "/mnt/disks/gcs/axolotl/outputs/gsm8k-out"

adapter: qlora
lora_model_dir:

sequence_len: 2048
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_fan_in_fan_out:
lora_target_modules:
  - gate_proj
  - down_proj
  - up_proj
  - q_proj
  - v_proj
  - k_proj
  - o_proj

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
# optimizer: adamw_bnb_8bit
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

# gradient_checkpointing: true
gradient_checkpointing: false
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
  - full_shard
  - auto_wrap  
fsdp_config:
  fsdp_limit_all_gathers: true
  fsdp_sync_module_states: true
  fsdp_offload_params: true
  fsdp_use_orig_params: false
  fsdp_cpu_ram_efficient_loading: true
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_activation_checkpointing: true  
special_tokens:
  bos_token: "<|begin_of_text|>"
  eos_token: "<|eot_id|>"
  pad_token: "<|finetune_right_pad_id|>"

Llama-3.2-1B-Instruct-gsm8k

This model is a fine-tuned version of meta-llama/Llama-3.2-1B-Instruct on the openai/gsm8k dataset. It achieves the following results on the evaluation set:

• Loss: 0.5915

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• total_eval_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
1.0727	0.0242	1	1.0837
1.0432	0.2667	11	1.0485
0.844	0.5333	22	0.8698
0.7058	0.8	33	0.7048
0.5912	1.0485	44	0.6477
0.6363	1.3152	55	0.6196
0.5919	1.5818	66	0.6064
0.5804	1.8485	77	0.5989
0.5718	2.0970	88	0.5945
0.5651	2.3636	99	0.5925
0.5808	2.6303	110	0.5915
0.5704	2.8970	121	0.5915

Framework versions

• PEFT 0.14.0
• Transformers 4.47.1
• Pytorch 2.3.1+cu121
• Datasets 3.1.0
• Tokenizers 0.21.0