See axolotl config

axolotl version: 0.8.0.dev0

save_total_limit: 2
mlflow_tracking_uri: http://127.0.0.1:7860
mlflow_experiment_name: Default

base_model: unsloth/Llama-3.3-70B-Instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: true
strict: false
sequence_len: 12288
bf16: auto
fp16:
tf32: false
flash_attention: true
tokens:
  - <hindsight>
  - </hindsight>
  - <|system|>
  - <|user|>
  - <|model|>

# Data
dataset_prepared_path: last_run_prepared
datasets:
  - path: TheDrummer/Hindsight-R1-v1
    type: custommetharme-masked
warmup_steps: 25

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

adapter: qlora
lora_model_dir:
lora_r: 64
lora_alpha: 64
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
lora_modules_to_save:
  - embed_tokens
  - lm_head

save_safetensors: true

# WandB
#wandb_project: hindsight-70b
#wandb_entity:

# Iterations
num_epochs: 2

# Output
output_dir: ./Hindsight-70B-v1a-Workspace
hub_model_id: BeaverAI/Hindsight-70B-v1a-WS
hub_strategy: "end"

# Sampling
sample_packing: false
pad_to_sequence_len: true

# Batching
gradient_accumulation_steps: 3
micro_batch_size: 1 # 2x gpu
gradient_checkpointing: true
gradient_checkpointing_kwargs:
    # use_reentrant: true

# Evaluation
val_set_size: 0.025
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 256
eval_sample_packing: true
eval_batch_size: 1

# Optimizer
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 0.00005
lr_scheduler: cosine_with_min_lr
lr_scheduler_kwargs:
    min_lr: 0.000005
weight_decay: 0.01
max_grad_norm: 5.0

# Misc
train_on_inputs: false
group_by_length: false
early_stopping_patience:
local_rank:
logging_steps: 1
xformers_attention:
debug:
deepspeed: deepspeed_configs/zero3_bf16.json
fsdp:
fsdp_config:

# Checkpoints
resume_from_checkpoint:
saves_per_epoch: 1

Hindsight-70B-v1a-WS

This model is a fine-tuned version of unsloth/Llama-3.3-70B-Instruct on the TheDrummer/Hindsight-R1-v1 dataset. It achieves the following results on the evaluation set:

• Loss: 0.8403

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 3
• total_train_batch_size: 6
• total_eval_batch_size: 2
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine_with_min_lr
• lr_scheduler_warmup_steps: 25
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
1.3642	0.0011	1	1.4614
0.7745	0.2508	236	0.9048
0.7786	0.5016	472	0.8718
0.6914	0.7524	708	0.8524
0.6159	1.0032	944	0.8359
0.5954	1.2540	1180	0.8505
0.6593	1.5048	1416	0.8416
0.567	1.7556	1652	0.8403

Framework versions

• PEFT 0.14.0
• Transformers 4.49.0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0