Hugging Face's logo

tyzhu
/
lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_lora2

PEFT
Safetensors
Generated from Trainer
Eval Results
Model card Files Community
lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_lora2 / README.md
tyzhu's picture
tyzhu
End of training
fb75b3a verified
preview code
|
raw
history blame contribute delete
3.28 kB
metadata
license: other
base_model: Qwen/Qwen1.5-4B
tags:
  - generated_from_trainer
datasets:
  - tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx
metrics:
  - accuracy
model-index:
  - name: lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_lora2
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx
          type: tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.7890842332613391
library_name: peft

lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx dataset. It achieves the following results on the evaluation set:

  • Loss: 0.7825
  • Accuracy: 0.7891

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Accuracy Validation Loss
1.6067 0.9997 839 0.7197 1.8440
1.5433 1.9994 1678 0.7247 1.7661
1.4167 2.9991 2517 0.7310 1.6455
1.2948 4.0 3357 0.7366 1.5394
1.1715 4.9997 4196 0.7422 1.4463
1.0458 5.9994 5035 0.7484 1.3537
0.9357 6.9991 5874 0.7546 1.2456
0.8269 8.0 6714 0.7598 1.1735
0.7262 8.9997 7553 0.7649 1.0966
0.6381 9.9970 8390 0.7692 1.0623
0.5784 10.9997 9229 1.0101 0.7731
0.5071 11.9994 10068 0.9538 0.7760
0.4734 12.9991 10907 0.9292 0.7791
0.4302 14.0 11747 0.8846 0.7809
0.3917 14.9997 12586 0.8536 0.7833
0.3632 15.9994 13425 0.8468 0.7846
0.3351 16.9991 14264 0.8244 0.7863
0.3186 18.0 15104 0.8096 0.7871
0.2957 18.9997 15943 0.7865 0.7885
0.2858 19.9970 16780 0.7825 0.7891

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1