metadata
library_name: transformers
datasets:
- mlabonne/orpo-dpo-mix-40k
base_model:
- meta-llama/Llama-3.2-1B-Instruct
ORPO-Tuned Llama2-1B-Instruct
NB: Done purely as a fine-tuning exercise. Not intedned for any practical use.
This model is a fine-tuned version of Meta's Llama-3.2-1B-Instruct using ORPO (Optimizing Reward with Policy Optimization). The model was trained to better align with human preferences using a curated preference dataset from mlabonne/orpo-dpo-mix-40k.
Model Details
- Base Model: meta-llama/Llama-3.2-1B-Instruct
- Training Method: ORPO (Optimizing Reward with Policy Optimization) with LoRA
- Training Dataset: mlabonne/orpo-dpo-mix-40k (subset of 100 examples)
- Framework: Hugging Face Transformers, TRL, PEFT
- Training Date: November 2024
- License: Same as base model (Llama 2)
Training Process
The model was fine-tuned using LoRA (Low-Rank Adaptation) with the following configuration:
LoRA Parameters
- r=16 (rank)
- lora_alpha=32
- lora_dropout=0.05
- bias="none"
- task_type="CAUSAL_LM"
Training Parameters
- Learning rate: 1e-5
- Batch size: 4
- Gradient accumulation steps: 4
- Maximum steps: 100
- Warmup steps: 10
- Gradient checkpointing: Enabled
- FP16 training: Enabled
- Maximum sequence length: 512
- Maximum prompt length: 512
- Optimizer: AdamW
Evaluation Results
The model was evaluated on the HellaSwag benchmark with the following configuration:
- Batch size: 64 (auto-detected)
- Full evaluation set
- Zero-shot setting
- FP16 precision
Results:
| Metric | Value | Standard Error |
|---|---|---|
| Accuracy | 45.20% | ±0.50% |
| Normalized Accuracy | 60.78% | ±0.49% |