|
--- |
|
model-index: |
|
- name: robinlee99/Pythia-2.8B-TLDR-Iterative-SamPO |
|
results: [] |
|
datasets: |
|
- webis/tldr-17 |
|
language: |
|
- en |
|
base_model: EleutherAI/pythia-2.8b |
|
license: apache-2.0 |
|
--- |
|
|
|
# Model Card for Pythia-2.8B-TLDR-Iterative-SamPO |
|
|
|
This repository provides a fine-tuned version of Pythia-2.8B, using our proposed [SamPO](https://github.com/LuJunru/SamPO) algorithm: Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence. |
|
|
|
## Performance |
|
| vs. SFT | wins | len / token | |
|
| ------ | ------ | ------ | |
|
| DPO | 60.98 | 53.8 | |
|
| Iterative DPO | **73.58** | 66.65 | |
|
| Length Normed DPO | 58.13 | 47.34 | |
|
| SimPO | 33.33 | **31.9** | |
|
| Iterative SamPO | **73.58** | 49.54 | |
|
|
|
## Evaluation Details |
|
We test our model with the same GPT-4 Win rate prompt template proposed by the [DPO paper](https://arxiv.org/pdf/2305.18290). The [sampled test set](https://huggingface.co/robinlee99/Pythia-2.8B-TLDR-Iterative-SamPO/blob/main/test_tldr.jsonl) is included in this repo. |
|
|
|
## Training hyperparameters |
|
|
|
The following hyperparameters were used during DPO/SamPO training: |
|
- DPO beta: 0.5 |
|
- learning_rate: 1e-6 |
|
- total_train_batch_size: 128 |
|
- optimizer: AdamW with beta1 0.9, beta2 0.999 and epsilon 1e-8 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_ratio: 0.1 |
|
- Weight Decay: 0.0 |
|
- num_epochs: 1.0 |
