chunwoolee0/distilgpt2_eli5_clm

This model is a fine-tuned version of distilgpt2 on an eli5 dataset. It achieves the following results on the evaluation set:

  • Train Loss: 3.7237
  • Validation Loss: 3.7528
  • Epoch: 2

Model description

DistilGPT2 is an English-language model pre-trained with the supervision of the 124 million parameter version of GPT-2. DistilGPT2, which has 82 million parameters, was developed using knowledge distillation and was designed to be a faster, lighter version of GPT-2.

Intended uses & limitations

This is an exercise for finetuning of the pretrained causal language model.

Training and evaluation data

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • optimizer: {'name': 'AdamWeightDecay', 'learning_rate': 2e-05, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False, 'weight_decay_rate': 0.01}
  • training_precision: float32

Training results

Train Loss Validation Loss Epoch
3.9048 3.7838 0
3.7853 3.7647 1
3.7237 3.7528 2

Framework versions

  • Transformers 4.31.0
  • TensorFlow 2.12.0
  • Datasets 2.14.4
  • Tokenizers 0.13.3
Downloads last month
19
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for chunwoolee0/distilgpt2_eli5_clm

Finetuned
(577)
this model