license: apache-2.0
base_model: Rijgersberg/GEITje-7B-chat-v2
tags:
- trl
- sft
- generated_from_trainer
model-index:
- name: AmsterdamDocClassificationGEITje200T3Epochs
results: []
datasets:
- FemkeBakker/AmsterdamBalancedFirst200Tokens
language:
- nl
AmsterdamDocClassificationGEITje200T3Epochs
As part of the Assessing Large Language Models for Document Classification project by the Municipality of Amsterdam, we fine-tune Mistral, Llama, and GEITje for document classification. The fine-tuning is performed using the AmsterdamBalancedFirst200Tokens dataset, which consists of documents truncated to the first 200 tokens. In our research, we evaluate the fine-tuning of these LLMs across one, two, and three epochs. This model is a fine-tuned version of Rijgersberg/GEITje-7B-chat-v2 and has been fine-tuned for three epochs.
It achieves the following results on the evaluation set:
- Loss: 0.5854
Training and evaluation data
- The training data consists of 9900 documents and their labels formatted into conversations.
- The evaluation data consists of 1100 documents and their labels formatted into conversations.
Training procedure
See the GitHub for specifics about the training and the code.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.7664 | 0.1988 | 123 | 0.6890 |
| 0.6617 | 0.3976 | 246 | 0.6347 |
| 0.3825 | 0.5964 | 369 | 0.6028 |
| 0.4427 | 0.7952 | 492 | 0.5913 |
| 0.6739 | 0.9939 | 615 | 0.5906 |
| 0.4407 | 1.1939 | 738 | 0.5918 |
| 0.6671 | 1.3927 | 861 | 0.5835 |
| 0.4845 | 1.5915 | 984 | 0.5802 |
| 0.4699 | 1.7903 | 1107 | 0.5796 |
| 0.5434 | 1.9891 | 1230 | 0.5796 |
| 0.6081 | 2.1891 | 1353 | 0.5886 |
| 0.2911 | 2.3879 | 1476 | 0.5862 |
| 0.3691 | 2.5867 | 1599 | 0.5853 |
| 0.6234 | 2.7855 | 1722 | 0.5853 |
| 0.653 | 2.9842 | 1845 | 0.5854 |
Training time: it took in total 2 hours and 26 minutes to fine-tune the model for three epochs.
Framework versions
- Transformers 4.41.1
- Pytorch 2.3.0+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1
Acknowledgements
This model was trained as part of [insert thesis info] in collaboration with Amsterdam Intelligence for the City of Amsterdam.