| # FLAN-T5-Base pre-trained on Historical Text Completion | |
| This model is a pre-trained version of [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) fine-tuned on historical text completion tasks. | |
| ## Model description | |
| This model has been trained to complete historical texts, enhancing its understanding of historical contexts and language. | |
| ## Intended uses & limitations | |
| This model is intended for further fine-tuning on specific historical NLP tasks or for generating historically-aware text completions. | |
| ## Training and evaluation data | |
| The model was trained on a subset of the dataset [ambrosfitz/just_history_xl_masked](https://huggingface.co/datasets/ambrosfitz/just_history_xl_masked), | |
| limited by available GPU memory. | |
| ## Training procedure | |
| The model was trained using the following hyperparameters: | |
| - Number of epochs: 3 | |
| - Batch size: 2 | |
| - Learning rate: 5e-05 | |
| - Gradient Accumulation Steps: 8 | |
| - Weight Decay: 0.01 | |
| ## Results | |
| Evaluation results: {'eval_loss': nan, 'eval_runtime': 0.7866, 'eval_samples_per_second': 38.139, 'eval_steps_per_second': 19.069, 'epoch': 2.909090909090909} | |