| language: | |
| - es | |
| license: gpl-3.0 | |
| tags: | |
| - masked-lm | |
| datasets: | |
| - jorgeortizfuentes/spanish_books | |
| - jorgeortizfuentes/small-chilean-spanish-corpus | |
| pipeline_tag: fill-mask | |
| base_model: dccuchile/bert-base-spanish-wwm-cased | |
| # Tulio | |
| Tulio is a BERT model trained with Chilean Spanish. This model is a fine-tuned version of [dccuchile/bert-base-spanish-wwm-cased](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased) on Spanish Books and Small Chilean Spanish Corpus. | |
| ## Training hyperparameters | |
| The following hyperparameters were used during training: | |
| - learning_rate: 5e-05 | |
| - train_batch_size: 20 | |
| - eval_batch_size: 20 | |
| - seed: 42 | |
| - distributed_type: multi-GPU | |
| - num_devices: 2 | |
| - total_train_batch_size: 20 | |
| - total_eval_batch_size: 20 | |
| - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 | |
| - lr_scheduler_type: linear | |
| - num_epochs: 2.0 | |
| ## Acknowledgments | |
| We are grateful for the servers provided by the Computer Science Department of the University of Chile and the ReLeLa (Representations for Learning and Language) study group for the training of the model. | |
| ## License Disclaimer | |
| The license gpl-3.0 best describes our intentions for our work. However we are not sure that all the datasets used to train the model have licenses compatible with gpl-3.0. Please use at your own discretion and verify that the licenses of the original text resources match your needs. | |
| ## Limitations | |
| The training dataset was not censored in any way. Therefore, the model may contain unwanted ideological representations. Use with caution. | |