| license: apache-2.0 | |
| datasets: | |
| - Fishfishfishfishfish/Synthetic_text.txt | |
| language: | |
| - en | |
| Each safetensors file represents a different hidden dim value. | |
| Each trained for 1 epoch. | |
| inference.py must be edited for each safetensors. | |
| sequence_length = 64 | |
| batch_size = 16 | |
| learning_rate = 0.0001 | |
| embedding_dim = 256 | |
| num_layers = 4 |