File size: 664 Bytes
16972a2 606be33 16972a2 606be33 852f1f9 16972a2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
---
datasets:
- JeanKaddour/minipile
language:
- en
license: mpl-2.0
---
# RWKV Minipile
## Model Specifications
- **Architecture**: RWKV
- **Vocabulary Size**: 65,536
- **Embedding Size**: 768
- **Number of Layers**: 12
- **Context Length**: 512
- **Data Type**: bfloat16
- **Dataset**: Minipile
- **Tokens**: 20,643,840 (20 Million)
The model underwent a rigorous training regimen, completing 32 epochs to optimize performance.
## Wandb
[Training progress](https://wandb.ai/evaldas-leliuga/Test/runs/30w2jwdp/overview)
## Verifying checksums
```bash
sha512sum -c sha512sums.txt
```
## Inference
```bash
pip install torch numpy
python inference.py
``` |