|
--- |
|
language: |
|
- en |
|
license: apache-2.0 |
|
datasets: |
|
- delphi-suite/stories |
|
library_name: delphi |
|
--- |
|
|
|
This is a part of `stories-llama2-*` model family: |
|
|
|
name | params | layers | hidden_size | query heads | key & value heads |
|
-|-|-|-|-|- |
|
stories-llama2-50k | 49,554 | 1 | 6 | 3 | 1 |
|
stories-llama2-100k | 99,924 | 1 | 12 | 2 | 1 |
|
stories-llama2-250k | 246,820 | 2 | 28 | 2 | 1 |
|
stories-llama2-500k | 527,912 | 2 | 56 | 4 | 2 |
|
stories-llama2-1m | 1,019,508 | 4 | 84 | 6 | 3 |
|
stories-llama2-2.5m | 2,437,280 | 4 | 160 | 8 | 4 |
|
stories-llama2-5m | 5,136,720 | 5 | 240 | 10 | 5 |
|
stories-llama2-10m | 10,421,340 | 6 | 340 | 10 | 5 |
|
stories-llama2-25m | 24,215,520 | 8 | 480 | 16 | 8 |
|
stories-llama2-50m | 49,387,712 | 8 | 704 | 16 | 8 |
|
|
|
You can access W&B logs [here](https://wandb.ai/delphi-suite/delphi). |
|
|
|
This model was trained using [delphi](https://github.com/delphi-suite/delphi). See `training_config.json` and `run_context.json` for details. |
|
|
|
|
|
|
|
|
|
|
|
|
