|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- EleutherAI/the_pile_deduplicated |
|
language: |
|
- en |
|
--- |
|
|
|
Pythia-2.8B Deduped 4K is a [Pythia-2.8B Deduped](https://huggingface.co/EleutherAI/pythia-2.8b-deduped) model fine-tuned with a 4096 context length. |
|
Training resumed from their 143,000 step checkpoint and continued on The Pile v1 Deduped (threshold=0.87). |
|
This particular model is from a checkpoint captured at step 175,500 for an extra 134,217,728,000 tokens of training. |
|
|
|
Note: Sequence length warmup was not used to move up from 2048 but, in hindsight, should have been applied. |
|
|
|
## Acknoweldgements |
|
|
|
This work would not have been possible without the support of [Stability AI](https://stability.ai/). |
