---
datasets:
- allenai/dolma
language:
- en
library_name: transformers
license: apache-2.0
tags:
- causal-lm
---

## Model Details
### Training
Models trained using [litgpt](https://github.com/Lightning-AI/litgpt) and [AxoNN](https://github.com/axonn-ai/litgpt) on AMD MI250 GPUs.

### Data
Train and validation data is taken from non-overlapping subsets of [dolma](https://huggingface.co/datasets/allenai/dolma).