---
datasets:
- EleutherAI/pile
language:
- en
---
# Model Card

This model is pretrained as a reference baseline to the Based model provided here: https://huggingface.co/hazyresearch/based-1b-50b. 

Both checkpoints are pretrained on **50Bn tokens** of the Pile in the exact same data order using next token prediction. 

A WandB report for training is here: https://api.wandb.ai/links/hazy-research/ggo9rst2


### Model Sources

The model is a standard Mamba model using the model code provided here: https://github.com/state-spaces/mamba/tree/main/mamba_ssm

The training code is provided here and can be used to reproduce training: https://github.com/HazyResearch/based

The paper for the work is here, and the appendix includes additional experimental details/hyperparameters: https://arxiv.org/abs/2402.18668


### Uses

The purpose of this work is to evaluate the language modeling quality of a new efficient architecture, Based. 

We include a series of benchmarks that you can use to evaluate quality: 
- FDA: https://huggingface.co/datasets/hazyresearch/based-fda
- SWDE: https://huggingface.co/datasets/hazyresearch/based-swde
- SQUAD: https://huggingface.co/datasets/hazyresearch/based-squad




## Citation

Please consider citing this paper if you use our work: 

```
@article{arora2024simple,
  title={Simple linear attention language models balance the recall-throughput tradeoff},
  author={Arora, Simran and Eyuboglu, Sabri and Zhang, Michael and Timalsina, Aman and Alberti, Silas and Zinsley, Dylan and Zou, James and Rudra, Atri and Ré, Christopher},
  journal={arXiv:2402.18668},
  year={2024}
}
```

Please reach out to simarora@stanford.edu, eyuboglu@stanford.edu, and mzhang20@stanford.edu with questions.