---
base_model: "JackFram/llama-68m"
language:
- en
license: apache-2.0
library_name: transformers
datasets:
- allenai/c4
---
## Model Description

This model is based of <a href="https://huggingface.co/JackFram/llama-68m">llama-68m</a> and was initialized from this model with tied embeddings and then further pretrained on the uncleaned c4 dataset. This model is obviously a beta based on the name and has not been trained on the full training set which is the first 1 million samples of the uncleaned c4 dataset as mentioned above.