|
--- |
|
base_model: |
|
- meta-llama/Llama-3.1-8B |
|
library_name: transformers |
|
license: cc-by-nc-sa-4.0 |
|
--- |
|
|
|
# An (untrained) pruned version of llama 3.1 8b with a bigger intermediate size |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
|
|
|
|
## Model Details |
|
#### The model has 4 layers which are the 1st, 2nd, 3rd and the last layers of llama 3.1 8b |
|
#### The embeddings and the lm_head have been also been copied over |
|
#### The intermediate size is 28672 (same as llama 3.1 70b) vs the 14336 of llama 3.1 8b |
|
#### The model still needs to be trained |
|
|
|
### This model should be used for research purposes only |
|
|
|
## Model Card Author |
|
#### Aarush Khilosia |
