|
--- |
|
license: cc-by-sa-4.0 |
|
--- |
|
|
|
# StableLM-3B-4E1T # |
|
|
|
* Model Creator: [Stability AI](https://huggingface.co/stabilityai) |
|
* original Model: [StableLM-3B-4E1T](https://huggingface.co/stabilityai/stablelm-3b-4e1t) |
|
|
|
## Description ## |
|
|
|
This repository contains the most relevant quantizations of Stability AI's |
|
[StableLM-3B-4E1T](https://huggingface.co/stabilityai/stablelm-3b-4e1t) model |
|
in GGUF format - ready to be used with |
|
[llama.cpp](https://github.com/ggerganov/llama.cpp) and similar applications. |
|
|
|
## About StableLM-3B-4E1T ## |
|
|
|
Stability AI claims: "_StableLM-3B-4E1T achieves |
|
state-of-the-art performance (September 2023) at the 3B parameter scale |
|
for open-source models and is competitive with many of the popular |
|
contemporary 7B models, even outperforming our most recent 7B |
|
StableLM-Base-Alpha-v2._" |
|
|
|
According to them "_The model is intended to be used as a foundational base |
|
model for application-specific fine-tuning. Developers must evaluate and |
|
fine-tune the model for safe performance in downstream applications._" |
|
|
|
## Files ## |
|
|
|
Right now, the following quantizations are available: |
|
|
|
* [stablelm-3b-4e1t-Q3_K_M](https://huggingface.co/rozek/StableLM-3B-4E1T_GGUF/blob/main/stablelm-3b-4e1t-Q3_K_M.bin) |
|
* [stablelm-3b-4e1t-Q4_K_M](https://huggingface.co/rozek/StableLM-3B-4E1T_GGUF/blob/main/stablelm-3b-4e1t-Q4_K_M.bin) |
|
* [stablelm-3b-4e1t-Q5_K_M](https://huggingface.co/rozek/StableLM-3B-4E1T_GGUF/blob/main/stablelm-3b-4e1t-Q5_K_M.bin) |
|
* [stablelm-3b-4e1t-Q6_K](https://huggingface.co/rozek/StableLM-3B-4E1T_GGUF/blob/main/stablelm-3b-4e1t-Q6_K.bin) |
|
* [stablelm-3b-4e1t-Q8_K](https://huggingface.co/rozek/StableLM-3B-4E1T_GGUF/blob/main/stablelm-3b-4e1t-Q8_K.bin) |
|
|
|
(tell me if you need more) |
|
|
|
These files are presented here with the written permission of Stability AI (although |
|
access to the model itself is still "gated"). |
|
|
|
## Usage Details ## |
|
|
|
Any technical details can be found on the |
|
[original model card](https://huggingface.co/stabilityai/stablelm-3b-4e1t) and in |
|
a paper on [StableLM-3B-4E1T](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo). |
|
The most important ones for using this model are |
|
|
|
* context length is 4096 |
|
* there does not seem to be a specific prompt structure - just provide the text |
|
you want to be completed |
|
|
|
### Text Completion with LLaMA.cpp ### |
|
|
|
For simple inferencing, use a command similar to |
|
|
|
``` |
|
./main -m stablelm-3b-4e1t-Q8_0.bin --temp 0 --top-k 4 --prompt "who was Joseph Weizenbaum?" |
|
``` |
|
|
|
### Text Tokenization with LLaMA.cpp ### |
|
|
|
To get a list of tokens, use a command similar to |
|
|
|
``` |
|
./tokenization -m stablelm-3b-4e1t-Q8_0.bin --prompt "who was Joseph Weizenbaum?" |
|
``` |
|
|
|
### Embeddings Calculation with LLaMA.cpp ### |
|
|
|
Text embeddings are calculated with a command similar to |
|
|
|
``` |
|
./embedding -m stablelm-3b-4e1t-Q8_0.bin --prompt "who was Joseph Weizenbaum?" |
|
``` |
|
|
|
## Conversion Details ## |
|
|
|
Conversion was done using a Docker container based on |
|
`python:3.10.13-slim-bookworm` |
|
|
|
After downloading the original model files into a separate directory, the |
|
container was started with |
|
|
|
``` |
|
docker run --interactive \ |
|
--mount type=bind,src=<local-folder>,dst=/llm \ |
|
python:3.10.13-slim-bookworm |
|
``` |
|
|
|
where `<local-folder>` was the path to the folder containing the downloaded |
|
model. |
|
|
|
Within the container's terminal, the following commands were issued: |
|
|
|
``` |
|
apt-get update |
|
apt-get install build-essential git -y |
|
|
|
git clone https://github.com/ggerganov/llama.cpp |
|
cd llama.cpp |
|
|
|
## Important: uncomment the make command that fits to your host computer! |
|
## on Apple Silicon machines: (see https://github.com/ggerganov/llama.cpp/issues/1655) |
|
# UNAME_M=arm64 UNAME_p=arm LLAMA_NO_METAL=1 make |
|
## otherwise |
|
# make |
|
|
|
python3 -m pip install -r requirements.txt |
|
pip install torch transformers |
|
|
|
# see https://github.com/ggerganov/llama.cpp/issues/3344 |
|
python3 convert-hf-to-gguf.py /llm |
|
mv /llm/ggml-model-f16.gguf /llm/stablelm-3b-4e1t.gguf |
|
|
|
# the following command is just an example, modify it as needed |
|
./quantize /llm/stablelm-3b-4e1t.gguf /llm/stablelm-3b-4e1t_Q3_K_M.gguf q3_k_m |
|
``` |
|
|
|
After conversion, the mounted folder (the one that originally contained the |
|
model only) now also contains all conversions. |
|
|
|
The container itself may now be safely deleted - the conversions will remain on |
|
disk. |
|
|
|
## License ## |
|
|
|
The original "_Model checkpoints are licensed under the Creative Commons license |
|
([CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/)). Under this |
|
license, you must give [credit](https://creativecommons.org/licenses/by/4.0/#) |
|
to Stability AI, provide a link to the license, and |
|
[indicate if changes were made](https://creativecommons.org/licenses/by/4.0/#). |
|
You may do so in any reasonable manner, but not in any way that suggests the Stability AI endorses you or your use._" |
|
|
|
So, in order to be fair and give credits to whom they belong: |
|
|
|
* the original model was created and published by [Stability AI](https://huggingface.co/stabilityai) |
|
* besides quantization, no changes were applied to the model itself |
|
|
