Llamacpp Quantizations of labradorite-13b

Using llama.cpp commit fa97464 for quantization.

Original model: https://huggingface.co/ibm/labradorite-13b

Download a file (not the whole branch) from below:

Filename Quant type File Size Description
labradorite-13b-Q8_0.gguf Q8_0 13.83GB Extremely high quality, generally unneeded but max available quant.
labradorite-13b-Q6_K.gguf Q6_K 10.67GB Very high quality, near perfect, recommended.
labradorite-13b-Q5_K_M.gguf Q5_K_M 9.22GB High quality, very usable.
labradorite-13b-Q5_K_S.gguf Q5_K_S 8.97GB High quality, very usable.
labradorite-13b-Q5_0.gguf Q5_0 8.97GB High quality, older format, generally not recommended.
labradorite-13b-Q4_K_M.gguf Q4_K_M 7.86GB Good quality, similar to 4.25 bpw.
labradorite-13b-Q4_K_S.gguf Q4_K_S 7.42GB Slightly lower quality with small space savings.
labradorite-13b-Q4_0.gguf Q4_0 7.36GB Decent quality, older format, generally not recommended.
labradorite-13b-Q3_K_L.gguf Q3_K_L 6.92GB Lower quality but usable, good for low RAM availability.
labradorite-13b-Q3_K_M.gguf Q3_K_M 6.33GB Even lower quality.
labradorite-13b-Q3_K_S.gguf Q3_K_S 5.65GB Low quality, not recommended.
labradorite-13b-Q2_K.gguf Q2_K 4.85GB Extremely low quality, not recommended.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Downloads last month
46
GGUF
Model size
13B params
Architecture
llama

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Examples
Unable to determine this model's library. Check the docs .