calcuis/olmo-gguf

GGUF quantized version of OLMo-7B-0724-Instruct Model

project original source: base model

Q_2_K (not nice)

Q_3_K_S (acceptable)

Q_3_K_M is acceptable (good for running with CPU)

Q_3_K_L (acceptable)

Q_4_K_S (okay)

Q_4_K_M is recommanded (balance)

Q_5_K_S (good)

Q_5_K_M (good in general)

Q_6_K is good also; if you want a better result; take this one instead of Q_5_K_M

Q_8_0 which is very good; need a reasonable size of RAM otherwise you might expect a long wait

f16 is similar to the original hf model; opt this one or hf also fine; make sure you have a good machine

how to run it

use any connector for interacting with gguf; i.e., gguf-connector

this picture is from the base model