File size: 1,140 Bytes
55d30f8 dae8e41 b9ee621 9942bd3 7c443c1 c49ee7e 23046d8 e541778 23046d8 e541778 7edcf39 e541778 23046d8 7edcf39 23046d8 20c0a40 23046d8 ee18348 20c0a40 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
---
license: apache-2.0
language:
- en
base_model:
- allenai/OLMo-7B-0724-Instruct-hf
pipeline_tag: text2text-generation
datasets:
- allenai/tulu-v3.1-mix-preview-4096-OLMoE
---
## GGUF quantized version of OLMo-7B-0724-Instruct Model
project original source: [base model](https://huggingface.co/allenai/OLMo-7B-0724-Instruct-hf)
Q_2_K (not nice)
Q_3_K_S (acceptable)
Q_3_K_M is acceptable (good for running with CPU)
Q_3_K_L (acceptable)
Q_4_K_S (okay)
Q_4_K_M is recommanded (balance)
Q_5_K_S (good)
Q_5_K_M (good in general)
Q_6_K is good also; if you want a better result; take this one instead of Q_5_K_M
Q_8_0 which is very good; need a reasonable size of RAM otherwise you might expect a long wait
f16 is similar to the original hf model; opt this one or hf also fine; make sure you have a good machine
### how to run it
use any connector for interacting with gguf; i.e., [gguf-connector](https://pypi.org/project/gguf-connector/)
<img src="https://allenai.org/olmo/olmo-7b-animation.gif" alt="OLMo Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
this picture is from the base model |