Q8_0 is as fast as Q2_* in ollama
Q8_0 is as fast as Q2_* in ollama on CPU which does not make any sense
In fact Q8_0 is faster than nearly everything else
that's.. quite interesting ! what are your specs..?
It is possible that the simplicity of Q8_0 is beneficial, can you try Q4_0?
it is a AMD Ryzen 5700U, running ollama version 0.5.4:
ollama run --verbose hf.co/bartowski/SmallThinker-3B-Preview-GGUF:Q8_0
hello thinker!
Hello! I'm here to help you with any questions or topics you'd like to discuss. Whether it's about technology, science, history, literature, philosophy, or
anything else, feel free to share your thoughts and we can explore them together.
So, since you started with "hello thinker!", it seems like you're interested in having someone to talk to about various subjects. I'm more than happy to
assist you. Please go ahead and let me know what's on your mind!
total duration: 11.946873s
load duration: 20.6196ms
prompt eval count: 11 token(s)
prompt eval duration: 520ms
prompt eval rate: 21.15 tokens/s
eval count: 99 token(s)
eval duration: 11.405s
eval rate: 8.68 tokens/s
ollama run --verbose hf.co/bartowski/SmallThinker-3B-Preview-GGUF:Q4_0
hello thinker!
Hello there! I'm just a large language model created by Alibaba, and I'm here to help you with any questions or topics you'd like to discuss. Whether it's
about technology, culture, history, science, or anything else, feel free to share your thoughts, and I'll do my best to provide insightful and informative
responses. Let's dive into a topic that interests you!
total duration: 12.2688759s
load duration: 20.6917ms
prompt eval count: 11 token(s)
prompt eval duration: 977ms
prompt eval rate: 11.26 tokens/s
eval count: 79 token(s)
eval duration: 11.27s
eval rate: 7.01 tokens/s