Which template shoud I use in Ollama
I tried ChatML and original Mistral-Nemo template , none of these template works, they both have infinite generation problem or model will generate some random unrelated code, so I am not sure it's templates problem or models problem.
I was using Q8 gguf
I'm running 8-bit GGUF from @bartowski in LM Studio, with ChatML format. It works fine on my end, not sure what the issue is.
I'm running 8-bit GGUF from @bartowski in LM Studio, with ChatML format. It works fine on my end, not sure what the issue is.
Are you using the template from bartowski's repo? I tried that and it also have these problems in ollama, I also have another Nemo model from @TheDrummer , which also was trained on ChatML template but it works fine
https://huggingface.co/TheDrummer/Rocinante-12B-v1.1
PS D:\LLM\gguf> ollama show --modelfile Rocinante-12B-v1.1-Q8_0:latest
# Modelfile generated by "ollama show"
# To build a new Modelfile based on this, replace FROM with:
# FROM Rocinante-12B-v1.1-Q8_0:latest
FROM D:\LLM\.ollama\models\blobs\sha256-5a0c3544bfc08067e55bf09575ac352baf31749bdbdb3c8ad7889fa1efec0a81
TEMPLATE "{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"
SYSTEM You are a helpful assistant
PARAMETER stop <|im_end|>
PARAMETER stop <|im_start|>
PS D:\LLM\gguf> ollama run Rocinante-12B-v1.1-Q8_0:latest
>>> how far is the moon
The average distance from Earth to the Moon is about 384,400 kilometers (238,855 miles). This distance is commonly referred to as "one lunar distance." However, due to the elliptical shape of the Moon's orbit around Earth, this distance can vary by about ±1%. At its
closest approach (perigee), the Moon is approximately 363,104 kilometers (225,622 miles) away from Earth. At its farthest point (apogee), it reaches a distance of about 405,754 kilometers (252,192 miles).