Template ? not able t use it.

#1
by NK-Spacewalk - opened

what should be the template ? what should i selected in lm studio ? could you provide ollama models too?

@bartowski could you provide better version of gguf model which can support lmstudio/jan etc...

Arcee AI org

chatml - and theres one on ollama as well.

Crystalcareai changed discussion status to closed

chatml is not working @Crystalcareai

image.png

Arcee AI org

I'll take a look though these ones were made by me so it's odd it's giving issues

Arcee AI org

@NK-Spacewalk are you using flash attention? Qwen models trend to prefer it and sometimes will misbehave without it

@bartowski Thank you for comment.
Im not using flash attention. im using latest lm studio.
this gguf file is not working with LM Studio, jan.

but ollama model is working , which i created manually by using same gguf file.

Arcee AI org

You should turn flash attention on, that will fix it. It's possible ollama is doing this by default

Arcee AI org

I've had good success with this spark variant in lm-studio using chatml template:
Screenshot 2024-07-01 at 1.17.38 PM.png

Sign up or log in to comment