Unable to load Q2_K
Can you do a quick check on the file integrity? I just fetched/pulled and rebuilding llama.cpp about 20 minutes ago.
~/llama.cpp/main -ngl 200 -m ./zixtral-8x22B-v0.1.Q2_K.gguf -p "What's the meaning of life? The meaning of life is "
Log start
main: build = 2668 (a4ec34e1)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed = 1713093326
llama_model_load: error loading model: invalid split file: ./zixtral-8x22B-v0.1.Q2_K.gguf
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model './zixtral-8x22B-v0.1.Q2_K.gguf'
main: error: unable to load model
ls -l
-rw-rw-r-- 1 seg seg 11843728352 Apr 10 09:03 'Mixtral-8x22B-v0.1.Q2_K-00001-of-00005.gguf?download=true'
-rw-rw-r-- 1 seg seg 12267708608 Apr 10 09:08 'Mixtral-8x22B-v0.1.Q2_K-00002-of-00005.gguf?download=true'
-rw-rw-r-- 1 seg seg 11188527232 Apr 10 09:14 'Mixtral-8x22B-v0.1.Q2_K-00003-of-00005.gguf?download=true'
-rw-rw-r-- 1 seg seg 12016738464 Apr 10 09:20 'Mixtral-8x22B-v0.1.Q2_K-00004-of-00005.gguf?download=true'
-rw-rw-r-- 1 seg seg 4786462016 Apr 10 09:26 'Mixtral-8x22B-v0.1.Q2_K-00005-of-00005.gguf?download=true'
-rw-rw-r-- 1 seg seg 52103164672 Apr 14 06:55 zixtral-8x22B-v0.1.Q2_K.gguf
cksum *
2418201643 11843728352 Mixtral-8x22B-v0.1.Q2_K-00001-of-00005.gguf?download=true
389371389 12267708608 Mixtral-8x22B-v0.1.Q2_K-00002-of-00005.gguf?download=true
1802665103 11188527232 Mixtral-8x22B-v0.1.Q2_K-00003-of-00005.gguf?download=true
1222997940 12016738464 Mixtral-8x22B-v0.1.Q2_K-00004-of-00005.gguf?download=true
3576333536 4786462016 Mixtral-8x22B-v0.1.Q2_K-00005-of-00005.gguf?download=true
3948285662 52103164672 zixtral-8x22B-v0.1.Q2_K.gguf
I didn't have the quants anymore, so I re-downloaded the Q2. It worked fine.
- what is this
./zixtral-8x22B-v0.1.Q2_K.gguf? - Please use the split support and just use
Mixtral-8x22B-v0.1.Q2_K-00001-of-00005.gguf, llama.cpp will load the rest. no need to manually merge them:
llama.cpp/main -m Mixtral-8x22B-v0.1.Q2_K-00001-of-00005.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 1024 -e
Thanks, I use to merge it with cat. It's good, I suppose they changed it. I can now load it without merging and if you need to merge then must use the gguf-split tool
~/llama.cpp/gguf-split --merge Mixtral-8x22B-v0.1.Q2_K-00001-of-00005.gguf output.gguf Thanks again