README.md · Nexesenex/Undi95_Miqu-70B-Alpaca-DPO-iMat.GGUF at main

GGUF Quants with iMatrix for : https://huggingface.co/Undi95/Miqu-70B-Alpaca-DPO

Q3_K_M to be uploaded shortly.

Q3_K_S, IQ3_XXS, Q2_K, Q2_K_S, IQ2_XS, IQ2_XXS to follow.

LlamaCPP Benchs on the Q3_K_M with iMatrix shared here :

Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,Hellaswag,84.5,,400,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,
Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,Hellaswag,83.6,,1000,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,
Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,Arc-Challenge,58.52842809,,299,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,
Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,Arc-Easy,77.36842105,,570,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,
Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,MMLU,49.84025559,,313,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,
Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,Thruthful-QA,42.83965728,,817,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,
Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,Winogrande,78.7687,,1267,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,
Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,wikitext,4.2963,512,512,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,81
Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-Q3_K_M.gguf,-,wikitext,3.8397,512,512,2024-02-07 00:00:00,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,Nexesenex,655

LlamaCPP Benchs on a non iMatrix Q3_K_M released by Undi95 :

Miqu-70B-DPO.q3_k_m.gguf,-,Hellaswag,84.5,400,,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
Miqu-70B-DPO.q3_k_m.gguf,-,Hellaswag,83.8,1000,,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
Miqu-70B-DPO.q3_k_m.gguf,-,Arc-Challenge,57.85953177,,299,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
Miqu-70B-DPO.q3_k_m.gguf,-,Arc-Easy,77.36842105,,570,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
Miqu-70B-DPO.q3_k_m.gguf,-,MMLU,50.15974441,,313,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
Miqu-70B-DPO.q3_k_m.gguf,-,Thruthful-QA,42.47246022,,817,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
Miqu-70B-DPO.q3_k_m.gguf,-,Winogrande,78.7687,,1267,2024-02-07 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,
Miqu-70B-DPO.q3_k_m.gguf,-,wikitext,4.3018,512,512,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,81
Miqu-70B-DPO.q3_k_m.gguf,-,wikitext,3.8469,512,512,2024-02-07 00:00:00,,70b,Mistral_Medium,32768,,,GGUF,NeverSleep,NeverSleep,655

Quite convincing compared to the original Miqu.. with iMatrix :

Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,Arc-Challenge,57.19063545,,299,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,Arc-Easy,77.19298246,,570,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,MMLU,50.15974441,,313,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,Thruthful-QA,41.49326805,,817,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,Winogrande,78.8477,,1267,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,
Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,wikitext,4.2957,512,512,2024-01-29 00:00:00,RBF1000000,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,81
Miqu-1-70b-Requant-b1989-iMat-c32_ch400-Q3_K_M.gguf,-,wikitext,3.8380,512,512,2024-01-29 00:00:00,RBF1000000,70b,Mistral_Medium,32768,,,GGUF,- Miqudev,Nexesenex,655

The TQA shows a slight bonus, thanks to the DPO training I believe. The slightly bonified ARC benchs (a rare thing on DPO releases!) and the respected perplexity show that the model was not dumbified by the DPO training. In ST, the models performs beautifully.