Method for quantizing and converting FluxDev to GGUF?

#27
by Melyn - opened

few days ago i came across a comment or discussion in either huggingface or github explaining how to do that in details & i just forgot where to find it.

does anyone please know how to do this? i have pretty slow internet & i can't download all models so i'd like to convert flux1-dev.safetensors to .gguf locally lets say i want to convert to flux1-dev-Q4_0.gguf, how can i achieve that?

thank you

Owner

Hi. The tools folder in the main repo here has some notes on how to do it. You'll have to apply a patch and build llama.cpp from source which is somewhat easier to do on linux than windows, but it works on both. LMK if you run into any issues and I can try to help out.

Thank you so much! would that work with prebuilt windows binaries provided at https://github.com/ggerganov/llama.cpp/releases/?

Owner

Sadly no because we have to patch in some image model specific things to make it recognize the Flux/etc architecture and to not add things like the tokenizer/prompt template (that we don't use) in these cases.

Alright thanks a lot, this was helpful

Melyn changed discussion status to closed

so i followed the steps there & im trying to use convert.py but it doesn't seem to recognize the input file. I'm on arch linux.

python /home/ComfyUI-GGUF/tools/convert.py --src /home/flux1-dev.safetensors --dst /home/flux1-dev.gguf

Console output:

usage: convert.py [-h] --src SRC [--dst DST]
convert.py: error: No input provided!
Owner

That means the path is invalid as it's failing this check here:

    if not os.path.isfile(args.src):
        parser.error("No input provided!")

I've had it work on linux before so not sure what you could be running into outside, well, an invalid path.

Sign up or log in to comment