Spaces:

---

ggml-org

/

gguf-my-repo

like 1.63k

Running on A10G

The problem with adding BF16 is that current we use convert_hf_to_gguf.py to convert HF model into F16, then use llama-quantize to quantize it.

So the conversion will be safetensors --> F16 --> BF16 which adds no benefit to the output model.

What we should do here is also modify the code that run convert_hf_to_gguf.py, so it outputs directly BF16 GGUF file

I would also like to see a direct conversion into 'Floating Point 16', especially now that I know how 'BrainFloat16' works.

TL;DR Basically, BF16 sucks.