Help Needed: Launching Ovis1.6-Gemma2-9B-bnb-4bit
I recently quantized this model into a 4-bit bnb format. Here's the link:
https://huggingface.co/ThetaCursed/Ovis1.6-Gemma2-9B-bnb-4bit
Need assistance with getting it to launch. If anyone has successfully run it, please let me know here.
The issue arises during the image conversion process for the visual tokenizer. The preprocess_image function in the modeling_ovis.py script fails to properly convert the images to the required format or type for the visual tokenizer.
The issue with running this model is solved, no more help is needed.
how did you do it
@ThetaCursed how much GPU is it consuming ?
@ThetaCursed kindly provide inference code snippet for loading 4bit quantized
@ThetaCursed can you share how you fixed the issue?
@ThetaCursed Don't just leave us hanging... :D