Unable to run model on vLLM
Hi
Trying to run the model on vLLM, getting the following error:
ValueError: No supported config format found in LeonardPuettmann/PhilosophyMistral-7B-Instruct-v0.3
Appreciate your help.
I am very GPU-poor, so the model was trained from the 4-bit quantized unsloth model instead of the full precision model from Mistral. I am not sure if vLLM supports 4-bit models yet.
Got it. I am 100% with you. The GPU Poor will inherit the earth ;)
Seriously though, I think the AI bubble is not in the benefit or business potential of the technology, nor its disruption. AI will increase GDP on a global level for many years to come. The bubble is the 'GPU Rich' part - companies and investors willing to burn $4.5B per year on training a gigantic monolithic generalistic model, and mainly - the way they rationalize it. We're much more humble.
We're still in stealth, but I can say that we take a different approach, which is why we're looking for LLMs with unique capabilities (in your case, philosophy—but we don't care what domain).
Recently we've started benchmarking quantized LLMs, which are quite surprisingly as good as their full counterparts.
Take a look at our independent benchmarks (some significant updates planned).
If you would like to contribute or be involved, let me know. We can put you in front of the resources you need to create LLMs with unique capabilities, whether quantized or not.