relaxml

/

Llama-2-7b-QTIP-2Bit

like 0

Follow

RelaxML 35

See https://github.com/Cornell-RelaxML/qtip/

not run nevr

https://github.com/werruww/Xqtip/blob/main/Xqtip.ipynb

colab t4

I cannot help you if you are unwilling or unable to read instructions. Our generation pipeline is contained here https://github.com/Cornell-RelaxML/qtip/blob/main/eval/interactive_gen.py. We have not tested our code with the transformers pipeline class and it most likely does not work there as you have experienced.

https://github.com/werruww/Xqtip/blob/main/%D9%92%D9%92%D9%92%D9%92%D9%92%D9%92%D9%92%D9%92%D9%92%D9%92%D9%92%D9%92XXXXXXXXXX.ipynb

not run in colab t4
colab 75

requset 80

It is not installed in any way basically
Do you have a Colab page that has been tested
Colab as it worked for you will work for others
The problem is if you follow the instructions it does not execute

I tried all the instructions in your code pages and it didn't work because the core libraries don't install and I doubt they depend on their cores that don't work in Colab.

I cannot help you if you are unwilling or unable to read instructions

Don't blame me for following instructions or not. If you have a modern collab page, provide it to me. It will be much better. Thank you for your effort, but I can't try it.

Install the fast-hadamard-transform package

python -m eval.interactive_gen --hf_path relaxml/Llama-2-13b-QTIP-3Bit --max_new_tokens 256 --streaming

How to run awesome compressed models in Colab t4 without installation complications

https://github.com/kim90000/QTIP-run-but/blob/main/suc_QTIP_tpu%20(3).ipynb