Hugging Face's logo

TheBloke
/
alpaca-lora-65B-GPTQ

Text Generation
Transformers
PyTorch
Safetensors
llama
text-generation-inference
4-bit precision
gptq
Model card Files Community
6
alpaca-lora-65B-GPTQ / README.md
TheBloke's picture
TheBloke
Update for Transformers GPTQ support
4483583
preview code
|
raw
history blame contribute delete
6.39 kB
metadata
license: other
inference: false
TheBlokeAI

TheBloke's LLM work is generously supported by a grant from andreessen horowitz (a16z)

Alpaca LoRA 65B GPTQ 4bit

This is a GPTQ-for-LLaMa 4bit quantisation of changsung's alpaca-lora-65B

I also have 4bit and 2bit GGML files for cPU inference available here: TheBloke/alpaca-lora-65B-GGML.

These files need a lot of VRAM!

I believe they will work on 2 x 24GB cards, and I hope that at least the 1024g file will work on an A100 40GB.

I can't guarantee that the two 128g files will work in only 40GB of VRAM.

I haven't specifically tested VRAM requirements yet but will aim to do so at some point. If you have any experiences to share, please do so in the comments.

If you want to try CPU inference instead, check out my GGML repo: TheBloke/alpaca-lora-65B-GGML.

Provided files

Three files are provided, in separate branches.

  • alpaca-lora-65B-GPTQ-4bit-128g.no-act-order.safetensors - branch main
    • Will require ~40GB of VRAM, meaning you'll need an A100 or 2 x 24GB cards.
    • Parameters: Groupsize = 128g. No act-order.
    • Command used to create the GPTQ:
      CUDA_VISIBLE_DEVICES=0 python3 llama.py alpaca-lora-65B-HF c4 --wbits 4 --true-sequential --groupsize 128 --save_safetensors alpaca-lora-65B-GPTQ-4bit-128g.no-act-order.safetensors
  • alpaca-lora-65B-GPTQ-4bit-128g.safetensors - branch gptq-4bit-128g-actorder_True
    • Parameters: Groupsize = 128g. act-order.
    • Command used to create the GPTQ:
      CUDA_VISIBLE_DEVICES=0 python3 llama.py alpaca-lora-65B-HF c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save_safetensors alpaca-lora-65B-GPTQ-4bit-128g.safetensors
  • alpaca-lora-65B-GPTQ-4bit-1024g.safetensors - branch gptq-4bit-1024g-actorder_True
    • Parameters: Groupsize = 1024g. act-order.
    • Command used to create the GPTQ:
      CUDA_VISIBLE_DEVICES=0 python3 llama.py alpaca-lora-65B-HF c4 --wbits 4 --true-sequential --act-order --groupsize 1024 --save_safetensors alpaca-lora-65B-GPTQ-4bit-1024g.safetensors

How to run in text-generation-webui

Please see one of my more recent repos for instructions on loading GPTQ models in text-generation-webui.

Discord

For further support, and discussions on these models and AI in general, join us at:

TheBloke AI's Discord server

Thanks, and how to contribute.

Thanks to the chirper.ai team!

I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.

If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.

Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.

Special thanks to: Aemon Algiz.

Patreon special mentions: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter

Thank you to all my generous patrons and donaters!

And thank you again to a16z for their generous grant.

Original model card not provided

No model card was provided in changsung's original repository.

Based on the name, I assume this is the result of fine tuning using the original GPT 3.5 Alpaca dataset. It is unknown as to whether the original Stanford data was used, or the cleaned tloen/alpaca-lora variant.