## About this repository
This is an attempt to create a GPTQ 4-bit version of [Galpaca 30B](https://huggingface.co/GeorgiaTechResearchInstitute/galpaca-30b).
I created these files on request. I have no previous experience of Galactica or Galpaca, and have not done much testing to confirm that the output is useful and usable.
You will need 18+ GB VRAM to load these models on a GPU.
## Provided files
* `galpaca-30B-4bit-128g.no-act-order.pt`
* Created with: `python3 opt.py /content/galpaca-30b c4 --wbits 4 --new-eval --groupsize 128 --save galpaca-30B-4bit-128g.no-act-order.pt`
* This file seems to create usable results, tested with [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
* `galpaca-30B-4bit-128g.pt`
* Created with: `python3 opt.py /content/galpaca-30b c4 --wbits 4 --new-eval --act-order --groupsize 128 --save galpaca-30B-4bit-128g.pt`
* In my testing so far, **this file does not work**. It produces garbage output.
* If you can get it working, please let me know!
## GPTQ
The GPTQ code used to create these models can be found at [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).
Note that during the GPTQ process, the following warning was seen:
`Token indices sequence length is longer than the specified maximum sequence length for this model (1915 > 512). Running this sequence through the model will result in indexing errors`
I do not know if this is indicates a potential problem in the GPTQ output, or if it can be ignored. If you know more about this, do let me know.
General model info below is as per the original [Galpaca 30B repository](https://huggingface.co/GeorgiaTechResearchInstitute/galpaca-30b).
For more information, example prompts and more, please see the original repository.
# GALPACA 30B (large)
GALACTICA 30B fine-tuned on the Alpaca dataset.
The model card from the original Galactica repo can be found [here](https://github.com/paperswithcode/galai/blob/main/docs/model_card.md), and the original paper [here](https://galactica.org/paper.pdf).
The dataset card for Alpaca can be found [here](https://huggingface.co/datasets/tatsu-lab/alpaca/blob/main/README.md), and the project homepage [here](https://crfm.stanford.edu/2023/03/13/alpaca.html).
The Alpaca dataset was collected with a modified version of the [Self-Instruct Framework](https://github.com/yizhongw/self-instruct), and was built using OpenAI's `text-davinci-003` model. As such it is subject to OpenAI's terms of service.
## Want to support my work?
I've had a lot of people ask if they can contribute. I love providing models and helping people, but it is starting to rack up pretty big cloud computing bills.
So if you're able and willing to contribute, it'd be most gratefully received and will help me to keep providing models, and work on various AI projects.
Donaters will get priority support on any and all AI/LLM/model questions, and I'll gladly quantise any model you'd like to try.
* Patreon: coming soon! (just awaiting approval)
* Ko-Fi: https://ko-fi.com/TheBlokeAI
* Discord: https://discord.gg/UBgz4VXf
## Model Details
The GALACTICA models are trained on a large-scale scientific corpus and are designed to perform scientific tasks.
The Alpaca dataset is a set of 52k instruct-response pairs designed to enhace the instruction following capabilites of pre-trained language models.
## Model Use
The GALACTICA model card specifies that the primary indended users of the GALACTICA models are researchers studying language models applied to the scientific domain, and it cautions against production use of GALACTICA without safeguards due to the potential for the model to produce inaccurate information.
The original GALACTICA models are available under a non-commercial CC BY-NC 4.0 license, and the GALPACA model is additionally subject to the [OpenAI Terms of Service](https://openai.com/policies/terms-of-use).
