Hugging Face's logo

TheBloke
/
galpaca-30B-GPTQ

Text Generation
Transformers
opt
galactica
alpaca
gptq
Model card Files Community
5
galpaca-30B-GPTQ / README.md
TheBloke's picture
TheBloke
Updating model files
dadc8c8
preview code
|
raw
history blame
4.73 kB
metadata
license: cc-by-nc-4.0
datasets:
  - tatsu-lab/alpaca
library_name: transformers
pipeline_tag: text-generation
tags:
  - galactica
  - alpaca
  - opt
  - gptq
inference: false
TheBlokeAI

About this repository

This is an attempt to create a GPTQ 4-bit version of Galpaca 30B.

I created these files on request. I have no previous experience of Galactica or Galpaca, and have not done much testing to confirm that the output is useful and usable.

You will need 18+ GB VRAM to load these models on a GPU.

Provided files

  • galpaca-30B-4bit-128g.no-act-order.pt

    • Created with: python3 opt.py /content/galpaca-30b c4 --wbits 4 --new-eval --groupsize 128 --save galpaca-30B-4bit-128g.no-act-order.pt
    • This file seems to create usable results, tested with text-generation-webui.

  • galpaca-30B-4bit-128g.pt

    • Created with: python3 opt.py /content/galpaca-30b c4 --wbits 4 --new-eval --act-order --groupsize 128 --save galpaca-30B-4bit-128g.pt
    • In my testing so far, this file does not work. It produces garbage output.
    • If you can get it working, please let me know!

GPTQ

The GPTQ code used to create these models can be found at GPTQ-for-LLaMa.

Note that during the GPTQ process, the following warning was seen:

Token indices sequence length is longer than the specified maximum sequence length for this model (1915 > 512). Running this sequence through the model will result in indexing errors

I do not know if this is indicates a potential problem in the GPTQ output, or if it can be ignored. If you know more about this, do let me know.

General model info below is as per the original Galpaca 30B repository.

For more information, example prompts and more, please see the original repository.

GALPACA 30B (large)

GALACTICA 30B fine-tuned on the Alpaca dataset.

The model card from the original Galactica repo can be found here, and the original paper here.

The dataset card for Alpaca can be found here, and the project homepage here. The Alpaca dataset was collected with a modified version of the Self-Instruct Framework, and was built using OpenAI's text-davinci-003 model. As such it is subject to OpenAI's terms of service.

Want to support my work?

I've had a lot of people ask if they can contribute. I love providing models and helping people, but it is starting to rack up pretty big cloud computing bills.

So if you're able and willing to contribute, it'd be most gratefully received and will help me to keep providing models, and work on various AI projects.

Donaters will get priority support on any and all AI/LLM/model questions, and I'll gladly quantise any model you'd like to try.

Model Details

The GALACTICA models are trained on a large-scale scientific corpus and are designed to perform scientific tasks. The Alpaca dataset is a set of 52k instruct-response pairs designed to enhace the instruction following capabilites of pre-trained language models.

Model Use

The GALACTICA model card specifies that the primary indended users of the GALACTICA models are researchers studying language models applied to the scientific domain, and it cautions against production use of GALACTICA without safeguards due to the potential for the model to produce inaccurate information. The original GALACTICA models are available under a non-commercial CC BY-NC 4.0 license, and the GALPACA model is additionally subject to the OpenAI Terms of Service.