File size: 3,420 Bytes
9df30db 02c68ec 0d57b8c 8e49fdb 0d57b8c 513cd61 48d184f 513cd61 0d57b8c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
---
license: cc-by-nc-4.0
datasets:
- tatsu-lab/alpaca
library_name: transformers
pipeline_tag: text-generation
tags:
- galactica
- alpaca
- opt
---
# GALPACA 30B (large)
GALACTICA 30B fine-tuned on the Alpaca dataset.
The model card from the original Galactica repo can be found [here](https://github.com/paperswithcode/galai/blob/main/docs/model_card.md), and the original paper [here](https://galactica.org/paper.pdf).
The dataset card for Alpaca can be found [here](https://huggingface.co/datasets/tatsu-lab/alpaca/blob/main/README.md), and the project homepage [here](https://crfm.stanford.edu/2023/03/13/alpaca.html).
The Alpaca dataset was collected with a modified version of the [Self-Instruct Framework](https://github.com/yizhongw/self-instruct), and was built using OpenAI's `text-davinci-003` model. As such it is subject to OpenAI's terms of service.
## Model Details
The GALACTICA models are trained on a large-scale scientific corpus and are designed to perform scientific tasks.
The Alpaca dataset is a set of 52k instruct-response pairs designed to enhace the instruction following capabilites of pre-trained language models.
## Model Use
The GALACTICA model card specifies that the primary indended users of the GALACTICA models are researchers studying language models applied to the scientific domain, and it cautions against production use of GALACTICA without safeguards due to the potential for the model to produce inaccurate information.
The original GALACTICA models are available under a non-commercial CC BY-NC 4.0 license, and the GALPACA model is additionally subject to the [OpenAI Terms of Service](https://openai.com/policies/terms-of-use).
## Training Data
The GALPACA models are trained by fine-tuning pre-trained GALACTICA models on the Alpaca dataset. GALACTICA models were trained on 106 billion tokens of open-access scientific text and data, including papers, textbooks, scientific websites, encyclopedias, and more.
Fine-tuning the base GALACTICA models on the 52k instruction-response pairs in the Alpaca dataset allows users to query the GALPACA models in an instruct-response fashion.
## How to Use
The GALPACA weights are made available for use with the `transformers` library.
TODO: add example inference usage.
## Training Resources
GALPACA 30B was fine-tuned in about 6 hours using 16 A100 80GB GPUS using 16-bit mixed-precision at an effective batch-size of 1024 and with a maximum context window of 384 tokens. This model was trained using DeepSpeed Stage 3 optimizations.
## Performance and Limitations
More information about the performance and limitations of this family of models can be found on the original GALACTICA model card.
## Works Cited
```bibtex
@inproceedings{GALACTICA,
title={GALACTICA: A Large Language Model for Science},
author={Ross Taylor and Marcin Kardas and Guillem Cucurull and Thomas Scialom and Anthony Hartshorn and Elvis Saravia and Andrew Poulton and Viktor Kerkez and Robert Stojnic},
year={2022}
}
```
```bibtex
@misc{alpaca,
author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
title = {Stanford Alpaca: An Instruction-following LLaMA model},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}
```
|