---
license: cc-by-4.0
---
# Model Card

We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model.



## Prompting

For instructions that **do not** require extraneous inputs, the recommended prompt is:

```
Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here --- >

### Response:
```

For instructions that **do** require extraneous inputs, the recommended prompt is:

```
Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here -- >

### Input:

< -- extraneous input goes here -- >

### Response:
```

Since the model performs [causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling), the model's response to the prompt is the text completing the sequence beginning with the prompt.


## Instruction-Tuning

This model was instruction-tuned on an internally-curated Alpaca-style dataset.

- **Epochs**: 3
- **Batch size**: 128
- **Cutoff length**: 2048
- **Learning rate**: 3.2e-5, cosine decay
- **LoRA _r_**: 8
- **LoRA _alpha_**: 16
- **LoRA _dropout_**: 0.0
- **LoRA target modules**: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
- **License**: The instruction-tuning data is subject to the [Creative Commons 4.0](https://creativecommons.org/licenses/by/4.0/) license.

## Base Model

This model was instruction-tuned from a 12B variant from the Pythia family. 

- **Repository:** [EleutherAI/pythia-12b](https://huggingface.co/EleutherAI/pythia-12b)
- **Paper:** [arxiv:2304.01373](https://arxiv.org/abs/2304.01373)
- **License**: The base model is subject to the Apache 2.0 license.
- **Model type**: Transformer-based Language Model

## Licensing Information

We release this adapter under the [Creative Commons NonCommercial (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/) license.