lucasmccabe-lmi
commited on
Commit
•
fcc49aa
1
Parent(s):
97ae314
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,70 @@
|
|
1 |
---
|
2 |
-
license:
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
license: cc-by-4.0
|
3 |
---
|
4 |
+
# Model Card
|
5 |
+
|
6 |
+
We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model.
|
7 |
+
|
8 |
+
|
9 |
+
|
10 |
+
## Prompting
|
11 |
+
|
12 |
+
For instructions that **do not** require extraneous inputs, the recommended prompt is:
|
13 |
+
|
14 |
+
```
|
15 |
+
Below is an instruction that describes a task. Write a response that appropriately completes the request.
|
16 |
+
|
17 |
+
|
18 |
+
### Instruction:
|
19 |
+
|
20 |
+
< -- instruction goes here --- >
|
21 |
+
|
22 |
+
### Response:
|
23 |
+
```
|
24 |
+
|
25 |
+
For instructions that **do** require extraneous inputs, the recommended prompt is:
|
26 |
+
|
27 |
+
```
|
28 |
+
Below is an instruction that describes a task. Write a response that appropriately completes the request.
|
29 |
+
|
30 |
+
|
31 |
+
### Instruction:
|
32 |
+
|
33 |
+
< -- instruction goes here -- >
|
34 |
+
|
35 |
+
### Input:
|
36 |
+
|
37 |
+
< -- extraneous input goes here -- >
|
38 |
+
|
39 |
+
### Response:
|
40 |
+
```
|
41 |
+
|
42 |
+
Since the model performs [causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling), the model's response to the prompt is the text completing the sequence beginning with the prompt.
|
43 |
+
|
44 |
+
|
45 |
+
## Instruction-Tuning
|
46 |
+
|
47 |
+
This model was instruction-tuned on an internally-curated Alpaca-style dataset.
|
48 |
+
|
49 |
+
- **Epochs**: 3
|
50 |
+
- **Batch size**: 128
|
51 |
+
- **Cutoff length**: 2048
|
52 |
+
- **Learning rate**: 3.2e-5, cosine decay
|
53 |
+
- **LoRA _r_**: 8
|
54 |
+
- **LoRA _alpha_**: 16
|
55 |
+
- **LoRA _dropout_**: 0.0
|
56 |
+
- **LoRA target modules**: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
|
57 |
+
- **License**: The instruction-tuning data is subject to the [Creative Commons 4.0](https://creativecommons.org/licenses/by/4.0/) license.
|
58 |
+
|
59 |
+
## Base Model
|
60 |
+
|
61 |
+
This model was instruction-tuned from a 12B variant from the Pythia family.
|
62 |
+
|
63 |
+
- **Repository:** [EleutherAI/pythia-12b](https://huggingface.co/EleutherAI/pythia-12b)
|
64 |
+
- **Paper:** [arxiv:2304.01373](https://arxiv.org/abs/2304.01373)
|
65 |
+
- **License**: The base model is subject to the Apache 2.0 license.
|
66 |
+
- **Model type**: Transformer-based Language Model
|
67 |
+
|
68 |
+
## Licensing Information
|
69 |
+
|
70 |
+
We release this adapter under the [Creative Commons NonCommercial (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/) license.
|