maorivgi
commited on
Commit
•
92db9b7
1
Parent(s):
b8c8c2e
updated the readme
Browse files
README.md
CHANGED
@@ -21,7 +21,18 @@ T5 v1.1 includes several improvments on top of the original checkpoint. see its
|
|
21 |
You can use the raw model for text infilling. However, the model is mostly meant to be fine-tuned on a supervised dataset.
|
22 |
|
23 |
### How to use
|
24 |
-
To use the model, you first
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
25 |
|
26 |
Here is how to use this model in PyTorch:
|
27 |
|
@@ -34,12 +45,26 @@ outputs = model(**inputs)
|
|
34 |
last_hidden_states = outputs.last_hidden_state
|
35 |
```
|
36 |
You can also replace SledModel by SledModelForConditionalGeneration for Seq2Seq generation
|
|
|
|
|
|
|
37 |
|
38 |
In case you wish to apply SLED on a task containing a prefix (e.g. question) which should be given as a context to
|
39 |
every chunk, you can pass the `prefix_length` tensor input as well (A LongTensor in the length of the batch size).
|
40 |
-
|
41 |
-
|
42 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
43 |
|
44 |
### BibTeX entry and citation info
|
45 |
|
|
|
21 |
You can use the raw model for text infilling. However, the model is mostly meant to be fine-tuned on a supervised dataset.
|
22 |
|
23 |
### How to use
|
24 |
+
To use the model, you first need to install `sled-py` in your environment (or clone the code from the [official repository](https://github.com/Mivg/SLED/blob/main/README.md))
|
25 |
+
```
|
26 |
+
pip install sled-py
|
27 |
+
```
|
28 |
+
For more installation instructions, see [here](https://github.com/Mivg/SLED#Installation).
|
29 |
+
|
30 |
+
Once installed, SLED is fully compatible with HuggingFace's AutoClasses (AutoTokenizer, AutoConfig, AutoModel
|
31 |
+
and AutoModelForCausalLM) and can be loaded using the from_pretrained methods
|
32 |
+
```python
|
33 |
+
import sled # *** required so that SledModels will be registered for the AutoClasses ***
|
34 |
+
model = AutoModel.from_pretrained('tau/t5-v1_1-base-sled')
|
35 |
+
```
|
36 |
|
37 |
Here is how to use this model in PyTorch:
|
38 |
|
|
|
45 |
last_hidden_states = outputs.last_hidden_state
|
46 |
```
|
47 |
You can also replace SledModel by SledModelForConditionalGeneration for Seq2Seq generation
|
48 |
+
```python
|
49 |
+
model = SledModelForConditionalGeneration.from_pretrained('tau/t5-v1_1-base-sled')
|
50 |
+
```
|
51 |
|
52 |
In case you wish to apply SLED on a task containing a prefix (e.g. question) which should be given as a context to
|
53 |
every chunk, you can pass the `prefix_length` tensor input as well (A LongTensor in the length of the batch size).
|
54 |
+
```python
|
55 |
+
import torch
|
56 |
+
import sled # *** required so that SledModels will be registered for the AutoClasses ***
|
57 |
+
tokenizer = AutoTokenizer.from_pretrained('tau/t5-v1_1-base-sled')
|
58 |
+
model = AutoModel.from_pretrained('tau/t5-v1_1-base-sled')
|
59 |
+
document_input_ids = tokenizer("Dogs are great for you.", return_tensors="pt").input_ids
|
60 |
+
prefix_input_ids = tokenizer("Are dogs good for you?", return_tensors="pt").input_ids
|
61 |
+
input_ids = torch.cat((prefix_input_ids, document_input_ids), dim=-1)
|
62 |
+
attention_mask = torch.ones_like(input_ids)
|
63 |
+
prefix_length = torch.LongTensor([[prefix_input_ids.size(1)]])
|
64 |
+
|
65 |
+
outputs = model(input_ids=input_ids, attention_mask=attention_mask, prefix_length=prefix_length)
|
66 |
+
last_hidden_states = outputs.last_hidden_state
|
67 |
+
```
|
68 |
|
69 |
### BibTeX entry and citation info
|
70 |
|