google
/

ul2

@@ -97,58 +97,70 @@ This model was contributed by [Daniel Hesslow](https://huggingface.co/Seledorn).
 ## Examples
 The following shows how one can predict masked passages using the different denoising strategies.
 ```python
 from transformers import T5ForConditionalGeneration, AutoTokenizer
-model = T5ForConditionalGeneration.from_pretrained("Seledorn/ul2", low_cpu_mem_usage=True, torch_dtype=torch.bfloat16)
-tokenizer = AutoTokenizer.from_pretrained("Seledorn/ul2")
 ```
-# Example usage
 ```python
-inps = ["""
-Mr. and Mrs. Dursley, of number four, Privet Drive, <extra_id_0> the last people you'd expect to be involved in anything strange or mysterious, because they just didn't hold with such nonsense.
-Mr. Dursley was the director of a firm called Grunnings, which made drills. He was a big, <extra_id_1>. Mrs. Dursley was thin and blonde and had nearly twice the usual amount of neck, which came in very useful as she spent so much of her time craning over garden fences, spying on the neighbours. The Dursleys had a small son called Dudley and in their opinion there was no finer boy anywhere.
-The Dursleys had everything they wanted, but they also had a secret, and their greatest fear was that somebody would discover it. They didn't think they could bear it if anyone found out about the Potters. Mrs. Potter was Mrs. Dursley's sister, but they hadn't met for several years; in fact, Mrs. Dursley pretended she didn't have a sister, because her sister and her good-for-nothing husband were as unDursleyish as it was possible to be. The Dursleys shuddered to think what the neighbours would say if the Potters arrived in the street. The Dursleys knew that the Potters had a small son, too, but they had never even seen him. This boy was another good reason for keeping the Potters away; they didn't want Dudley mixing with a child like that."""
-]
-Note use `[NLG]` for X-denoisers, `[NLU]` for R-denoisers and `[S2S]` for S-Denoisers.
-model.cuda()
-model.eval()
-with torch.no_grad():
-    for inp in inps:
-        inputs = tokenizer(inp, return_tensors="pt").input_ids
-        inputs_ = inputs.cuda()
-        outputs = model.generate(inputs_, max_length = 200, do_sample=True, temperature = 0.9, num_return_sequences=4)
-        for output in outputs:
-            out = tokenizer.decode(output)
-            inps = re.split(pattern, inp)
-            outs = re.split(pattern, out)
-            l = [z for (x,y) in zip(inps, outs[1:len(inps)]+ [""]) for z in (x,"*"+y+"*" if y != "" else "")]
-            print("".join(l))
-            print("-------------------------------")
 ```
-Example output
-Mr. and Mrs. Dursley, of number four, Privet Drive, **were** the last people you'd expect to be involved in anything strange or mysterious, because they just didn't hold with such nonsense.
-Mr. Dursley was the director of a firm called Grunnings, which made drills. He was a big, **solid man with a short, brown beard, an enormous head full of brains and a round, fat face. He had a voice, too, that was so deep that it was nearly always accompanied by a slight tremor..**. Mrs. Dursley was thin and blonde and had nearly twice the usual amount of neck, which came in very useful as she spent so much of her time craning over garden fences, spying on the neighbours. The Dursleys had a small son called Dudley and in their opinion there was no finer boy anywhere.
-The Dursleys had everything they wanted, but they also had a secret, and their greatest fear was that somebody would discover it. They didn't think they could bear it if anyone found out about the Potters. Mrs. Potter was Mrs. Dursley's sister, but they hadn't met for several years; in fact, Mrs. Dursley pretended she didn't have a sister, because her sister and her good-for-nothing husband were as unDursleyish as it was possible to be. The Dursleys shuddered to think what the neighbours would say if the Potters arrived in the street. The Dursleys knew that the Potters had a small son, too, but they had never even seen him. This boy was another good reason for keeping the Potters away; they didn't want Dudley mixing with a child like that.
-Where bold is the completion of the model.

 ## Examples
 The following shows how one can predict masked passages using the different denoising strategies.
+Given the size of the model the following examples need to be run on at least a 40GB A100 GPU.
+### R-Denoising
+For *R-Denoising*, please make sure to prompt the text with the prefix `[S2S]` as shown below.
 ```python
 from transformers import T5ForConditionalGeneration, AutoTokenizer
+import torch
+model = T5ForConditionalGeneration.from_pretrained("google/ul2", low_cpu_mem_usage=True, torch_dtype=torch.bfloat16).to("cuda")
+tokenizer = AutoTokenizer.from_pretrained("google/ul2")
+input_string = "[S2S] Mr. Dursley was the director of a firm called Grunnings, which made drills. He was a big, solid man with a bald head. Mrs. Dursley was thin and blonde and more than the usual amount of neck, which came in very useful as she spent so much of her time craning over garden fences, spying on the neighbours. The Dursleys had a small son called Dudley and in their opinion there was no finer boy anywhere <extra_id_0>"
+inputs = tokenizer(input_string, return_tensors="pt").input_ids.to("cuda")
+outputs = model.generate(inputs, max_length=200)
+print(tokenizer.decode(outputs[0]))
+# -> <pad>. Dudley was a very good boy, but he was also very stupid.</s>
 ```
+### S-Denoising
+For *S-Denoising*, please make sure to prompt the text with the prefix `[NLU]` as shown below.
 ```python
+from transformers import T5ForConditionalGeneration, AutoTokenizer
+import torch
+model = T5ForConditionalGeneration.from_pretrained("google/ul2", low_cpu_mem_usage=True, torch_dtype=torch.bfloat16).to("cuda")
+tokenizer = AutoTokenizer.from_pretrained("google/ul2")
+input_string = "[NLU] Mr. Dursley was the director of a firm called <extra_id_0>, which made <extra_id_1>. He was a big, solid man with a bald head. Mrs. Dursley was thin and <extra_id_2> of neck, which came in very useful as she spent so much of her time <extra_id_3>. The Dursleys had a small son called Dudley and <extra_id_4>"
+inputs = tokenizer(input_string, return_tensors="pt", add_special_tokens=False).input_ids.to("cuda")
+outputs = model.generate(inputs, max_length=200)
+print(tokenizer.decode(outputs[0]))
+# -> "<pad><extra_id_0> Burrows<extra_id_1> brooms for witches and wizards<extra_id_2> had a lot<extra_id_3> scolding Dudley<extra_id_4> a daughter called Petunia. Dudley was a nasty, spoiled little boy who was always getting into trouble. He was very fond of his pet rat, Scabbers.<extra_id_5> Burrows<extra_id_3> screaming at him<extra_id_4> a daughter called Petunia</s>
+"
 ```
+### X-Denoising
+For *X-Denoising*, please make sure to prompt the text with the prefix `[NLG]` as shown below.
+```python
+from transformers import T5ForConditionalGeneration, AutoTokenizer
+import torch
+model = T5ForConditionalGeneration.from_pretrained("google/ul2", low_cpu_mem_usage=True, torch_dtype=torch.bfloat16).to("cuda")
+tokenizer = AutoTokenizer.from_pretrained("google/ul2")
+input_string = "[NLG] Mr. Dursley was the director of a firm called Grunnings, which made drills. He was a big, solid man wiht a bald head. Mrs. Dursley was thin and blonde and more than the usual amount of neck, which came in very useful as she
+spent so much of her time craning over garden fences, spying on the neighbours. The Dursleys had a small son called Dudley and in their opinion there was no finer boy anywhere. <extra_id_0>"
+model.cuda()
+inputs = tokenizer(input_string, return_tensors="pt", add_special_tokens=False).input_ids.to("cuda")
+outputs = model.generate(inputs, max_length=200)
+print(tokenizer.decode(outputs[0]))
+# -> "<pad><extra_id_0> Burrows<extra_id_1> a lot of money from the manufacture of a product called '' Burrows'''s ''<extra_id_2> had a lot<extra_id_3> looking down people's throats<extra_id_4> a daughter called Petunia. Dudley was a very stupid boy who was always getting into trouble. He was a big, fat, ugly boy who was always getting into trouble. He was a big, fat, ugly boy who was always getting into trouble. He was a big, fat, ugly boy who was always getting into trouble. He was a big, fat, ugly boy who was always getting into trouble. He was a big, fat, ugly boy who was always getting into trouble. He was a big, fat, ugly boy who was always getting into trouble. He was a big, fat, ugly boy who was always getting into trouble. He was a big, fat,"
+```