Parrot
1. What is Parrot?
Parrot is a paraphrase based utterance augmentation framework purpose built to accelerate training NLU models. A paraphrase framework is more than just a paraphrasing model. Please refer to the github page
Installation
pip install git+https://github.com/PrithivirajDamodaran/Parrot.git
Quickstart
from parrot import Parrot
import torch
import warnings
warnings.filterwarnings("ignore")
'''
uncomment to get reproducable paraphrase generations
def random_state(seed):
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed)
random_state(1234)
'''
#Init models (make sure you init ONLY once if you integrate this to your code)
parrot = Parrot(model_tag="prithivida/parrot_paraphraser_on_T5", use_gpu=False)
phrases = ["Can you recommed some upscale restaurants in Rome?",
"What are the famous places we should not miss in Russia?"
]
for phrase in phrases:
print("-"*100)
print("Input_phrase: ", phrase)
print("-"*100)
para_phrases = parrot.augment(input_phrase=phrase)
for para_phrase in para_phrases:
print(para_phrase)
-----------------------------------------------------------------------------
Input_phrase: Can you recommed some upscale restaurants in Rome?
-----------------------------------------------------------------------------
"which upscale restaurants are recommended in rome?"
"which are the best restaurants in rome?"
"are there any upscale restaurants near rome?"
"can you recommend a good restaurant in rome?"
"can you recommend some of the best restaurants in rome?"
"can you recommend some best restaurants in rome?"
"can you recommend some upscale restaurants in rome?"
-----------------------------------------------------------------------------
Input_phrase: What are the famous places we should not miss in Russia
-----------------------------------------------------------------------------
"which are the must do places for tourists to visit in russia?"
"what are the best places to visit in russia?"
"what are some of the most visited sights in russia?"
"what are some of the most beautiful places in russia that tourists should not miss?"
"which are some of the most beautiful places to visit in russia?"
"what are some of the most important places to visit in russia?"
"what are some of the most famous places of russia?"
"what are some places we should not miss in russia?"
How to get syntactic and phrasal diversity/variety in your paraphrases using parrot?
You can play with the do_diverse knob (check out the next section for more knobs). Consider this example: do_diverse = False (default)
------------------------------------------------------------------------------
Input_phrase: The ultimate test of your knowledge is your capacity to convey it to another.
------------------------------------------------------------------------------
'the final test of knowledge is your capacity to impart it '
'the ultimate test of a person's knowledge is his ability to transmit it to another '
'the ultimate test of knowledge is the ability to communicate it to another '
'the ultimate test of knowledge is your ability to communicate it to others '
'the test of your knowledge is your capacity to communicate it to others '
'the ultimate test for knowledge is the capacity to show it to another '
'the ultimate test of your knowledge is your ability to transmit to others '
'the ultimate test of a knowledge is your capacity to communicate it to another '
'the ultimate test of knowledge is your capacity to transmit it to another '
'the final test of your knowledge is your ability to convey it to another '
do_diverse = True ```
Input_phrase: The ultimate test of your knowledge is your capacity to convey it to another.
'one of the ultimate tests of knowledge is your ability to communicate it to another person ' 'one of the ultimate tests of knowledge is your ability to transmit it to another person ' 'one of the ultimate tests of knowledge is your ability to communicate it to another ' 'one of the greatest tests of knowledge is your ability to convey it to another ' 'one of the ultimate tests of knowledge is your ability to transmit it to another ' 'one of the ultimate tests of knowledge is your ability to convey it to another person ' 'one of the ultimate tests of knowledge is the ability to convey it to another ' 'the ultimate test of your knowledge is your ability to communicate it to another ' 'the ultimate test of your knowledge is your ability to transmit it to another '
### Knobs
```python
para_phrases = parrot.augment(input_phrase=phrase,
diversity_ranker="levenshtein",
do_diverse=False,
max_return_phrases = 10,
max_length=32,
adequacy_threshold = 0.99,
fluency_threshold = 0.90)
2. Why Parrot?
Huggingface lists 12 paraphrase models, RapidAPI lists 7 fremium and commercial paraphrasers like QuillBot, Rasa has discussed an experimental paraphraser for augmenting text data here, Sentence-transfomers offers a paraphrase mining utility and NLPAug offers word level augmentation with a PPDB (a multi-million paraphrase database). While these attempts at paraphrasing are great, there are still some gaps and paraphrasing is NOT yet a mainstream option for text augmentation in building NLU models....Parrot is a humble attempt to fill some of these gaps.
What is a good paraphrase? Almost all conditioned text generation models are validated on 2 factors, (1) if the generated text conveys the same meaning as the original context (Adequacy) (2) if the text is fluent / grammatically correct english (Fluency). For instance Neural Machine Translation outputs are tested for Adequacy and Fluency. But a good paraphrase should be adequate and fluent while being as different as possible on the surface lexical form. With respect to this definition, the 3 key metrics that measures the quality of paraphrases are:
- Adequacy (Is the meaning preserved adequately?)
- Fluency (Is the paraphrase fluent English?)
- Diversity (Lexical / Phrasal / Syntactical) (How much has the paraphrase changed the original sentence?)
Parrot offers knobs to control Adequacy, Fluency and Diversity as per your needs.
What makes a paraphraser a good augmentor? For training a NLU model we just don't need a lot of utterances but utterances with intents and slots/entities annotated. Typical flow would be:
- Given an input utterance + input annotations a good augmentor spits out N output paraphrases while preserving the intent and slots.
- The output paraphrases are then converted into annotated data using the input annotations that we got in step 1.
- The annotated data created out of the output paraphrases then makes the training dataset for your NLU model.
But in general being a generative model paraphrasers doesn't guarantee to preserve the slots/entities. So the ability to generate high quality paraphrases in a constrained fashion without trading off the intents and slots for lexical dissimilarity makes a paraphraser a good augmentor. More on this in section 3 below
3. Scope
In the space of conversational engines, knowledge bots are to which we ask questions like "when was the Berlin wall teared down?", transactional bots are to which we give commands like "Turn on the music please" and voice assistants are the ones which can do both answer questions and action our commands. Parrot mainly foucses on augmenting texts typed-into or spoken-to conversational interfaces for building robust NLU models. (So usually people neither type out or yell out long paragraphs to conversational interfaces. Hence the pre-trained model is trained on text samples of maximum length of 32.)
While Parrot predominantly aims to be a text augmentor for building good NLU models, it can also be used as a pure-play paraphraser.