Testing upload

README.md

@@ -11,117 +11,54 @@ tags:
 - evaluate
 - metric
 description: >-
-  BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference
-  sentences by cosine similarity.
-  It has been shown to correlate with human judgment on sentence-level and system-level evaluation.
-  Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language
-  generation tasks.
+  ParaScore is a new metric to scoring the performance of paraphrase generation tasks
   
-  See the project's README at https://github.com/Tiiiger/bert_score#readme for more information.
+  See the project at https://github.com/shadowkiller33/ParaScore for more information.
 ---
 
-# Metric Card for BERT Score
+# Metric Card for ParaScore
 
 ## Metric description
 
-BERTScore is an automatic evaluation metric for text generation that computes a similarity score for each token in the candidate sentence with each token in the reference sentence. It leverages the pre-trained contextual embeddings from [BERT](https://huggingface.co/bert-base-uncased) models and matches words in candidate and reference sentences by cosine similarity. 
-
-Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks. 
+ParaScore is a new metric to scoring the performance of paraphrase generation tasks
 
 ## How to use 
 
-BERTScore takes 3 mandatory arguments : `predictions` (a list of string of candidate sentences), `references` (a list of strings or list of list of strings of reference sentences) and either `lang` (a string of two letters indicating the language of the sentences, in [ISO 639-1 format](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)) or `model_type` (a string specififying which model to use, according to the BERT specification). The default behavior of the metric is to use the suggested model for the target language when one is specified, otherwise to use the `model_type` indicated.
-
 ```python
 from evaluate import load
-bertscore = load("bertscore")
+bertscore = load("transZ/test_parascore")
 predictions = ["hello there", "general kenobi"]
 references = ["hello there", "general kenobi"]
 results = bertscore.compute(predictions=predictions, references=references, lang="en")
 ```
 
-BERTScore also accepts multiple optional arguments: 
-
-
-`num_layers` (int): The layer of representation to use. The default is the number of layers tuned on WMT16 correlation data, which depends on the `model_type` used.
-
-`verbose` (bool): Turn on intermediate status update. The default value is `False`. 
-
-`idf` (bool or dict): Use idf weighting; can also be a precomputed idf_dict. 
-`device` (str): On which the contextual embedding model will be allocated on. If this argument is `None`, the model lives on `cuda:0` if cuda is available.
-`nthreads` (int): Number of threads used for computation. The default value is `4`. 
-`rescale_with_baseline` (bool): Rescale BERTScore with the  pre-computed baseline. The default value is `False`. 
-`batch_size` (int): BERTScore processing batch size, at least one of `model_type` or `lang`. `lang` needs to be specified when `rescale_with_baseline` is `True`.
-`baseline_path` (str): Customized baseline file.
-
-`use_fast_tokenizer` (bool): `use_fast` parameter passed to HF tokenizer. The default value is `False`. 
-
-
 ## Output values
 
-BERTScore outputs a dictionary with the following values:
-
-`precision`: The [precision](https://huggingface.co/metrics/precision) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0. 
-
-`recall`: The [recall](https://huggingface.co/metrics/recall) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0.
-
-`f1`: The [F1 score](https://huggingface.co/metrics/f1) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0.
- 
-`hashcode:` The hashcode of the library.
-
-
-### Values from popular papers
-The [original BERTScore paper](https://openreview.net/pdf?id=SkeHuCVFDr) reported average model selection accuracies (Hits@1) on WMT18 hybrid systems for different language pairs, which ranged from 0.004 for `en<->tr` to 0.824 for `en<->de`.
+ParaScore outputs a dictionary with the following values:
 
-For more recent model performance, see the [metric leaderboard](https://paperswithcode.com/paper/bertscore-evaluating-text-generation-with).
-
-## Examples 
-
-Maximal values with the `distilbert-base-uncased` model:
-
-```python
-from evaluate import load
-bertscore = load("bertscore")
-predictions = ["hello world", "general kenobi"]
-references = ["hello world", "general kenobi"]
-results = bertscore.compute(predictions=predictions, references=references, model_type="distilbert-base-uncased")
-print(results)
-{'precision': [1.0, 1.0], 'recall': [1.0, 1.0], 'f1': [1.0, 1.0], 'hashcode': 'distilbert-base-uncased_L5_no-idf_version=0.3.10(hug_trans=4.10.3)'}
-```
-
-Partial match with the `distilbert-base-uncased` model:
-
-```python
-from evaluate import load
-bertscore = load("bertscore")
-predictions = ["hello world", "general kenobi"]
-references = ["goodnight moon", "the sun is shining"]
-results = bertscore.compute(predictions=predictions, references=references, model_type="distilbert-base-uncased")
-print(results)
-{'precision': [0.7380737066268921, 0.5584042072296143], 'recall': [0.7380737066268921, 0.5889028906822205], 'f1': [0.7380737066268921, 0.5732481479644775], 'hashcode': 'bert-base-uncased_L5_no-idf_version=0.3.10(hug_trans=4.10.3)'}
-```
+`score`: Range from 0.0 to 1.0
 
 ## Limitations and bias
 
-The [original BERTScore paper](https://openreview.net/pdf?id=SkeHuCVFDr) showed that BERTScore correlates well with human judgment on sentence-level and system-level evaluation, but this depends on the model and language pair selected.
-
-Furthermore, not all languages are supported by the metric -- see the [BERTScore supported language list](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages) for more information.
-
-Finally, calculating the BERTScore metric involves downloading the BERT model that is used to compute the score-- the default model for `en`, `roberta-large`, takes over 1.4GB of storage space and downloading it can take a significant amount of time depending on the speed of your internet connection. If this is an issue, choose a smaller model; for instance `distilbert-base-uncased` is 268MB. A full list of compatible models can be found [here](https://docs.google.com/spreadsheets/d/1RKOVpselB98Nnh_EOC4A2BYn8_201tmPODpNWu4w7xI/edit#gid=0).  
-
+The [original ParaScore paper](https://arxiv.org/abs/2202.08479) showed that ParaScore correlates well with human judgment on sentence-level and system-level evaluation, but this depends on the model and language pair selected.
 
 ## Citation
 
 ```bibtex
-@inproceedings{bert-score,
-  title={BERTScore: Evaluating Text Generation with BERT},
-  author={Tianyi Zhang* and Varsha Kishore* and Felix Wu* and Kilian Q. Weinberger and Yoav Artzi},
-  booktitle={International Conference on Learning Representations},
-  year={2020},
-  url={https://openreview.net/forum?id=SkeHuCVFDr}
+@article{Shen2022,
+    archivePrefix = {arXiv},
+    arxivId = {2202.08479},
+    author = {Shen, Lingfeng and Liu, Lemao and Jiang, Haiyun and Shi, Shuming},
+    journal = {EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings},
+    eprint = {2202.08479},
+    month = {feb},
+    number = {1},
+    pages = {3178--3190},
+    title = {{On the Evaluation Metrics for Paraphrase Generation}},
+    url = {http://arxiv.org/abs/2202.08479},
+    year = {2022}
 }
 ```
     
 ## Further References 
-- [BERTScore Project README](https://github.com/Tiiiger/bert_score#readme)
-- [BERTScore ICLR 2020 Poster Presentation](https://iclr.cc/virtual_2020/poster_SkeHuCVFDr.html)
+- [Offcial implementation](https://github.com/shadowkiller33/parascore_toolkit)

test_parascore.py

@@ -15,54 +15,59 @@
 
 import evaluate
 import datasets
+import nltk
 
 
-# TODO: Add BibTeX citation
 _CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
+@article{Shen2022,
+archivePrefix = {arXiv},
+arxivId = {2202.08479},
+author = {Shen, Lingfeng and Liu, Lemao and Jiang, Haiyun and Shi, Shuming},
+journal = {EMNLP 2022 - 2022 Conference on Empirical Methods in Natural Language Processing, Proceedings},
+eprint = {2202.08479},
+month = {feb},
+number = {1},
+pages = {3178--3190},
+title = {{On the Evaluation Metrics for Paraphrase Generation}},
+url = {http://arxiv.org/abs/2202.08479},
+year = {2022}
 }
 """
 
-# TODO: Add description of the module here
 _DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
+ParaScore is a new metric to scoring the performance of paraphrase generation tasks
 """
 
 
 # TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates how good are predictions given some references, using certain scores
+Calculates how good the paraphrase is
 Args:
     predictions: list of predictions to score. Each predictions
         should be a string with tokens separated by spaces.
     references: list of reference for each prediction. Each
         reference should be a string with tokens separated by spaces.
 Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
+    score: description of the first score,
 Examples:
     Examples should be written in doctest format, and should illustrate how
     to use the function.
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
+    >>> metrics = evaluate.load("transZ/test_parascore")
+    >>> results = my_new_module.compute(references=["They work for 6 months"], predictions=["They have working for 6 months"])
     >>> print(results)
-    {'accuracy': 1.0}
+    {'score': 0.85}
 """
 
 # TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
+BAD_WORDS_URL = "https://github.com/shadowkiller33/parascore_toolkit"
 
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class test_parascore(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
+    """ParaScore is a new metric to scoring the performance of paraphrase generation tasks"""
 
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
             # This is the description that will appear on the modules page.
             module_type="metric",
@@ -70,26 +75,60 @@ class test_parascore(evaluate.Metric):
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
             # This defines the format of each prediction and reference
-            features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
-            }),
+            features=[
+                datasets.Features(
+                    {
+                        "predictions": datasets.Value("string", id="sequence"),
+                        "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
+                    }
+                ),
+                datasets.Features(
+                    {
+                        "predictions": datasets.Value("string", id="sequence"),
+                        "references": datasets.Value("string", id="sequence"),
+                    }
+                ),
+            ],
             # Homepage of the module for documentation
-            homepage="http://module.homepage",
+            homepage="https://github.com/shadowkiller33/ParaScore",
             # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            codebase_urls=["https://github.com/shadowkiller33/ParaScore"],
+            reference_urls=["https://github.com/shadowkiller33/ParaScore"]
         )
 
     def _download_and_prepare(self, dl_manager):
         """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
-        pass
+        self.score = evaluate.load('bertscore')
 
-    def _compute(self, predictions, references):
+    def _edit(self, x, y, lang='en'):
+        if lang == 'zh':
+            x = x.replace(" ", "")
+            y = y.replace(" ", "")
+        a = len(x)
+        b = len(y)
+        dis = nltk.edit_distance(x,y)
+        return dis/max(a,b)
+
+    def _diverse(self, cands, sources, lang='en'):
+        diversity = []
+        thresh = 0.35
+        for x, y in zip(cands, sources):
+            div = self._edit(x, y, lang)
+            if div >= thresh:
+                ss = thresh
+            elif div < thresh:
+                ss = -1 + ((thresh + 1) / thresh) * div
+            diversity.append(ss)
+        return diversity[0]
+
+    def _compute(self, predictions, references, lang='en'):
         """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
+        
+        score = self.score.compute(predictions=predictions, references=references, lang=lang)
+        bert_score = round(score['f1'], 2)
+        diversity = self._diverse(predictions, references, lang)
+
+        score = bert_score + 0.05 * diversity
         return {
-            "accuracy": accuracy,
+            "accuracy": score,
         }