Adding jaccard similarity

README.md

@@ -1,12 +1,118 @@
 ---
 title: Jaccard Similarity
-emoji: 📈
-colorFrom: red
+emoji: 🤗 
+colorFrom: blue
 colorTo: red
 sdk: gradio
-sdk_version: 4.44.0
+sdk_version: 3.19.1
 app_file: app.py
 pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+  Jaccard similarity coefficient score is defined as the size of the intersection divided by the size of the union of two label sets. It is used to compare the set of predicted labels for a sample to the corresponding set of true labels.
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Metric Card for Jaccard Similarity
+
+## Metric Description
+
+The Jaccard similarity coefficient score, also known as the Jaccard index, is defined as the size of the intersection divided by the size of the union of two label sets. It is used to compare the set of predicted labels for a sample to the corresponding set of true labels.
+
+For binary classification, it can be computed as:
+Jaccard = TP / (TP + FP + FN)
+Where:
+TP: True positive
+FP: False positive
+FN: False negative
+
+The metric supports multiclass and multilabel classification by treating it as a collection of binary problems, one for each label.
+
+## How to Use
+
+At minimum, this metric requires predictions and references as inputs.
+
+```python
+>>> jaccard_metric = evaluate.load("jaccard_similarity")
+>>> results = jaccard_metric.compute(references=[0, 1, 2, 0, 1, 2], predictions=[0, 2, 1, 0, 0, 1])
+>>> print(results)
+{'jaccard_similarity': 0.3333333333333333}
+```
+
+### Inputs
+- **predictions** (`list` of `int` or `array-like` of shape (n_samples,) or (n_samples, n_classes)): Predicted labels or label indicators.
+- **references** (`list` of `int` or `array-like` of shape (n_samples,) or (n_samples, n_classes)): Ground truth labels or label indicators.
+- **average** (`string`, default='binary'): This parameter is required for multiclass/multilabel targets. Options are ['binary', 'micro', 'macro', 'samples', 'weighted', None].
+- **labels** (`list` of `int`, default=None): The set of labels to include when `average != 'binary'`.
+- **pos_label** (`int`, `float`, `bool` or `str`, default=1): The class to report if `average='binary'` and the data is binary.
+- **sample_weight** (`list` of `float`, default=None): Sample weights.
+- **zero_division** (`"warn"`, `0` or `1`, default="warn"): Sets the value to return when there is a zero division.
+
+### Output Values
+- **jaccard_similarity** (`float` or `ndarray` of `float64`): Jaccard similarity score. Minimum possible value is 0. Maximum possible value is 1.0. A higher score means higher similarity.
+
+Output Example:
+```python
+{'jaccard_similarity': 0.3333333333333333}
+```
+
+This metric outputs a dictionary containing the Jaccard similarity score.
+
+#### Values from Popular Papers
+
+Jaccard similarity is often used in information retrieval and text similarity tasks. For example, it's used to evaluate the performance of named entity recognition systems or in plagiarism detection.
+
+### Examples
+
+Example 1 - Binary classification:
+```python
+>>> jaccard_metric = evaluate.load("jaccard_similarity")
+>>> results = jaccard_metric.compute(references=[0, 1, 1, 1], predictions=[1, 1, 0, 1])
+>>> print(results)
+{'jaccard_similarity': 0.6666666666666666}
+```
+
+Example 2 - Multiclass classification:
+```python
+>>> jaccard_metric = evaluate.load("jaccard_similarity")
+>>> results = jaccard_metric.compute(references=[0, 1, 2, 3], predictions=[0, 2, 1, 3], average='macro')
+>>> print(results)
+{'jaccard_similarity': 0.5}
+```
+
+Example 3 - Multilabel classification:
+```python
+>>> jaccard_metric = evaluate.load("jaccard_similarity")
+>>> results = jaccard_metric.compute(
+...     references=[[0, 1, 1], [0, 1, 1]],
+...     predictions=[[1, 1, 0], [0, 1, 0]],
+...     average='samples'
+... )
+>>> print(results)
+{'jaccard_similarity': 0.41666666666666663}
+```
+
+## Limitations and Bias
+Jaccard similarity may be a poor metric if there are no positives for some samples or classes. It is undefined if there are no true or predicted labels, and our implementation will return a score of 0 with a warning in such cases.
+
+For imbalanced datasets, Jaccard similarity might not provide a complete picture of the model's performance. In such cases, it's often beneficial to use it in conjunction with other metrics like precision, recall, and F1-score.
+
+## Citation
+```bibtex
+@article{scikit-learn,
+  title={Scikit-learn: Machine Learning in {P}ython},
+  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  journal={Journal of Machine Learning Research},
+  volume={12},
+  pages={2825--2830},
+  year={2011}
+}
+```
+
+## Further References
+- [Scikit-learn documentation on Jaccard similarity score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html)
+- [Wikipedia entry for the Jaccard index](https://en.wikipedia.org/wiki/Jaccard_index)

app.py

@@ -0,0 +1,6 @@
+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("Ruchin/jaccard_similarity")
+launch_gradio_widget(module)

jaccard_similarity.py

@@ -0,0 +1,102 @@
+# Copyright 2023 The HuggingFace Evaluate Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Jaccard similarity metric."""
+
+import evaluate
+import datasets
+from sklearn.metrics import jaccard_score
+import numpy as np
+
+
+_CITATION = """\
+@article{jaccard1912distribution,
+  title={The distribution of the flora in the alpine zone},
+  author={Jaccard, Paul},
+  journal={New phytologist},
+  volume={11},
+  number={2},
+  pages={37--50},
+  year={1912},
+  publisher={Wiley Online Library}
+}
+"""
+
+_DESCRIPTION = """\
+Jaccard similarity is a statistic used for gauging the similarity and diversity of sample sets. 
+The Jaccard coefficient measures similarity between finite sample sets, and is defined as the size of 
+the intersection divided by the size of the union of the sample sets. This implementation uses 
+scikit-learn's jaccard_score function.
+"""
+
+_KWARGS_DESCRIPTION = """
+Calculates the Jaccard similarity between predictions and references using scikit-learn.
+Args:
+    predictions: 1d array-like, or label indicator array / sparse matrix
+        Predicted labels, as returned by a classifier.
+    references: 1d array-like, or label indicator array / sparse matrix
+        Ground truth (correct) labels.
+    labels: array-like of shape (n_classes,), default=None
+        The set of labels to include when average != 'binary', and their order if average is None.
+    pos_label: int, float, bool or str, default=1
+        The class to report if average='binary' and the data is binary.
+    average: {'micro', 'macro', 'samples', 'weighted', 'binary'} or None, default='binary'
+        This parameter is required for multiclass/multilabel targets.
+    sample_weight: array-like of shape (n_samples,), default=None
+        Sample weights.
+    zero_division: "warn", {0.0, 1.0}, default="warn"
+        Sets the value to return when there is a zero division.
+Returns:
+    jaccard_similarity: float or ndarray of shape (n_unique_labels,)
+        Jaccard similarity score.
+Examples:
+    >>> jaccard_metric = evaluate.load("jaccard_similarity")
+    >>> predictions = [0, 2, 1, 3]
+    >>> references = [0, 1, 2, 3]
+    >>> results = jaccard_metric.compute(predictions=predictions, references=references, average='macro')
+    >>> print(results)
+    {'jaccard_similarity': 0.5}
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class JaccardSimilarity(evaluate.Metric):
+    def _info(self):
+        return evaluate.MetricInfo(
+            module_type="metric",
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features({
+                'predictions': datasets.Value('int32'),
+                'references': datasets.Value('int32'),
+            }),
+            reference_urls=[
+                "https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html",
+                "https://en.wikipedia.org/wiki/Jaccard_index"
+            ],
+        )
+
+    def _compute(self, predictions, references, labels=None, pos_label=1, average='binary', sample_weight=None, zero_division='warn'):
+        """Returns the Jaccard similarity score using scikit-learn"""
+        return {
+            "jaccard_similarity": jaccard_score(
+                references, 
+                predictions, 
+                labels=labels, 
+                pos_label=pos_label, 
+                average=average, 
+                sample_weight=sample_weight, 
+                zero_division=zero_division
+            )
+        }

requirements.txt

@@ -0,0 +1 @@
+git+https://github.com/huggingface/evaluate@main

tests.py

@@ -0,0 +1,73 @@
+test_cases = [
+    {
+        "predictions": [0, 0],
+        "references": [1, 1],
+        "result": {"jaccard_similarity": 0.0}
+    },
+    {
+        "predictions": [1, 1],
+        "references": [1, 1],
+        "result": {"jaccard_similarity": 1.0}
+    },
+    {
+        "predictions": [1, 0],
+        "references": [1, 1],
+        "result": {"jaccard_similarity": 0.5}
+    },
+    {
+        "predictions": [0, 1, 2, 3],
+        "references": [0, 1, 2, 3],
+        "result": {"jaccard_similarity": 1.0}
+    },
+    {
+        "predictions": [0, 1, 2, 3],
+        "references": [3, 2, 1, 0],
+        "result": {"jaccard_similarity": 1.0}
+    },
+    {
+        "predictions": [0, 0, 1, 1],
+        "references": [1, 1, 1, 1],
+        "result": {"jaccard_similarity": 0.5}
+    },
+    {
+        "predictions": [0, 0, 0, 1],
+        "references": [1, 1, 1, 1],
+        "result": {"jaccard_similarity": 0.25}
+    },
+    {
+        "predictions": [0, 1, 2, 3],
+        "references": [0, 1, 2, 3],
+        "result": {"jaccard_similarity": 1.0},
+        "kwargs": {"average": "macro"}
+    },
+    {
+        "predictions": [0, 0, 1, 1],
+        "references": [1, 1, 1, 1],
+        "result": {"jaccard_similarity": 0.5},
+        "kwargs": {"average": "binary", "pos_label": 1}
+    },
+    {
+        "predictions": [0, 1, 2, 0],
+        "references": [0, 1, 1, 2],
+        "result": {"jaccard_similarity": 0.375},
+        "kwargs": {"average": "weighted"}
+    },
+    {
+        "predictions": [[1, 1, 1], [1, 0, 0]],
+        "references": [[0, 1, 1], [1, 1, 0]],
+        "result": {"jaccard_similarity": 0.5833333333333334},
+        "kwargs": {"average": "samples"}
+    },
+    {
+        "predictions": [[1, 1, 1], [1, 0, 0]],
+        "references": [[0, 1, 1], [1, 1, 0]],
+        "result": {"jaccard_similarity": 0.6666666666666666},
+        "kwargs": {"average": "macro"}
+    },
+    {
+        "predictions": [[1, 1, 1], [1, 0, 0]],
+        "references": [[0, 1, 1], [1, 1, 0]],
+        "result": {"jaccard_similarity": [0.5, 0.5, 1.0]},
+        "kwargs": {"average": None}
+    },
+]