ROC Curve

README.md

@@ -1,50 +1,83 @@
 ---
-title: Roc Curve
-datasets:
--  
-tags:
-- evaluate
-- metric
-description: "TODO: add a description here"
+title: ROC Curve
+emoji: 📉
+colorFrom: yellow
+colorTo: green
 sdk: gradio
-sdk_version: 3.0.2
+sdk_version: 3.17.0
 app_file: app.py
 pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+ Compute Receiver operating characteristic (ROC).
+ Note: this implementation is restricted to the binary classification task.
 ---
 
-# Metric Card for Roc Curve
+# Metric Card for Confusion Matrix
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
 
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
 
-## How to Use
-*Give general statement of how to use the metric*
+Compute Receiver operating characteristic (ROC).
+Note: this implementation is restricted to the binary classification task.
 
-*Provide simplest possible example for using the metric*
 
-### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+## How to Use
 
-### Output Values
+At minimum, this metric requires predictions and references as inputs.
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+```python
+>>> cfm_metric = evaluate.load("BucketHeadP65/roc_curve")
+>>> results = cfm_metric.compute(references=[1, 0, 1, 1, 0], prediction_scores=[0.1, 0.4, 0.6, 0.7, 0.1])
+>>> print(results)
+{'roc_curve': (array([0. , 0. , 0. , 0.5, 1. ]), array([0.        , 0.33333333, 0.66666667, 0.66666667, 1.        ]), array([1.69999999, 0.69999999, 0.60000002, 0.40000001, 0.1       ]))}
+```
 
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
 
-#### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
+### Inputs
+- **prediction_scores** (`list` of `float`): Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers).
+- **references** (`list` of `int`): Ground truth labels.
+- **pos_label** (`int` or `str`): default=None True binary labels. If labels are not either {-1, 1} or {0, 1}, then pos_label should be explicitly given.
+- **sample_weight** (`list` of `float`): Sample weights Defaults to None.
+- **drop_intermediate** (`bool`): default=True
+        Whether to drop some suboptimal thresholds which would not appear
+        on a plotted ROC curve. This is useful in order to create lighter
+        ROC curves.
 
-### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+### Output Values
+- **fpr** (`ndarray`): Increasing false positive rates such that element i is the false
+        positive rate of predictions with score >= `thresholds[i]`.
+- **tpr** (`ndarray`): Increasing true positive rates such that element `i` is the true
+        positive rate of predictions with score >= `thresholds[i]`.
+- **thresholds** (`ndarray`): Decreasing thresholds on the decision function used to compute
+        `fpr` and `tpr`. `thresholds[0]` represents no instances being predicted
+        and is arbitrarily set to `max(y_score) + 1`.
 
-## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+Output Example(s):
+```python
+'roc_curve': (array([0. , 0. , 0. , 0.5, 1. ]), array([0.        , 0.33333333, 0.66666667, 0.66666667, 1.        ]), array([1.69999999, 0.69999999, 0.60000002, 0.40000001, 0.1       ]))}
 
-## Citation
-*Cite the source where this metric was introduced.*
+```
+This metric outputs a dictionary, containing the confusion matrix.
 
+## Citation(s)
+```bibtex
+@article{scikit-learn,
+  title={Scikit-learn: Machine Learning in {P}ython},
+  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  journal={Journal of Machine Learning Research},
+  volume={12},
+  pages={2825--2830},
+  year={2011}
+}
+```
 ## Further References
-*Add any useful further references.*
+Wikipedia entry for the Confusion matrix
+<https://en.wikipedia.org/wiki/Confusion_matrix>`_
+(Wikipedia and other references may use a different
+convention for axes).

app.py

@@ -3,4 +3,4 @@ from evaluate.utils import launch_gradio_widget
 
 
 module = evaluate.load("BucketHeadP65/roc_curve")
-launch_gradio_widget(module)
+launch_gradio_widget(module)

requirements.txt

@@ -1 +1,2 @@
-git+https://github.com/huggingface/evaluate@main
+git+https://github.com/huggingface/evaluate@75c09ff6cd599744d0641e4ef1d4cfedbd35eec9
+scikit-learn

roc_curve.py

@@ -1,95 +1,155 @@
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""TODO: Add a description here."""
+"""Confusion Matrix metric."""
 
-import evaluate
 import datasets
+import evaluate
+from sklearn.metrics import roc_curve
 
-
-# TODO: Add BibTeX citation
-_CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
-}
-"""
-
-# TODO: Add description of the module here
-_DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
+_DESCRIPTION = """
+Compute Receiver operating characteristic (ROC).
+Note: this implementation is restricted to the binary classification task.
 """
 
 
-# TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates how good are predictions given some references, using certain scores
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
+
+    y_true : ndarray of shape (n_samples,)
+        True binary labels. If labels are not either {-1, 1} or {0, 1}, then
+        pos_label should be explicitly given.
+
+    y_score : ndarray of shape (n_samples,)
+        Target scores, can either be probability estimates of the positive
+        class, confidence values, or non-thresholded measure of decisions
+        (as returned by "decision_function" on some classifiers).
+
+    pos_label : int or str, default=None
+        The label of the positive class.
+        When ``pos_label=None``, if `y_true` is in {-1, 1} or {0, 1},
+        ``pos_label`` is set to 1, otherwise an error will be raised.
+
+    sample_weight : array-like of shape (n_samples,), default=None
+        Sample weights.
+
+    drop_intermediate : bool, default=True
+        Whether to drop some suboptimal thresholds which would not appear
+        on a plotted ROC curve. This is useful in order to create lighter
+        ROC curves.
+
+        .. versionadded:: 0.17
+           parameter *drop_intermediate*.
+
 Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
+
+    fpr : ndarray of shape (>2,)
+        Increasing false positive rates such that element i is the false
+        positive rate of predictions with score >= `thresholds[i]`.
+
+    tpr : ndarray of shape (>2,)
+        Increasing true positive rates such that element `i` is the true
+        positive rate of predictions with score >= `thresholds[i]`.
+
+    thresholds : ndarray of shape = (n_thresholds,)
+        Decreasing thresholds on the decision function used to compute
+        fpr and tpr. `thresholds[0]` represents no instances being predicted
+        and is arbitrarily set to `max(y_score) + 1`.
+
+See Also:
+
+    RocCurveDisplay.from_estimator : Plot Receiver Operating Characteristic
+        (ROC) curve given an estimator and some data.
+    RocCurveDisplay.from_predictions : Plot Receiver Operating Characteristic
+        (ROC) curve given the true and predicted values.
+    det_curve: Compute error rates for different probability thresholds.
+    roc_auc_score : Compute the area under the ROC curve.
+
+Notes:
+
+    Since the thresholds are sorted from low to high values, they
+    are reversed upon returning them to ensure they correspond to both ``fpr``
+    and ``tpr``, which are sorted in reversed order during their calculation.
+
+References:
+
+    .. [1] `Wikipedia entry for the Receiver operating characteristic
+            <https://en.wikipedia.org/wiki/Receiver_operating_characteristic>`_
+
+    .. [2] Fawcett T. An introduction to ROC analysis[J]. Pattern Recognition
+           Letters, 2006, 27(8):861-874.
+
 Examples:
-    Examples should be written in doctest format, and should illustrate how
-    to use the function.
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'accuracy': 1.0}
+   >>> import numpy as np
+    >>> from sklearn import metrics
+    >>> y = np.array([1, 1, 2, 2])
+    >>> scores = np.array([0.1, 0.4, 0.35, 0.8])
+    >>> fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=2)
+    >>> fpr
+    array([0. , 0. , 0.5, 0.5, 1. ])
+    >>> tpr
+    array([0. , 0.5, 0.5, 1. , 1. ])
+    >>> thresholds
+    array([1.8 , 0.8 , 0.4 , 0.35, 0.1 ])
 """
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
 
+_CITATION = """
+@article{scikit-learn,
+  title={Scikit-learn: Machine Learning in {P}ython},
+  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  journal={Journal of Machine Learning Research},
+  volume={12},
+  pages={2825--2830},
+  year={2011}
+}
+"""
 
-@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
-class RocCurve(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
 
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class ConfusionMatrix(evaluate.Metric):
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
-            # This is the description that will appear on the modules page.
-            module_type="metric",
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
-            # This defines the format of each prediction and reference
-            features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
-            }),
-            # Homepage of the module for documentation
-            homepage="http://module.homepage",
-            # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            features=datasets.Features(
+                {
+                    "prediction_scores": datasets.Sequence(datasets.Value("float")),
+                    "references": datasets.Value("int32"),
+                }
+                if self.config_name == "multiclass"
+                else {
+                    "references": datasets.Sequence(datasets.Value("int32")),
+                    "prediction_scores": datasets.Sequence(datasets.Value("float")),
+                }
+                if self.config_name == "multilabel"
+                else {
+                    "references": datasets.Value("int32"),
+                    "prediction_scores": datasets.Value("float"),
+                }
+            ),
+            reference_urls=[
+                "https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html"
+            ],
         )
 
-    def _download_and_prepare(self, dl_manager):
-        """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
-        pass
-
-    def _compute(self, predictions, references):
-        """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
+    def _compute(
+        self,
+        references,
+        prediction_scores,
+        *,
+        pos_label=None,
+        sample_weight=None,
+        drop_intermediate=True
+    ):
         return {
-            "accuracy": accuracy,
-        }
+            "roc_curve": roc_curve(
+                y_true=references,
+                y_score=prediction_scores,
+                pos_label=pos_label,
+                sample_weight=sample_weight,
+                drop_intermediate=drop_intermediate,
+            )
+        }

tests.py

@@ -1,17 +0,0 @@
-test_cases = [
-    {
-        "predictions": [0, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0}
-    },
-    {
-        "predictions": [1, 1],
-        "references": [1, 1],
-        "result": {"metric_score": 1}
-    },
-    {
-        "predictions": [1, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0.5}
-    }
-]