different size can be input and python version is retreated

README.md

@@ -12,19 +12,28 @@ pinned: false
 
 # Metric Card for matching_series
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
-
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+Matching Series is a metric for evaluating time-series generation models. It is based on the idea of matching the generated time-series with the original time-series. The metric calculates the Mean Squared Error (MSE) between the generated time-series and the original time-series between matched instances. The metric outputs a score greater or equal to 0, where 0 indicates a perfect generation.
 
 ## How to Use
-*Give general statement of how to use the metric*
-
-*Provide simplest possible example for using the metric*
+At minium, the metric requires the original time-series and the generated time-series as input. The metric can be used to evaluate the performance of time-series generation models.
+
+```python
+>>> num_generation = 100
+>>> num_reference = 10
+>>> seq_len = 100
+>>> num_features = 10
+>>> references = np.random.rand(num_reference, seq_len, num_features)
+>>> predictions = np.random.rand(num_generation, seq_len, num_features)
+>>> metric = evaluate.load("bowdbeg/matching_series")
+>>> results = metric.compute(references=references, predictions=predictions, batch_size=1000)
+>>> print(results)
+{'matching_mse': 0.15250070138019745, 'harmonic_mean': 0.15246672297315564, 'covered_mse': 0.15243275970407652, 'index_mse': 0.16772539808686357, 'matching_mse_features': [0.11976368411913872, 0.1238622735860897, 0.1235259257706047, 0.12385236248438022, 0.12241466736218365, 0.12328439290438079, 0.1232240061707885, 0.12342319803028035, 0.12235222572924524, 0.12437865819262514], 'harmonic_mean_features': [0.12010478503934609, 0.12379899085819131, 0.12321441761307182, 0.12273884163905005, 0.12256126537300535, 0.12323289686030311, 0.12323847434641247, 0.12333469339243568, 0.12273530480438972, 0.12390254295493403], 'covered_mse_features': [0.12044783449951382, 0.1237357727610885, 0.12290447662839017, 0.12164516506865233, 0.12270821492248948, 0.12318144381818667, 0.12325294591995689, 0.12324631559392285, 0.12312079021887229, 0.12343005890751833], 'index_mse_features': [0.16331894487549958, 0.1679797859239729, 0.16904075114728268, 0.16962427920551068, 0.16915910655024802, 0.16686197230602684, 0.17056311327206022, 0.1638796919248867, 0.16736730842643857, 0.16945902723670975], 'macro_matching_mse': 0.1230081394349717, 'macro_covered_mse': 0.12276730183385913, 'macro_harmonic_mean': 0.12288622128811397}
+```
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+- **predictions**: (list of list of list of float or numpy.ndarray): The generated time-series. The shape of the array should be `(num_generation, seq_len, num_features)`.
+- **references**: (list of list of list of float or numpy.ndarray): The original time-series. The shape of the array should be `(num_reference, seq_len, num_features)`.
 
 ### Output Values
 

dicrect_compute_metric.py

@@ -0,0 +1,49 @@
+from typing import Optional
+
+import evaluate
+
+
+class DirectComputeMetric(evaluate.Metric):
+    """
+    Base class for metrics that directly compute the score from the predictions and references without add_batch
+    """
+
+    def compute(self, *, predictions=None, references=None, **kwargs) -> Optional[dict]:
+        """Compute the evaluation module.
+
+        Usage of positional arguments is not allowed to prevent mistakes.
+
+        Args:
+            predictions (`list/array/tensor`, *optional*):
+                Predictions.
+            references (`list/array/tensor`, *optional*):
+                References.
+            **kwargs (optional):
+                Keyword arguments that will be forwarded to the evaluation module [`~evaluate.EvaluationModule.compute`]
+                method (see details in the docstring).
+
+        Return:
+            `dict` or `None`
+
+            - Dictionary with the results if this evaluation module is run on the main process (`process_id == 0`).
+            - `None` if the evaluation module is not run on the main process (`process_id != 0`).
+
+        ```py
+        >>> import evaluate
+        >>> accuracy =  evaluate.load("accuracy")
+        >>> accuracy.compute(predictions=[0, 1, 1, 0], references=[0, 1, 0, 1])
+        ```
+        """
+        all_kwargs = {"predictions": predictions, "references": references, **kwargs}
+        if predictions is None and references is None:
+            missing_kwargs = {k: None for k in self._feature_names() if k not in all_kwargs}
+            all_kwargs.update(missing_kwargs)
+        else:
+            missing_inputs = [k for k in self._feature_names() if k not in all_kwargs]
+            if missing_inputs:
+                raise ValueError(
+                    f"Evaluation module inputs are missing: {missing_inputs}. All required inputs are {list(self._feature_names())}"
+                )
+        inputs = {input_name: all_kwargs[input_name] for input_name in self._feature_names()}
+        compute_kwargs = {k: kwargs[k] for k in kwargs if k not in self._feature_names()}
+        return self._compute(**inputs, **compute_kwargs)

matching_series.py

@@ -14,11 +14,14 @@
 """TODO: Add a description here."""
 
 import statistics
+from typing import Optional, Union
 
 import datasets
 import evaluate
 import numpy as np
 
+from dicrect_compute_metric import DirectComputeMetric
+
 # TODO: Add BibTeX citation
 _CITATION = """\
 @InProceedings{huggingface:module,
@@ -55,7 +58,7 @@ Examples:
 
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
-class matching_series(evaluate.Metric):
+class matching_series(DirectComputeMetric):
     """TODO: Short description of my evaluation module."""
 
     def _info(self):
@@ -74,7 +77,7 @@ class matching_series(evaluate.Metric):
                 }
             ),
             # Homepage of the module for documentation
-            homepage="http://module.homepage",
+            homepage="https://huggingface.co/spaces/bowdbeg/matching_series",
             # Additional links to the codebase or references
             codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
             reference_urls=["http://path.to.reference.url/new_module"],
@@ -84,7 +87,12 @@ class matching_series(evaluate.Metric):
         """Optional: download external resources useful to compute the scores"""
         pass
 
-    def _compute(self, predictions: list | np.ndarray, references: list | np.ndarray, batch_size: None | int = None):
+    def _compute(
+        self,
+        predictions: Union[list, np.ndarray],
+        references: Union[list, np.ndarray],
+        batch_size: Optional[int] = None,
+    ):
         """
         Compute the scores of the module given the predictions and references
         Args: