Update README.md

README.md

@@ -1,50 +1,47 @@
 ---
-title: apps-metric
+title: apps_metric
 datasets:
 - 
 tags:
 - evaluate
 - metric
-description: "TODO: add a description here"
+description: "Evaluation metric for the APPS benchmark"
 sdk: gradio
 sdk_version: 3.0.2
 app_file: app.py
 pinned: false
 ---
 
-# Metric Card for apps-metric
-
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
+# Metric Card for apps-metric [WIP]
 
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+This metric is used to evaluate code generation on the [APPS benchmark](https://huggingface.co/datasets/codeparrot/apps).
 
 ## How to Use
-*Give general statement of how to use the metric*
-
-*Provide simplest possible example for using the metric*
+You can load the metric and use it with the following commands:
+```
+from evaluate import load
+glue_metric = load('loubnabnl/apps_metric')
+results = apps_metric.compute(predictions=generations)
+```
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+**generations** (list(str)): List of code generations, each sub-list corresponds to the generation for a problem in APPS dataset, the order of the samples in the dataset must be kept.
 
 ### Output Values
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
-
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
+**average accuracy**: when a single solution is generated, average accuracy computes the average of test cases that are passed.
 
-#### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
+**strict accuracy**: when a single solution is generated, strict accuracy computes the average number of problems that pass all their test cases.
 
-### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
-
-## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+**pass@k**: when multiple solutions are generated per problem, pass@k is the metric originally used for the [HumanEval](https://huggingface.co/datasets/openai_humaneval) benchmark. For more details please refer to the [metric space](https://huggingface.co/spaces/evaluate-metric/code_eval) and [Codex paper](https://arxiv.org/pdf/2107.03374v2.pdf).
 
 ## Citation
-*Cite the source where this metric was introduced.*
-
-## Further References
-*Add any useful further references.*
+```
+@article{hendrycksapps2021,
+ title={Measuring Coding Challenge Competence With APPS},
+ author={Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt},
+ journal={NeurIPS},
+ year={2021}
+}
+```