Spaces:
Running
Running
| title: det-metrics | |
| tags: | |
| - evaluate | |
| - metric | |
| description: >- | |
| Modified cocoevals.py which is wrapped into torchmetrics' mAP metric with | |
| numpy instead of torch dependency. | |
| sdk: gradio | |
| sdk_version: 4.44.0 | |
| app_file: app.py | |
| pinned: false | |
| emoji: 🕵️ | |
| # SEA-AI/det-metrics | |
| This hugging face metric uses `seametrics.detection.PrecisionRecallF1Support` under the hood to compute coco-like metrics for object detection tasks. It is a [modified cocoeval.py](https://github.com/SEA-AI/seametrics/blob/develop/seametrics/detection/cocoeval.py) wrapped inside [torchmetrics' mAP metric](https://lightning.ai/docs/torchmetrics/stable/detection/mean_average_precision.html) but with numpy arrays instead of torch tensors. | |
| ## Getting Started | |
| To get started with det-metrics, make sure you have the necessary dependencies installed. This metric relies on the `evaluate` and `seametrics` libraries for metric calculation and integration with FiftyOne datasets. | |
| ### Installation | |
| First, ensure you have Python 3.8 or later installed. Then, install det-metrics using pip: | |
| ```sh | |
| pip install evaluate git+https://github.com/SEA-AI/seametrics@develop | |
| ``` | |
| ## Basic Usage | |
| Here's how to quickly evaluate your object detection models using SEA-AI/det-metrics: | |
| ```python | |
| import evaluate | |
| # Define your predictions and references (dict values can also by numpy arrays) | |
| predictions = [ | |
| { | |
| "boxes": [[449.3, 197.75390625, 6.25, 7.03125], [334.3, 181.58203125, 11.5625, 6.85546875]], | |
| "labels": [0, 0], | |
| "scores": [0.153076171875, 0.72314453125], | |
| } | |
| ] | |
| references = [ | |
| { | |
| "boxes": [[449.3, 197.75390625, 6.25, 7.03125], [334.3, 181.58203125, 11.5625, 6.85546875]], | |
| "labels": [0, 0], | |
| "area": [132.2, 83.8], | |
| } | |
| ] | |
| # Load SEA-AI/det-metrics and evaluate | |
| module = evaluate.load("SEA-AI/det-metrics") | |
| module.add(prediction=predictions, reference=references) | |
| results = module.compute() | |
| print(results) | |
| ``` | |
| This will output the following dictionary containing metrics for the detection model. The key of the dictionary will be the model name or "custom" if no model names are available like in this case. | |
| ```json | |
| { | |
| "custom": { | |
| "metrics": ..., | |
| "eval": ..., | |
| "params": ... | |
| } | |
| } | |
| ``` | |
| - `metrics`: A dictionary containing performance metrics for each area range | |
| - `eval`: Output of COCOeval.accumulate() | |
| - `params`: COCOeval parameters object | |
| See [Output Values](#output-values) for more detailed information about the returned results structure, which includes metrics, eval, and params fields for each model passed as input. | |
| ## FiftyOne Integration | |
| Integrate SEA-AI/det-metrics with FiftyOne datasets for enhanced analysis and visualization: | |
| ```python | |
| import evaluate | |
| import logging | |
| from seametrics.payload.processor import PayloadProcessor | |
| logging.basicConfig(level=logging.WARNING) | |
| # Configure your dataset and model details | |
| processor = PayloadProcessor( | |
| dataset_name="SAILING_DATASET_QA", | |
| gt_field="ground_truth_det", | |
| models=["yolov5n6_RGB_D2304-v1_9C", "tf1zoo_ssd-mobilenet-v2_agnostic_D2207"], | |
| sequence_list=["Trip_14_Seq_1"], | |
| data_type="rgb", | |
| slices=["rgb"] | |
| ) | |
| # Evaluate using SEA-AI/det-metrics | |
| module = evaluate.load("SEA-AI/det-metrics", payload=processor.payload) | |
| results = module.compute() | |
| print(results) | |
| ``` | |
| This will output the following dictionary containing metrics for the detection model. The key of the dictionary will be the model name. | |
| ```json | |
| { | |
| "yolov5n6_RGB_D2304-v1_9C": { | |
| "metrics": ..., | |
| "eval": ..., | |
| "params": ... | |
| }, | |
| "tf1zoo_ssd-mobilenet-v2_agnostic_D2207": { | |
| "metrics": ..., | |
| "eval": ..., | |
| "params": ... | |
| } | |
| } | |
| ``` | |
| - `metrics`: A dictionary containing performance metrics for each area range | |
| - `eval`: Output of COCOeval.accumulate() | |
| - `params`: COCOeval parameters object | |
| See [Output Values](#output-values) for more detailed information about the returned results structure, which includes metrics, eval, and params fields for each model passed as input. | |
| ## Metric Settings | |
| Customize your evaluation by specifying various parameters when loading SEA-AI/det-metrics: | |
| - **area_ranges_tuples**: Define different area ranges for metrics calculation. | |
| - **bbox_format**: Set the bounding box format (e.g., `"xywh"`). | |
| - **iou_threshold**: Choose the IOU threshold for determining correct detections. | |
| - **class_agnostic**: Specify whether to calculate metrics disregarding class labels. | |
| ```python | |
| area_ranges_tuples = [ | |
| ("all", [0, 1e5**2]), | |
| ("small", [0**2, 6**2]), | |
| ("medium", [6**2, 12**2]), | |
| ("large", [12**2, 1e5**2]), | |
| ] | |
| module = evaluate.load( | |
| "SEA-AI/det-metrics", | |
| iou_threshold=[0.00001], | |
| area_ranges_tuples=area_ranges_tuples, | |
| ) | |
| ``` | |
| ## Output Values | |
| For every model passed as input, the results contain the metrics, eval, and params fields. If no specific model was passed (usage without payload), the default model name “custom” will be used. | |
| ```json | |
| { | |
| "model_1": { | |
| "metrics": ..., | |
| "eval": ..., | |
| "params": ... | |
| }, | |
| "model_2": { | |
| "metrics": ..., | |
| "eval": ..., | |
| "params": ... | |
| }, | |
| "model_3": { | |
| "metrics": ..., | |
| "eval": ..., | |
| "params": ... | |
| }, | |
| ... | |
| } | |
| ``` | |
| ### Metrics | |
| SEA-AI/det-metrics metrics dictionary provides a detailed breakdown of performance metrics for each specified area range: | |
| - **range**: The area range considered. | |
| - **iouThr**: The IOU threshold applied. | |
| - **maxDets**: The maximum number of detections evaluated. | |
| - **tp/fp/fn**: Counts of true positives, false positives, and false negatives. | |
| - **duplicates**: Number of duplicate detections. | |
| - **precision/recall/f1**: Calculated precision, recall, and F1 score. | |
| - **support**: Number of ground truth boxes considered. | |
| - **fpi**: Number of images with predictions but no ground truths. | |
| - **nImgs**: Total number of images evaluated. | |
| ### Eval | |
| The SEA-AI/det-metrics evaluation dictionary provides details about evaluation metrics and results. Below is a description of each field: | |
| - **params**: Parameters used for evaluation, defining settings and conditions. | |
| - **counts**: Dimensions of parameters used in evaluation, represented as a list [T, R, K, A, M]: | |
| - T: IoU threshold (default: [1e-10]) | |
| - R: Recall threshold (not used) | |
| - K: Class index (class-agnostic, so only 0) | |
| - A: Area range (0=all, 1=valid_n, 2=valid_w, 3=tiny, 4=small, 5=medium, 6=large) | |
| - M: Max detections (default: [100]) | |
| - **date**: The date when the evaluation was performed. | |
| - **precision**: A multi-dimensional array [TxRxKxAxM] storing precision values for each evaluation setting. | |
| - **recall**: A multi-dimensional array [TxKxAxM] storing maximum recall values for each evaluation setting. | |
| - **scores**: Scores for each detection. | |
| - **TP**: True Positives - correct detections matching ground truth. | |
| - **FP**: False Positives - incorrect detections not matching ground truth. | |
| - **FN**: False Negatives - ground truth objects not detected. | |
| - **duplicates**: Duplicate detections of the same object. | |
| - **support**: Number of ground truth objects for each category. | |
| - **FPI**: False Positives per Image. | |
| - **TPC**: True Positives per Category. | |
| - **FPC**: False Positives per Category. | |
| - **sorted_conf**: Confidence scores of detections sorted in descending order. | |
| > Note: | |
| > **precision** and **recall** are set to -1 for settings with no ground truth objects. | |
| ### Params | |
| The params return value of the COCO evaluation parameters in PyCOCO represents a dictionary with various evaluation settings that can be customized. Here’s a breakdown of what each parameter means: | |
| - **imgIds**: List of image IDs to use for evaluation. By default, it evaluates on all images. | |
| - **catIds**: List of category IDs to use for evaluation. By default, it evaluates on all categories. | |
| - **iouThrs**: List of IoU (Intersection over Union) thresholds for evaluation. By default, it uses thresholds from 0.5 to 0.95 with a step of 0.05 (i.e., [0.5, 0.55, …, 0.95]). | |
| - **recThrs**: List of recall thresholds for evaluation. By default, it uses 101 thresholds from 0 to 1 with a step of 0.01 (i.e., [0, 0.01, …, 1]). | |
| - **areaRng**: Object area ranges for evaluation. This parameter defines the sizes of objects to evaluate. It is specified as a list of tuples, where each tuple represents a range of area in square pixels. | |
| - **maxDets**: List of thresholds on maximum detections per image for evaluation. By default, it evaluates with thresholds of 1, 10, and 100 detections per image. | |
| - **iouType**: Type of IoU calculation used for evaluation. It can be ‘segm’ (segmentation), ‘bbox’ (bounding box), or ‘keypoints’. | |
| - **useCats**: Boolean flag indicating whether to use category labels for evaluation (default is 1, meaning true). | |
| > Note: | |
| > If useCats=0 category labels are ignored as in proposal scoring. | |
| > Multiple areaRngs [Ax2] and maxDets [Mx1] can be specified. | |
| ## Confidence Curves | |
| When you use **module.generate_confidence_curves()**, it creates a graph that shows how metrics like **precision, recall, and f1 score** change as you adjust confidence thresholds. This helps you see the trade-offs between precision (how accurate positive predictions are) and recall (how well the model finds all positive instances) at different confidence levels. As confidence scores go up, models usually have higher precision but may find fewer positive instances, reflecting their certainty in making correct predictions. | |
| #### Confidence Config | |
| The `confidence_config` dictionary is set as `{"T": 0, "R": 0, "K": 0, "A": 0, "M": 0}`, where: | |
| - `T = 0`: represents the IoU (Intersection over Union) threshold. | |
| - `R = 0`: is the recall threshold, although it's currently not used. | |
| - `K = 0`: indicates a class index for class-agnostic mean Average Precision (mAP), with only one class indexed at 0. | |
| - `A = 0`: signifies that all object sizes are considered for evaluation. (0=all, 1=small, 2=medium, 3=large, ... depending on area ranges) | |
| - `M = 0`: sets the default maximum detections (`maxDets`) to 100 in precision_recall_f1_support calculations. | |
| ```python | |
| import evaluate | |
| import logging | |
| from seametrics.payload.processor import PayloadProcessor | |
| logging.basicConfig(level=logging.WARNING) | |
| # Configure your dataset and model details | |
| processor = PayloadProcessor( | |
| dataset_name="SAILING_DATASET_QA", | |
| gt_field="ground_truth_det", | |
| models=["yolov5n6_RGB_D2304-v1_9C"], | |
| sequence_list=["Trip_14_Seq_1"], | |
| data_type="rgb", | |
| ) | |
| # Evaluate using SEA-AI/det-metrics | |
| module = evaluate.load("SEA-AI/det-metrics", payload=processor.payload) | |
| results = module.compute() | |
| # Plot confidence curves | |
| confidence_config={"T": 0, "R": 0, "K": 0, "A": 0, "M": 0} | |
| fig = module.generate_confidence_curves(results, confidence_config) | |
| fig.show() | |
| ``` | |
|  | |
| ## W&B logging | |
| When you use **module.wandb()**, it is possible to log the detection metrics values for each model in Weights and Bias (W&B). The W&B key is stored as a Secret in this repository. | |
| ### Params | |
| - **wandb_project** - Name of the W&B project (Default: `'detection_metrics'`) | |
| ```python | |
| import evaluate | |
| import logging | |
| from seametrics.payload.processor import PayloadProcessor | |
| logging.basicConfig(level=logging.WARNING) | |
| # Configure your dataset and model details | |
| processor = PayloadProcessor( | |
| dataset_name="SAILING_PANOPTIC_DATASET_QA_REVIEWED", | |
| gt_field="ground_truth_det", | |
| models=["ahoy-b2-IR"], | |
| data_type="thermal", | |
| slices = ['thermal_right', 'thermal_left', 'thermal_narrow', 'thermal_wide'], | |
| ) | |
| AREA_RNGS = [[0, 1e5**2], | |
| [0**2, 6**2], | |
| [6**2, 12**2], | |
| [12**2, 1e5**2]] | |
| AREA_NAMES = ["all", "small", "medium", "large"] | |
| area_ranges_tuples = list(zip(AREA_NAMES, AREA_RNGS)) | |
| # Evaluate using SEA-AI/det-metrics | |
| module = evaluate.load("SEA-AI/det-metrics", payload=processor.payload, area_ranges_tuples=area_ranges_tuples) | |
| results = module.compute() | |
| # Log the results on W&B. | |
| module.wandb(results) | |
| ``` | |
|  | |
| ## Further References | |
| - **seametrics Library**: Explore the [seametrics GitHub repository](https://github.com/SEA-AI/seametrics/tree/main) for more details on the underlying library. | |
| - **Pycoco Tools**: SEA-AI/det-metrics calculations are based on [pycoco tools](https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools), a widely used library for COCO dataset evaluation. | |
| - **Understanding Metrics**: For a deeper understanding of precision, recall, and other metrics, read [this comprehensive guide](https://www.analyticsvidhya.com/blog/2020/09/precision-recall-machine-learning/). | |
| ## Contribution | |
| Your contributions are welcome! If you'd like to improve SEA-AI/det-metrics or add new features, please feel free to fork the repository, make your changes, and submit a pull request. |