metadata

title: PanopticQuality
tags:
  - evaluate
  - metric
description: PanopticQuality score
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
emoji: 🖼️

SEA-AI/PanopticQuality

This hugging face metric uses seametrics.segmentation.PanopticQuality under the hood to compute a panoptic quality score. It is a wrapper class for the torchmetrics class torchmetrics.detection.PanopticQuality.

Getting Started

To get started with PanopticQuality, make sure you have the necessary dependencies installed. This metric relies on the evaluate, seametrics and seametrics[segmentation]libraries for metric calculation and integration with FiftyOne datasets.

Basic Usage

>>> import evaluate
>>> from seametrics.fo_utils.utils import fo_to_payload
>>> MODEL_FIELD = ["maskformer-27k-100ep"]
>>> payload = fo_to_payload("SAILING_PANOPTIC_DATASET_QA",
>>>                         gt_field="ground_truth_det",
>>>                         models=MODEL_FIELD,
>>>                         sequence_list=["Trip_55_Seq_2", "Trip_197_Seq_1", "Trip_197_Seq_68"],
>>>                         excluded_classes=[""])
>>> module = evaluate.load("SEA-AI/PanopticQuality")
>>> module.add_payload(payload, model_name=MODEL_FIELD[0])
>>> module.compute()
100%|██████████| 3/3 [00:03<00:00,  1.30s/it]
Added data ...
Start computing ...
Finished!
tensor(0.2082, dtype=torch.float64)

Metric Settings

The metric takes two optional input parameters: label2id and stuff.

label2id: Dict[str, int]: this dictionary is used to map string labels to an integer representation. if not provided a default setting will be used: {'WATER': 0, 'SKY': 1, 'LAND': 2, 'MOTORBOAT': 3, 'FAR_AWAY_OBJECT': 4, 'SAILING_BOAT_WITH_CLOSED_SAILS': 5, 'SHIP': 6, 'WATERCRAFT': 7, 'SPHERICAL_BUOY': 8, 'CONSTRUCTION': 9, 'FLOTSAM': 10, 'SAILING_BOAT_WITH_OPEN_SAILS': 11, 'CONTAINER': 12, 'PILLAR_BUOY': 13, 'AERIAL_ANIMAL': 14, 'HUMAN_IN_WATER': 15, 'OWN_BOAT': 16, 'WOODEN_LOG': 17, 'MARITIME_ANIMAL': 18}
stuff: List[str]: this list holds all string labels that belong to stuff. if not provided a default setting will be used: ["WATER", "SKY", "LAND", "CONSTRUCTION", "ICE", "OWN_BOAT"]

Output Values

A single float number between 0 and 1 is returned, which represents the PQ score. The bigger the number the better the PQ score, and vice versa.

Further References

seametrics Library: Explore the seametrics GitHub repository for more details on the underlying library.
Torchmetrics: https://lightning.ai/docs/torchmetrics/stable/detection/panoptic_quality.html
Understanding Metrics: The Panoptic Segmentation task, as well as Panoptic Quality as the evaluation metric, were introduced in this paper.

Contribution

Your contributions are welcome! If you'd like to improve SEA-AI/PanopticQuality or add new features, please feel free to fork the repository, make your changes, and submit a pull request.