Spaces:
Running
Running
AtsuMiyai
commited on
Commit
•
4d0571a
1
Parent(s):
5b772ef
update explanations on MM-UPD Bench
Browse files- constants.py +16 -13
constants.py
CHANGED
@@ -37,24 +37,27 @@ LEADERBORAD_INTRODUCTION = """
|
|
37 |
|
38 |
## About MM-UPD Bench
|
39 |
### What is MM-UPD Bench?
|
40 |
-
MM-UPD Bench
|
|
|
41 |
Our MM-UPD Bench encompasses three benchmarks: MM-AAD, MM-IASD, and MM-IVQD.
|
42 |
-
- **MM-AAD:** Benchmark for Absent Answer Detection (AAD). MM-AAD Bench is a dataset where the correct answer
|
43 |
-
option for each question is removed. MM-AAD tests the model's capability
|
44 |
-
to recognize when the correct answer is absent from the provided choices.
|
45 |
|
46 |
-
- **MM-
|
47 |
-
is
|
48 |
-
MM-
|
49 |
-
|
50 |
-
MM-
|
51 |
-
|
|
|
|
|
|
|
|
|
|
|
52 |
|
53 |
### Characteristics of MM-UPD Bench
|
54 |
We design MM-UPD Bench to provide a comprehensive evaluation of VLMs across multiple senarios.
|
55 |
-
- **Multiple Senario Evaluation:** We carefully design prompts choices and examine the three senario:
|
56 |
-
|
57 |
-
- **
|
58 |
|
59 |
|
60 |
## About Evaluation Metrics
|
|
|
37 |
|
38 |
## About MM-UPD Bench
|
39 |
### What is MM-UPD Bench?
|
40 |
+
MM-UPD Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Vision Language Models (VLMs) in the Context of Unsolvable Problem Detection (UPD)
|
41 |
+
|
42 |
Our MM-UPD Bench encompasses three benchmarks: MM-AAD, MM-IASD, and MM-IVQD.
|
|
|
|
|
|
|
43 |
|
44 |
+
- **MM-AAD:** Benchmark for Absent Answer Detection (AAD).
|
45 |
+
MM-AAD Bench is a dataset where the correct answer option for each question is removed.
|
46 |
+
MM-AAD tests the model's capability to recognize when the correct answer is absent from the provided choices.
|
47 |
+
|
48 |
+
- **MM-IASD:** Benchmark for Incompatible Answer Set Detection (IASD).
|
49 |
+
MM-IASD Bench is a dataset where the answer set is completely incompatible with the context specified by the question and the image.
|
50 |
+
MM-IASD tests the model's capability to recognize when the answer set is incompatible with the context.
|
51 |
+
|
52 |
+
- **MM-IVQD:** Benchmark for Incompatible Visual Question Detection (IVQD).
|
53 |
+
MM-IVQD Bench is a dataset where the question is incompatible with the image.
|
54 |
+
MM-IVQD evaluates the VLMs' capability to discern when a question and image are irrelevant or inappropriate.
|
55 |
|
56 |
### Characteristics of MM-UPD Bench
|
57 |
We design MM-UPD Bench to provide a comprehensive evaluation of VLMs across multiple senarios.
|
58 |
+
- **Multiple Senario Evaluation:** We carefully design prompts choices and examine the three senario:
|
59 |
+
(i) Base (w/o instruction), (ii) Option (w/ additional option), (iii) Instruction (w/ additional instruction).
|
60 |
+
- **Ability-Wise Evaluation:** We carefully decompose each benchmark into various abilities to reveal individual model's strengths and weaknesses.
|
61 |
|
62 |
|
63 |
## About Evaluation Metrics
|