AtsuMiyai commited on
Commit
4d0571a
1 Parent(s): 5b772ef

update explanations on MM-UPD Bench

Browse files
Files changed (1) hide show
  1. constants.py +16 -13
constants.py CHANGED
@@ -37,24 +37,27 @@ LEADERBORAD_INTRODUCTION = """
37
 
38
  ## About MM-UPD Bench
39
  ### What is MM-UPD Bench?
40
- MM-UPD Bench is a comprehensive benchmark for evaluating the trustworthiness of Vision Language Models (VLMs) in the context of Unsolvable Problem Detection (UPD).
 
41
  Our MM-UPD Bench encompasses three benchmarks: MM-AAD, MM-IASD, and MM-IVQD.
42
- - **MM-AAD:** Benchmark for Absent Answer Detection (AAD). MM-AAD Bench is a dataset where the correct answer
43
- option for each question is removed. MM-AAD tests the model's capability
44
- to recognize when the correct answer is absent from the provided choices.
45
 
46
- - **MM-IASD:** Benchmark for Incompatible Answer Set Detection (IASD). MM-IASD Bench is a dataset where the answer set
47
- is completely incompatible with the context specified by the question and the image.
48
- MM-IASD tests the model's capability to recognize when the answer set is incompatible with the context.
49
- - **MM-IVQD:** Benchmark for Incompatible Visual Question Detection (IVQD). MM-IVQD Bench is a dataset where the question is incompatible with the image.
50
- MM-IVQD evaluates the VLMs' capability to discern when a question and image are irrelevant or
51
- inappropriate.
 
 
 
 
 
52
 
53
  ### Characteristics of MM-UPD Bench
54
  We design MM-UPD Bench to provide a comprehensive evaluation of VLMs across multiple senarios.
55
- - **Multiple Senario Evaluation:** We carefully design prompts choices and examine the three senario: (i) base (no instruction), (ii) option (add an additional option), (iii) instruction (add an instruction).
56
- - **Ability-Wise Evaluation:** We carefully decompose each benchmark into more than 10 abilities to reveal individual model's strengths and weaknesses.
57
- - **Valuable Insights:** MM-UPD Bench provides multi-perspective insights on trustworthiness and reliablitity for the community.
58
 
59
 
60
  ## About Evaluation Metrics
 
37
 
38
  ## About MM-UPD Bench
39
  ### What is MM-UPD Bench?
40
+ MM-UPD Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Vision Language Models (VLMs) in the Context of Unsolvable Problem Detection (UPD)
41
+
42
  Our MM-UPD Bench encompasses three benchmarks: MM-AAD, MM-IASD, and MM-IVQD.
 
 
 
43
 
44
+ - **MM-AAD:** Benchmark for Absent Answer Detection (AAD).
45
+ MM-AAD Bench is a dataset where the correct answer option for each question is removed.
46
+ MM-AAD tests the model's capability to recognize when the correct answer is absent from the provided choices.
47
+
48
+ - **MM-IASD:** Benchmark for Incompatible Answer Set Detection (IASD).
49
+ MM-IASD Bench is a dataset where the answer set is completely incompatible with the context specified by the question and the image.
50
+ MM-IASD tests the model's capability to recognize when the answer set is incompatible with the context.
51
+
52
+ - **MM-IVQD:** Benchmark for Incompatible Visual Question Detection (IVQD).
53
+ MM-IVQD Bench is a dataset where the question is incompatible with the image.
54
+ MM-IVQD evaluates the VLMs' capability to discern when a question and image are irrelevant or inappropriate.
55
 
56
  ### Characteristics of MM-UPD Bench
57
  We design MM-UPD Bench to provide a comprehensive evaluation of VLMs across multiple senarios.
58
+ - **Multiple Senario Evaluation:** We carefully design prompts choices and examine the three senario:
59
+ (i) Base (w/o instruction), (ii) Option (w/ additional option), (iii) Instruction (w/ additional instruction).
60
+ - **Ability-Wise Evaluation:** We carefully decompose each benchmark into various abilities to reveal individual model's strengths and weaknesses.
61
 
62
 
63
  ## About Evaluation Metrics