Model	Open Ended VQA: % Human Rating	Multiple Choice VQA: % Accuracy	Hints-Multiple Choice VQA: % Accuracy 	Attributions-Multiple Choice VQA: % Accuracy 	Refernce Based-Automatic Evaluation: Accuracy of Judge Prediction Compared to Human Ratings	Refernce Free-Automatic Evaluation: Accuracy of Judge Prediction Compared to Human Ratings	Automatic Evaluation: % Auto-Rater Ratings	Hints-Automatic Evaluation: % Auto-Rater Ratings	Attributions-Automatic Evaluation: % Auto-Rater Ratings
Humans	82	*	*	*	*	*	78	*	*
Gemini Pro 1.5	40	38	66	72	87	52	53	62	29
Gemini Pro Vision	30	41	62	*	75	38	34	47	
GPT4	34	45	69	82	86	51	38	61	25
LlaVA-1.6-34B	15	24	30	*	76	43	21	16	*
LlaVA-1.5-7B	13	17	29	*	70	35	19	30	*
InstructBlip	13	*	*	*	*	*	20	28	*
Gemini Pro 1.5 Caption _ Gemini Pro 1.5	23	*	*	*	*	*	*	*	*
Human (Oracle) Caption _ Gemini Pro 1.5	50	*	*	*	*	*	*	*	*
Claude 3.5 Sonnet	*	46	45	*	*	*	39	*	*
GPT4o	*	55	83	*	*	*	50	*	*
Qwen-VL-Max	*	35	53	*	*	*	26	*	*
Molmo-7B	*	34	42	*	*	*	36	*	*