ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities
Paper
•
2412.06745
•
Published
•
6
Ameya Prabhu
AmeyaPrabhu
AI & ML interests
None yet
Recent Activity
authored
a paper
9 days ago
ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended
Capabilities
upvoted
a
paper
21 days ago
MALT: Improving Reasoning with Multi-Agent LLM Training
Organizations
AmeyaPrabhu's activity
authored
a
paper
9 days ago
upvoted
a
paper
21 days ago
updated
a
dataset
2 months ago
updated
a
dataset
3 months ago
Update README.md
1
#1 opened 3 months ago
by
oripress
authored
a
paper
5 months ago
Added Contamination Evidence on MMLU of ChatGPT/GPT4 from "Investigating data contamination in modern benchmarks for large language models"
7
#10 opened 8 months ago
by
AmeyaPrabhu
Mistral 7B Arc Easy Contamination based on "Proving Test Set Contamination in Black Box Language Models"
1
#14 opened 8 months ago
by
AmeyaPrabhu
Added Contamination Evidence from GPT4 Tech Report using String matching on GPT-4
9
#11 opened 8 months ago
by
AmeyaPrabhu
Added Contamination Info on Old Models: GPT3, FLAN, GLaM, PaLM, PaLM 2
3
#13 opened 8 months ago
by
AmeyaPrabhu
Code contamination in HumanEval and MBPP
1
#12 opened 8 months ago
by
AmeyaPrabhu
authored
a
paper
9 months ago
updated
a
dataset
9 months ago
authored
3
papers
9 months ago
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
Paper
•
2404.04125
•
Published
•
27
Lifelong Benchmarks: Efficient Model Evaluation in an Era of Rapid Progress
Paper
•
2402.19472
•
Published
•
2
Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right?
Paper
•
2305.09275
•
Published
•
1
reacted to
akhaliq's
post with 🚀
9 months ago
Post
2619
No "Zero-Shot" Without Exponential Data
Pretraining Concept Frequency Determines Multimodal Model Performance
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (2404.04125)
Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by the frequency of these concepts in their pretraining datasets? We comprehensively investigate this question across 34 models and five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics), generating over 300GB of data artifacts. We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly. We contribute this long-tail test set as the "Let it Wag!" benchmark to further research in this direction. Taken together, our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.
Pretraining Concept Frequency Determines Multimodal Model Performance
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (2404.04125)
Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by the frequency of these concepts in their pretraining datasets? We comprehensively investigate this question across 34 models and five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics), generating over 300GB of data artifacts. We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly. We contribute this long-tail test set as the "Let it Wag!" benchmark to further research in this direction. Taken together, our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.
updated
a
dataset
10 months ago