GAIA: a benchmark for General AI Assistants
Paper
•
2311.12983
•
Published
•
185
Benchmarks
-
-
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Paper • 2311.16502 • Published • 35 -
BLINK: Multimodal Large Language Models Can See but Not Perceive
Paper • 2404.12390 • Published • 24 -
RULER: What's the Real Context Size of Your Long-Context Language Models?
Paper • 2404.06654 • Published • 34 -
CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialogues
Paper • 2404.03820 • Published • 24 -
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Paper • 2404.03543 • Published • 15 -
Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings
Paper • 2404.16820 • Published • 15 -
SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
Paper • 2404.16790 • Published • 7 -
On the Planning Abilities of Large Language Models -- A Critical Investigation
Paper • 2305.15771 • Published • 1 -
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
Paper • 2405.21075 • Published • 20 -
Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Paper • 2406.09170 • Published • 24 -
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Paper • 2406.09411 • Published • 18 -
CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery
Paper • 2406.08587 • Published • 15 -
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
Paper • 2406.11833 • Published • 61 -
ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation
Paper • 2406.09961 • Published • 54 -
Needle In A Multimodal Haystack
Paper • 2406.07230 • Published • 53 -
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Paper • 2406.10149 • Published • 48 -
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Paper • 2407.18961 • Published • 39 -
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Paper • 2407.18901 • Published • 32 -
WebArena: A Realistic Web Environment for Building Autonomous Agents
Paper • 2307.13854 • Published • 23 -
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Paper • 2408.03361 • Published • 85 -
SWE-bench-java: A GitHub Issue Resolving Benchmark for Java
Paper • 2408.14354 • Published • 40 -
AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments
Paper • 2405.07960 • Published • 1 -
MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
Paper • 2310.16049 • Published • 4 -
MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines
Paper • 2409.12959 • Published • 36 -
DSBench: How Far Are Data Science Agents to Becoming Data Science Experts?
Paper • 2409.07703 • Published • 66 -
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Paper • 2409.16191 • Published • 41 -
OmniBench: Towards The Future of Universal Omni-Language Models
Paper • 2409.15272 • Published • 26 -
Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
Paper • 2410.07985 • Published • 28 -
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Paper • 2404.07972 • Published • 46