Data-Juicer

Interaction

Data-Juicer is a one-stop data processing system to make data higher-quality, juicier, and more digestible for (multimodal) LLMs!

Data-Juicer 是一个一站式数据处理系统，为大模型提供更高质量、更丰富、更易”消化“的数据！

News

• [2024-07-17] We utilized the Data-Juicer Sandbox Laboratory Suite to systematically optimize data and models through an co-development workflow between data and models, achieving a new top spot on the VBench text-to-video leaderboard. The related achievements have been compiled and published in a paper, and the model has been released on the ModelScope and HuggingFace platforms.
• [2024-07-12] Our awesome list of MLLM-Data has evolved into a systemic survey from model-data co-development perspective. Welcome to explore and contribute!
• [2024-06-01] ModelScope-Sora "Data Directors" creative sprint—Our third data-centric LLM competition has kicked off! Please visit the competition's official website for more information.
• [2024-03-07] We release Data-Juicer v0.2.0 now! In this new version, we support more features for multimodal data (including video now), and introduce DJ-SORA to provide open large-scale, high-quality datasets for SORA-like models.
• [2024-02-20] We have actively maintained an awesome list of LLM-Data, welcome to visit and contribute!
• [2024-02-05] Our paper has been accepted by SIGMOD'24 industrial track!
• [2024-01-10] Discover new horizons in "Data Mixture"—Our second data-centric LLM competition has kicked off! Please visit the competition's official website for more information.
• [2024-01-05] We release Data-Juicer v0.1.3 now! In this new version, we support more Python versions (3.8-3.10), and support multimodal dataset converting/processing (Including texts, images, and audios. More modalities will be supported in the future). Besides, our paper is also updated to v3.