docs/description.md

LIBRA: Long Input Benchmark for Russian Analysis

Dataset Summary

LIBRA (Long Input Benchmark for Russian Analysis) is designed to evaluate the capabilities of large language models (LLMs) in understanding and processing long texts in Russian. This benchmark includes 21 datasets adapted for different tasks and complexities. The tasks are divided into four complexity groups and allow evaluation across various context lengths ranging from 4k up to 128k tokens.

Tasks and Complexity Groups

Group I: Simple Information Retrieval

• Passkey: Extract a relevant piece of code number from a long text fragment. Based on the original PassKey test from the m LongLLaMA’s GitHub repo.
• PasskeyWithLibrusec: Similar to Passkey but with added noise from Librusec texts.

Group II: Question Answering and Multiple Choice

• MatreshkaNames: Identify the person in dialogues based on the discussed topic. We used Matreshka dataset and Russian Names dataset to create this and the next task.
• MatreshkaYesNo: Indicate whether a specific topic was mentioned in the dialog.
• LibrusecHistory: Answer questions based on historical texts. Ideologically similiar to the PassageRetrieval dataset from LongBench.
• ruTREC: Few-shot in-context learning for topic classification. Created by translating the TREC dataset from LongBench.
• ruSciFi: Answer true/false based on context and general world knowledge. Translation of SciFi dataset from L-Eval which originally was based on SF-Gram.
• ruSciAbstractRetrieval: Retrieve relevant paragraphs from scientific abstracts.
• ruTPO: Multiple-choice questions similar to TOEFL exams. Translation of the TPO dataset from L-Eval.
• ruQuALITY: Multiple-choice QA tasks based on detailed texts. Created by translating the QuALITY dataset from L-Eval.

Group III: Multi-hop Question Answering

• ruBABILongQA: 5 long-context reasoning tasks for QA using facts hidden among irrelevant information.
• LongContextMultiQ: Multi-hop QA based on Wikidata and Wikipedia.
• LibrusecMHQA: Multi-hop QA requiring information distributed across several text parts.
• ru2WikiMultihopQA: Translation of the 2WikiMultihopQA dataset from LongBench.

Group IV: Complex Reasoning and Mathematical Problems

• ruSciPassageCount: Count unique paragraphs in a long text. Uses the basic idea of the original PassageCount dataset from LongBench.
• ruQasper: Question Answering over academic research papers. Created by translating the Qasper dataset from LongBench.
• ruGSM100: Solve math problems using Chain-of-Thought reasoning. Created by translating the GSM100 dataset from L-Eval.

Dataset Structure

The datasets are divided into subsets based on context lengths: 4k, 8k, 16k, 32k, 64k, and 128k tokens. Each subset contains a different number of samples depending on the task complexity.

Add your model

For placing your model to leaderboard you need to:

1. Score the model on our repository.
2. Add the result in the format "<model_name>.json" in the "results" folder.
3. Create a Pull Request.

*GPT-4o

Because of limited resources, we assessed GPT-4o on just 10% of each dataset in our benchmark, including each context length. Consequently, the results might not be exact.

Citation

TODO

@article{LIBRA2024,
title={Long Input Benchmark for Russian Analysis},
author={Anonymous},
journal={ACL},
year={2024}
}

License

The datasets are published under the MIT license.

Acknowledgments

For more details and code, please visit our GitHub repository.