docs/description.md

# LIBRA: Long Input Benchmark for Russian Analysis

<img src="https://i.imgur.com/BNleRrG.png" width="800" />

## Dataset Summary

LIBRA (Long Input Benchmark for Russian Analysis) is designed to evaluate the capabilities of large language models (LLMs) in understanding and processing long texts in Russian. This benchmark includes 21 datasets adapted for different tasks and complexities. The tasks are divided into four complexity groups and allow evaluation across various context lengths ranging from 4,000 up to 128,000 tokens.

## Tasks and Complexity Groups

### Group I: Simple Information Retrieval
- **Passkey**: Extract a relevant piece of code number from a long text fragment.
- **PasskeyWithLibrusec**: Similar to Passkey but with added noise from Librusec texts.

### Group II: Question Answering and Multiple Choice
- **MatreshkaNames**: Identify the person in dialogues based on the discussed topic.
- **MatreshkaYesNo**: Indicate whether a specific topic was mentioned in the dialog.
- **LibrusecHistory**: Answer questions based on historical texts.
- **ruTREC**: Few-shot in-context learning for topic classification. Created by translating the TREC dataset from LongBench.
- **ruSciFi**: Answer true/false based on context and general world knowledge. Translation of SciFi dataset from L-Eval.
- **ruSciAbstractRetrieval**: Retrieve relevant paragraphs from scientific abstracts.
- **ruTPO**: Multiple-choice questions similar to TOEFL exams. Translation of the TPO dataset from L-Eval.
- **ruQuALITY**: Multiple-choice QA tasks based on detailed texts. Created by translating the QuALITY dataset from L-Eval.

### Group III: Multi-hop Question Answering
- **ruBABILongQA**: 5 long-context reasoning tasks for QA using facts hidden among irrelevant information.
- **LongContextMultiQ**: Multi-hop QA based on Wikidata and Wikipedia.
- **LibrusecMHQA**: Multi-hop QA requiring information distributed across several text parts.
- **ru2WikiMultihopQA**: Translation of the 2WikiMultihopQA dataset from LongBench.

### Group IV: Complex Reasoning and Mathematical Problems
- **ruSciPassageCount**: Count unique paragraphs in a long text.
- **ruQasper**: Question Answering over academic research papers. Created by translating the Qasper dataset from LongBench.
- **ruGSM100**: Solve math problems using Chain-of-Thought reasoning.

## Dataset Structure

The datasets are divided into subsets based on context lengths: 4k, 8k, 16k, 32k, 64k, and 128k tokens. Each subset contains a different number of samples depending on the task complexity.

## Usage

The LIBRA benchmark is available under the MIT license. Researchers and developers can use these datasets to evaluate the long-context understanding abilities of various LLMs. The datasets, codebase, and public leaderboard are open-source to guide forthcoming research in this area.

## Citation

_TODO_

@article{LIBRA2024,
title={Long Input Benchmark for Russian Analysis},
author={Anonymous},
journal={ACL},
year={2024}
}

## License

The datasets are published under the MIT license.

## Acknowledgments

_TODO_

For more details and code, please visit our [GitHub repository](#).