File size: 1,876 Bytes
13f81d0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
---
library_name: transformers
datasets:
- WebOrganizer/Corpus-200B
---
# WebOrganizer/LM-1b_1x-Sampling_over_Formats_for_HellaSwag
[[Paper](https://arxiv.org/abs/2502.10341)] [[Website](https://weborganizer.allenai.org)] [[GitHub](https://github.com/CodeCreator/WebOrganizer)]
A 1.4B parameter model trained for 29B tokens from [WebOrganizer/Corpus-200B](https://huggingface.co/datasets/WebOrganizer/Corpus-200B).
The training data for this model was selected via:
1. **Selection method**: Random sampling
2. **Domain definition**: 24 [WebOrganizer Formats](https://huggingface.co/WebOrganizer/FormatClassifier)
3. **Domain mixture**: n/a
## Repository Contents
Besides the HuggingFace model and tokenizer, the repository contains:
- `open_lm/`: Contains the OpenLM config and final checkpoint
- `evals/`: Evaluation results for various benchmarks
- `core_9mcqa/`: Results of 9 multiple choice QA tasks with the OLMES evaluation framework
- `mmlu/`: MMLU results with the OLMES evaluation framework
- `dclm/`: Results using the DCLM evaluation framework
- `perplexity/`: Perplexity results using the huggingface trainer
- `indices.tar.zst`: The indices for the selected documents in each shard of the Corpus-200B dataset used for training. The indices can be extracted with `tar --use-compress-program "zstd" -xf indices.tar.zst`.
## Usage
To use this model, you need to install the [open_lm](https://github.com/mlfoundations/open_lm) library and add `from open_lm.hf import *` before loading the model with `AutoModel.from_pretrained(...)`.
## Citation
```bibtex
@article{wettig2025organize,
title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},
author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},
journal={arXiv preprint arXiv:2502.10341},
year={2025}
}
```
|