|
--- |
|
license: mit |
|
--- |
|
Fasttext model used for filtering in [DataComp-LM](https://arxiv.org/abs/2406.11794) to produce [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0). |
|
|
|
The model classifies between `__label__hq` and `__label__cc` which correspond to "high-quality" (i.e., OH2.5 and Reddit ELI5 data) and "low-quality" (i.e., web-crawled data from Common Crawl) respectively. We use the score given to `__label__hq` to filter our documents via a percentile-based threshold. |
|
|
|
See our [dclm](https://github.com/mlfoundations/dclm/tree/main/baselines#fasttext-filtering) repo for documentation about how we applied to to filter data in our experiments. |
|
|
|
See [fasttext documentation](https://fasttext.cc/docs/en/python-module.html) for general documentation on fasttext classifiers and how to use them with python. |
|
|
