Create README.md

README.md

@@ -0,0 +1,56 @@
+---
+license: mit
+datasets:
+- kenhktsui/llm-data-quality
+---
+# llm-data-textbook-quality-fasttext-classifer-v1
+Model is built on fasttext. It is an optimisation of llm-data-textbook-quality-classifer-v1.  
+It can classify more than 2000 examples per second in CPU.  
+This model can classify if a text is of textbook quality data. It can be used as a filter for data curation when training a LLM.  
+Please note textbook quality is a subset of high quality.  
+
+
+## Model Performance
+|Dataset | F1 Score |
+|-------|-------|
+|Train | 0.8695|
+|Test | 0.8485|
+
+
+# Usage
+```python
+from typing import List
+import re
+import fasttext
+
+
+model = fasttext.load_model("model_textbook_quality.bin")
+
+
+def replace_newlines(text):
+  return re.sub("\n+", " ", text)
+
+
+def predict(text_list: List[str]):
+  text_list = [replace_newlines(text) for text in text_list]
+  pred = model.predict(text_list)
+  return [{"label": l[0].lstrip("__label__"), "score": s[0]} for l, s in zip(*pred)]
+
+
+predict(["Hi"])
+# Output: {'label': 'LOW_QUALITY', 'score': 1.00001}
+
+```
+
+## Benchmark
+
+|Dataset | Sampling | Average Quality Score |
+|--------------------------------------|---|-------------------|
+|[nampdn-ai/tiny-orca-textbooks](https://huggingface.co/datasets/nampdn-ai/tiny-orca-textbooks) |First 10,000| 0.8356|
+|[nampdn-ai/tiny-textbooks](https://huggingface.co/datasets/nampdn-ai/tiny-textbooks) |First 10,000| 0.7488|
+|[SciPhi/textbooks-are-all-you-need-lite](https://huggingface.co/datasets/SciPhi/textbooks-are-all-you-need-lite) |First 10,000| 0.7182|
+|[vikp/textbook_quality_programming](https://huggingface.co/datasets/vikp/textbook_quality_programming) |First 10,000| 0.5410|
+|[BEE-spoke-data/fineweb-100k_en-med](https://huggingface.co/datasets/BEE-spoke-data/fineweb-100k_en-med)| First 10,000| 0.4760|
+|[pszemraj/simple_wikipedia_LM](https://huggingface.co/datasets/pszemraj/simple_wikipedia_LM) | First 10,000| 0.4670|
+|[mattymchen/refinedweb-3m](https://huggingface.co/datasets/mattymchen/refinedweb-3m)| First 10,000| 0.2916|
+|[JeanKaddour/minipile](https://huggingface.co/datasets/JeanKaddour/minipile)| First 10,000 | 0.2525|