SmolLM - blazingly fast and remarkably powerful
•
292
Loubna Ben Allal
loubnabnl
AI & ML interests
SmolLMs, ML for code, data
Recent Activity
new activity
1 day ago
HuggingFaceTB/finemath:Why did you use CC rather than FineWeb to create FineMath?
updated
a dataset
3 days ago
loubnabnl/mmlu-evals-smollm-360m
updated
a dataset
3 days ago
loubnabnl/code_data
Articles
Organizations
Posts
4
Post
1632
Making SmolLM2 reproducible: open-sourcing our training & evaluation toolkit 🛠️ https://github.com/huggingface/smollm/
- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents
Apache 2.0 licensed. V2 pre-training data mix coming soon!
Which other tools should we add next?
- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents
Apache 2.0 licensed. V2 pre-training data mix coming soon!
Which other tools should we add next?
Post
5327
🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.
Technical report: HuggingFaceFW/blogpost-fineweb-v1
Dataset: HuggingFaceFW/fineweb-edu
We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens.
You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
Enjoy!
Technical report: HuggingFaceFW/blogpost-fineweb-v1
Dataset: HuggingFaceFW/fineweb-edu
We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens.
You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
Enjoy!
spaces
7
models
32
loubnabnl/test
Updated
loubnabnl/smollm-135M-instruct-v2
Updated
•
9
loubnabnl/smollm-1.7B-instruct-v2
Updated
•
11
loubnabnl/smollm-360M-instruct-add-basics-w-math
Updated
•
7
loubnabnl/smollm-135M-instruct-add-basics-w-math
Updated
•
8
loubnabnl/smollm-350M-instruct-add-basics-eq
Updated
•
12
loubnabnl/smollm-350M-instruct-add-basics-only
Updated
•
11
loubnabnl/SmolLM-360M-4bit
Text Generation
•
Updated
•
7
loubnabnl/smollm-350M-instruct-test2-noOH
Updated
•
7
loubnabnl/smollm-350M-instruct-test2-ep2
Updated
•
11
datasets
98
loubnabnl/mmlu-evals-smollm-360m
Viewer
•
Updated
•
1
•
11
loubnabnl/code_data
Viewer
•
Updated
•
1k
•
38
loubnabnl/english-web-100k
Viewer
•
Updated
•
100k
•
14
loubnabnl/generations_dataset_sysprompt
Viewer
•
Updated
•
41
•
51
loubnabnl/gens-360M-temp7-v2
Viewer
•
Updated
•
41
•
34
loubnabnl/gens-360M-v2
Viewer
•
Updated
•
41
•
37
loubnabnl/gens-135M-v2
Viewer
•
Updated
•
41
•
44
loubnabnl/generations_dataset
Viewer
•
Updated
•
40
•
35
loubnabnl/example-generations
Viewer
•
Updated
•
547
•
14
loubnabnl/bisac_subset
Viewer
•
Updated
•
150
•
35