Update README.md

README.md

@@ -12,18 +12,20 @@ base_model:
 
 ## Model summary
 
-This model is part of the 📐 [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) ablations, we continue pretraining [Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B) base on different math datasets for 60B tokens.
-The model has 3.21B parameters and 4096 context length. It was trained on **160B tokens** using a mix of 40% [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and 30% FineMath-4+ and 30% InfiWebMath-4+ from the  📐 [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) dataset.
+This is a continual-pre-training of [Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B) on a mix of  📐 [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) (our new high quality math dataset) and [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).
+
+The model demonstrates superior math performance compared to Llama 3.2 3B, while having similar performance on Knowledge, reasoning and Common sense benchmarks:
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/HZ6KOc8IVXXOABrdv0dyK.png)
+
+It was trained on **160B tokens** using a mix of 40% FineWeb-Edu and 30% FineMath-4+ and 30% InfiWebMath-4+ from FineMath. We use [nanotron](https://github.com/huggingface/smollm/tree/main/pre-training/continual-pretraining) for the training. You can find the training scripts in our [SmolLM2 GitHub repo](https://github.com/huggingface/smollm).
 
-- **License**: Apache-2
-- **Languages**: English
 
 ## Use
 
 ### Intended use
 
-This model was trained on English math data and is not instruction-tuned, making it intended for text completion in English with a focus on math.
-It is important to note that the primary intended use case of this model is to compare its performance with other models trained under the same conditions. This model is not necessarily the best possible outcome achievable with the given dataset.
+This model was trained on English math data and is not instruction-tuned, making it intended for text completion in English.
 
 ### Generation
 
@@ -31,7 +33,7 @@ It is important to note that the primary intended use case of this model is to c
 # pip install -q transformers
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model = "HuggingFaceTB/finemath-ablation-4plus-160B"
+model = "HuggingFaceTB/FineMath-Llama-3B"
 device = "cuda" # for GPU usage or "cpu" for CPU usage
 
 tokenizer = AutoTokenizer.from_pretrained(model)
@@ -48,20 +50,20 @@ We are releasing intermediate checkpoints for this model at intervals of every 1
 
 You can load a specific model revision with `transformers` using the argument `revision`:
 ```python
-model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/finemath-ablation-4plus-160B", revision="10B")
+model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/FineMath-Llama-3B", revision="10B")
 ```
 You can access all the revisions for the models via the following code:
 ```python
 from huggingface_hub import list_repo_refs
-out = list_repo_refs("HuggingFaceTB/finemath-ablation-4plus-160B")
+out = list_repo_refs("HuggingFaceTB/FineMath-Llama-3B")
 print([b.name for b in out.branches])
 ```
 
 ## Training
 ### Model
 - **Architecture**: Llama3  
-- **Pretraining steps**: 60k
-- **Pretraining tokens**: 60B
+- **Pretraining steps**: 160k
+- **Pretraining tokens**: 160B
 - **Precision**: bfloat16
 
 ### Hardware