update README

.gitattributes

@@ -34,3 +34,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/costs-1.5b.png filter=lfs diff=lfs merge=lfs -text
+assets/costs-7b.png filter=lfs diff=lfs merge=lfs -text
+assets/performances.png filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Knovel Engineering
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,56 @@
+---
+pipeline_tag: text-generation
+inference: true
+license: mit
+datasets:
+  - knoveleng/open-rs
+  - knoveleng/open-s1
+  - knoveleng/open-deepscaler
+base_model:
+  - deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
+---
+
+# Model Summary
+
+This repository hosts model for the **Open RS** project, accompanying the paper *Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn’t*. The project explores enhancing reasoning capabilities in small large language models (LLMs) using reinforcement learning (RL) under resource-constrained conditions.
+
+We focus on a 1.5-billion-parameter model, `DeepSeek-R1-Distill-Qwen-1.5B`, trained on 4 NVIDIA A40 GPUs (48 GB VRAM each) within 24 hours. By adapting the Group Relative Policy Optimization (GRPO) algorithm and leveraging a curated, compact mathematical reasoning dataset, we conducted three experiments to assess performance and behavior. Key findings include:
+
+- Significant reasoning improvements, e.g., AMC23 accuracy rising from 63% to 80% and AIME24 reaching 46.7%, outperforming `o1-preview`.
+- Efficient training with just 7,000 samples at a cost of $42, compared to thousands of dollars for baseline models.
+- Challenges like optimization instability and length constraints with extended training.
+
+These results showcase RL-based fine-tuning as a cost-effective approach for small LLMs, making reasoning capabilities accessible in resource-limited settings. We open-source our code, models, and datasets to support further research.
+
+For more details, please refer our [github](https://github.com/knoveleng/open-rs).
+
+## Evaluation
+### Performance Highlights
+- **Open-RS1**: 53.0% avg. score
+- **Open-RS2**: 55.7% avg. score, 80.0% on AMC23
+- **Open-RS3**: 56.3% avg. score, 46.7% on AIME24 (outperforms `o1-preview` at 44.6%)
+- Competitive MATH-500 scores; Minerva lags behind 7B models.
+
+![Performance Metrics](assets/performances.png)
+
+### Cost Efficiency
+Our approach uses 7,000 samples (42,000 total outputs) and costs ~$42 on 4x A40 GPUs in 24 hours, compared to:
+- 7B models: `Qwen2.5-7B-SimpleRL` ($1,633), `Eurus-2-7B-PRIME` ($1,088)
+- 1.5B models: `DeepScaleR-1.5B-Preview` ($3,629), `Still-3-1.5B-Preview` ($2,268)
+
+![7B Model Costs](assets/costs-7b.png)  
+![1.5B Model Costs](assets/costs-1.5b.png)
+
+
+## Citation
+If this project aids your work, please cite it as:
+```
+@misc{open-rs,
+  title = {Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't},
+  url = {https://github.com/knoveleng/open-rs},
+  author = {Quy-Anh Dang, Chris Ngo},
+  month = {March},
+  year = {2025}
+}
+```
+