Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# MuMath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models
|
2 |
+
|
3 |
+
## Introduction
|
4 |
+
|
5 |
+
We have amalgamated and further refined these strengths while broadening the scope of augmentation methods to construct a multi-perspective augmentation dataset for mathematics—termed [MuMath (μ-Math) Dataset](https://huggingface.co/datasets/weihao1/MuMath). Subsequently, we finetune LLaMA-2 on the MuMath dataset to derive the MuMath model.
|
6 |
+
|
7 |
+
|
8 |
+
| Model | Size | GSM8k | MATH |
|
9 |
+
|---|---|---|---|
|
10 |
+
| WizardMath-7B | 7B | 54.9 | 10.7 |
|
11 |
+
| MetaMath-7B | 7B | 66.3 | 19.7 |
|
12 |
+
| MuggleMath-7B | 7B | 68.4 | - |
|
13 |
+
| [MuMath-7B](https://huggingface.co/weihao1/MuMath-7B) | 7B | **79.1** | **30.0** |
|
14 |
+
||
|
15 |
+
| WizardMath-13B | 13B | 63.9 | 14 |
|
16 |
+
| MetaMath-13B | 13B | 72.3 | 22.4 |
|
17 |
+
| MuggleMath-13B | 13B | 74 | - |s
|
18 |
+
| [MuMath-13B](https://huggingface.co/weihao1/MuMath-13B) | 13B | **83.6** | **33.3** |
|
19 |
+
||
|
20 |
+
| WizardMath-70B | 70B | 81.6 | 22.7 |
|
21 |
+
| MetaMath-70B | 70B | 82.3 | 26.6 |
|
22 |
+
| MuggleMath-70B | 70B | 82.3 | - |
|
23 |
+
| [MuMath-70B](https://huggingface.co/weihao1/MuMath-70B) | 70B | **88.5** | **41.2** |
|
24 |
+
|
25 |
+
> The best results are bolded.
|
26 |
+
|
27 |
+
|
28 |
+
## Augmentation Methods
|
29 |
+
<p align="center">
|
30 |
+
<img src="./images/MuMath.png" width="800">
|
31 |
+
<br>
|
32 |
+
<em>Overview of the augmentation methods our MuMath employs, which can be divided into four categories: (1) Data Reformulation includes solution reorganization and question rephrasing; (2) Backward Creation includes Backward-Forward Transformation (BF-Trans) and FOBAR; (3) Question Alteration includes expression replacement and difficulty enhancement; (4) Nested Multi-task construction includes data of the auxiliary tasks, i.e., Problem Outline and Solution Plan.</em>
|
33 |
+
</p>
|
34 |
+
|
35 |
+
|
36 |
+
### Setup
|
37 |
+
|
38 |
+
We recommend using [Conda](https://docs.conda.io/projects/miniconda) to manage your environment. We use [vLLM](https://github.com/vllm-project/vllm) to accelerate inference. Run the following commands to setup your environment:
|
39 |
+
|
40 |
+
```sh
|
41 |
+
conda create -n mumath python=3.10
|
42 |
+
conda activate mumath
|
43 |
+
cd MuMath-src
|
44 |
+
pip install -r requirements.txt
|
45 |
+
```
|
46 |
+
|
47 |
+
|
48 |
+
## Training
|
49 |
+
|
50 |
+
We also open [MuMath Dataset](https://huggingface.co/datasets/weihao1/MuMath) for the training stage.
|
51 |
+
|
52 |
+
To train a model, after specifying `MODEL_PATH`, `SAVE_PATH`, `DATA_PATH`, the conda environment and so on, run the following command:
|
53 |
+
|
54 |
+
```sh
|
55 |
+
|
56 |
+
# 7B or 13B
|
57 |
+
bash train_7b.sh
|
58 |
+
|
59 |
+
# 34B
|
60 |
+
bash train_13b_70b.sh
|
61 |
+
|
62 |
+
```
|
63 |
+
|
64 |
+
### Inference and Evaluation
|
65 |
+
|
66 |
+
We provide scripts for inference and evaluation, which are called in `train_7b.sh` and `train_13b_70b.sh` as mentioned above.
|
67 |
+
|
68 |
+
```sh
|
69 |
+
python eval_gsm8k.py --model $SAVE_PATH --data_file ./data/test/GSM8K_test.jsonl
|
70 |
+
python eval_math.py --model $SAVE_PATH --data_path ./data/test/MATH_test.jsonl
|
71 |
+
```
|
72 |
+
|
73 |
+
## Citation
|
74 |
+
Please cite the paper if you refer to our model, code, data or paper from MuMath.
|
75 |
+
|
76 |
+
```
|
77 |
+
@inproceedings{you-etal-2024-mumath,
|
78 |
+
title = "{M}u{M}ath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models",
|
79 |
+
author = "You, Weihao and Yin, Shuo and Zhao, Xudong and Ji, Zhilong and Zhong, Guoqiang and Bai, Jinfeng",
|
80 |
+
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
|
81 |
+
month = jun,
|
82 |
+
year = "2024",
|
83 |
+
pages = "2932--2958",
|
84 |
+
}
|
85 |
+
```
|
86 |
+
|
87 |
+
## Credits
|
88 |
+
|
89 |
+
This project has adopted the [MeteMath](https://github.com/meta-math/MetaMath) and [MuggleMath](https://github.com/OFA-Sys/gsm8k-ScRel).
|
90 |
+
|