# Train

## Environment

```bash
cd scripts
python -m venv venv
source venv/bin/activate
pip install -U -r requirements.in
```

## Tokenizer

```bash
python -B train_tokenizer.py
```

## Dataset

```bash
python -B prepare_pretrain_dataset.py
```

```python
from litdata import StreamingDataset, StreamingDataLoader, TokensLoader

dataset = StreamingDataset(
  input_dir='../pretrain-data/',
  item_loader=TokensLoader(block_size=2048 + 1),
)

print(len(dataset))
```

## Model

### Pretrain

```bash
litgpt pretrain --config ./pretrain-model.yaml
```

```bash
litgpt convert_from_litgpt out/pretrain/final/ out/converted_model
cp config.json out/pretrain/final/
cp config.json out/converted_model/
```

```python
import torch
from safetensors.torch import save_file

state_dict = torch.load('out/converted_model/model.pth', map_location='cpu')
save_file(state_dict, 'out/converted_model/model.safetensors')
```

## Evaluate

```bash
litgpt evaluate --tasks 'hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge' --out_dir 'evaluate-quick/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

|                 Tasks                 |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|---------------------------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|arc_challenge                          |      1|none            |     0|acc        |↑  |0.1962|±  |0.0116|
|                                       |       |none            |     0|acc_norm   |↑  |0.2304|±  |0.0123|
|gsm8k                                  |      3|flexible-extract|     5|exact_match|↑  |0.0144|±  |0.0033|
|                                       |       |strict-match    |     5|exact_match|↑  |0.0015|±  |0.0011|
|hellaswag                              |      1|none            |     0|acc        |↑  |0.2631|±  |0.0044|
|                                       |       |none            |     0|acc_norm   |↑  |0.2758|±  |0.0045|
|mmlu                                   |      2|none            |      |acc        |↑  |0.2473|±  |0.0036|
| - humanities                          |      2|none            |      |acc        |↑  |0.2351|±  |0.0062|
|  - formal_logic                       |      1|none            |     0|acc        |↑  |0.2857|±  |0.0404|
|  - high_school_european_history       |      1|none            |     0|acc        |↑  |0.2667|±  |0.0345|
|  - high_school_us_history             |      1|none            |     0|acc        |↑  |0.2696|±  |0.0311|
|  - high_school_world_history          |      1|none            |     0|acc        |↑  |0.2110|±  |0.0266|
|  - international_law                  |      1|none            |     0|acc        |↑  |0.1653|±  |0.0339|
|  - jurisprudence                      |      1|none            |     0|acc        |↑  |0.2870|±  |0.0437|
|  - logical_fallacies                  |      1|none            |     0|acc        |↑  |0.2331|±  |0.0332|
|  - moral_disputes                     |      1|none            |     0|acc        |↑  |0.2283|±  |0.0226|
|  - moral_scenarios                    |      1|none            |     0|acc        |↑  |0.2425|±  |0.0143|
|  - philosophy                         |      1|none            |     0|acc        |↑  |0.2186|±  |0.0235|
|  - prehistory                         |      1|none            |     0|acc        |↑  |0.2099|±  |0.0227|
|  - professional_law                   |      1|none            |     0|acc        |↑  |0.2314|±  |0.0108|
|  - world_religions                    |      1|none            |     0|acc        |↑  |0.2632|±  |0.0338|
| - other                               |      2|none            |      |acc        |↑  |0.2485|±  |0.0078|
|  - business_ethics                    |      1|none            |     0|acc        |↑  |0.2600|±  |0.0441|
|  - clinical_knowledge                 |      1|none            |     0|acc        |↑  |0.2528|±  |0.0267|
|  - college_medicine                   |      1|none            |     0|acc        |↑  |0.2254|±  |0.0319|
|  - global_facts                       |      1|none            |     0|acc        |↑  |0.2700|±  |0.0446|
|  - human_aging                        |      1|none            |     0|acc        |↑  |0.2377|±  |0.0286|
|  - management                         |      1|none            |     0|acc        |↑  |0.2816|±  |0.0445|
|  - marketing                          |      1|none            |     0|acc        |↑  |0.2692|±  |0.0291|
|  - medical_genetics                   |      1|none            |     0|acc        |↑  |0.2600|±  |0.0441|
|  - miscellaneous                      |      1|none            |     0|acc        |↑  |0.2350|±  |0.0152|
|  - nutrition                          |      1|none            |     0|acc        |↑  |0.2549|±  |0.0250|
|  - professional_accounting            |      1|none            |     0|acc        |↑  |0.2801|±  |0.0268|
|  - professional_medicine              |      1|none            |     0|acc        |↑  |0.2610|±  |0.0267|
|  - virology                           |      1|none            |     0|acc        |↑  |0.1807|±  |0.0300|
| - social sciences                     |      2|none            |      |acc        |↑  |0.2658|±  |0.0080|
|  - econometrics                       |      1|none            |     0|acc        |↑  |0.1930|±  |0.0371|
|  - high_school_geography              |      1|none            |     0|acc        |↑  |0.2172|±  |0.0294|
|  - high_school_government_and_politics|      1|none            |     0|acc        |↑  |0.3212|±  |0.0337|
|  - high_school_macroeconomics         |      1|none            |     0|acc        |↑  |0.2923|±  |0.0231|
|  - high_school_microeconomics         |      1|none            |     0|acc        |↑  |0.3025|±  |0.0298|
|  - high_school_psychology             |      1|none            |     0|acc        |↑  |0.2752|±  |0.0191|
|  - human_sexuality                    |      1|none            |     0|acc        |↑  |0.2290|±  |0.0369|
|  - professional_psychology            |      1|none            |     0|acc        |↑  |0.2386|±  |0.0172|
|  - public_relations                   |      1|none            |     0|acc        |↑  |0.2636|±  |0.0422|
|  - security_studies                   |      1|none            |     0|acc        |↑  |0.3143|±  |0.0297|
|  - sociology                          |      1|none            |     0|acc        |↑  |0.2338|±  |0.0299|
|  - us_foreign_policy                  |      1|none            |     0|acc        |↑  |0.2600|±  |0.0441|
| - stem                                |      2|none            |      |acc        |↑  |0.2464|±  |0.0077|
|  - abstract_algebra                   |      1|none            |     0|acc        |↑  |0.2500|±  |0.0435|
|  - anatomy                            |      1|none            |     0|acc        |↑  |0.2148|±  |0.0355|
|  - astronomy                          |      1|none            |     0|acc        |↑  |0.1908|±  |0.0320|
|  - college_biology                    |      1|none            |     0|acc        |↑  |0.2569|±  |0.0365|
|  - college_chemistry                  |      1|none            |     0|acc        |↑  |0.2700|±  |0.0446|
|  - college_computer_science           |      1|none            |     0|acc        |↑  |0.3500|±  |0.0479|
|  - college_mathematics                |      1|none            |     0|acc        |↑  |0.2700|±  |0.0446|
|  - college_physics                    |      1|none            |     0|acc        |↑  |0.2745|±  |0.0444|
|  - computer_security                  |      1|none            |     0|acc        |↑  |0.3000|±  |0.0461|
|  - conceptual_physics                 |      1|none            |     0|acc        |↑  |0.2766|±  |0.0292|
|  - electrical_engineering             |      1|none            |     0|acc        |↑  |0.2345|±  |0.0353|
|  - elementary_mathematics             |      1|none            |     0|acc        |↑  |0.2566|±  |0.0225|
|  - high_school_biology                |      1|none            |     0|acc        |↑  |0.2226|±  |0.0237|
|  - high_school_chemistry              |      1|none            |     0|acc        |↑  |0.2217|±  |0.0292|
|  - high_school_computer_science       |      1|none            |     0|acc        |↑  |0.2000|±  |0.0402|
|  - high_school_mathematics            |      1|none            |     0|acc        |↑  |0.2370|±  |0.0259|
|  - high_school_physics                |      1|none            |     0|acc        |↑  |0.2517|±  |0.0354|
|  - high_school_statistics             |      1|none            |     0|acc        |↑  |0.2685|±  |0.0302|
|  - machine_learning                   |      1|none            |     0|acc        |↑  |0.1786|±  |0.0364|
|truthfulqa_mc2                         |      2|none            |     0|acc        |↑  |0.4668|±  |0.0161|
|winogrande                             |      1|none            |     0|acc        |↑  |0.5012|±  |0.0141|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.2473|±  |0.0036|
| - humanities     |      2|none  |      |acc   |↑  |0.2351|±  |0.0062|
| - other          |      2|none  |      |acc   |↑  |0.2485|±  |0.0078|
| - social sciences|      2|none  |      |acc   |↑  |0.2658|±  |0.0080|
| - stem           |      2|none  |      |acc   |↑  |0.2464|±  |0.0077|

```bash
litgpt evaluate --tasks 'leaderboard' --out_dir 'evaluate-leaderboard/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'bbh_zeroshot,bbh_fewshot,bbh_cot_fewshot,bbh_cot_zeroshot' --out_dir 'evaluate-bigbenchhard/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'mmlu,mmlu_pro' --out_dir 'evaluate-mmlu/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'arc_challenge,boolq,gpqa,hellaswag,openbookqa,piqa,truthfulqa_mc2,winogrande' --out_dir 'evaluate-reasoning/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'mmlu_multilingual,mgsm' --out_dir 'evaluate-multilinguals/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'gsm8k,mathqa' --out_dir 'evaluate-math/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'qasper' --out_dir 'evaluate-long/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```