scripts/TRAIN.md

# Train

## Environment

```bash
cd scripts
python -m venv venv
source venv/bin/activate
pip install -U -r requirements.in
```

## Tokenizer

```bash
python -B train_tokenizer.py
```

## Dataset

```bash
python -B prepare_pretrain_dataset.py
```

```python
from litdata import StreamingDataset, StreamingDataLoader, TokensLoader

dataset = StreamingDataset(
 input_dir='../pretrain-data/',
 item_loader=TokensLoader(block_size=2048 + 1),
)

print(len(dataset))
```

## Model

### Pretrain

```bash
litgpt pretrain --config ./pretrain-model.yaml
```

```bash
litgpt convert_from_litgpt out/pretrain/final/ out/converted_model
cp config.json out/pretrain/final/
cp config.json out/converted_model/
```

```python
import torch
from safetensors.torch import save_file

state_dict = torch.load('out/converted_model/model.pth', map_location='cpu')
save_file(state_dict, 'out/converted_model/model.safetensors')
```

## Evaluate

```bash
litgpt evaluate --tasks 'hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge' --out_dir 'evaluate-quick/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------------------------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|arc_challenge | 1|none | 0|acc |↑ |0.1962|± |0.0116|
| | |none | 0|acc_norm |↑ |0.2304|± |0.0123|
|gsm8k | 3|flexible-extract| 5|exact_match|↑ |0.0144|± |0.0033|
| | |strict-match | 5|exact_match|↑ |0.0015|± |0.0011|
|hellaswag | 1|none | 0|acc |↑ |0.2631|± |0.0044|
| | |none | 0|acc_norm |↑ |0.2758|± |0.0045|
|mmlu | 2|none | |acc |↑ |0.2473|± |0.0036|
| - humanities | 2|none | |acc |↑ |0.2351|± |0.0062|
| - formal_logic | 1|none | 0|acc |↑ |0.2857|± |0.0404|
| - high_school_european_history | 1|none | 0|acc |↑ |0.2667|± |0.0345|
| - high_school_us_history | 1|none | 0|acc |↑ |0.2696|± |0.0311|
| - high_school_world_history | 1|none | 0|acc |↑ |0.2110|± |0.0266|
| - international_law | 1|none | 0|acc |↑ |0.1653|± |0.0339|
| - jurisprudence | 1|none | 0|acc |↑ |0.2870|± |0.0437|
| - logical_fallacies | 1|none | 0|acc |↑ |0.2331|± |0.0332|
| - moral_disputes | 1|none | 0|acc |↑ |0.2283|± |0.0226|
| - moral_scenarios | 1|none | 0|acc |↑ |0.2425|± |0.0143|
| - philosophy | 1|none | 0|acc |↑ |0.2186|± |0.0235|
| - prehistory | 1|none | 0|acc |↑ |0.2099|± |0.0227|
| - professional_law | 1|none | 0|acc |↑ |0.2314|± |0.0108|
| - world_religions | 1|none | 0|acc |↑ |0.2632|± |0.0338|
| - other | 2|none | |acc |↑ |0.2485|± |0.0078|
| - business_ethics | 1|none | 0|acc |↑ |0.2600|± |0.0441|
| - clinical_knowledge | 1|none | 0|acc |↑ |0.2528|± |0.0267|
| - college_medicine | 1|none | 0|acc |↑ |0.2254|± |0.0319|
| - global_facts | 1|none | 0|acc |↑ |0.2700|± |0.0446|
| - human_aging | 1|none | 0|acc |↑ |0.2377|± |0.0286|
| - management | 1|none | 0|acc |↑ |0.2816|± |0.0445|
| - marketing | 1|none | 0|acc |↑ |0.2692|± |0.0291|
| - medical_genetics | 1|none | 0|acc |↑ |0.2600|± |0.0441|
| - miscellaneous | 1|none | 0|acc |↑ |0.2350|± |0.0152|
| - nutrition | 1|none | 0|acc |↑ |0.2549|± |0.0250|
| - professional_accounting | 1|none | 0|acc |↑ |0.2801|± |0.0268|
| - professional_medicine | 1|none | 0|acc |↑ |0.2610|± |0.0267|
| - virology | 1|none | 0|acc |↑ |0.1807|± |0.0300|
| - social sciences | 2|none | |acc |↑ |0.2658|± |0.0080|
| - econometrics | 1|none | 0|acc |↑ |0.1930|± |0.0371|
| - high_school_geography | 1|none | 0|acc |↑ |0.2172|± |0.0294|
| - high_school_government_and_politics| 1|none | 0|acc |↑ |0.3212|± |0.0337|
| - high_school_macroeconomics | 1|none | 0|acc |↑ |0.2923|± |0.0231|
| - high_school_microeconomics | 1|none | 0|acc |↑ |0.3025|± |0.0298|
| - high_school_psychology | 1|none | 0|acc |↑ |0.2752|± |0.0191|
| - human_sexuality | 1|none | 0|acc |↑ |0.2290|± |0.0369|
| - professional_psychology | 1|none | 0|acc |↑ |0.2386|± |0.0172|
| - public_relations | 1|none | 0|acc |↑ |0.2636|± |0.0422|
| - security_studies | 1|none | 0|acc |↑ |0.3143|± |0.0297|
| - sociology | 1|none | 0|acc |↑ |0.2338|± |0.0299|
| - us_foreign_policy | 1|none | 0|acc |↑ |0.2600|± |0.0441|
| - stem | 2|none | |acc |↑ |0.2464|± |0.0077|
| - abstract_algebra | 1|none | 0|acc |↑ |0.2500|± |0.0435|
| - anatomy | 1|none | 0|acc |↑ |0.2148|± |0.0355|
| - astronomy | 1|none | 0|acc |↑ |0.1908|± |0.0320|
| - college_biology | 1|none | 0|acc |↑ |0.2569|± |0.0365|
| - college_chemistry | 1|none | 0|acc |↑ |0.2700|± |0.0446|
| - college_computer_science | 1|none | 0|acc |↑ |0.3500|± |0.0479|
| - college_mathematics | 1|none | 0|acc |↑ |0.2700|± |0.0446|
| - college_physics | 1|none | 0|acc |↑ |0.2745|± |0.0444|
| - computer_security | 1|none | 0|acc |↑ |0.3000|± |0.0461|
| - conceptual_physics | 1|none | 0|acc |↑ |0.2766|± |0.0292|
| - electrical_engineering | 1|none | 0|acc |↑ |0.2345|± |0.0353|
| - elementary_mathematics | 1|none | 0|acc |↑ |0.2566|± |0.0225|
| - high_school_biology | 1|none | 0|acc |↑ |0.2226|± |0.0237|
| - high_school_chemistry | 1|none | 0|acc |↑ |0.2217|± |0.0292|
| - high_school_computer_science | 1|none | 0|acc |↑ |0.2000|± |0.0402|
| - high_school_mathematics | 1|none | 0|acc |↑ |0.2370|± |0.0259|
| - high_school_physics | 1|none | 0|acc |↑ |0.2517|± |0.0354|
| - high_school_statistics | 1|none | 0|acc |↑ |0.2685|± |0.0302|
| - machine_learning | 1|none | 0|acc |↑ |0.1786|± |0.0364|
|truthfulqa_mc2 | 2|none | 0|acc |↑ |0.4668|± |0.0161|
|winogrande | 1|none | 0|acc |↑ |0.5012|± |0.0141|

| Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu | 2|none | |acc |↑ |0.2473|± |0.0036|
| - humanities | 2|none | |acc |↑ |0.2351|± |0.0062|
| - other | 2|none | |acc |↑ |0.2485|± |0.0078|
| - social sciences| 2|none | |acc |↑ |0.2658|± |0.0080|
| - stem | 2|none | |acc |↑ |0.2464|± |0.0077|

```bash
litgpt evaluate --tasks 'leaderboard' --out_dir 'evaluate-leaderboard/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'bbh_zeroshot,bbh_fewshot,bbh_cot_fewshot,bbh_cot_zeroshot' --out_dir 'evaluate-bigbenchhard/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'mmlu,mmlu_pro' --out_dir 'evaluate-mmlu/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'arc_challenge,boolq,gpqa,hellaswag,openbookqa,piqa,truthfulqa_mc2,winogrande' --out_dir 'evaluate-reasoning/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'mmlu_multilingual,mgsm' --out_dir 'evaluate-multilinguals/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'gsm8k,mathqa' --out_dir 'evaluate-math/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```

```bash
litgpt evaluate --tasks 'qasper' --out_dir 'evaluate-long/' --batch_size 4 --dtype 'bfloat16' out/pretrain/final/
```