metadata
datasets:
- HuggingFaceFW/fineweb
base_model:
- openai-community/gpt2
NanoGPT Speedrun
Following https://github.com/KellerJordan/modded-nanogpt for fun (learning).
Run Info
baseline/
- Run on lightning cloud, using one L40S
- Batch size set to 32
- VRAM usage: 26.95GB (25698MB reported in
nvidia-smi) - 4 seconds per step, total 3200 steps
- Checkpoint saved every 320 steps