--- library_name: transformers license: apache-2.0 datasets: - Open-Orca/SlimOrca pipeline_tag: text-generation base_model: Na0s/Llama-3.1-8b-Pruned-4-Layers --- <a href="https://ibb.co/0Yhg31Q"><img src="https://i.ibb.co/F8gStcn/Model-card-peft-lora.webp" alt="Model-card-peft-lora" align="center"></a> # Model Card for Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT ## Model Details ### Model Description - **Finetuned from model:[Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT]** ## Training Details LoRA BF16, batch_size=2, steps=10000, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 10000 learning_rate = 2e-4, fp16 = not is_bfloat16_supported(), bf16 = is_bfloat16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407 ### Training Data <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> [Open-Orca/SlimOrca] ## Evaluation MMLU Pro 0-shot: 0.2937 #### Evaluation Data <!-- This should link to a Dataset Card if possible. --> [TIGER-AI-Lab/MMLU-Pro] ## Environmental Impact <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).