--- base_model: microsoft/Phi-3-mini-4k-instruct library_name: peft license: mit tags: - trl - sft - generated_from_trainer model-index: - name: phi-3-mini-LoRA results: [] --- [Visualize in Weights & Biases](https://wandb.ai/alizaidi/Phi3-mini-ft-goud-summarization/runs/4r2wc9jx) # phi-3-mini-LoRA This model is a fine-tuned version of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.2015 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0001 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 32 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_ratio: 0.1 - num_epochs: 3 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:-----:|:---------------:| | 1.9715 | 0.0242 | 100 | 1.9517 | | 1.8637 | 0.0484 | 200 | 1.7875 | | 1.7019 | 0.0725 | 300 | 1.6473 | | 1.6127 | 0.0967 | 400 | 1.5828 | | 1.5545 | 0.1209 | 500 | 1.5389 | | 1.5144 | 0.1451 | 600 | 1.5045 | | 1.4823 | 0.1693 | 700 | 1.4746 | | 1.4535 | 0.1935 | 800 | 1.4502 | | 1.4293 | 0.2176 | 900 | 1.4270 | | 1.4132 | 0.2418 | 1000 | 1.4073 | | 1.388 | 0.2660 | 1100 | 1.3880 | | 1.3757 | 0.2902 | 1200 | 1.3706 | | 1.3594 | 0.3144 | 1300 | 1.3543 | | 1.3399 | 0.3386 | 1400 | 1.3410 | | 1.3314 | 0.3627 | 1500 | 1.3284 | | 1.3161 | 0.3869 | 1600 | 1.3167 | | 1.3005 | 0.4111 | 1700 | 1.3084 | | 1.2937 | 0.4353 | 1800 | 1.2987 | | 1.2824 | 0.4595 | 1900 | 1.2920 | | 1.2806 | 0.4836 | 2000 | 1.2859 | | 1.2773 | 0.5078 | 2100 | 1.2793 | | 1.2717 | 0.5320 | 2200 | 1.2738 | | 1.2654 | 0.5562 | 2300 | 1.2692 | | 1.2597 | 0.5804 | 2400 | 1.2644 | | 1.2536 | 0.6046 | 2500 | 1.2601 | | 1.2486 | 0.6287 | 2600 | 1.2560 | | 1.2416 | 0.6529 | 2700 | 1.2527 | | 1.2462 | 0.6771 | 2800 | 1.2494 | | 1.2402 | 0.7013 | 2900 | 1.2465 | | 1.2353 | 0.7255 | 3000 | 1.2434 | | 1.2285 | 0.7497 | 3100 | 1.2410 | | 1.2314 | 0.7738 | 3200 | 1.2384 | | 1.2342 | 0.7980 | 3300 | 1.2357 | | 1.2195 | 0.8222 | 3400 | 1.2339 | | 1.2306 | 0.8464 | 3500 | 1.2316 | | 1.2225 | 0.8706 | 3600 | 1.2301 | | 1.2174 | 0.8947 | 3700 | 1.2281 | | 1.2293 | 0.9189 | 3800 | 1.2267 | | 1.2194 | 0.9431 | 3900 | 1.2250 | | 1.2169 | 0.9673 | 4000 | 1.2234 | | 1.2138 | 0.9915 | 4100 | 1.2224 | | 1.2105 | 1.0157 | 4200 | 1.2214 | | 1.2081 | 1.0398 | 4300 | 1.2201 | | 1.2129 | 1.0640 | 4400 | 1.2188 | | 1.1995 | 1.0882 | 4500 | 1.2177 | | 1.196 | 1.1124 | 4600 | 1.2167 | | 1.2041 | 1.1366 | 4700 | 1.2163 | | 1.2104 | 1.1608 | 4800 | 1.2151 | | 1.205 | 1.1849 | 4900 | 1.2144 | | 1.2055 | 1.2091 | 5000 | 1.2135 | | 1.1966 | 1.2333 | 5100 | 1.2128 | | 1.2017 | 1.2575 | 5200 | 1.2120 | | 1.1995 | 1.2817 | 5300 | 1.2117 | | 1.2015 | 1.3058 | 5400 | 1.2108 | | 1.1978 | 1.3300 | 5500 | 1.2103 | | 1.2017 | 1.3542 | 5600 | 1.2098 | | 1.196 | 1.3784 | 5700 | 1.2093 | | 1.1976 | 1.4026 | 5800 | 1.2089 | | 1.2057 | 1.4268 | 5900 | 1.2082 | | 1.2012 | 1.4509 | 6000 | 1.2079 | | 1.2067 | 1.4751 | 6100 | 1.2074 | | 1.2048 | 1.4993 | 6200 | 1.2071 | | 1.2011 | 1.5235 | 6300 | 1.2068 | | 1.1911 | 1.5477 | 6400 | 1.2064 | | 1.1974 | 1.5719 | 6500 | 1.2061 | | 1.1934 | 1.5960 | 6600 | 1.2059 | | 1.1896 | 1.6202 | 6700 | 1.2057 | | 1.1895 | 1.6444 | 6800 | 1.2052 | | 1.203 | 1.6686 | 6900 | 1.2051 | | 1.191 | 1.6928 | 7000 | 1.2048 | | 1.1995 | 1.7169 | 7100 | 1.2045 | | 1.1979 | 1.7411 | 7200 | 1.2043 | | 1.1918 | 1.7653 | 7300 | 1.2042 | | 1.1969 | 1.7895 | 7400 | 1.2040 | | 1.1869 | 1.8137 | 7500 | 1.2038 | | 1.1871 | 1.8379 | 7600 | 1.2036 | | 1.1988 | 1.8620 | 7700 | 1.2035 | | 1.1942 | 1.8862 | 7800 | 1.2034 | | 1.1931 | 1.9104 | 7900 | 1.2033 | | 1.1947 | 1.9346 | 8000 | 1.2030 | | 1.1932 | 1.9588 | 8100 | 1.2030 | | 1.1922 | 1.9830 | 8200 | 1.2028 | | 1.192 | 2.0071 | 8300 | 1.2027 | | 1.1997 | 2.0313 | 8400 | 1.2027 | | 1.1945 | 2.0555 | 8500 | 1.2026 | | 1.1934 | 2.0797 | 8600 | 1.2026 | | 1.1955 | 2.1039 | 8700 | 1.2024 | | 1.1901 | 2.1280 | 8800 | 1.2024 | | 1.1898 | 2.1522 | 8900 | 1.2023 | | 1.186 | 2.1764 | 9000 | 1.2022 | | 1.1858 | 2.2006 | 9100 | 1.2022 | | 1.1965 | 2.2248 | 9200 | 1.2021 | | 1.1835 | 2.2490 | 9300 | 1.2021 | | 1.1983 | 2.2731 | 9400 | 1.2020 | | 1.1813 | 2.2973 | 9500 | 1.2020 | | 1.1903 | 2.3215 | 9600 | 1.2019 | | 1.1952 | 2.3457 | 9700 | 1.2019 | | 1.1899 | 2.3699 | 9800 | 1.2018 | | 1.2011 | 2.3941 | 9900 | 1.2018 | | 1.1936 | 2.4182 | 10000 | 1.2018 | | 1.1931 | 2.4424 | 10100 | 1.2018 | | 1.1991 | 2.4666 | 10200 | 1.2017 | | 1.19 | 2.4908 | 10300 | 1.2017 | | 1.1913 | 2.5150 | 10400 | 1.2016 | | 1.1886 | 2.5391 | 10500 | 1.2017 | | 1.1848 | 2.5633 | 10600 | 1.2016 | | 1.1875 | 2.5875 | 10700 | 1.2016 | | 1.1887 | 2.6117 | 10800 | 1.2016 | | 1.1866 | 2.6359 | 10900 | 1.2016 | | 1.188 | 2.6601 | 11000 | 1.2016 | | 1.1952 | 2.6842 | 11100 | 1.2015 | | 1.1947 | 2.7084 | 11200 | 1.2015 | | 1.1905 | 2.7326 | 11300 | 1.2015 | | 1.1838 | 2.7568 | 11400 | 1.2015 | | 1.1893 | 2.7810 | 11500 | 1.2015 | | 1.1808 | 2.8052 | 11600 | 1.2015 | | 1.1909 | 2.8293 | 11700 | 1.2015 | | 1.1858 | 2.8535 | 11800 | 1.2015 | | 1.185 | 2.8777 | 11900 | 1.2015 | | 1.1947 | 2.9019 | 12000 | 1.2015 | | 1.1868 | 2.9261 | 12100 | 1.2014 | | 1.1872 | 2.9502 | 12200 | 1.2015 | | 1.1852 | 2.9744 | 12300 | 1.2015 | | 1.185 | 2.9986 | 12400 | 1.2015 | ### Framework versions - PEFT 0.11.1 - Transformers 4.43.1 - Pytorch 2.3.1+cu121 - Datasets 2.20.0 - Tokenizers 0.19.1