--- license: gemma base_model: google/gemma-2-2b tags: - trl - sft - generated_from_trainer model-index: - name: collapse_gemma-2-2b_hs2_accumulate_iter15_sftsd2 results: [] --- # collapse_gemma-2-2b_hs2_accumulate_iter15_sftsd2 This model is a fine-tuned version of [google/gemma-2-2b](https://huggingface.co/google/gemma-2-2b) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.1131 - Num Input Tokens Seen: 77355600 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 8e-06 - train_batch_size: 8 - eval_batch_size: 16 - seed: 2 - gradient_accumulation_steps: 16 - total_train_batch_size: 128 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: constant_with_warmup - lr_scheduler_warmup_ratio: 0.05 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen | |:-------------:|:------:|:----:|:---------------:|:-----------------:| | No log | 0 | 0 | 1.3909 | 0 | | 1.5857 | 0.0035 | 5 | 1.3905 | 274072 | | 1.5223 | 0.0070 | 10 | 1.3796 | 541744 | | 1.4477 | 0.0105 | 15 | 1.3488 | 817496 | | 1.4617 | 0.0139 | 20 | 1.3054 | 1085248 | | 1.3413 | 0.0174 | 25 | 1.2629 | 1353288 | | 1.291 | 0.0209 | 30 | 1.2367 | 1623248 | | 1.2206 | 0.0244 | 35 | 1.1999 | 1889576 | | 1.2079 | 0.0279 | 40 | 1.1918 | 2157840 | | 1.0084 | 0.0314 | 45 | 1.2110 | 2428448 | | 0.8869 | 0.0349 | 50 | 1.2244 | 2699856 | | 0.9091 | 0.0384 | 55 | 1.2313 | 2973208 | | 0.7925 | 0.0418 | 60 | 1.2519 | 3236816 | | 0.6403 | 0.0453 | 65 | 1.2698 | 3508448 | | 0.6131 | 0.0488 | 70 | 1.2591 | 3784352 | | 0.4917 | 0.0523 | 75 | 1.3081 | 4051368 | | 0.5493 | 0.0558 | 80 | 1.2474 | 4316016 | | 0.4799 | 0.0593 | 85 | 1.2648 | 4582616 | | 0.5019 | 0.0628 | 90 | 1.2398 | 4853864 | | 0.4845 | 0.0663 | 95 | 1.2529 | 5124936 | | 0.2863 | 0.0697 | 100 | 1.2257 | 5394776 | | 0.3814 | 0.0732 | 105 | 1.2353 | 5662064 | | 0.2254 | 0.0767 | 110 | 1.2212 | 5927976 | | 0.4051 | 0.0802 | 115 | 1.2220 | 6205320 | | 0.3712 | 0.0837 | 120 | 1.2122 | 6474344 | | 0.3397 | 0.0872 | 125 | 1.2214 | 6747360 | | 0.2313 | 0.0907 | 130 | 1.2138 | 7019848 | | 0.3128 | 0.0942 | 135 | 1.2125 | 7279560 | | 0.2283 | 0.0976 | 140 | 1.2016 | 7541824 | | 0.2835 | 0.1011 | 145 | 1.2145 | 7810224 | | 0.2073 | 0.1046 | 150 | 1.2051 | 8072048 | | 0.3416 | 0.1081 | 155 | 1.2066 | 8347104 | | 0.2624 | 0.1116 | 160 | 1.1970 | 8622296 | | 0.1935 | 0.1151 | 165 | 1.1999 | 8897864 | | 0.2294 | 0.1186 | 170 | 1.1998 | 9165896 | | 0.2058 | 0.1221 | 175 | 1.1989 | 9440512 | | 0.2686 | 0.1255 | 180 | 1.1982 | 9706224 | | 0.186 | 0.1290 | 185 | 1.2053 | 9975248 | | 0.1737 | 0.1325 | 190 | 1.1959 | 10240736 | | 0.2474 | 0.1360 | 195 | 1.1953 | 10505488 | | 0.1694 | 0.1395 | 200 | 1.1968 | 10774672 | | 0.2778 | 0.1430 | 205 | 1.2009 | 11047008 | | 0.2827 | 0.1465 | 210 | 1.1941 | 11314056 | | 0.2641 | 0.1500 | 215 | 1.1896 | 11581288 | | 0.2102 | 0.1534 | 220 | 1.2018 | 11850656 | | 0.2089 | 0.1569 | 225 | 1.1883 | 12120104 | | 0.1844 | 0.1604 | 230 | 1.1941 | 12392624 | | 0.1579 | 0.1639 | 235 | 1.1996 | 12665456 | | 0.221 | 0.1674 | 240 | 1.1860 | 12937024 | | 0.1537 | 0.1709 | 245 | 1.1936 | 13204896 | | 0.2092 | 0.1744 | 250 | 1.1877 | 13472088 | | 0.2015 | 0.1779 | 255 | 1.1879 | 13746752 | | 0.1777 | 0.1813 | 260 | 1.1869 | 14012632 | | 0.1181 | 0.1848 | 265 | 1.1869 | 14281208 | | 0.1912 | 0.1883 | 270 | 1.1827 | 14552360 | | 0.2131 | 0.1918 | 275 | 1.1783 | 14825984 | | 0.1601 | 0.1953 | 280 | 1.1827 | 15099872 | | 0.1679 | 0.1988 | 285 | 1.1785 | 15369728 | | 0.2369 | 0.2023 | 290 | 1.1773 | 15641640 | | 0.1621 | 0.2058 | 295 | 1.1815 | 15913448 | | 0.2069 | 0.2092 | 300 | 1.1750 | 16185080 | | 0.1399 | 0.2127 | 305 | 1.1779 | 16452968 | | 0.1589 | 0.2162 | 310 | 1.1837 | 16721400 | | 0.1577 | 0.2197 | 315 | 1.1731 | 16993344 | | 0.1535 | 0.2232 | 320 | 1.1725 | 17259176 | | 0.1642 | 0.2267 | 325 | 1.1773 | 17528016 | | 0.2407 | 0.2302 | 330 | 1.1709 | 17792488 | | 0.1973 | 0.2337 | 335 | 1.1699 | 18063896 | | 0.1771 | 0.2371 | 340 | 1.1730 | 18331584 | | 0.1094 | 0.2406 | 345 | 1.1719 | 18598880 | | 0.1967 | 0.2441 | 350 | 1.1702 | 18867056 | | 0.201 | 0.2476 | 355 | 1.1742 | 19137192 | | 0.1584 | 0.2511 | 360 | 1.1715 | 19413432 | | 0.2602 | 0.2546 | 365 | 1.1671 | 19679304 | | 0.1579 | 0.2581 | 370 | 1.1742 | 19947504 | | 0.1705 | 0.2616 | 375 | 1.1675 | 20220776 | | 0.1731 | 0.2650 | 380 | 1.1655 | 20489056 | | 0.1427 | 0.2685 | 385 | 1.1659 | 20761480 | | 0.1754 | 0.2720 | 390 | 1.1667 | 21033896 | | 0.1246 | 0.2755 | 395 | 1.1647 | 21308192 | | 0.2133 | 0.2790 | 400 | 1.1653 | 21580496 | | 0.189 | 0.2825 | 405 | 1.1605 | 21852880 | | 0.201 | 0.2860 | 410 | 1.1617 | 22125800 | | 0.1375 | 0.2895 | 415 | 1.1603 | 22387808 | | 0.1831 | 0.2929 | 420 | 1.1595 | 22658384 | | 0.1702 | 0.2964 | 425 | 1.1611 | 22933712 | | 0.153 | 0.2999 | 430 | 1.1620 | 23202632 | | 0.1521 | 0.3034 | 435 | 1.1625 | 23472840 | | 0.0915 | 0.3069 | 440 | 1.1611 | 23747432 | | 0.223 | 0.3104 | 445 | 1.1592 | 24019904 | | 0.1467 | 0.3139 | 450 | 1.1577 | 24298624 | | 0.1539 | 0.3173 | 455 | 1.1584 | 24574280 | | 0.1824 | 0.3208 | 460 | 1.1545 | 24839104 | | 0.1814 | 0.3243 | 465 | 1.1549 | 25113392 | | 0.1265 | 0.3278 | 470 | 1.1562 | 25389512 | | 0.1726 | 0.3313 | 475 | 1.1575 | 25664664 | | 0.1663 | 0.3348 | 480 | 1.1542 | 25940040 | | 0.1412 | 0.3383 | 485 | 1.1545 | 26204272 | | 0.2201 | 0.3418 | 490 | 1.1544 | 26475360 | | 0.172 | 0.3452 | 495 | 1.1536 | 26746912 | | 0.2073 | 0.3487 | 500 | 1.1586 | 27019352 | | 0.2455 | 0.3522 | 505 | 1.1489 | 27289256 | | 0.1789 | 0.3557 | 510 | 1.1500 | 27558280 | | 0.1252 | 0.3592 | 515 | 1.1532 | 27825544 | | 0.168 | 0.3627 | 520 | 1.1511 | 28090352 | | 0.1569 | 0.3662 | 525 | 1.1479 | 28353368 | | 0.1293 | 0.3697 | 530 | 1.1505 | 28621776 | | 0.207 | 0.3731 | 535 | 1.1506 | 28894096 | | 0.2029 | 0.3766 | 540 | 1.1465 | 29161496 | | 0.1751 | 0.3801 | 545 | 1.1494 | 29430320 | | 0.2447 | 0.3836 | 550 | 1.1484 | 29703696 | | 0.1782 | 0.3871 | 555 | 1.1446 | 29972224 | | 0.1726 | 0.3906 | 560 | 1.1479 | 30237496 | | 0.1688 | 0.3941 | 565 | 1.1448 | 30511616 | | 0.1342 | 0.3976 | 570 | 1.1455 | 30780112 | | 0.1458 | 0.4010 | 575 | 1.1485 | 31046152 | | 0.1455 | 0.4045 | 580 | 1.1436 | 31320584 | | 0.1433 | 0.4080 | 585 | 1.1468 | 31594488 | | 0.2107 | 0.4115 | 590 | 1.1521 | 31863232 | | 0.168 | 0.4150 | 595 | 1.1438 | 32136200 | | 0.1955 | 0.4185 | 600 | 1.1424 | 32402328 | | 0.1586 | 0.4220 | 605 | 1.1446 | 32673480 | | 0.1551 | 0.4255 | 610 | 1.1428 | 32945984 | | 0.2038 | 0.4289 | 615 | 1.1429 | 33213344 | | 0.168 | 0.4324 | 620 | 1.1440 | 33483904 | | 0.1486 | 0.4359 | 625 | 1.1430 | 33754856 | | 0.1428 | 0.4394 | 630 | 1.1430 | 34019248 | | 0.0953 | 0.4429 | 635 | 1.1427 | 34290664 | | 0.1611 | 0.4464 | 640 | 1.1411 | 34560112 | | 0.2 | 0.4499 | 645 | 1.1417 | 34828392 | | 0.1332 | 0.4534 | 650 | 1.1433 | 35097200 | | 0.1407 | 0.4568 | 655 | 1.1397 | 35372584 | | 0.1316 | 0.4603 | 660 | 1.1410 | 35642384 | | 0.1168 | 0.4638 | 665 | 1.1429 | 35912368 | | 0.1899 | 0.4673 | 670 | 1.1433 | 36183144 | | 0.1515 | 0.4708 | 675 | 1.1393 | 36450368 | | 0.15 | 0.4743 | 680 | 1.1386 | 36726152 | | 0.1255 | 0.4778 | 685 | 1.1410 | 36993864 | | 0.2125 | 0.4813 | 690 | 1.1396 | 37262768 | | 0.1353 | 0.4847 | 695 | 1.1371 | 37532080 | | 0.2373 | 0.4882 | 700 | 1.1361 | 37804320 | | 0.1203 | 0.4917 | 705 | 1.1400 | 38068112 | | 0.1384 | 0.4952 | 710 | 1.1418 | 38332664 | | 0.1105 | 0.4987 | 715 | 1.1382 | 38603384 | | 0.1218 | 0.5022 | 720 | 1.1382 | 38880896 | | 0.1842 | 0.5057 | 725 | 1.1380 | 39140680 | | 0.1953 | 0.5092 | 730 | 1.1363 | 39407688 | | 0.1655 | 0.5126 | 735 | 1.1368 | 39678184 | | 0.1442 | 0.5161 | 740 | 1.1367 | 39950552 | | 0.2163 | 0.5196 | 745 | 1.1372 | 40220840 | | 0.1383 | 0.5231 | 750 | 1.1355 | 40484976 | | 0.119 | 0.5266 | 755 | 1.1371 | 40760568 | | 0.1839 | 0.5301 | 760 | 1.1388 | 41026904 | | 0.1105 | 0.5336 | 765 | 1.1363 | 41295648 | | 0.2016 | 0.5371 | 770 | 1.1345 | 41571256 | | 0.1222 | 0.5405 | 775 | 1.1334 | 41837520 | | 0.1108 | 0.5440 | 780 | 1.1368 | 42108784 | | 0.1396 | 0.5475 | 785 | 1.1352 | 42379248 | | 0.1336 | 0.5510 | 790 | 1.1325 | 42647536 | | 0.1388 | 0.5545 | 795 | 1.1343 | 42913200 | | 0.1104 | 0.5580 | 800 | 1.1338 | 43180288 | | 0.2462 | 0.5615 | 805 | 1.1294 | 43452368 | | 0.1217 | 0.5650 | 810 | 1.1332 | 43718184 | | 0.1668 | 0.5684 | 815 | 1.1339 | 43987296 | | 0.1155 | 0.5719 | 820 | 1.1329 | 44258096 | | 0.1311 | 0.5754 | 825 | 1.1338 | 44531712 | | 0.1013 | 0.5789 | 830 | 1.1357 | 44793288 | | 0.1882 | 0.5824 | 835 | 1.1326 | 45057712 | | 0.1579 | 0.5859 | 840 | 1.1327 | 45328280 | | 0.1741 | 0.5894 | 845 | 1.1328 | 45602560 | | 0.0777 | 0.5929 | 850 | 1.1309 | 45877784 | | 0.1954 | 0.5963 | 855 | 1.1318 | 46142784 | | 0.155 | 0.5998 | 860 | 1.1311 | 46408176 | | 0.1422 | 0.6033 | 865 | 1.1308 | 46682208 | | 0.2115 | 0.6068 | 870 | 1.1330 | 46947496 | | 0.1239 | 0.6103 | 875 | 1.1329 | 47209064 | | 0.1788 | 0.6138 | 880 | 1.1310 | 47480896 | | 0.1399 | 0.6173 | 885 | 1.1318 | 47754464 | | 0.1178 | 0.6207 | 890 | 1.1311 | 48024448 | | 0.2156 | 0.6242 | 895 | 1.1319 | 48293792 | | 0.1503 | 0.6277 | 900 | 1.1304 | 48568472 | | 0.1432 | 0.6312 | 905 | 1.1302 | 48833328 | | 0.1372 | 0.6347 | 910 | 1.1301 | 49099528 | | 0.1597 | 0.6382 | 915 | 1.1291 | 49366152 | | 0.1637 | 0.6417 | 920 | 1.1317 | 49634592 | | 0.1749 | 0.6452 | 925 | 1.1327 | 49902552 | | 0.1394 | 0.6486 | 930 | 1.1310 | 50172504 | | 0.1735 | 0.6521 | 935 | 1.1319 | 50443656 | | 0.2064 | 0.6556 | 940 | 1.1311 | 50700592 | | 0.1581 | 0.6591 | 945 | 1.1295 | 50966920 | | 0.1516 | 0.6626 | 950 | 1.1310 | 51234848 | | 0.134 | 0.6661 | 955 | 1.1291 | 51501296 | | 0.1841 | 0.6696 | 960 | 1.1255 | 51768648 | | 0.118 | 0.6731 | 965 | 1.1275 | 52044776 | | 0.1642 | 0.6765 | 970 | 1.1278 | 52314744 | | 0.1485 | 0.6800 | 975 | 1.1257 | 52586176 | | 0.1001 | 0.6835 | 980 | 1.1292 | 52853256 | | 0.1443 | 0.6870 | 985 | 1.1281 | 53118936 | | 0.1721 | 0.6905 | 990 | 1.1270 | 53385208 | | 0.1803 | 0.6940 | 995 | 1.1268 | 53659240 | | 0.1446 | 0.6975 | 1000 | 1.1286 | 53936672 | | 0.1397 | 0.7010 | 1005 | 1.1259 | 54211392 | | 0.1688 | 0.7044 | 1010 | 1.1247 | 54485344 | | 0.1743 | 0.7079 | 1015 | 1.1250 | 54759912 | | 0.1671 | 0.7114 | 1020 | 1.1241 | 55023480 | | 0.2176 | 0.7149 | 1025 | 1.1251 | 55300088 | | 0.0987 | 0.7184 | 1030 | 1.1275 | 55566656 | | 0.1689 | 0.7219 | 1035 | 1.1262 | 55841368 | | 0.0993 | 0.7254 | 1040 | 1.1242 | 56109104 | | 0.1464 | 0.7289 | 1045 | 1.1269 | 56375096 | | 0.1196 | 0.7323 | 1050 | 1.1262 | 56648304 | | 0.1293 | 0.7358 | 1055 | 1.1265 | 56919840 | | 0.2165 | 0.7393 | 1060 | 1.1264 | 57192200 | | 0.1204 | 0.7428 | 1065 | 1.1251 | 57462304 | | 0.1277 | 0.7463 | 1070 | 1.1250 | 57736096 | | 0.1889 | 0.7498 | 1075 | 1.1250 | 58012208 | | 0.0892 | 0.7533 | 1080 | 1.1229 | 58275376 | | 0.156 | 0.7568 | 1085 | 1.1234 | 58544656 | | 0.198 | 0.7602 | 1090 | 1.1223 | 58809944 | | 0.1886 | 0.7637 | 1095 | 1.1231 | 59084120 | | 0.0769 | 0.7672 | 1100 | 1.1271 | 59360272 | | 0.1228 | 0.7707 | 1105 | 1.1247 | 59626856 | | 0.1821 | 0.7742 | 1110 | 1.1218 | 59901240 | | 0.1788 | 0.7777 | 1115 | 1.1219 | 60174472 | | 0.1447 | 0.7812 | 1120 | 1.1252 | 60443200 | | 0.1558 | 0.7847 | 1125 | 1.1236 | 60715824 | | 0.1888 | 0.7881 | 1130 | 1.1204 | 60992976 | | 0.1044 | 0.7916 | 1135 | 1.1221 | 61257496 | | 0.1546 | 0.7951 | 1140 | 1.1231 | 61526448 | | 0.1096 | 0.7986 | 1145 | 1.1216 | 61798776 | | 0.1492 | 0.8021 | 1150 | 1.1220 | 62071136 | | 0.1426 | 0.8056 | 1155 | 1.1234 | 62340416 | | 0.0687 | 0.8091 | 1160 | 1.1214 | 62618184 | | 0.103 | 0.8126 | 1165 | 1.1210 | 62891344 | | 0.14 | 0.8160 | 1170 | 1.1194 | 63160328 | | 0.1646 | 0.8195 | 1175 | 1.1201 | 63435072 | | 0.1694 | 0.8230 | 1180 | 1.1192 | 63706128 | | 0.1759 | 0.8265 | 1185 | 1.1183 | 63971872 | | 0.2014 | 0.8300 | 1190 | 1.1208 | 64244560 | | 0.1704 | 0.8335 | 1195 | 1.1198 | 64513712 | | 0.1256 | 0.8370 | 1200 | 1.1198 | 64787280 | | 0.1241 | 0.8405 | 1205 | 1.1218 | 65061456 | | 0.1498 | 0.8439 | 1210 | 1.1214 | 65323328 | | 0.1587 | 0.8474 | 1215 | 1.1206 | 65590088 | | 0.0935 | 0.8509 | 1220 | 1.1200 | 65856416 | | 0.1773 | 0.8544 | 1225 | 1.1192 | 66133352 | | 0.1088 | 0.8579 | 1230 | 1.1186 | 66407160 | | 0.1253 | 0.8614 | 1235 | 1.1199 | 66679424 | | 0.0902 | 0.8649 | 1240 | 1.1203 | 66952240 | | 0.1381 | 0.8684 | 1245 | 1.1186 | 67222368 | | 0.1402 | 0.8718 | 1250 | 1.1201 | 67496928 | | 0.1335 | 0.8753 | 1255 | 1.1190 | 67767944 | | 0.0875 | 0.8788 | 1260 | 1.1199 | 68032696 | | 0.2325 | 0.8823 | 1265 | 1.1213 | 68307792 | | 0.1242 | 0.8858 | 1270 | 1.1167 | 68580000 | | 0.1802 | 0.8893 | 1275 | 1.1168 | 68841768 | | 0.1417 | 0.8928 | 1280 | 1.1179 | 69111664 | | 0.1716 | 0.8963 | 1285 | 1.1196 | 69381096 | | 0.2164 | 0.8997 | 1290 | 1.1170 | 69641296 | | 0.1582 | 0.9032 | 1295 | 1.1163 | 69904696 | | 0.2125 | 0.9067 | 1300 | 1.1171 | 70172256 | | 0.1373 | 0.9102 | 1305 | 1.1181 | 70438992 | | 0.1768 | 0.9137 | 1310 | 1.1187 | 70711416 | | 0.1272 | 0.9172 | 1315 | 1.1173 | 70983104 | | 0.1771 | 0.9207 | 1320 | 1.1167 | 71260568 | | 0.1259 | 0.9241 | 1325 | 1.1173 | 71537792 | | 0.1775 | 0.9276 | 1330 | 1.1158 | 71811624 | | 0.1705 | 0.9311 | 1335 | 1.1153 | 72081128 | | 0.0856 | 0.9346 | 1340 | 1.1183 | 72342520 | | 0.1641 | 0.9381 | 1345 | 1.1211 | 72614760 | | 0.1664 | 0.9416 | 1350 | 1.1194 | 72890136 | | 0.1798 | 0.9451 | 1355 | 1.1157 | 73156960 | | 0.1163 | 0.9486 | 1360 | 1.1154 | 73435888 | | 0.1608 | 0.9520 | 1365 | 1.1181 | 73703192 | | 0.1355 | 0.9555 | 1370 | 1.1174 | 73977432 | | 0.1911 | 0.9590 | 1375 | 1.1159 | 74248368 | | 0.1568 | 0.9625 | 1380 | 1.1158 | 74509400 | | 0.209 | 0.9660 | 1385 | 1.1138 | 74774824 | | 0.1531 | 0.9695 | 1390 | 1.1158 | 75047224 | | 0.1059 | 0.9730 | 1395 | 1.1161 | 75315032 | | 0.1128 | 0.9765 | 1400 | 1.1144 | 75579816 | | 0.184 | 0.9799 | 1405 | 1.1148 | 75847120 | | 0.1303 | 0.9834 | 1410 | 1.1159 | 76120640 | | 0.1457 | 0.9869 | 1415 | 1.1129 | 76392480 | | 0.1718 | 0.9904 | 1420 | 1.1142 | 76652728 | | 0.1126 | 0.9939 | 1425 | 1.1175 | 76921848 | | 0.1228 | 0.9974 | 1430 | 1.1165 | 77192504 | ### Framework versions - Transformers 4.44.0 - Pytorch 2.4.0+cu121 - Datasets 2.20.0 - Tokenizers 0.19.1