Hugging Face's logo

avuhong
/
AAVesm2_650M_800_v1

Fill-Mask
Transformers
PyTorch
TensorBoard
esm
Generated from Trainer
Model card Files
xet
Metrics Community
avuhong commited on
Commit
817fe09
·
1 Parent(s): 905008d

Upload with huggingface_hub

Browse files
Files changed (38) hide show
  1. README.md +100 -0
  2. all_results.json +15 -0
  3. checkpoint-13650/config.json +30 -0
  4. checkpoint-13650/optimizer.pt +3 -0
  5. checkpoint-13650/pytorch_model.bin +3 -0
  6. checkpoint-13650/rng_state_0.pth +3 -0
  7. checkpoint-13650/rng_state_1.pth +3 -0
  8. checkpoint-13650/scaler.pt +3 -0
  9. checkpoint-13650/scheduler.pt +3 -0
  10. checkpoint-13650/special_tokens_map.json +7 -0
  11. checkpoint-13650/tokenizer_config.json +6 -0
  12. checkpoint-13650/trainer_state.json +493 -0
  13. checkpoint-13650/training_args.bin +3 -0
  14. checkpoint-13650/vocab.txt +33 -0
  15. checkpoint-14040/config.json +30 -0
  16. checkpoint-14040/optimizer.pt +3 -0
  17. checkpoint-14040/pytorch_model.bin +3 -0
  18. checkpoint-14040/rng_state_0.pth +3 -0
  19. checkpoint-14040/rng_state_1.pth +3 -0
  20. checkpoint-14040/scaler.pt +3 -0
  21. checkpoint-14040/scheduler.pt +3 -0
  22. checkpoint-14040/special_tokens_map.json +7 -0
  23. checkpoint-14040/tokenizer_config.json +6 -0
  24. checkpoint-14040/trainer_state.json +508 -0
  25. checkpoint-14040/training_args.bin +3 -0
  26. checkpoint-14040/vocab.txt +33 -0
  27. config.json +30 -0
  28. eval_results.json +10 -0
  29. pytorch_model.bin +3 -0
  30. runs/Apr26_12-00-12_srvgpu/1682510421.8712294/events.out.tfevents.1682510421.srvgpu.3077809.1 +3 -0
  31. runs/Apr26_12-00-12_srvgpu/events.out.tfevents.1682510421.srvgpu.3077809.0 +3 -0
  32. runs/Apr26_12-00-12_srvgpu/events.out.tfevents.1682524144.srvgpu.3077809.2 +3 -0
  33. special_tokens_map.json +7 -0
  34. tokenizer_config.json +6 -0
  35. train_results.json +8 -0
  36. trainer_state.json +517 -0
  37. training_args.bin +3 -0
  38. vocab.txt +33 -0
README.md ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - generated_from_trainer
5
+ metrics:
6
+ - accuracy
7
+ model-index:
8
+ - name: output_AAVESM2_650M_v1
9
+ results: []
10
+ ---
11
+
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ # output_AAVESM2_650M_v1
16
+
17
+ This model is a fine-tuned version of [facebook/esm2_t33_650M_UR50D](https://huggingface.co/facebook/esm2_t33_650M_UR50D) on an unknown dataset.
18
+ It achieves the following results on the evaluation set:
19
+ - Loss: 0.3707
20
+ - Accuracy: 0.8905
21
+
22
+ ## Model description
23
+
24
+ More information needed
25
+
26
+ ## Intended uses & limitations
27
+
28
+ More information needed
29
+
30
+ ## Training and evaluation data
31
+
32
+ More information needed
33
+
34
+ ## Training procedure
35
+
36
+ ### Training hyperparameters
37
+
38
+ The following hyperparameters were used during training:
39
+ - learning_rate: 1e-05
40
+ - train_batch_size: 1
41
+ - eval_batch_size: 1
42
+ - seed: 42
43
+ - distributed_type: multi-GPU
44
+ - num_devices: 2
45
+ - gradient_accumulation_steps: 4
46
+ - total_train_batch_size: 8
47
+ - total_eval_batch_size: 2
48
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
49
+ - lr_scheduler_type: linear
50
+ - num_epochs: 36.0
51
+ - mixed_precision_training: Native AMP
52
+
53
+ ### Training results
54
+
55
+ | Training Loss | Epoch | Step | Validation Loss | Accuracy |
56
+ |:-------------:|:-----:|:-----:|:---------------:|:--------:|
57
+ | No log | 1.0 | 390 | 1.3347 | 0.5993 |
58
+ | 1.5408 | 2.0 | 780 | 1.0699 | 0.6796 |
59
+ | 1.1283 | 3.0 | 1170 | 0.8751 | 0.7373 |
60
+ | 0.9078 | 4.0 | 1560 | 0.7534 | 0.7711 |
61
+ | 0.9078 | 5.0 | 1950 | 0.6711 | 0.8022 |
62
+ | 0.7705 | 6.0 | 2340 | 0.6078 | 0.8169 |
63
+ | 0.6863 | 7.0 | 2730 | 0.5668 | 0.8318 |
64
+ | 0.6277 | 8.0 | 3120 | 0.5461 | 0.8386 |
65
+ | 0.5863 | 9.0 | 3510 | 0.5143 | 0.8514 |
66
+ | 0.5863 | 10.0 | 3900 | 0.4992 | 0.8522 |
67
+ | 0.5564 | 11.0 | 4290 | 0.4940 | 0.8533 |
68
+ | 0.5199 | 12.0 | 4680 | 0.4727 | 0.8633 |
69
+ | 0.5025 | 13.0 | 5070 | 0.4586 | 0.8638 |
70
+ | 0.5025 | 14.0 | 5460 | 0.4549 | 0.8673 |
71
+ | 0.4814 | 15.0 | 5850 | 0.4442 | 0.8698 |
72
+ | 0.4746 | 16.0 | 6240 | 0.4306 | 0.8750 |
73
+ | 0.4527 | 17.0 | 6630 | 0.4291 | 0.8742 |
74
+ | 0.4382 | 18.0 | 7020 | 0.4213 | 0.8751 |
75
+ | 0.4382 | 19.0 | 7410 | 0.4193 | 0.8751 |
76
+ | 0.4328 | 20.0 | 7800 | 0.4143 | 0.8760 |
77
+ | 0.4191 | 21.0 | 8190 | 0.4071 | 0.8836 |
78
+ | 0.4106 | 22.0 | 8580 | 0.3980 | 0.8819 |
79
+ | 0.4106 | 23.0 | 8970 | 0.3987 | 0.8822 |
80
+ | 0.4037 | 24.0 | 9360 | 0.4027 | 0.8819 |
81
+ | 0.3893 | 25.0 | 9750 | 0.3868 | 0.8893 |
82
+ | 0.3991 | 26.0 | 10140 | 0.3882 | 0.8846 |
83
+ | 0.3786 | 27.0 | 10530 | 0.3939 | 0.8859 |
84
+ | 0.3786 | 28.0 | 10920 | 0.3959 | 0.8848 |
85
+ | 0.38 | 29.0 | 11310 | 0.3950 | 0.8850 |
86
+ | 0.3764 | 30.0 | 11700 | 0.3783 | 0.8893 |
87
+ | 0.3708 | 31.0 | 12090 | 0.3799 | 0.8891 |
88
+ | 0.3708 | 32.0 | 12480 | 0.3915 | 0.8867 |
89
+ | 0.3656 | 33.0 | 12870 | 0.3780 | 0.8903 |
90
+ | 0.3617 | 34.0 | 13260 | 0.3805 | 0.8874 |
91
+ | 0.361 | 35.0 | 13650 | 0.3776 | 0.8920 |
92
+ | 0.3595 | 36.0 | 14040 | 0.3712 | 0.8888 |
93
+
94
+
95
+ ### Framework versions
96
+
97
+ - Transformers 4.26.1
98
+ - Pytorch 1.13.1+cu117
99
+ - Datasets 2.9.0
100
+ - Tokenizers 0.13.2
all_results.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 36.0,
3
+ "eval_accuracy": 0.8905318152693578,
4
+ "eval_loss": 0.3707260489463806,
5
+ "eval_runtime": 5.8253,
6
+ "eval_samples": 171,
7
+ "eval_samples_per_second": 29.355,
8
+ "eval_steps_per_second": 14.763,
9
+ "perplexity": 1.448786122531838,
10
+ "train_loss": 0.5379996796958467,
11
+ "train_runtime": 13714.1823,
12
+ "train_samples": 3120,
13
+ "train_samples_per_second": 8.19,
14
+ "train_steps_per_second": 1.024
15
+ }
checkpoint-13650/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "facebook/esm2_t33_650M_UR50D",
3
+ "architectures": [
4
+ "EsmForMaskedLM"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.0,
7
+ "classifier_dropout": null,
8
+ "emb_layer_norm_before": false,
9
+ "esmfold_config": null,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.0,
12
+ "hidden_size": 1280,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 5120,
15
+ "is_folding_model": false,
16
+ "layer_norm_eps": 1e-05,
17
+ "mask_token_id": 32,
18
+ "max_position_embeddings": 1026,
19
+ "model_type": "esm",
20
+ "num_attention_heads": 20,
21
+ "num_hidden_layers": 33,
22
+ "pad_token_id": 1,
23
+ "position_embedding_type": "rotary",
24
+ "token_dropout": true,
25
+ "torch_dtype": "float32",
26
+ "transformers_version": "4.26.1",
27
+ "use_cache": true,
28
+ "vocab_list": null,
29
+ "vocab_size": 33
30
+ }
checkpoint-13650/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:47906c5bdfd21f3c1def1fff088870f34a611e9072ba44f92f8945901a5ed84b
3
+ size 5219162243
checkpoint-13650/pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72ea0df553d4c0d01a0dcb8c038c6a302c744033b0b2326f16875b92199db24f
3
+ size 2609524829
checkpoint-13650/rng_state_0.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6a90d7656620c27b5f0a3cc9bf82045add3872287b52b0e4d7a821d9cbc93563
3
+ size 14583
checkpoint-13650/rng_state_1.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cc77f294d04ff7f88102ed8463d22301bd8233d77afd2d90cad8d2babaf94fb6
3
+ size 14583
checkpoint-13650/scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ea5476f54bdb6a23fa5620ad0532c467c086b58ae9fe7eafc8ff2d62a37b1b89
3
+ size 557
checkpoint-13650/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:55022d9dcbbef78486034062d55f0f0896988b0f3ec0147598f1ed7a8e3734d6
3
+ size 627
checkpoint-13650/special_tokens_map.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "<cls>",
3
+ "eos_token": "<eos>",
4
+ "mask_token": "<mask>",
5
+ "pad_token": "<pad>",
6
+ "unk_token": "<unk>"
7
+ }
checkpoint-13650/tokenizer_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "model_max_length": 1000000000000000019884624838656,
3
+ "name_or_path": "facebook/esm2_t33_650M_UR50D",
4
+ "special_tokens_map_file": "/home/avuhong/.cache/huggingface/hub/models--facebook--esm2_t33_650M_UR50D/snapshots/08e4846e537177426273712802403f7ba8261b6c/special_tokens_map.json",
5
+ "tokenizer_class": "EsmTokenizer"
6
+ }
checkpoint-13650/trainer_state.json ADDED
@@ -0,0 +1,493 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 35.0,
5
+ "global_step": 13650,
6
+ "is_hyper_param_search": false,
7
+ "is_local_process_zero": true,
8
+ "is_world_process_zero": true,
9
+ "log_history": [
10
+ {
11
+ "epoch": 1.0,
12
+ "eval_accuracy": 0.5992906078421846,
13
+ "eval_loss": 1.33469557762146,
14
+ "eval_runtime": 5.8169,
15
+ "eval_samples_per_second": 29.397,
16
+ "eval_steps_per_second": 14.785,
17
+ "step": 390
18
+ },
19
+ {
20
+ "epoch": 1.28,
21
+ "learning_rate": 9.644586894586896e-06,
22
+ "loss": 1.5408,
23
+ "step": 500
24
+ },
25
+ {
26
+ "epoch": 2.0,
27
+ "eval_accuracy": 0.6796235478653008,
28
+ "eval_loss": 1.0698518753051758,
29
+ "eval_runtime": 5.8146,
30
+ "eval_samples_per_second": 29.409,
31
+ "eval_steps_per_second": 14.79,
32
+ "step": 780
33
+ },
34
+ {
35
+ "epoch": 2.56,
36
+ "learning_rate": 9.28917378917379e-06,
37
+ "loss": 1.1283,
38
+ "step": 1000
39
+ },
40
+ {
41
+ "epoch": 3.0,
42
+ "eval_accuracy": 0.7372856151482106,
43
+ "eval_loss": 0.8751375675201416,
44
+ "eval_runtime": 5.8077,
45
+ "eval_samples_per_second": 29.444,
46
+ "eval_steps_per_second": 14.808,
47
+ "step": 1170
48
+ },
49
+ {
50
+ "epoch": 3.85,
51
+ "learning_rate": 8.933048433048434e-06,
52
+ "loss": 0.9078,
53
+ "step": 1500
54
+ },
55
+ {
56
+ "epoch": 4.0,
57
+ "eval_accuracy": 0.7711176956331666,
58
+ "eval_loss": 0.7534294128417969,
59
+ "eval_runtime": 5.8229,
60
+ "eval_samples_per_second": 29.367,
61
+ "eval_steps_per_second": 14.769,
62
+ "step": 1560
63
+ },
64
+ {
65
+ "epoch": 5.0,
66
+ "eval_accuracy": 0.802166420482521,
67
+ "eval_loss": 0.6710610389709473,
68
+ "eval_runtime": 5.6976,
69
+ "eval_samples_per_second": 30.013,
70
+ "eval_steps_per_second": 15.094,
71
+ "step": 1950
72
+ },
73
+ {
74
+ "epoch": 5.13,
75
+ "learning_rate": 8.576923076923077e-06,
76
+ "loss": 0.7705,
77
+ "step": 2000
78
+ },
79
+ {
80
+ "epoch": 6.0,
81
+ "eval_accuracy": 0.8168786692759296,
82
+ "eval_loss": 0.6077755689620972,
83
+ "eval_runtime": 5.8114,
84
+ "eval_samples_per_second": 29.425,
85
+ "eval_steps_per_second": 14.798,
86
+ "step": 2340
87
+ },
88
+ {
89
+ "epoch": 6.41,
90
+ "learning_rate": 8.220797720797722e-06,
91
+ "loss": 0.6863,
92
+ "step": 2500
93
+ },
94
+ {
95
+ "epoch": 7.0,
96
+ "eval_accuracy": 0.831802051431627,
97
+ "eval_loss": 0.5667673945426941,
98
+ "eval_runtime": 5.8128,
99
+ "eval_samples_per_second": 29.418,
100
+ "eval_steps_per_second": 14.795,
101
+ "step": 2730
102
+ },
103
+ {
104
+ "epoch": 7.69,
105
+ "learning_rate": 7.864672364672366e-06,
106
+ "loss": 0.6277,
107
+ "step": 3000
108
+ },
109
+ {
110
+ "epoch": 8.0,
111
+ "eval_accuracy": 0.8386034912718204,
112
+ "eval_loss": 0.546131432056427,
113
+ "eval_runtime": 5.8073,
114
+ "eval_samples_per_second": 29.446,
115
+ "eval_steps_per_second": 14.809,
116
+ "step": 3120
117
+ },
118
+ {
119
+ "epoch": 8.97,
120
+ "learning_rate": 7.508547008547009e-06,
121
+ "loss": 0.5863,
122
+ "step": 3500
123
+ },
124
+ {
125
+ "epoch": 9.0,
126
+ "eval_accuracy": 0.8513571943310113,
127
+ "eval_loss": 0.5142761468887329,
128
+ "eval_runtime": 5.8058,
129
+ "eval_samples_per_second": 29.453,
130
+ "eval_steps_per_second": 14.813,
131
+ "step": 3510
132
+ },
133
+ {
134
+ "epoch": 10.0,
135
+ "eval_accuracy": 0.8522174818930449,
136
+ "eval_loss": 0.49920225143432617,
137
+ "eval_runtime": 5.8205,
138
+ "eval_samples_per_second": 29.379,
139
+ "eval_steps_per_second": 14.775,
140
+ "step": 3900
141
+ },
142
+ {
143
+ "epoch": 10.26,
144
+ "learning_rate": 7.152421652421653e-06,
145
+ "loss": 0.5564,
146
+ "step": 4000
147
+ },
148
+ {
149
+ "epoch": 11.0,
150
+ "eval_accuracy": 0.8533209429717304,
151
+ "eval_loss": 0.49400192499160767,
152
+ "eval_runtime": 5.8142,
153
+ "eval_samples_per_second": 29.411,
154
+ "eval_steps_per_second": 14.791,
155
+ "step": 4290
156
+ },
157
+ {
158
+ "epoch": 11.54,
159
+ "learning_rate": 6.796296296296296e-06,
160
+ "loss": 0.5199,
161
+ "step": 4500
162
+ },
163
+ {
164
+ "epoch": 12.0,
165
+ "eval_accuracy": 0.8632922665875019,
166
+ "eval_loss": 0.4726846218109131,
167
+ "eval_runtime": 5.8155,
168
+ "eval_samples_per_second": 29.404,
169
+ "eval_steps_per_second": 14.788,
170
+ "step": 4680
171
+ },
172
+ {
173
+ "epoch": 12.82,
174
+ "learning_rate": 6.440170940170941e-06,
175
+ "loss": 0.5025,
176
+ "step": 5000
177
+ },
178
+ {
179
+ "epoch": 13.0,
180
+ "eval_accuracy": 0.8637540927527733,
181
+ "eval_loss": 0.4585917592048645,
182
+ "eval_runtime": 5.8153,
183
+ "eval_samples_per_second": 29.405,
184
+ "eval_steps_per_second": 14.789,
185
+ "step": 5070
186
+ },
187
+ {
188
+ "epoch": 14.0,
189
+ "eval_accuracy": 0.8673048600883653,
190
+ "eval_loss": 0.45485442876815796,
191
+ "eval_runtime": 5.8178,
192
+ "eval_samples_per_second": 29.393,
193
+ "eval_steps_per_second": 14.782,
194
+ "step": 5460
195
+ },
196
+ {
197
+ "epoch": 14.1,
198
+ "learning_rate": 6.084045584045585e-06,
199
+ "loss": 0.4814,
200
+ "step": 5500
201
+ },
202
+ {
203
+ "epoch": 15.0,
204
+ "eval_accuracy": 0.8698035411108416,
205
+ "eval_loss": 0.44424179196357727,
206
+ "eval_runtime": 5.7973,
207
+ "eval_samples_per_second": 29.496,
208
+ "eval_steps_per_second": 14.834,
209
+ "step": 5850
210
+ },
211
+ {
212
+ "epoch": 15.38,
213
+ "learning_rate": 5.727920227920228e-06,
214
+ "loss": 0.4746,
215
+ "step": 6000
216
+ },
217
+ {
218
+ "epoch": 16.0,
219
+ "eval_accuracy": 0.8749573607523999,
220
+ "eval_loss": 0.43056586384773254,
221
+ "eval_runtime": 5.6925,
222
+ "eval_samples_per_second": 30.04,
223
+ "eval_steps_per_second": 15.108,
224
+ "step": 6240
225
+ },
226
+ {
227
+ "epoch": 16.67,
228
+ "learning_rate": 5.371794871794872e-06,
229
+ "loss": 0.4527,
230
+ "step": 6500
231
+ },
232
+ {
233
+ "epoch": 17.0,
234
+ "eval_accuracy": 0.874227107665129,
235
+ "eval_loss": 0.42905324697494507,
236
+ "eval_runtime": 5.8566,
237
+ "eval_samples_per_second": 29.198,
238
+ "eval_steps_per_second": 14.684,
239
+ "step": 6630
240
+ },
241
+ {
242
+ "epoch": 17.95,
243
+ "learning_rate": 5.016381766381767e-06,
244
+ "loss": 0.4382,
245
+ "step": 7000
246
+ },
247
+ {
248
+ "epoch": 18.0,
249
+ "eval_accuracy": 0.8751032706419789,
250
+ "eval_loss": 0.4213222861289978,
251
+ "eval_runtime": 5.8093,
252
+ "eval_samples_per_second": 29.435,
253
+ "eval_steps_per_second": 14.804,
254
+ "step": 7020
255
+ },
256
+ {
257
+ "epoch": 19.0,
258
+ "eval_accuracy": 0.8751214299591995,
259
+ "eval_loss": 0.41926833987236023,
260
+ "eval_runtime": 5.8022,
261
+ "eval_samples_per_second": 29.472,
262
+ "eval_steps_per_second": 14.822,
263
+ "step": 7410
264
+ },
265
+ {
266
+ "epoch": 19.23,
267
+ "learning_rate": 4.6602564102564106e-06,
268
+ "loss": 0.4328,
269
+ "step": 7500
270
+ },
271
+ {
272
+ "epoch": 20.0,
273
+ "eval_accuracy": 0.8759944995580002,
274
+ "eval_loss": 0.41431769728660583,
275
+ "eval_runtime": 5.8137,
276
+ "eval_samples_per_second": 29.413,
277
+ "eval_steps_per_second": 14.793,
278
+ "step": 7800
279
+ },
280
+ {
281
+ "epoch": 20.51,
282
+ "learning_rate": 4.304131054131054e-06,
283
+ "loss": 0.4191,
284
+ "step": 8000
285
+ },
286
+ {
287
+ "epoch": 21.0,
288
+ "eval_accuracy": 0.8835775987576434,
289
+ "eval_loss": 0.4071265459060669,
290
+ "eval_runtime": 5.8164,
291
+ "eval_samples_per_second": 29.4,
292
+ "eval_steps_per_second": 14.786,
293
+ "step": 8190
294
+ },
295
+ {
296
+ "epoch": 21.79,
297
+ "learning_rate": 3.948717948717949e-06,
298
+ "loss": 0.4106,
299
+ "step": 8500
300
+ },
301
+ {
302
+ "epoch": 22.0,
303
+ "eval_accuracy": 0.881939052795031,
304
+ "eval_loss": 0.3980366587638855,
305
+ "eval_runtime": 5.875,
306
+ "eval_samples_per_second": 29.106,
307
+ "eval_steps_per_second": 14.638,
308
+ "step": 8580
309
+ },
310
+ {
311
+ "epoch": 23.0,
312
+ "eval_accuracy": 0.8821786026625207,
313
+ "eval_loss": 0.39872363209724426,
314
+ "eval_runtime": 5.8105,
315
+ "eval_samples_per_second": 29.43,
316
+ "eval_steps_per_second": 14.801,
317
+ "step": 8970
318
+ },
319
+ {
320
+ "epoch": 23.08,
321
+ "learning_rate": 3.592592592592593e-06,
322
+ "loss": 0.4037,
323
+ "step": 9000
324
+ },
325
+ {
326
+ "epoch": 24.0,
327
+ "eval_accuracy": 0.8819395993222366,
328
+ "eval_loss": 0.40265128016471863,
329
+ "eval_runtime": 5.8012,
330
+ "eval_samples_per_second": 29.477,
331
+ "eval_steps_per_second": 14.824,
332
+ "step": 9360
333
+ },
334
+ {
335
+ "epoch": 24.36,
336
+ "learning_rate": 3.2364672364672365e-06,
337
+ "loss": 0.3893,
338
+ "step": 9500
339
+ },
340
+ {
341
+ "epoch": 25.0,
342
+ "eval_accuracy": 0.8892501819946614,
343
+ "eval_loss": 0.3867790400981903,
344
+ "eval_runtime": 5.8209,
345
+ "eval_samples_per_second": 29.377,
346
+ "eval_steps_per_second": 14.774,
347
+ "step": 9750
348
+ },
349
+ {
350
+ "epoch": 25.64,
351
+ "learning_rate": 2.8803418803418804e-06,
352
+ "loss": 0.3991,
353
+ "step": 10000
354
+ },
355
+ {
356
+ "epoch": 26.0,
357
+ "eval_accuracy": 0.8846003326484688,
358
+ "eval_loss": 0.3882477581501007,
359
+ "eval_runtime": 5.8184,
360
+ "eval_samples_per_second": 29.389,
361
+ "eval_steps_per_second": 14.781,
362
+ "step": 10140
363
+ },
364
+ {
365
+ "epoch": 26.92,
366
+ "learning_rate": 2.5242165242165246e-06,
367
+ "loss": 0.3786,
368
+ "step": 10500
369
+ },
370
+ {
371
+ "epoch": 27.0,
372
+ "eval_accuracy": 0.8858714334822964,
373
+ "eval_loss": 0.3939129710197449,
374
+ "eval_runtime": 5.8222,
375
+ "eval_samples_per_second": 29.37,
376
+ "eval_steps_per_second": 14.771,
377
+ "step": 10530
378
+ },
379
+ {
380
+ "epoch": 28.0,
381
+ "eval_accuracy": 0.8847989764283254,
382
+ "eval_loss": 0.39587706327438354,
383
+ "eval_runtime": 5.8063,
384
+ "eval_samples_per_second": 29.451,
385
+ "eval_steps_per_second": 14.812,
386
+ "step": 10920
387
+ },
388
+ {
389
+ "epoch": 28.21,
390
+ "learning_rate": 2.168803418803419e-06,
391
+ "loss": 0.38,
392
+ "step": 11000
393
+ },
394
+ {
395
+ "epoch": 29.0,
396
+ "eval_accuracy": 0.8849687976020835,
397
+ "eval_loss": 0.3949810862541199,
398
+ "eval_runtime": 5.8138,
399
+ "eval_samples_per_second": 29.413,
400
+ "eval_steps_per_second": 14.792,
401
+ "step": 11310
402
+ },
403
+ {
404
+ "epoch": 29.49,
405
+ "learning_rate": 1.8126780626780629e-06,
406
+ "loss": 0.3764,
407
+ "step": 11500
408
+ },
409
+ {
410
+ "epoch": 30.0,
411
+ "eval_accuracy": 0.8893153879792042,
412
+ "eval_loss": 0.3783101439476013,
413
+ "eval_runtime": 5.8039,
414
+ "eval_samples_per_second": 29.463,
415
+ "eval_steps_per_second": 14.818,
416
+ "step": 11700
417
+ },
418
+ {
419
+ "epoch": 30.77,
420
+ "learning_rate": 1.4565527065527065e-06,
421
+ "loss": 0.3708,
422
+ "step": 12000
423
+ },
424
+ {
425
+ "epoch": 31.0,
426
+ "eval_accuracy": 0.8890792500856626,
427
+ "eval_loss": 0.3798995912075043,
428
+ "eval_runtime": 5.7021,
429
+ "eval_samples_per_second": 29.989,
430
+ "eval_steps_per_second": 15.082,
431
+ "step": 12090
432
+ },
433
+ {
434
+ "epoch": 32.0,
435
+ "eval_accuracy": 0.8867172306495527,
436
+ "eval_loss": 0.39150363206863403,
437
+ "eval_runtime": 5.8791,
438
+ "eval_samples_per_second": 29.086,
439
+ "eval_steps_per_second": 14.628,
440
+ "step": 12480
441
+ },
442
+ {
443
+ "epoch": 32.05,
444
+ "learning_rate": 1.1004273504273506e-06,
445
+ "loss": 0.3656,
446
+ "step": 12500
447
+ },
448
+ {
449
+ "epoch": 33.0,
450
+ "eval_accuracy": 0.8902530694061639,
451
+ "eval_loss": 0.3780055344104767,
452
+ "eval_runtime": 5.8784,
453
+ "eval_samples_per_second": 29.09,
454
+ "eval_steps_per_second": 14.63,
455
+ "step": 12870
456
+ },
457
+ {
458
+ "epoch": 33.33,
459
+ "learning_rate": 7.443019943019944e-07,
460
+ "loss": 0.3617,
461
+ "step": 13000
462
+ },
463
+ {
464
+ "epoch": 34.0,
465
+ "eval_accuracy": 0.8873723487824038,
466
+ "eval_loss": 0.38049712777137756,
467
+ "eval_runtime": 5.8779,
468
+ "eval_samples_per_second": 29.092,
469
+ "eval_steps_per_second": 14.631,
470
+ "step": 13260
471
+ },
472
+ {
473
+ "epoch": 34.62,
474
+ "learning_rate": 3.8817663817663825e-07,
475
+ "loss": 0.361,
476
+ "step": 13500
477
+ },
478
+ {
479
+ "epoch": 35.0,
480
+ "eval_accuracy": 0.8919778767559101,
481
+ "eval_loss": 0.3775971233844757,
482
+ "eval_runtime": 5.8063,
483
+ "eval_samples_per_second": 29.451,
484
+ "eval_steps_per_second": 14.812,
485
+ "step": 13650
486
+ }
487
+ ],
488
+ "max_steps": 14040,
489
+ "num_train_epochs": 36,
490
+ "total_flos": 3.412286906774323e+17,
491
+ "trial_name": null,
492
+ "trial_params": null
493
+ }
checkpoint-13650/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e62f0cca5fa131673e262469dc4be8f9f3fb2048d33c3baf0cda18865991b15f
3
+ size 3579
checkpoint-13650/vocab.txt ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <cls>
2
+ <pad>
3
+ <eos>
4
+ <unk>
5
+ L
6
+ A
7
+ G
8
+ V
9
+ S
10
+ E
11
+ R
12
+ T
13
+ I
14
+ D
15
+ P
16
+ K
17
+ Q
18
+ N
19
+ F
20
+ Y
21
+ M
22
+ H
23
+ W
24
+ C
25
+ X
26
+ B
27
+ U
28
+ Z
29
+ O
30
+ .
31
+ -
32
+ <null_1>
33
+ <mask>
checkpoint-14040/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "facebook/esm2_t33_650M_UR50D",
3
+ "architectures": [
4
+ "EsmForMaskedLM"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.0,
7
+ "classifier_dropout": null,
8
+ "emb_layer_norm_before": false,
9
+ "esmfold_config": null,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.0,
12
+ "hidden_size": 1280,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 5120,
15
+ "is_folding_model": false,
16
+ "layer_norm_eps": 1e-05,
17
+ "mask_token_id": 32,
18
+ "max_position_embeddings": 1026,
19
+ "model_type": "esm",
20
+ "num_attention_heads": 20,
21
+ "num_hidden_layers": 33,
22
+ "pad_token_id": 1,
23
+ "position_embedding_type": "rotary",
24
+ "token_dropout": true,
25
+ "torch_dtype": "float32",
26
+ "transformers_version": "4.26.1",
27
+ "use_cache": true,
28
+ "vocab_list": null,
29
+ "vocab_size": 33
30
+ }
checkpoint-14040/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:74539603eff61ec2403f3fa04875cce68c8f7c190776b90de13c53f01dff31e2
3
+ size 5219162243
checkpoint-14040/pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a1f288729c792c2f6a5e683c9206c7652333c3fbc7bdae07606ec895c87b5a03
3
+ size 2609524829
checkpoint-14040/rng_state_0.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:537ade68a6593552161018bf2051b9024127d35978be29ce8334872cdba5fa4a
3
+ size 14583
checkpoint-14040/rng_state_1.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15dbfdcf2c44f910e294ab5ba8b338eddffa03ec039d7237837bd239d5a739c1
3
+ size 14583
checkpoint-14040/scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d74e9e7f7e2d62262547d85c561b50718c1cb2c1ab608d095515ea413ac7b069
3
+ size 557
checkpoint-14040/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:275a4bdd257f5c2b2e9c249486eb13b8c14a0a886a1a497af5c19b0dfe32cfd8
3
+ size 627
checkpoint-14040/special_tokens_map.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "<cls>",
3
+ "eos_token": "<eos>",
4
+ "mask_token": "<mask>",
5
+ "pad_token": "<pad>",
6
+ "unk_token": "<unk>"
7
+ }
checkpoint-14040/tokenizer_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "model_max_length": 1000000000000000019884624838656,
3
+ "name_or_path": "facebook/esm2_t33_650M_UR50D",
4
+ "special_tokens_map_file": "/home/avuhong/.cache/huggingface/hub/models--facebook--esm2_t33_650M_UR50D/snapshots/08e4846e537177426273712802403f7ba8261b6c/special_tokens_map.json",
5
+ "tokenizer_class": "EsmTokenizer"
6
+ }
checkpoint-14040/trainer_state.json ADDED
@@ -0,0 +1,508 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 36.0,
5
+ "global_step": 14040,
6
+ "is_hyper_param_search": false,
7
+ "is_local_process_zero": true,
8
+ "is_world_process_zero": true,
9
+ "log_history": [
10
+ {
11
+ "epoch": 1.0,
12
+ "eval_accuracy": 0.5992906078421846,
13
+ "eval_loss": 1.33469557762146,
14
+ "eval_runtime": 5.8169,
15
+ "eval_samples_per_second": 29.397,
16
+ "eval_steps_per_second": 14.785,
17
+ "step": 390
18
+ },
19
+ {
20
+ "epoch": 1.28,
21
+ "learning_rate": 9.644586894586896e-06,
22
+ "loss": 1.5408,
23
+ "step": 500
24
+ },
25
+ {
26
+ "epoch": 2.0,
27
+ "eval_accuracy": 0.6796235478653008,
28
+ "eval_loss": 1.0698518753051758,
29
+ "eval_runtime": 5.8146,
30
+ "eval_samples_per_second": 29.409,
31
+ "eval_steps_per_second": 14.79,
32
+ "step": 780
33
+ },
34
+ {
35
+ "epoch": 2.56,
36
+ "learning_rate": 9.28917378917379e-06,
37
+ "loss": 1.1283,
38
+ "step": 1000
39
+ },
40
+ {
41
+ "epoch": 3.0,
42
+ "eval_accuracy": 0.7372856151482106,
43
+ "eval_loss": 0.8751375675201416,
44
+ "eval_runtime": 5.8077,
45
+ "eval_samples_per_second": 29.444,
46
+ "eval_steps_per_second": 14.808,
47
+ "step": 1170
48
+ },
49
+ {
50
+ "epoch": 3.85,
51
+ "learning_rate": 8.933048433048434e-06,
52
+ "loss": 0.9078,
53
+ "step": 1500
54
+ },
55
+ {
56
+ "epoch": 4.0,
57
+ "eval_accuracy": 0.7711176956331666,
58
+ "eval_loss": 0.7534294128417969,
59
+ "eval_runtime": 5.8229,
60
+ "eval_samples_per_second": 29.367,
61
+ "eval_steps_per_second": 14.769,
62
+ "step": 1560
63
+ },
64
+ {
65
+ "epoch": 5.0,
66
+ "eval_accuracy": 0.802166420482521,
67
+ "eval_loss": 0.6710610389709473,
68
+ "eval_runtime": 5.6976,
69
+ "eval_samples_per_second": 30.013,
70
+ "eval_steps_per_second": 15.094,
71
+ "step": 1950
72
+ },
73
+ {
74
+ "epoch": 5.13,
75
+ "learning_rate": 8.576923076923077e-06,
76
+ "loss": 0.7705,
77
+ "step": 2000
78
+ },
79
+ {
80
+ "epoch": 6.0,
81
+ "eval_accuracy": 0.8168786692759296,
82
+ "eval_loss": 0.6077755689620972,
83
+ "eval_runtime": 5.8114,
84
+ "eval_samples_per_second": 29.425,
85
+ "eval_steps_per_second": 14.798,
86
+ "step": 2340
87
+ },
88
+ {
89
+ "epoch": 6.41,
90
+ "learning_rate": 8.220797720797722e-06,
91
+ "loss": 0.6863,
92
+ "step": 2500
93
+ },
94
+ {
95
+ "epoch": 7.0,
96
+ "eval_accuracy": 0.831802051431627,
97
+ "eval_loss": 0.5667673945426941,
98
+ "eval_runtime": 5.8128,
99
+ "eval_samples_per_second": 29.418,
100
+ "eval_steps_per_second": 14.795,
101
+ "step": 2730
102
+ },
103
+ {
104
+ "epoch": 7.69,
105
+ "learning_rate": 7.864672364672366e-06,
106
+ "loss": 0.6277,
107
+ "step": 3000
108
+ },
109
+ {
110
+ "epoch": 8.0,
111
+ "eval_accuracy": 0.8386034912718204,
112
+ "eval_loss": 0.546131432056427,
113
+ "eval_runtime": 5.8073,
114
+ "eval_samples_per_second": 29.446,
115
+ "eval_steps_per_second": 14.809,
116
+ "step": 3120
117
+ },
118
+ {
119
+ "epoch": 8.97,
120
+ "learning_rate": 7.508547008547009e-06,
121
+ "loss": 0.5863,
122
+ "step": 3500
123
+ },
124
+ {
125
+ "epoch": 9.0,
126
+ "eval_accuracy": 0.8513571943310113,
127
+ "eval_loss": 0.5142761468887329,
128
+ "eval_runtime": 5.8058,
129
+ "eval_samples_per_second": 29.453,
130
+ "eval_steps_per_second": 14.813,
131
+ "step": 3510
132
+ },
133
+ {
134
+ "epoch": 10.0,
135
+ "eval_accuracy": 0.8522174818930449,
136
+ "eval_loss": 0.49920225143432617,
137
+ "eval_runtime": 5.8205,
138
+ "eval_samples_per_second": 29.379,
139
+ "eval_steps_per_second": 14.775,
140
+ "step": 3900
141
+ },
142
+ {
143
+ "epoch": 10.26,
144
+ "learning_rate": 7.152421652421653e-06,
145
+ "loss": 0.5564,
146
+ "step": 4000
147
+ },
148
+ {
149
+ "epoch": 11.0,
150
+ "eval_accuracy": 0.8533209429717304,
151
+ "eval_loss": 0.49400192499160767,
152
+ "eval_runtime": 5.8142,
153
+ "eval_samples_per_second": 29.411,
154
+ "eval_steps_per_second": 14.791,
155
+ "step": 4290
156
+ },
157
+ {
158
+ "epoch": 11.54,
159
+ "learning_rate": 6.796296296296296e-06,
160
+ "loss": 0.5199,
161
+ "step": 4500
162
+ },
163
+ {
164
+ "epoch": 12.0,
165
+ "eval_accuracy": 0.8632922665875019,
166
+ "eval_loss": 0.4726846218109131,
167
+ "eval_runtime": 5.8155,
168
+ "eval_samples_per_second": 29.404,
169
+ "eval_steps_per_second": 14.788,
170
+ "step": 4680
171
+ },
172
+ {
173
+ "epoch": 12.82,
174
+ "learning_rate": 6.440170940170941e-06,
175
+ "loss": 0.5025,
176
+ "step": 5000
177
+ },
178
+ {
179
+ "epoch": 13.0,
180
+ "eval_accuracy": 0.8637540927527733,
181
+ "eval_loss": 0.4585917592048645,
182
+ "eval_runtime": 5.8153,
183
+ "eval_samples_per_second": 29.405,
184
+ "eval_steps_per_second": 14.789,
185
+ "step": 5070
186
+ },
187
+ {
188
+ "epoch": 14.0,
189
+ "eval_accuracy": 0.8673048600883653,
190
+ "eval_loss": 0.45485442876815796,
191
+ "eval_runtime": 5.8178,
192
+ "eval_samples_per_second": 29.393,
193
+ "eval_steps_per_second": 14.782,
194
+ "step": 5460
195
+ },
196
+ {
197
+ "epoch": 14.1,
198
+ "learning_rate": 6.084045584045585e-06,
199
+ "loss": 0.4814,
200
+ "step": 5500
201
+ },
202
+ {
203
+ "epoch": 15.0,
204
+ "eval_accuracy": 0.8698035411108416,
205
+ "eval_loss": 0.44424179196357727,
206
+ "eval_runtime": 5.7973,
207
+ "eval_samples_per_second": 29.496,
208
+ "eval_steps_per_second": 14.834,
209
+ "step": 5850
210
+ },
211
+ {
212
+ "epoch": 15.38,
213
+ "learning_rate": 5.727920227920228e-06,
214
+ "loss": 0.4746,
215
+ "step": 6000
216
+ },
217
+ {
218
+ "epoch": 16.0,
219
+ "eval_accuracy": 0.8749573607523999,
220
+ "eval_loss": 0.43056586384773254,
221
+ "eval_runtime": 5.6925,
222
+ "eval_samples_per_second": 30.04,
223
+ "eval_steps_per_second": 15.108,
224
+ "step": 6240
225
+ },
226
+ {
227
+ "epoch": 16.67,
228
+ "learning_rate": 5.371794871794872e-06,
229
+ "loss": 0.4527,
230
+ "step": 6500
231
+ },
232
+ {
233
+ "epoch": 17.0,
234
+ "eval_accuracy": 0.874227107665129,
235
+ "eval_loss": 0.42905324697494507,
236
+ "eval_runtime": 5.8566,
237
+ "eval_samples_per_second": 29.198,
238
+ "eval_steps_per_second": 14.684,
239
+ "step": 6630
240
+ },
241
+ {
242
+ "epoch": 17.95,
243
+ "learning_rate": 5.016381766381767e-06,
244
+ "loss": 0.4382,
245
+ "step": 7000
246
+ },
247
+ {
248
+ "epoch": 18.0,
249
+ "eval_accuracy": 0.8751032706419789,
250
+ "eval_loss": 0.4213222861289978,
251
+ "eval_runtime": 5.8093,
252
+ "eval_samples_per_second": 29.435,
253
+ "eval_steps_per_second": 14.804,
254
+ "step": 7020
255
+ },
256
+ {
257
+ "epoch": 19.0,
258
+ "eval_accuracy": 0.8751214299591995,
259
+ "eval_loss": 0.41926833987236023,
260
+ "eval_runtime": 5.8022,
261
+ "eval_samples_per_second": 29.472,
262
+ "eval_steps_per_second": 14.822,
263
+ "step": 7410
264
+ },
265
+ {
266
+ "epoch": 19.23,
267
+ "learning_rate": 4.6602564102564106e-06,
268
+ "loss": 0.4328,
269
+ "step": 7500
270
+ },
271
+ {
272
+ "epoch": 20.0,
273
+ "eval_accuracy": 0.8759944995580002,
274
+ "eval_loss": 0.41431769728660583,
275
+ "eval_runtime": 5.8137,
276
+ "eval_samples_per_second": 29.413,
277
+ "eval_steps_per_second": 14.793,
278
+ "step": 7800
279
+ },
280
+ {
281
+ "epoch": 20.51,
282
+ "learning_rate": 4.304131054131054e-06,
283
+ "loss": 0.4191,
284
+ "step": 8000
285
+ },
286
+ {
287
+ "epoch": 21.0,
288
+ "eval_accuracy": 0.8835775987576434,
289
+ "eval_loss": 0.4071265459060669,
290
+ "eval_runtime": 5.8164,
291
+ "eval_samples_per_second": 29.4,
292
+ "eval_steps_per_second": 14.786,
293
+ "step": 8190
294
+ },
295
+ {
296
+ "epoch": 21.79,
297
+ "learning_rate": 3.948717948717949e-06,
298
+ "loss": 0.4106,
299
+ "step": 8500
300
+ },
301
+ {
302
+ "epoch": 22.0,
303
+ "eval_accuracy": 0.881939052795031,
304
+ "eval_loss": 0.3980366587638855,
305
+ "eval_runtime": 5.875,
306
+ "eval_samples_per_second": 29.106,
307
+ "eval_steps_per_second": 14.638,
308
+ "step": 8580
309
+ },
310
+ {
311
+ "epoch": 23.0,
312
+ "eval_accuracy": 0.8821786026625207,
313
+ "eval_loss": 0.39872363209724426,
314
+ "eval_runtime": 5.8105,
315
+ "eval_samples_per_second": 29.43,
316
+ "eval_steps_per_second": 14.801,
317
+ "step": 8970
318
+ },
319
+ {
320
+ "epoch": 23.08,
321
+ "learning_rate": 3.592592592592593e-06,
322
+ "loss": 0.4037,
323
+ "step": 9000
324
+ },
325
+ {
326
+ "epoch": 24.0,
327
+ "eval_accuracy": 0.8819395993222366,
328
+ "eval_loss": 0.40265128016471863,
329
+ "eval_runtime": 5.8012,
330
+ "eval_samples_per_second": 29.477,
331
+ "eval_steps_per_second": 14.824,
332
+ "step": 9360
333
+ },
334
+ {
335
+ "epoch": 24.36,
336
+ "learning_rate": 3.2364672364672365e-06,
337
+ "loss": 0.3893,
338
+ "step": 9500
339
+ },
340
+ {
341
+ "epoch": 25.0,
342
+ "eval_accuracy": 0.8892501819946614,
343
+ "eval_loss": 0.3867790400981903,
344
+ "eval_runtime": 5.8209,
345
+ "eval_samples_per_second": 29.377,
346
+ "eval_steps_per_second": 14.774,
347
+ "step": 9750
348
+ },
349
+ {
350
+ "epoch": 25.64,
351
+ "learning_rate": 2.8803418803418804e-06,
352
+ "loss": 0.3991,
353
+ "step": 10000
354
+ },
355
+ {
356
+ "epoch": 26.0,
357
+ "eval_accuracy": 0.8846003326484688,
358
+ "eval_loss": 0.3882477581501007,
359
+ "eval_runtime": 5.8184,
360
+ "eval_samples_per_second": 29.389,
361
+ "eval_steps_per_second": 14.781,
362
+ "step": 10140
363
+ },
364
+ {
365
+ "epoch": 26.92,
366
+ "learning_rate": 2.5242165242165246e-06,
367
+ "loss": 0.3786,
368
+ "step": 10500
369
+ },
370
+ {
371
+ "epoch": 27.0,
372
+ "eval_accuracy": 0.8858714334822964,
373
+ "eval_loss": 0.3939129710197449,
374
+ "eval_runtime": 5.8222,
375
+ "eval_samples_per_second": 29.37,
376
+ "eval_steps_per_second": 14.771,
377
+ "step": 10530
378
+ },
379
+ {
380
+ "epoch": 28.0,
381
+ "eval_accuracy": 0.8847989764283254,
382
+ "eval_loss": 0.39587706327438354,
383
+ "eval_runtime": 5.8063,
384
+ "eval_samples_per_second": 29.451,
385
+ "eval_steps_per_second": 14.812,
386
+ "step": 10920
387
+ },
388
+ {
389
+ "epoch": 28.21,
390
+ "learning_rate": 2.168803418803419e-06,
391
+ "loss": 0.38,
392
+ "step": 11000
393
+ },
394
+ {
395
+ "epoch": 29.0,
396
+ "eval_accuracy": 0.8849687976020835,
397
+ "eval_loss": 0.3949810862541199,
398
+ "eval_runtime": 5.8138,
399
+ "eval_samples_per_second": 29.413,
400
+ "eval_steps_per_second": 14.792,
401
+ "step": 11310
402
+ },
403
+ {
404
+ "epoch": 29.49,
405
+ "learning_rate": 1.8126780626780629e-06,
406
+ "loss": 0.3764,
407
+ "step": 11500
408
+ },
409
+ {
410
+ "epoch": 30.0,
411
+ "eval_accuracy": 0.8893153879792042,
412
+ "eval_loss": 0.3783101439476013,
413
+ "eval_runtime": 5.8039,
414
+ "eval_samples_per_second": 29.463,
415
+ "eval_steps_per_second": 14.818,
416
+ "step": 11700
417
+ },
418
+ {
419
+ "epoch": 30.77,
420
+ "learning_rate": 1.4565527065527065e-06,
421
+ "loss": 0.3708,
422
+ "step": 12000
423
+ },
424
+ {
425
+ "epoch": 31.0,
426
+ "eval_accuracy": 0.8890792500856626,
427
+ "eval_loss": 0.3798995912075043,
428
+ "eval_runtime": 5.7021,
429
+ "eval_samples_per_second": 29.989,
430
+ "eval_steps_per_second": 15.082,
431
+ "step": 12090
432
+ },
433
+ {
434
+ "epoch": 32.0,
435
+ "eval_accuracy": 0.8867172306495527,
436
+ "eval_loss": 0.39150363206863403,
437
+ "eval_runtime": 5.8791,
438
+ "eval_samples_per_second": 29.086,
439
+ "eval_steps_per_second": 14.628,
440
+ "step": 12480
441
+ },
442
+ {
443
+ "epoch": 32.05,
444
+ "learning_rate": 1.1004273504273506e-06,
445
+ "loss": 0.3656,
446
+ "step": 12500
447
+ },
448
+ {
449
+ "epoch": 33.0,
450
+ "eval_accuracy": 0.8902530694061639,
451
+ "eval_loss": 0.3780055344104767,
452
+ "eval_runtime": 5.8784,
453
+ "eval_samples_per_second": 29.09,
454
+ "eval_steps_per_second": 14.63,
455
+ "step": 12870
456
+ },
457
+ {
458
+ "epoch": 33.33,
459
+ "learning_rate": 7.443019943019944e-07,
460
+ "loss": 0.3617,
461
+ "step": 13000
462
+ },
463
+ {
464
+ "epoch": 34.0,
465
+ "eval_accuracy": 0.8873723487824038,
466
+ "eval_loss": 0.38049712777137756,
467
+ "eval_runtime": 5.8779,
468
+ "eval_samples_per_second": 29.092,
469
+ "eval_steps_per_second": 14.631,
470
+ "step": 13260
471
+ },
472
+ {
473
+ "epoch": 34.62,
474
+ "learning_rate": 3.8817663817663825e-07,
475
+ "loss": 0.361,
476
+ "step": 13500
477
+ },
478
+ {
479
+ "epoch": 35.0,
480
+ "eval_accuracy": 0.8919778767559101,
481
+ "eval_loss": 0.3775971233844757,
482
+ "eval_runtime": 5.8063,
483
+ "eval_samples_per_second": 29.451,
484
+ "eval_steps_per_second": 14.812,
485
+ "step": 13650
486
+ },
487
+ {
488
+ "epoch": 35.9,
489
+ "learning_rate": 3.205128205128205e-08,
490
+ "loss": 0.3595,
491
+ "step": 14000
492
+ },
493
+ {
494
+ "epoch": 36.0,
495
+ "eval_accuracy": 0.8888084202394747,
496
+ "eval_loss": 0.3711872100830078,
497
+ "eval_runtime": 5.8621,
498
+ "eval_samples_per_second": 29.171,
499
+ "eval_steps_per_second": 14.671,
500
+ "step": 14040
501
+ }
502
+ ],
503
+ "max_steps": 14040,
504
+ "num_train_epochs": 36,
505
+ "total_flos": 3.509780816886497e+17,
506
+ "trial_name": null,
507
+ "trial_params": null
508
+ }
checkpoint-14040/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e62f0cca5fa131673e262469dc4be8f9f3fb2048d33c3baf0cda18865991b15f
3
+ size 3579
checkpoint-14040/vocab.txt ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <cls>
2
+ <pad>
3
+ <eos>
4
+ <unk>
5
+ L
6
+ A
7
+ G
8
+ V
9
+ S
10
+ E
11
+ R
12
+ T
13
+ I
14
+ D
15
+ P
16
+ K
17
+ Q
18
+ N
19
+ F
20
+ Y
21
+ M
22
+ H
23
+ W
24
+ C
25
+ X
26
+ B
27
+ U
28
+ Z
29
+ O
30
+ .
31
+ -
32
+ <null_1>
33
+ <mask>
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "facebook/esm2_t33_650M_UR50D",
3
+ "architectures": [
4
+ "EsmForMaskedLM"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.0,
7
+ "classifier_dropout": null,
8
+ "emb_layer_norm_before": false,
9
+ "esmfold_config": null,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.0,
12
+ "hidden_size": 1280,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 5120,
15
+ "is_folding_model": false,
16
+ "layer_norm_eps": 1e-05,
17
+ "mask_token_id": 32,
18
+ "max_position_embeddings": 1026,
19
+ "model_type": "esm",
20
+ "num_attention_heads": 20,
21
+ "num_hidden_layers": 33,
22
+ "pad_token_id": 1,
23
+ "position_embedding_type": "rotary",
24
+ "token_dropout": true,
25
+ "torch_dtype": "float32",
26
+ "transformers_version": "4.26.1",
27
+ "use_cache": true,
28
+ "vocab_list": null,
29
+ "vocab_size": 33
30
+ }
eval_results.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 36.0,
3
+ "eval_accuracy": 0.8905318152693578,
4
+ "eval_loss": 0.3707260489463806,
5
+ "eval_runtime": 5.8253,
6
+ "eval_samples": 171,
7
+ "eval_samples_per_second": 29.355,
8
+ "eval_steps_per_second": 14.763,
9
+ "perplexity": 1.448786122531838
10
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a1f288729c792c2f6a5e683c9206c7652333c3fbc7bdae07606ec895c87b5a03
3
+ size 2609524829
runs/Apr26_12-00-12_srvgpu/1682510421.8712294/events.out.tfevents.1682510421.srvgpu.3077809.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:91cf0149c9cdf0c5878eea2813b3bc0b71f0577fc423468e593e5588cee6aff3
3
+ size 5688
runs/Apr26_12-00-12_srvgpu/events.out.tfevents.1682510421.srvgpu.3077809.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a4765859d81060ca2c805656dcd79226754725821890cc8b27f964a16487e0b6
3
+ size 20340
runs/Apr26_12-00-12_srvgpu/events.out.tfevents.1682524144.srvgpu.3077809.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5dbeb63262a0cd34567d2e14c7ab41e7712ba596938b712994795fae04637926
3
+ size 363
special_tokens_map.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "<cls>",
3
+ "eos_token": "<eos>",
4
+ "mask_token": "<mask>",
5
+ "pad_token": "<pad>",
6
+ "unk_token": "<unk>"
7
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "model_max_length": 1000000000000000019884624838656,
3
+ "name_or_path": "facebook/esm2_t33_650M_UR50D",
4
+ "special_tokens_map_file": "/home/avuhong/.cache/huggingface/hub/models--facebook--esm2_t33_650M_UR50D/snapshots/08e4846e537177426273712802403f7ba8261b6c/special_tokens_map.json",
5
+ "tokenizer_class": "EsmTokenizer"
6
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 36.0,
3
+ "train_loss": 0.5379996796958467,
4
+ "train_runtime": 13714.1823,
5
+ "train_samples": 3120,
6
+ "train_samples_per_second": 8.19,
7
+ "train_steps_per_second": 1.024
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,517 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 36.0,
5
+ "global_step": 14040,
6
+ "is_hyper_param_search": false,
7
+ "is_local_process_zero": true,
8
+ "is_world_process_zero": true,
9
+ "log_history": [
10
+ {
11
+ "epoch": 1.0,
12
+ "eval_accuracy": 0.5992906078421846,
13
+ "eval_loss": 1.33469557762146,
14
+ "eval_runtime": 5.8169,
15
+ "eval_samples_per_second": 29.397,
16
+ "eval_steps_per_second": 14.785,
17
+ "step": 390
18
+ },
19
+ {
20
+ "epoch": 1.28,
21
+ "learning_rate": 9.644586894586896e-06,
22
+ "loss": 1.5408,
23
+ "step": 500
24
+ },
25
+ {
26
+ "epoch": 2.0,
27
+ "eval_accuracy": 0.6796235478653008,
28
+ "eval_loss": 1.0698518753051758,
29
+ "eval_runtime": 5.8146,
30
+ "eval_samples_per_second": 29.409,
31
+ "eval_steps_per_second": 14.79,
32
+ "step": 780
33
+ },
34
+ {
35
+ "epoch": 2.56,
36
+ "learning_rate": 9.28917378917379e-06,
37
+ "loss": 1.1283,
38
+ "step": 1000
39
+ },
40
+ {
41
+ "epoch": 3.0,
42
+ "eval_accuracy": 0.7372856151482106,
43
+ "eval_loss": 0.8751375675201416,
44
+ "eval_runtime": 5.8077,
45
+ "eval_samples_per_second": 29.444,
46
+ "eval_steps_per_second": 14.808,
47
+ "step": 1170
48
+ },
49
+ {
50
+ "epoch": 3.85,
51
+ "learning_rate": 8.933048433048434e-06,
52
+ "loss": 0.9078,
53
+ "step": 1500
54
+ },
55
+ {
56
+ "epoch": 4.0,
57
+ "eval_accuracy": 0.7711176956331666,
58
+ "eval_loss": 0.7534294128417969,
59
+ "eval_runtime": 5.8229,
60
+ "eval_samples_per_second": 29.367,
61
+ "eval_steps_per_second": 14.769,
62
+ "step": 1560
63
+ },
64
+ {
65
+ "epoch": 5.0,
66
+ "eval_accuracy": 0.802166420482521,
67
+ "eval_loss": 0.6710610389709473,
68
+ "eval_runtime": 5.6976,
69
+ "eval_samples_per_second": 30.013,
70
+ "eval_steps_per_second": 15.094,
71
+ "step": 1950
72
+ },
73
+ {
74
+ "epoch": 5.13,
75
+ "learning_rate": 8.576923076923077e-06,
76
+ "loss": 0.7705,
77
+ "step": 2000
78
+ },
79
+ {
80
+ "epoch": 6.0,
81
+ "eval_accuracy": 0.8168786692759296,
82
+ "eval_loss": 0.6077755689620972,
83
+ "eval_runtime": 5.8114,
84
+ "eval_samples_per_second": 29.425,
85
+ "eval_steps_per_second": 14.798,
86
+ "step": 2340
87
+ },
88
+ {
89
+ "epoch": 6.41,
90
+ "learning_rate": 8.220797720797722e-06,
91
+ "loss": 0.6863,
92
+ "step": 2500
93
+ },
94
+ {
95
+ "epoch": 7.0,
96
+ "eval_accuracy": 0.831802051431627,
97
+ "eval_loss": 0.5667673945426941,
98
+ "eval_runtime": 5.8128,
99
+ "eval_samples_per_second": 29.418,
100
+ "eval_steps_per_second": 14.795,
101
+ "step": 2730
102
+ },
103
+ {
104
+ "epoch": 7.69,
105
+ "learning_rate": 7.864672364672366e-06,
106
+ "loss": 0.6277,
107
+ "step": 3000
108
+ },
109
+ {
110
+ "epoch": 8.0,
111
+ "eval_accuracy": 0.8386034912718204,
112
+ "eval_loss": 0.546131432056427,
113
+ "eval_runtime": 5.8073,
114
+ "eval_samples_per_second": 29.446,
115
+ "eval_steps_per_second": 14.809,
116
+ "step": 3120
117
+ },
118
+ {
119
+ "epoch": 8.97,
120
+ "learning_rate": 7.508547008547009e-06,
121
+ "loss": 0.5863,
122
+ "step": 3500
123
+ },
124
+ {
125
+ "epoch": 9.0,
126
+ "eval_accuracy": 0.8513571943310113,
127
+ "eval_loss": 0.5142761468887329,
128
+ "eval_runtime": 5.8058,
129
+ "eval_samples_per_second": 29.453,
130
+ "eval_steps_per_second": 14.813,
131
+ "step": 3510
132
+ },
133
+ {
134
+ "epoch": 10.0,
135
+ "eval_accuracy": 0.8522174818930449,
136
+ "eval_loss": 0.49920225143432617,
137
+ "eval_runtime": 5.8205,
138
+ "eval_samples_per_second": 29.379,
139
+ "eval_steps_per_second": 14.775,
140
+ "step": 3900
141
+ },
142
+ {
143
+ "epoch": 10.26,
144
+ "learning_rate": 7.152421652421653e-06,
145
+ "loss": 0.5564,
146
+ "step": 4000
147
+ },
148
+ {
149
+ "epoch": 11.0,
150
+ "eval_accuracy": 0.8533209429717304,
151
+ "eval_loss": 0.49400192499160767,
152
+ "eval_runtime": 5.8142,
153
+ "eval_samples_per_second": 29.411,
154
+ "eval_steps_per_second": 14.791,
155
+ "step": 4290
156
+ },
157
+ {
158
+ "epoch": 11.54,
159
+ "learning_rate": 6.796296296296296e-06,
160
+ "loss": 0.5199,
161
+ "step": 4500
162
+ },
163
+ {
164
+ "epoch": 12.0,
165
+ "eval_accuracy": 0.8632922665875019,
166
+ "eval_loss": 0.4726846218109131,
167
+ "eval_runtime": 5.8155,
168
+ "eval_samples_per_second": 29.404,
169
+ "eval_steps_per_second": 14.788,
170
+ "step": 4680
171
+ },
172
+ {
173
+ "epoch": 12.82,
174
+ "learning_rate": 6.440170940170941e-06,
175
+ "loss": 0.5025,
176
+ "step": 5000
177
+ },
178
+ {
179
+ "epoch": 13.0,
180
+ "eval_accuracy": 0.8637540927527733,
181
+ "eval_loss": 0.4585917592048645,
182
+ "eval_runtime": 5.8153,
183
+ "eval_samples_per_second": 29.405,
184
+ "eval_steps_per_second": 14.789,
185
+ "step": 5070
186
+ },
187
+ {
188
+ "epoch": 14.0,
189
+ "eval_accuracy": 0.8673048600883653,
190
+ "eval_loss": 0.45485442876815796,
191
+ "eval_runtime": 5.8178,
192
+ "eval_samples_per_second": 29.393,
193
+ "eval_steps_per_second": 14.782,
194
+ "step": 5460
195
+ },
196
+ {
197
+ "epoch": 14.1,
198
+ "learning_rate": 6.084045584045585e-06,
199
+ "loss": 0.4814,
200
+ "step": 5500
201
+ },
202
+ {
203
+ "epoch": 15.0,
204
+ "eval_accuracy": 0.8698035411108416,
205
+ "eval_loss": 0.44424179196357727,
206
+ "eval_runtime": 5.7973,
207
+ "eval_samples_per_second": 29.496,
208
+ "eval_steps_per_second": 14.834,
209
+ "step": 5850
210
+ },
211
+ {
212
+ "epoch": 15.38,
213
+ "learning_rate": 5.727920227920228e-06,
214
+ "loss": 0.4746,
215
+ "step": 6000
216
+ },
217
+ {
218
+ "epoch": 16.0,
219
+ "eval_accuracy": 0.8749573607523999,
220
+ "eval_loss": 0.43056586384773254,
221
+ "eval_runtime": 5.6925,
222
+ "eval_samples_per_second": 30.04,
223
+ "eval_steps_per_second": 15.108,
224
+ "step": 6240
225
+ },
226
+ {
227
+ "epoch": 16.67,
228
+ "learning_rate": 5.371794871794872e-06,
229
+ "loss": 0.4527,
230
+ "step": 6500
231
+ },
232
+ {
233
+ "epoch": 17.0,
234
+ "eval_accuracy": 0.874227107665129,
235
+ "eval_loss": 0.42905324697494507,
236
+ "eval_runtime": 5.8566,
237
+ "eval_samples_per_second": 29.198,
238
+ "eval_steps_per_second": 14.684,
239
+ "step": 6630
240
+ },
241
+ {
242
+ "epoch": 17.95,
243
+ "learning_rate": 5.016381766381767e-06,
244
+ "loss": 0.4382,
245
+ "step": 7000
246
+ },
247
+ {
248
+ "epoch": 18.0,
249
+ "eval_accuracy": 0.8751032706419789,
250
+ "eval_loss": 0.4213222861289978,
251
+ "eval_runtime": 5.8093,
252
+ "eval_samples_per_second": 29.435,
253
+ "eval_steps_per_second": 14.804,
254
+ "step": 7020
255
+ },
256
+ {
257
+ "epoch": 19.0,
258
+ "eval_accuracy": 0.8751214299591995,
259
+ "eval_loss": 0.41926833987236023,
260
+ "eval_runtime": 5.8022,
261
+ "eval_samples_per_second": 29.472,
262
+ "eval_steps_per_second": 14.822,
263
+ "step": 7410
264
+ },
265
+ {
266
+ "epoch": 19.23,
267
+ "learning_rate": 4.6602564102564106e-06,
268
+ "loss": 0.4328,
269
+ "step": 7500
270
+ },
271
+ {
272
+ "epoch": 20.0,
273
+ "eval_accuracy": 0.8759944995580002,
274
+ "eval_loss": 0.41431769728660583,
275
+ "eval_runtime": 5.8137,
276
+ "eval_samples_per_second": 29.413,
277
+ "eval_steps_per_second": 14.793,
278
+ "step": 7800
279
+ },
280
+ {
281
+ "epoch": 20.51,
282
+ "learning_rate": 4.304131054131054e-06,
283
+ "loss": 0.4191,
284
+ "step": 8000
285
+ },
286
+ {
287
+ "epoch": 21.0,
288
+ "eval_accuracy": 0.8835775987576434,
289
+ "eval_loss": 0.4071265459060669,
290
+ "eval_runtime": 5.8164,
291
+ "eval_samples_per_second": 29.4,
292
+ "eval_steps_per_second": 14.786,
293
+ "step": 8190
294
+ },
295
+ {
296
+ "epoch": 21.79,
297
+ "learning_rate": 3.948717948717949e-06,
298
+ "loss": 0.4106,
299
+ "step": 8500
300
+ },
301
+ {
302
+ "epoch": 22.0,
303
+ "eval_accuracy": 0.881939052795031,
304
+ "eval_loss": 0.3980366587638855,
305
+ "eval_runtime": 5.875,
306
+ "eval_samples_per_second": 29.106,
307
+ "eval_steps_per_second": 14.638,
308
+ "step": 8580
309
+ },
310
+ {
311
+ "epoch": 23.0,
312
+ "eval_accuracy": 0.8821786026625207,
313
+ "eval_loss": 0.39872363209724426,
314
+ "eval_runtime": 5.8105,
315
+ "eval_samples_per_second": 29.43,
316
+ "eval_steps_per_second": 14.801,
317
+ "step": 8970
318
+ },
319
+ {
320
+ "epoch": 23.08,
321
+ "learning_rate": 3.592592592592593e-06,
322
+ "loss": 0.4037,
323
+ "step": 9000
324
+ },
325
+ {
326
+ "epoch": 24.0,
327
+ "eval_accuracy": 0.8819395993222366,
328
+ "eval_loss": 0.40265128016471863,
329
+ "eval_runtime": 5.8012,
330
+ "eval_samples_per_second": 29.477,
331
+ "eval_steps_per_second": 14.824,
332
+ "step": 9360
333
+ },
334
+ {
335
+ "epoch": 24.36,
336
+ "learning_rate": 3.2364672364672365e-06,
337
+ "loss": 0.3893,
338
+ "step": 9500
339
+ },
340
+ {
341
+ "epoch": 25.0,
342
+ "eval_accuracy": 0.8892501819946614,
343
+ "eval_loss": 0.3867790400981903,
344
+ "eval_runtime": 5.8209,
345
+ "eval_samples_per_second": 29.377,
346
+ "eval_steps_per_second": 14.774,
347
+ "step": 9750
348
+ },
349
+ {
350
+ "epoch": 25.64,
351
+ "learning_rate": 2.8803418803418804e-06,
352
+ "loss": 0.3991,
353
+ "step": 10000
354
+ },
355
+ {
356
+ "epoch": 26.0,
357
+ "eval_accuracy": 0.8846003326484688,
358
+ "eval_loss": 0.3882477581501007,
359
+ "eval_runtime": 5.8184,
360
+ "eval_samples_per_second": 29.389,
361
+ "eval_steps_per_second": 14.781,
362
+ "step": 10140
363
+ },
364
+ {
365
+ "epoch": 26.92,
366
+ "learning_rate": 2.5242165242165246e-06,
367
+ "loss": 0.3786,
368
+ "step": 10500
369
+ },
370
+ {
371
+ "epoch": 27.0,
372
+ "eval_accuracy": 0.8858714334822964,
373
+ "eval_loss": 0.3939129710197449,
374
+ "eval_runtime": 5.8222,
375
+ "eval_samples_per_second": 29.37,
376
+ "eval_steps_per_second": 14.771,
377
+ "step": 10530
378
+ },
379
+ {
380
+ "epoch": 28.0,
381
+ "eval_accuracy": 0.8847989764283254,
382
+ "eval_loss": 0.39587706327438354,
383
+ "eval_runtime": 5.8063,
384
+ "eval_samples_per_second": 29.451,
385
+ "eval_steps_per_second": 14.812,
386
+ "step": 10920
387
+ },
388
+ {
389
+ "epoch": 28.21,
390
+ "learning_rate": 2.168803418803419e-06,
391
+ "loss": 0.38,
392
+ "step": 11000
393
+ },
394
+ {
395
+ "epoch": 29.0,
396
+ "eval_accuracy": 0.8849687976020835,
397
+ "eval_loss": 0.3949810862541199,
398
+ "eval_runtime": 5.8138,
399
+ "eval_samples_per_second": 29.413,
400
+ "eval_steps_per_second": 14.792,
401
+ "step": 11310
402
+ },
403
+ {
404
+ "epoch": 29.49,
405
+ "learning_rate": 1.8126780626780629e-06,
406
+ "loss": 0.3764,
407
+ "step": 11500
408
+ },
409
+ {
410
+ "epoch": 30.0,
411
+ "eval_accuracy": 0.8893153879792042,
412
+ "eval_loss": 0.3783101439476013,
413
+ "eval_runtime": 5.8039,
414
+ "eval_samples_per_second": 29.463,
415
+ "eval_steps_per_second": 14.818,
416
+ "step": 11700
417
+ },
418
+ {
419
+ "epoch": 30.77,
420
+ "learning_rate": 1.4565527065527065e-06,
421
+ "loss": 0.3708,
422
+ "step": 12000
423
+ },
424
+ {
425
+ "epoch": 31.0,
426
+ "eval_accuracy": 0.8890792500856626,
427
+ "eval_loss": 0.3798995912075043,
428
+ "eval_runtime": 5.7021,
429
+ "eval_samples_per_second": 29.989,
430
+ "eval_steps_per_second": 15.082,
431
+ "step": 12090
432
+ },
433
+ {
434
+ "epoch": 32.0,
435
+ "eval_accuracy": 0.8867172306495527,
436
+ "eval_loss": 0.39150363206863403,
437
+ "eval_runtime": 5.8791,
438
+ "eval_samples_per_second": 29.086,
439
+ "eval_steps_per_second": 14.628,
440
+ "step": 12480
441
+ },
442
+ {
443
+ "epoch": 32.05,
444
+ "learning_rate": 1.1004273504273506e-06,
445
+ "loss": 0.3656,
446
+ "step": 12500
447
+ },
448
+ {
449
+ "epoch": 33.0,
450
+ "eval_accuracy": 0.8902530694061639,
451
+ "eval_loss": 0.3780055344104767,
452
+ "eval_runtime": 5.8784,
453
+ "eval_samples_per_second": 29.09,
454
+ "eval_steps_per_second": 14.63,
455
+ "step": 12870
456
+ },
457
+ {
458
+ "epoch": 33.33,
459
+ "learning_rate": 7.443019943019944e-07,
460
+ "loss": 0.3617,
461
+ "step": 13000
462
+ },
463
+ {
464
+ "epoch": 34.0,
465
+ "eval_accuracy": 0.8873723487824038,
466
+ "eval_loss": 0.38049712777137756,
467
+ "eval_runtime": 5.8779,
468
+ "eval_samples_per_second": 29.092,
469
+ "eval_steps_per_second": 14.631,
470
+ "step": 13260
471
+ },
472
+ {
473
+ "epoch": 34.62,
474
+ "learning_rate": 3.8817663817663825e-07,
475
+ "loss": 0.361,
476
+ "step": 13500
477
+ },
478
+ {
479
+ "epoch": 35.0,
480
+ "eval_accuracy": 0.8919778767559101,
481
+ "eval_loss": 0.3775971233844757,
482
+ "eval_runtime": 5.8063,
483
+ "eval_samples_per_second": 29.451,
484
+ "eval_steps_per_second": 14.812,
485
+ "step": 13650
486
+ },
487
+ {
488
+ "epoch": 35.9,
489
+ "learning_rate": 3.205128205128205e-08,
490
+ "loss": 0.3595,
491
+ "step": 14000
492
+ },
493
+ {
494
+ "epoch": 36.0,
495
+ "eval_accuracy": 0.8888084202394747,
496
+ "eval_loss": 0.3711872100830078,
497
+ "eval_runtime": 5.8621,
498
+ "eval_samples_per_second": 29.171,
499
+ "eval_steps_per_second": 14.671,
500
+ "step": 14040
501
+ },
502
+ {
503
+ "epoch": 36.0,
504
+ "step": 14040,
505
+ "total_flos": 3.509780816886497e+17,
506
+ "train_loss": 0.5379996796958467,
507
+ "train_runtime": 13714.1823,
508
+ "train_samples_per_second": 8.19,
509
+ "train_steps_per_second": 1.024
510
+ }
511
+ ],
512
+ "max_steps": 14040,
513
+ "num_train_epochs": 36,
514
+ "total_flos": 3.509780816886497e+17,
515
+ "trial_name": null,
516
+ "trial_params": null
517
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e62f0cca5fa131673e262469dc4be8f9f3fb2048d33c3baf0cda18865991b15f
3
+ size 3579
vocab.txt ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <cls>
2
+ <pad>
3
+ <eos>
4
+ <unk>
5
+ L
6
+ A
7
+ G
8
+ V
9
+ S
10
+ E
11
+ R
12
+ T
13
+ I
14
+ D
15
+ P
16
+ K
17
+ Q
18
+ N
19
+ F
20
+ Y
21
+ M
22
+ H
23
+ W
24
+ C
25
+ X
26
+ B
27
+ U
28
+ Z
29
+ O
30
+ .
31
+ -
32
+ <null_1>
33
+ <mask>