Hugging Face's logo

Mardiyyah
/
bioformer-ner-model

Token Classification
Transformers
Safetensors
English
bert
Generated from Trainer
Inference Endpoints
Model card Files Community
Mardiyyah commited on
Commit
a4a1e96
•
1 Parent(s): 82abc73

uploading custom model to hub

Browse files
Files changed (18) hide show
  1. README.md +78 -0
  2. config.json +51 -0
  3. model.safetensors +3 -0
  4. special_tokens_map.json +7 -0
  5. tokenizer.json +0 -0
  6. tokenizer_config.json +57 -0
  7. training_args.bin +3 -0
  8. vocab.txt +0 -0
  9. wandb/debug-internal.log +670 -0
  10. wandb/debug.log +40 -0
  11. wandb/run-20240917_122726-z0ntw6ob/files/config.yaml +776 -0
  12. wandb/run-20240917_122726-z0ntw6ob/files/output.log +115 -0
  13. wandb/run-20240917_122726-z0ntw6ob/files/requirements.txt +313 -0
  14. wandb/run-20240917_122726-z0ntw6ob/files/wandb-metadata.json +281 -0
  15. wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json +1 -0
  16. wandb/run-20240917_122726-z0ntw6ob/logs/debug-internal.log +670 -0
  17. wandb/run-20240917_122726-z0ntw6ob/logs/debug.log +40 -0
  18. wandb/run-20240917_122726-z0ntw6ob/run-z0ntw6ob.wandb +0 -0
README.md ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: bioformers/bioformer-16L
4
+ tags:
5
+ - generated_from_trainer
6
+ metrics:
7
+ - f1
8
+ - precision
9
+ - recall
10
+ - accuracy
11
+ model-index:
12
+ - name: cl_ct_custom_model
13
+ results: []
14
+ ---
15
+
16
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
17
+ should probably proofread and complete it, then remove this comment. -->
18
+
19
+ # cl_ct_custom_model
20
+
21
+ This model is a fine-tuned version of [bioformers/bioformer-16L](https://huggingface.co/bioformers/bioformer-16L) on an unknown dataset.
22
+ It achieves the following results on the evaluation set:
23
+ - Loss: 0.2590
24
+ - F1: 0.7609
25
+ - Precision: 0.7112
26
+ - Recall: 0.8181
27
+ - Accuracy: 0.9229
28
+
29
+ ## Model description
30
+
31
+ More information needed
32
+
33
+ ## Intended uses & limitations
34
+
35
+ More information needed
36
+
37
+ ## Training and evaluation data
38
+
39
+ More information needed
40
+
41
+ ## Training procedure
42
+
43
+ ### Training hyperparameters
44
+
45
+ The following hyperparameters were used during training:
46
+ - learning_rate: 2e-05
47
+ - train_batch_size: 16
48
+ - eval_batch_size: 8
49
+ - seed: 3407
50
+ - gradient_accumulation_steps: 4
51
+ - total_train_batch_size: 64
52
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
53
+ - lr_scheduler_type: linear
54
+ - num_epochs: 10
55
+ - mixed_precision_training: Native AMP
56
+
57
+ ### Training results
58
+
59
+ | Training Loss | Epoch | Step | Validation Loss | F1 | Precision | Recall | Accuracy |
60
+ |:-------------:|:------:|:----:|:---------------:|:------:|:---------:|:------:|:--------:|
61
+ | 0.4568 | 0.9971 | 259 | 0.2146 | 0.8139 | 0.7920 | 0.8370 | 0.9326 |
62
+ | 0.2115 | 1.9981 | 519 | 0.1907 | 0.8349 | 0.8125 | 0.8586 | 0.9379 |
63
+ | 0.1802 | 2.9990 | 779 | 0.1912 | 0.8407 | 0.8178 | 0.8650 | 0.9394 |
64
+ | 0.164 | 4.0 | 1039 | 0.1869 | 0.8449 | 0.8255 | 0.8652 | 0.9401 |
65
+ | 0.1518 | 4.9971 | 1298 | 0.1819 | 0.8525 | 0.8348 | 0.8710 | 0.9428 |
66
+ | 0.1424 | 5.9981 | 1558 | 0.1842 | 0.8506 | 0.8351 | 0.8666 | 0.9422 |
67
+ | 0.134 | 6.9990 | 1818 | 0.1869 | 0.8539 | 0.8373 | 0.8712 | 0.9428 |
68
+ | 0.128 | 8.0 | 2078 | 0.1889 | 0.8540 | 0.8374 | 0.8712 | 0.9429 |
69
+ | 0.1241 | 8.9971 | 2337 | 0.1892 | 0.8559 | 0.8401 | 0.8724 | 0.9432 |
70
+ | 0.1199 | 9.9711 | 2590 | 0.1899 | 0.8552 | 0.8392 | 0.8718 | 0.9431 |
71
+
72
+
73
+ ### Framework versions
74
+
75
+ - Transformers 4.43.4
76
+ - Pytorch 2.4.1+cu121
77
+ - Datasets 2.20.0
78
+ - Tokenizers 0.19.1
config.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "bioformers/bioformer-16L",
3
+ "architectures": [
4
+ "BertForTokenClassification"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "classifier_dropout": null,
8
+ "hidden_act": "gelu",
9
+ "hidden_dropout_prob": 0.1,
10
+ "hidden_size": 384,
11
+ "id2label": {
12
+ "0": "O",
13
+ "1": "B-DNA",
14
+ "2": "I-DNA",
15
+ "3": "B-protein",
16
+ "4": "I-protein",
17
+ "5": "B-cell_type",
18
+ "6": "I-cell_type",
19
+ "7": "B-cell_line",
20
+ "8": "I-cell_line",
21
+ "9": "B-RNA",
22
+ "10": "I-RNA"
23
+ },
24
+ "initializer_range": 0.02,
25
+ "intermediate_size": 1536,
26
+ "label2id": {
27
+ "B-DNA": 1,
28
+ "B-RNA": 9,
29
+ "B-cell_line": 7,
30
+ "B-cell_type": 5,
31
+ "B-protein": 3,
32
+ "I-DNA": 2,
33
+ "I-RNA": 10,
34
+ "I-cell_line": 8,
35
+ "I-cell_type": 6,
36
+ "I-protein": 4,
37
+ "O": 0
38
+ },
39
+ "layer_norm_eps": 1e-12,
40
+ "max_position_embeddings": 1024,
41
+ "model_type": "bert",
42
+ "num_attention_heads": 6,
43
+ "num_hidden_layers": 16,
44
+ "pad_token_id": 0,
45
+ "position_embedding_type": "absolute",
46
+ "torch_dtype": "float32",
47
+ "transformers_version": "4.43.4",
48
+ "type_vocab_size": 2,
49
+ "use_cache": true,
50
+ "vocab_size": 32768
51
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:05f9592f249a6ca5d6030a72fc179ab0a60e6c8101ece2f7add4a72f978de1a1
3
+ size 165524020
special_tokens_map.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "[CLS]",
3
+ "mask_token": "[MASK]",
4
+ "pad_token": "[PAD]",
5
+ "sep_token": "[SEP]",
6
+ "unk_token": "[UNK]"
7
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "[PAD]",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "100": {
12
+ "content": "[UNK]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "101": {
20
+ "content": "[CLS]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "102": {
28
+ "content": "[SEP]",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "103": {
36
+ "content": "[MASK]",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "clean_up_tokenization_spaces": true,
45
+ "cls_token": "[CLS]",
46
+ "do_basic_tokenize": true,
47
+ "do_lower_case": false,
48
+ "mask_token": "[MASK]",
49
+ "model_max_length": 1000000000000000019884624838656,
50
+ "never_split": null,
51
+ "pad_token": "[PAD]",
52
+ "sep_token": "[SEP]",
53
+ "strip_accents": null,
54
+ "tokenize_chinese_chars": true,
55
+ "tokenizer_class": "BertTokenizer",
56
+ "unk_token": "[UNK]"
57
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e4a970661a5acc1bd9d22d153ad785b90c3948e053bcee1b96658fabd72aabf7
3
+ size 5304
vocab.txt ADDED
The diff for this file is too large to render. See raw diff
 
wandb/debug-internal.log ADDED
@@ -0,0 +1,670 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2024-09-17 12:27:26,830 INFO StreamThr :4132313 [internal.py:wandb_internal():85] W&B internal server running at pid: 4132313, started at: 2024-09-17 12:27:26.826981
2
+ 2024-09-17 12:27:26,834 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status
3
+ 2024-09-17 12:27:26,838 INFO WriterThread:4132313 [datastore.py:open_for_write():87] open: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/run-z0ntw6ob.wandb
4
+ 2024-09-17 12:27:26,844 DEBUG SenderThread:4132313 [sender.py:send():391] send: header
5
+ 2024-09-17 12:27:26,848 DEBUG SenderThread:4132313 [sender.py:send():391] send: run
6
+ 2024-09-17 12:27:27,198 INFO SenderThread:4132313 [dir_watcher.py:__init__():211] watching files in: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files
7
+ 2024-09-17 12:27:27,198 INFO SenderThread:4132313 [sender.py:_start_run_threads():1200] run started: z0ntw6ob with start time 1726572446.82874
8
+ 2024-09-17 12:27:27,214 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: run_start
9
+ 2024-09-17 12:27:27,243 DEBUG HandlerThread:4132313 [system_info.py:__init__():26] System info init
10
+ 2024-09-17 12:27:27,243 DEBUG HandlerThread:4132313 [system_info.py:__init__():41] System info init done
11
+ 2024-09-17 12:27:27,244 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor
12
+ 2024-09-17 12:27:27,244 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads
13
+ 2024-09-17 12:27:27,244 INFO HandlerThread:4132313 [system_monitor.py:probe():214] Collecting system info
14
+ 2024-09-17 12:27:27,247 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring
15
+ 2024-09-17 12:27:27,250 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring
16
+ 2024-09-17 12:27:27,252 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring
17
+ 2024-09-17 12:27:27,253 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring
18
+ 2024-09-17 12:27:27,257 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring
19
+ 2024-09-17 12:27:27,284 DEBUG HandlerThread:4132313 [system_info.py:probe():152] Probing system
20
+ 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [gitlib.py:_init_repo():56] git repository is invalid
21
+ 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [system_info.py:probe():200] Probing system done
22
+ 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28', 'python': '3.11.4', 'heartbeatAt': '2024-09-17T11:27:27.284628', 'startedAt': '2024-09-17T11:27:26.786345', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '<python with no main file>', 'codePathLocal': None, 'host': 'codon-gpu-006.ebi.ac.uk', 'username': 'amrufai', 'executable': '/nfs/production/literature/amina-mardiyyah/envs/llm-prompt/bin/python', 'cpu_count': 48, 'cpu_count_logical': 48, 'cpu_freq': {'current': 2758.33775, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 1157.924, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 47.760292053222656, 'used': 15.848060607910156}}, 'gpu': 'NVIDIA A100 80GB PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}], 'memory': {'total': 502.8375930786133}}
23
+ 2024-09-17 12:27:27,291 INFO HandlerThread:4132313 [system_monitor.py:probe():224] Finished collecting system info
24
+ 2024-09-17 12:27:27,291 INFO HandlerThread:4132313 [system_monitor.py:probe():227] Publishing system info
25
+ 2024-09-17 12:27:27,296 INFO HandlerThread:4132313 [system_monitor.py:probe():229] Finished publishing system info
26
+ 2024-09-17 12:27:27,320 DEBUG SenderThread:4132313 [sender.py:send():391] send: files
27
+ 2024-09-17 12:27:27,320 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-metadata.json with policy now
28
+ 2024-09-17 12:27:27,845 INFO wandb-upload_0:4132313 [upload_job.py:push():130] Uploaded file /tmp/tmpr_ev51fvwandb/7xtqy0my-wandb-metadata.json
29
+ 2024-09-17 12:27:27,876 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: python_packages
30
+ 2024-09-17 12:27:27,877 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
31
+ 2024-09-17 12:27:27,877 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: python_packages
32
+ 2024-09-17 12:27:27,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
33
+ 2024-09-17 12:27:27,883 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
34
+ 2024-09-17 12:27:27,923 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause
35
+ 2024-09-17 12:27:27,923 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread
36
+ 2024-09-17 12:27:27,923 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor
37
+ 2024-09-17 12:27:27,924 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop
38
+ 2024-09-17 12:27:27,925 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor
39
+ 2024-09-17 12:27:27,926 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop
40
+ 2024-09-17 12:27:27,927 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics
41
+ 2024-09-17 12:27:27,927 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor
42
+ 2024-09-17 12:27:28,000 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor
43
+ 2024-09-17 12:27:28,000 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor
44
+ 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor
45
+ 2024-09-17 12:27:28,001 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume
46
+ 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread
47
+ 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor
48
+ 2024-09-17 12:27:28,002 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads
49
+ 2024-09-17 12:27:28,005 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring
50
+ 2024-09-17 12:27:28,008 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring
51
+ 2024-09-17 12:27:28,008 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring
52
+ 2024-09-17 12:27:28,010 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring
53
+ 2024-09-17 12:27:28,011 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring
54
+ 2024-09-17 12:27:28,066 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
55
+ 2024-09-17 12:27:28,067 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
56
+ 2024-09-17 12:27:28,204 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/requirements.txt
57
+ 2024-09-17 12:27:28,205 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
58
+ 2024-09-17 12:27:28,205 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-metadata.json
59
+ 2024-09-17 12:27:30,204 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
60
+ 2024-09-17 12:27:32,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
61
+ 2024-09-17 12:27:32,935 DEBUG SenderThread:4132313 [sender.py:send():391] send: config
62
+ 2024-09-17 12:27:32,937 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
63
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
64
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
65
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
66
+ 2024-09-17 12:27:32,938 WARNING SenderThread:4132313 [sender.py:send_metric():1417] Seen metric with glob (shouldn't happen)
67
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
68
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
69
+ 2024-09-17 12:27:32,939 DEBUG SenderThread:4132313 [sender.py:send():391] send: config
70
+ 2024-09-17 12:27:34,211 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
71
+ 2024-09-17 12:27:37,878 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
72
+ 2024-09-17 12:27:37,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
73
+ 2024-09-17 12:27:42,877 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
74
+ 2024-09-17 12:27:42,878 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
75
+ 2024-09-17 12:27:43,034 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
76
+ 2024-09-17 12:27:47,879 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
77
+ 2024-09-17 12:27:48,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
78
+ 2024-09-17 12:27:54,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
79
+ 2024-09-17 12:27:57,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
80
+ 2024-09-17 12:27:57,882 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
81
+ 2024-09-17 12:27:57,922 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
82
+ 2024-09-17 12:28:00,105 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
83
+ 2024-09-17 12:28:01,242 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml
84
+ 2024-09-17 12:28:05,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
85
+ 2024-09-17 12:28:07,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
86
+ 2024-09-17 12:28:10,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
87
+ 2024-09-17 12:28:12,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
88
+ 2024-09-17 12:28:12,882 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
89
+ 2024-09-17 12:28:16,038 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
90
+ 2024-09-17 12:28:17,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
91
+ 2024-09-17 12:28:21,884 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
92
+ 2024-09-17 12:28:27,884 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
93
+ 2024-09-17 12:28:27,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
94
+ 2024-09-17 12:28:27,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
95
+ 2024-09-17 12:28:27,886 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
96
+ 2024-09-17 12:28:28,011 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop
97
+ 2024-09-17 12:28:28,056 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
98
+ 2024-09-17 12:28:31,893 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
99
+ 2024-09-17 12:28:31,897 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
100
+ 2024-09-17 12:28:31,897 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
101
+ 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
102
+ 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
103
+ 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
104
+ 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
105
+ 2024-09-17 12:28:31,904 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
106
+ 2024-09-17 12:28:32,281 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
107
+ 2024-09-17 12:28:32,889 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
108
+ 2024-09-17 12:28:33,282 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml
109
+ 2024-09-17 12:28:37,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
110
+ 2024-09-17 12:28:37,167 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
111
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
112
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
113
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
114
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
115
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
116
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
117
+ 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
118
+ 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
119
+ 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
120
+ 2024-09-17 12:28:37,174 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
121
+ 2024-09-17 12:28:37,287 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
122
+ 2024-09-17 12:28:38,159 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
123
+ 2024-09-17 12:28:38,160 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
124
+ 2024-09-17 12:28:39,291 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
125
+ 2024-09-17 12:28:42,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
126
+ 2024-09-17 12:28:42,886 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
127
+ 2024-09-17 12:28:44,160 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
128
+ 2024-09-17 12:28:48,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
129
+ 2024-09-17 12:28:49,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
130
+ 2024-09-17 12:28:55,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
131
+ 2024-09-17 12:28:57,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
132
+ 2024-09-17 12:28:57,887 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
133
+ 2024-09-17 12:28:58,095 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
134
+ 2024-09-17 12:28:58,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
135
+ 2024-09-17 12:29:00,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
136
+ 2024-09-17 12:29:05,170 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
137
+ 2024-09-17 12:29:06,343 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml
138
+ 2024-09-17 12:29:08,159 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
139
+ 2024-09-17 12:29:11,164 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
140
+ 2024-09-17 12:29:12,889 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
141
+ 2024-09-17 12:29:12,890 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
142
+ 2024-09-17 12:29:17,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
143
+ 2024-09-17 12:29:18,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
144
+ 2024-09-17 12:29:22,164 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
145
+ 2024-09-17 12:29:27,888 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
146
+ 2024-09-17 12:29:27,889 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
147
+ 2024-09-17 12:29:28,113 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
148
+ 2024-09-17 12:29:28,118 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
149
+ 2024-09-17 12:29:28,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
150
+ 2024-09-17 12:29:33,168 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
151
+ 2024-09-17 12:29:38,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
152
+ 2024-09-17 12:29:38,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
153
+ 2024-09-17 12:29:38,188 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
154
+ 2024-09-17 12:29:38,189 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
155
+ 2024-09-17 12:29:38,189 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
156
+ 2024-09-17 12:29:38,196 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
157
+ 2024-09-17 12:29:38,383 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
158
+ 2024-09-17 12:29:42,913 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
159
+ 2024-09-17 12:29:42,914 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
160
+ 2024-09-17 12:29:43,192 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
161
+ 2024-09-17 12:29:43,343 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
162
+ 2024-09-17 12:29:43,345 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
163
+ 2024-09-17 12:29:43,345 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
164
+ 2024-09-17 12:29:43,349 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
165
+ 2024-09-17 12:29:43,389 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
166
+ 2024-09-17 12:29:45,554 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
167
+ 2024-09-17 12:29:48,168 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
168
+ 2024-09-17 12:29:49,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
169
+ 2024-09-17 12:29:54,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
170
+ 2024-09-17 12:29:57,893 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
171
+ 2024-09-17 12:29:57,894 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
172
+ 2024-09-17 12:29:58,026 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
173
+ 2024-09-17 12:29:58,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
174
+ 2024-09-17 12:30:00,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
175
+ 2024-09-17 12:30:05,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
176
+ 2024-09-17 12:30:08,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
177
+ 2024-09-17 12:30:11,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
178
+ 2024-09-17 12:30:12,894 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
179
+ 2024-09-17 12:30:12,894 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
180
+ 2024-09-17 12:30:16,175 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
181
+ 2024-09-17 12:30:18,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
182
+ 2024-09-17 12:30:22,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
183
+ 2024-09-17 12:30:27,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
184
+ 2024-09-17 12:30:27,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
185
+ 2024-09-17 12:30:27,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
186
+ 2024-09-17 12:30:28,105 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
187
+ 2024-09-17 12:30:28,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
188
+ 2024-09-17 12:30:32,176 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
189
+ 2024-09-17 12:30:37,176 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
190
+ 2024-09-17 12:30:38,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
191
+ 2024-09-17 12:30:42,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
192
+ 2024-09-17 12:30:42,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
193
+ 2024-09-17 12:30:43,087 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
194
+ 2024-09-17 12:30:44,559 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
195
+ 2024-09-17 12:30:44,561 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
196
+ 2024-09-17 12:30:44,561 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
197
+ 2024-09-17 12:30:44,568 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
198
+ 2024-09-17 12:30:44,626 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
199
+ 2024-09-17 12:30:48,189 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
200
+ 2024-09-17 12:30:48,214 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
201
+ 2024-09-17 12:30:49,050 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
202
+ 2024-09-17 12:30:49,053 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
203
+ 2024-09-17 12:30:49,053 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
204
+ 2024-09-17 12:30:49,057 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
205
+ 2024-09-17 12:30:49,633 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
206
+ 2024-09-17 12:30:51,690 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
207
+ 2024-09-17 12:30:54,178 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
208
+ 2024-09-17 12:30:57,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
209
+ 2024-09-17 12:30:57,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
210
+ 2024-09-17 12:30:58,125 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
211
+ 2024-09-17 12:30:58,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
212
+ 2024-09-17 12:31:00,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
213
+ 2024-09-17 12:31:06,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
214
+ 2024-09-17 12:31:08,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
215
+ 2024-09-17 12:31:11,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
216
+ 2024-09-17 12:31:12,899 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
217
+ 2024-09-17 12:31:12,900 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
218
+ 2024-09-17 12:31:17,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
219
+ 2024-09-17 12:31:18,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
220
+ 2024-09-17 12:31:22,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
221
+ 2024-09-17 12:31:27,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
222
+ 2024-09-17 12:31:27,899 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
223
+ 2024-09-17 12:31:27,900 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
224
+ 2024-09-17 12:31:28,115 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
225
+ 2024-09-17 12:31:28,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
226
+ 2024-09-17 12:31:33,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
227
+ 2024-09-17 12:31:38,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
228
+ 2024-09-17 12:31:38,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
229
+ 2024-09-17 12:31:42,546 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
230
+ 2024-09-17 12:31:42,548 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
231
+ 2024-09-17 12:31:42,548 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
232
+ 2024-09-17 12:31:42,553 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
233
+ 2024-09-17 12:31:42,750 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
234
+ 2024-09-17 12:31:42,900 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
235
+ 2024-09-17 12:31:42,901 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
236
+ 2024-09-17 12:31:43,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
237
+ 2024-09-17 12:31:47,028 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
238
+ 2024-09-17 12:31:47,030 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
239
+ 2024-09-17 12:31:47,030 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
240
+ 2024-09-17 12:31:47,034 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
241
+ 2024-09-17 12:31:47,755 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
242
+ 2024-09-17 12:31:48,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
243
+ 2024-09-17 12:31:48,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
244
+ 2024-09-17 12:31:49,759 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
245
+ 2024-09-17 12:31:54,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
246
+ 2024-09-17 12:31:57,903 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
247
+ 2024-09-17 12:31:57,903 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
248
+ 2024-09-17 12:31:58,101 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
249
+ 2024-09-17 12:31:58,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
250
+ 2024-09-17 12:31:59,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
251
+ 2024-09-17 12:32:04,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
252
+ 2024-09-17 12:32:08,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
253
+ 2024-09-17 12:32:09,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
254
+ 2024-09-17 12:32:12,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
255
+ 2024-09-17 12:32:12,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
256
+ 2024-09-17 12:32:15,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
257
+ 2024-09-17 12:32:18,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
258
+ 2024-09-17 12:32:21,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
259
+ 2024-09-17 12:32:26,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
260
+ 2024-09-17 12:32:27,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
261
+ 2024-09-17 12:32:27,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
262
+ 2024-09-17 12:32:28,122 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
263
+ 2024-09-17 12:32:28,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
264
+ 2024-09-17 12:32:32,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
265
+ 2024-09-17 12:32:38,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
266
+ 2024-09-17 12:32:38,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
267
+ 2024-09-17 12:32:42,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
268
+ 2024-09-17 12:32:42,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
269
+ 2024-09-17 12:32:43,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
270
+ 2024-09-17 12:32:45,990 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
271
+ 2024-09-17 12:32:45,992 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
272
+ 2024-09-17 12:32:45,992 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
273
+ 2024-09-17 12:32:45,997 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
274
+ 2024-09-17 12:32:46,827 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
275
+ 2024-09-17 12:32:48,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
276
+ 2024-09-17 12:32:48,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
277
+ 2024-09-17 12:32:50,431 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
278
+ 2024-09-17 12:32:50,433 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
279
+ 2024-09-17 12:32:50,433 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
280
+ 2024-09-17 12:32:50,437 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
281
+ 2024-09-17 12:32:50,832 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
282
+ 2024-09-17 12:32:51,907 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
283
+ 2024-09-17 12:32:54,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
284
+ 2024-09-17 12:32:57,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
285
+ 2024-09-17 12:32:57,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
286
+ 2024-09-17 12:32:58,050 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
287
+ 2024-09-17 12:32:58,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
288
+ 2024-09-17 12:32:59,188 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
289
+ 2024-09-17 12:33:05,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
290
+ 2024-09-17 12:33:08,196 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
291
+ 2024-09-17 12:33:10,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
292
+ 2024-09-17 12:33:12,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
293
+ 2024-09-17 12:33:12,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
294
+ 2024-09-17 12:33:16,193 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
295
+ 2024-09-17 12:33:18,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
296
+ 2024-09-17 12:33:21,193 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
297
+ 2024-09-17 12:33:26,199 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
298
+ 2024-09-17 12:33:27,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
299
+ 2024-09-17 12:33:27,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
300
+ 2024-09-17 12:33:28,135 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
301
+ 2024-09-17 12:33:28,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
302
+ 2024-09-17 12:33:31,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
303
+ 2024-09-17 12:33:37,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
304
+ 2024-09-17 12:33:38,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
305
+ 2024-09-17 12:33:42,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
306
+ 2024-09-17 12:33:42,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
307
+ 2024-09-17 12:33:42,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
308
+ 2024-09-17 12:33:48,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
309
+ 2024-09-17 12:33:48,197 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
310
+ 2024-09-17 12:33:50,740 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
311
+ 2024-09-17 12:33:50,741 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
312
+ 2024-09-17 12:33:50,742 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
313
+ 2024-09-17 12:33:50,745 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
314
+ 2024-09-17 12:33:50,980 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
315
+ 2024-09-17 12:33:53,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
316
+ 2024-09-17 12:33:55,232 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
317
+ 2024-09-17 12:33:55,235 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
318
+ 2024-09-17 12:33:55,235 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
319
+ 2024-09-17 12:33:55,239 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
320
+ 2024-09-17 12:33:55,987 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
321
+ 2024-09-17 12:33:56,989 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
322
+ 2024-09-17 12:33:57,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
323
+ 2024-09-17 12:33:57,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
324
+ 2024-09-17 12:33:58,051 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
325
+ 2024-09-17 12:33:58,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
326
+ 2024-09-17 12:33:59,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
327
+ 2024-09-17 12:34:04,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
328
+ 2024-09-17 12:34:08,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
329
+ 2024-09-17 12:34:09,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
330
+ 2024-09-17 12:34:12,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
331
+ 2024-09-17 12:34:12,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
332
+ 2024-09-17 12:34:15,208 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
333
+ 2024-09-17 12:34:18,205 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
334
+ 2024-09-17 12:34:20,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
335
+ 2024-09-17 12:34:26,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
336
+ 2024-09-17 12:34:27,906 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
337
+ 2024-09-17 12:34:27,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
338
+ 2024-09-17 12:34:28,050 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
339
+ 2024-09-17 12:34:28,208 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
340
+ 2024-09-17 12:34:31,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
341
+ 2024-09-17 12:34:37,210 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
342
+ 2024-09-17 12:34:38,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
343
+ 2024-09-17 12:34:42,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
344
+ 2024-09-17 12:34:42,906 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
345
+ 2024-09-17 12:34:42,907 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
346
+ 2024-09-17 12:34:48,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
347
+ 2024-09-17 12:34:48,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
348
+ 2024-09-17 12:34:50,281 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
349
+ 2024-09-17 12:34:50,283 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
350
+ 2024-09-17 12:34:50,283 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
351
+ 2024-09-17 12:34:50,290 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
352
+ 2024-09-17 12:34:51,047 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
353
+ 2024-09-17 12:34:54,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
354
+ 2024-09-17 12:34:55,374 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
355
+ 2024-09-17 12:34:55,376 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
356
+ 2024-09-17 12:34:55,376 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
357
+ 2024-09-17 12:34:55,381 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
358
+ 2024-09-17 12:34:56,054 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
359
+ 2024-09-17 12:34:57,070 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
360
+ 2024-09-17 12:34:57,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
361
+ 2024-09-17 12:34:57,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
362
+ 2024-09-17 12:34:58,118 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
363
+ 2024-09-17 12:34:58,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
364
+ 2024-09-17 12:35:00,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
365
+ 2024-09-17 12:35:06,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
366
+ 2024-09-17 12:35:08,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
367
+ 2024-09-17 12:35:11,215 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
368
+ 2024-09-17 12:35:12,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
369
+ 2024-09-17 12:35:12,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
370
+ 2024-09-17 12:35:16,217 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
371
+ 2024-09-17 12:35:18,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
372
+ 2024-09-17 12:35:21,217 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
373
+ 2024-09-17 12:35:26,224 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
374
+ 2024-09-17 12:35:27,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
375
+ 2024-09-17 12:35:27,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
376
+ 2024-09-17 12:35:28,144 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
377
+ 2024-09-17 12:35:28,212 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
378
+ 2024-09-17 12:35:32,223 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
379
+ 2024-09-17 12:35:37,223 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
380
+ 2024-09-17 12:35:38,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
381
+ 2024-09-17 12:35:42,225 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
382
+ 2024-09-17 12:35:42,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
383
+ 2024-09-17 12:35:42,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
384
+ 2024-09-17 12:35:48,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
385
+ 2024-09-17 12:35:48,222 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
386
+ 2024-09-17 12:35:49,141 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
387
+ 2024-09-17 12:35:49,143 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
388
+ 2024-09-17 12:35:49,144 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
389
+ 2024-09-17 12:35:49,146 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
390
+ 2024-09-17 12:35:50,122 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
391
+ 2024-09-17 12:35:53,263 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
392
+ 2024-09-17 12:35:53,590 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
393
+ 2024-09-17 12:35:53,592 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
394
+ 2024-09-17 12:35:53,593 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
395
+ 2024-09-17 12:35:53,597 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
396
+ 2024-09-17 12:35:54,127 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
397
+ 2024-09-17 12:35:55,221 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
398
+ 2024-09-17 12:35:57,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
399
+ 2024-09-17 12:35:57,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
400
+ 2024-09-17 12:35:58,079 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
401
+ 2024-09-17 12:35:58,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
402
+ 2024-09-17 12:35:59,228 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
403
+ 2024-09-17 12:36:04,228 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
404
+ 2024-09-17 12:36:08,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
405
+ 2024-09-17 12:36:09,236 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
406
+ 2024-09-17 12:36:12,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
407
+ 2024-09-17 12:36:12,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
408
+ 2024-09-17 12:36:15,235 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
409
+ 2024-09-17 12:36:18,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
410
+ 2024-09-17 12:36:20,241 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
411
+ 2024-09-17 12:36:25,244 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
412
+ 2024-09-17 12:36:27,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
413
+ 2024-09-17 12:36:27,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
414
+ 2024-09-17 12:36:28,173 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
415
+ 2024-09-17 12:36:28,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
416
+ 2024-09-17 12:36:31,242 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
417
+ 2024-09-17 12:36:36,242 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
418
+ 2024-09-17 12:36:38,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
419
+ 2024-09-17 12:36:41,243 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
420
+ 2024-09-17 12:36:42,957 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
421
+ 2024-09-17 12:36:42,958 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
422
+ 2024-09-17 12:36:47,089 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
423
+ 2024-09-17 12:36:47,091 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
424
+ 2024-09-17 12:36:47,091 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
425
+ 2024-09-17 12:36:47,092 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
426
+ 2024-09-17 12:36:47,096 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
427
+ 2024-09-17 12:36:47,279 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
428
+ 2024-09-17 12:36:48,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
429
+ 2024-09-17 12:36:51,557 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
430
+ 2024-09-17 12:36:51,560 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
431
+ 2024-09-17 12:36:51,560 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
432
+ 2024-09-17 12:36:51,565 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
433
+ 2024-09-17 12:36:52,285 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
434
+ 2024-09-17 12:36:52,566 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
435
+ 2024-09-17 12:36:53,287 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
436
+ 2024-09-17 12:36:57,956 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
437
+ 2024-09-17 12:36:57,957 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
438
+ 2024-09-17 12:36:58,101 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
439
+ 2024-09-17 12:36:58,107 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
440
+ 2024-09-17 12:36:58,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
441
+ 2024-09-17 12:37:03,344 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
442
+ 2024-09-17 12:37:08,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
443
+ 2024-09-17 12:37:08,344 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
444
+ 2024-09-17 12:37:12,956 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
445
+ 2024-09-17 12:37:12,957 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
446
+ 2024-09-17 12:37:14,124 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
447
+ 2024-09-17 12:37:18,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
448
+ 2024-09-17 12:37:19,349 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
449
+ 2024-09-17 12:37:24,350 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
450
+ 2024-09-17 12:37:27,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
451
+ 2024-09-17 12:37:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
452
+ 2024-09-17 12:37:28,152 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
453
+ 2024-09-17 12:37:28,221 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
454
+ 2024-09-17 12:37:29,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
455
+ 2024-09-17 12:37:35,356 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
456
+ 2024-09-17 12:37:38,221 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
457
+ 2024-09-17 12:37:40,356 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
458
+ 2024-09-17 12:37:42,962 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
459
+ 2024-09-17 12:37:42,963 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
460
+ 2024-09-17 12:37:45,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
461
+ 2024-09-17 12:37:49,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
462
+ 2024-09-17 12:37:49,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
463
+ 2024-09-17 12:37:49,167 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
464
+ 2024-09-17 12:37:49,167 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
465
+ 2024-09-17 12:37:49,170 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
466
+ 2024-09-17 12:37:49,351 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
467
+ 2024-09-17 12:37:50,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
468
+ 2024-09-17 12:37:53,890 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
469
+ 2024-09-17 12:37:53,892 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
470
+ 2024-09-17 12:37:53,892 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
471
+ 2024-09-17 12:37:53,898 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
472
+ 2024-09-17 12:37:54,358 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
473
+ 2024-09-17 12:37:55,453 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
474
+ 2024-09-17 12:37:55,795 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
475
+ 2024-09-17 12:37:57,183 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
476
+ 2024-09-17 12:37:57,226 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
477
+ 2024-09-17 12:37:57,227 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
478
+ 2024-09-17 12:37:57,231 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
479
+ 2024-09-17 12:37:57,232 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
480
+ 2024-09-17 12:37:57,236 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
481
+ 2024-09-17 12:37:57,237 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
482
+ 2024-09-17 12:37:57,241 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
483
+ 2024-09-17 12:37:57,241 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
484
+ 2024-09-17 12:37:57,245 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
485
+ 2024-09-17 12:37:57,245 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
486
+ 2024-09-17 12:37:57,248 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
487
+ 2024-09-17 12:37:57,249 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
488
+ 2024-09-17 12:37:57,249 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
489
+ 2024-09-17 12:37:57,252 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
490
+ 2024-09-17 12:37:57,455 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
491
+ 2024-09-17 12:37:57,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
492
+ 2024-09-17 12:37:57,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
493
+ 2024-09-17 12:37:58,128 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
494
+ 2024-09-17 12:37:59,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
495
+ 2024-09-17 12:38:01,794 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
496
+ 2024-09-17 12:38:05,499 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
497
+ 2024-09-17 12:38:05,502 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
498
+ 2024-09-17 12:38:05,503 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause
499
+ 2024-09-17 12:38:05,503 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread
500
+ 2024-09-17 12:38:05,504 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor
501
+ 2024-09-17 12:38:05,504 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
502
+ 2024-09-17 12:38:05,506 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop
503
+ 2024-09-17 12:38:05,507 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
504
+ 2024-09-17 12:38:05,511 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor
505
+ 2024-09-17 12:38:05,511 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics
506
+ 2024-09-17 12:38:05,512 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor
507
+ 2024-09-17 12:38:05,517 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
508
+ 2024-09-17 12:38:05,517 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
509
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
510
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
511
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
512
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
513
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
514
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
515
+ 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
516
+ 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
517
+ 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
518
+ 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
519
+ 2024-09-17 12:38:05,522 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
520
+ 2024-09-17 12:38:05,592 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor
521
+ 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor
522
+ 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor
523
+ 2024-09-17 12:38:05,593 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume
524
+ 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread
525
+ 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor
526
+ 2024-09-17 12:38:05,594 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads
527
+ 2024-09-17 12:38:05,595 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
528
+ 2024-09-17 12:38:05,595 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring
529
+ 2024-09-17 12:38:05,597 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring
530
+ 2024-09-17 12:38:05,599 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring
531
+ 2024-09-17 12:38:05,602 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring
532
+ 2024-09-17 12:38:05,603 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring
533
+ 2024-09-17 12:38:05,619 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause
534
+ 2024-09-17 12:38:05,620 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread
535
+ 2024-09-17 12:38:05,620 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor
536
+ 2024-09-17 12:38:05,621 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop
537
+ 2024-09-17 12:38:05,621 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor
538
+ 2024-09-17 12:38:05,621 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop
539
+ 2024-09-17 12:38:05,622 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor
540
+ 2024-09-17 12:38:05,622 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics
541
+ 2024-09-17 12:38:05,679 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor
542
+ 2024-09-17 12:38:05,680 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor
543
+ 2024-09-17 12:38:05,680 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor
544
+ 2024-09-17 12:38:05,681 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
545
+ 2024-09-17 12:38:06,464 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
546
+ 2024-09-17 12:38:07,468 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
547
+ 2024-09-17 12:38:07,683 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
548
+ 2024-09-17 12:38:09,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
549
+ 2024-09-17 12:38:12,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
550
+ 2024-09-17 12:38:12,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
551
+ 2024-09-17 12:38:12,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
552
+ 2024-09-17 12:38:17,809 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
553
+ 2024-09-17 12:38:18,480 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml
554
+ 2024-09-17 12:38:19,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
555
+ 2024-09-17 12:38:23,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
556
+ 2024-09-17 12:38:27,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
557
+ 2024-09-17 12:38:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
558
+ 2024-09-17 12:38:29,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
559
+ 2024-09-17 12:38:29,152 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
560
+ 2024-09-17 12:38:34,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
561
+ 2024-09-17 12:38:39,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
562
+ 2024-09-17 12:38:39,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
563
+ 2024-09-17 12:38:42,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
564
+ 2024-09-17 12:38:42,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
565
+ 2024-09-17 12:38:44,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
566
+ 2024-09-17 12:38:49,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
567
+ 2024-09-17 12:38:50,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
568
+ 2024-09-17 12:38:55,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
569
+ 2024-09-17 12:38:57,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
570
+ 2024-09-17 12:38:57,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
571
+ 2024-09-17 12:38:59,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
572
+ 2024-09-17 12:39:01,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
573
+ 2024-09-17 12:39:06,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
574
+ 2024-09-17 12:39:09,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
575
+ 2024-09-17 12:39:12,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
576
+ 2024-09-17 12:39:12,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
577
+ 2024-09-17 12:39:12,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
578
+ 2024-09-17 12:39:17,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
579
+ 2024-09-17 12:39:19,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
580
+ 2024-09-17 12:39:22,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
581
+ 2024-09-17 12:39:27,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
582
+ 2024-09-17 12:39:27,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
583
+ 2024-09-17 12:39:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
584
+ 2024-09-17 12:39:29,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
585
+ 2024-09-17 12:39:33,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
586
+ 2024-09-17 12:39:38,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
587
+ 2024-09-17 12:39:39,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
588
+ 2024-09-17 12:39:42,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
589
+ 2024-09-17 12:39:42,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
590
+ 2024-09-17 12:39:44,149 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
591
+ 2024-09-17 12:39:49,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
592
+ 2024-09-17 12:39:49,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
593
+ 2024-09-17 12:39:54,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
594
+ 2024-09-17 12:39:57,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
595
+ 2024-09-17 12:39:57,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
596
+ 2024-09-17 12:39:59,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
597
+ 2024-09-17 12:39:59,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
598
+ 2024-09-17 12:40:04,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
599
+ 2024-09-17 12:40:09,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
600
+ 2024-09-17 12:40:09,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
601
+ 2024-09-17 12:40:12,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
602
+ 2024-09-17 12:40:12,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
603
+ 2024-09-17 12:40:14,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
604
+ 2024-09-17 12:40:19,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
605
+ 2024-09-17 12:40:19,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
606
+ 2024-09-17 12:40:24,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
607
+ 2024-09-17 12:40:27,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
608
+ 2024-09-17 12:40:27,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
609
+ 2024-09-17 12:40:29,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
610
+ 2024-09-17 12:40:29,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
611
+ 2024-09-17 12:40:34,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
612
+ 2024-09-17 12:40:39,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
613
+ 2024-09-17 12:40:39,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
614
+ 2024-09-17 12:40:42,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
615
+ 2024-09-17 12:40:42,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
616
+ 2024-09-17 12:40:44,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
617
+ 2024-09-17 12:40:49,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
618
+ 2024-09-17 12:40:49,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
619
+ 2024-09-17 12:40:54,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
620
+ 2024-09-17 12:40:57,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
621
+ 2024-09-17 12:40:57,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
622
+ 2024-09-17 12:40:59,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
623
+ 2024-09-17 12:40:59,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
624
+ 2024-09-17 12:41:04,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
625
+ 2024-09-17 12:41:09,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
626
+ 2024-09-17 12:41:10,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
627
+ 2024-09-17 12:41:12,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
628
+ 2024-09-17 12:41:12,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
629
+ 2024-09-17 12:41:15,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
630
+ 2024-09-17 12:41:19,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
631
+ 2024-09-17 12:41:20,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
632
+ 2024-09-17 12:41:26,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
633
+ 2024-09-17 12:41:27,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
634
+ 2024-09-17 12:41:27,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
635
+ 2024-09-17 12:41:29,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
636
+ 2024-09-17 12:41:31,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
637
+ 2024-09-17 12:41:36,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
638
+ 2024-09-17 12:41:39,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
639
+ 2024-09-17 12:41:41,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
640
+ 2024-09-17 12:41:42,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
641
+ 2024-09-17 12:41:42,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
642
+ 2024-09-17 12:41:47,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
643
+ 2024-09-17 12:41:49,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
644
+ 2024-09-17 12:41:52,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
645
+ 2024-09-17 12:41:57,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
646
+ 2024-09-17 12:41:57,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
647
+ 2024-09-17 12:41:57,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
648
+ 2024-09-17 12:41:59,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
649
+ 2024-09-17 12:42:03,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
650
+ 2024-09-17 12:42:08,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
651
+ 2024-09-17 12:42:09,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
652
+ 2024-09-17 12:42:12,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
653
+ 2024-09-17 12:42:12,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
654
+ 2024-09-17 12:42:14,152 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
655
+ 2024-09-17 12:42:17,599 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume
656
+ 2024-09-17 12:42:17,599 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread
657
+ 2024-09-17 12:42:17,599 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor
658
+ 2024-09-17 12:42:17,600 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads
659
+ 2024-09-17 12:42:17,601 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring
660
+ 2024-09-17 12:42:17,605 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring
661
+ 2024-09-17 12:42:17,606 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring
662
+ 2024-09-17 12:42:17,608 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring
663
+ 2024-09-17 12:42:17,610 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring
664
+ 2024-09-17 12:42:19,079 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
665
+ 2024-09-17 12:42:19,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
666
+ 2024-09-17 12:42:24,807 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
667
+ 2024-09-17 12:42:27,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
668
+ 2024-09-17 12:42:27,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
669
+ 2024-09-17 12:42:29,079 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
670
+ 2024-09-17 12:42:30,807 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
wandb/debug.log ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2024-09-17 12:27:26,817 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Current SDK version is 0.17.9
2
+ 2024-09-17 12:27:26,817 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Configure stats pid to 4132266
3
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Loading settings from /homes/amrufai/.config/wandb/settings
4
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Loading settings from /nfs/production/literature/amina-mardiyyah/notebooks/wandb/settings
5
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Loading settings from environment variables: {}
6
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Inferring run settings from compute environment: {'program': '<python with no main file>'}
7
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Applying login settings: {}
8
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Applying login settings: {}
9
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_init.py:_log_setup():524] Logging user logs to /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/logs/debug.log
10
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_init.py:_log_setup():525] Logging internal logs to /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/logs/debug-internal.log
11
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_init.py:_jupyter_setup():470] configuring jupyter hooks <wandb.sdk.wandb_init._WandbInit object at 0x7fecc6548b90>
12
+ 2024-09-17 12:27:26,821 INFO MainThread:4132266 [wandb_init.py:init():608] calling init triggers
13
+ 2024-09-17 12:27:26,821 INFO MainThread:4132266 [wandb_init.py:init():615] wandb.init called with sweep_config: {}
14
+ config: {}
15
+ 2024-09-17 12:27:26,821 INFO MainThread:4132266 [wandb_init.py:init():658] starting backend
16
+ 2024-09-17 12:27:26,821 INFO MainThread:4132266 [wandb_init.py:init():662] setting up manager
17
+ 2024-09-17 12:27:26,823 INFO MainThread:4132266 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
18
+ 2024-09-17 12:27:26,828 INFO MainThread:4132266 [wandb_init.py:init():670] backend started and connected
19
+ 2024-09-17 12:27:26,836 INFO MainThread:4132266 [wandb_run.py:_label_probe_notebook():1344] probe notebook
20
+ 2024-09-17 12:27:26,837 INFO MainThread:4132266 [wandb_run.py:_label_probe_notebook():1354] Unable to probe notebook: 'NoneType' object has no attribute 'get'
21
+ 2024-09-17 12:27:26,837 INFO MainThread:4132266 [wandb_init.py:init():768] updated telemetry
22
+ 2024-09-17 12:27:26,846 INFO MainThread:4132266 [wandb_init.py:init():801] communicating run to backend with 90.0 second timeout
23
+ 2024-09-17 12:27:27,208 INFO MainThread:4132266 [wandb_init.py:init():852] starting run threads in backend
24
+ 2024-09-17 12:27:27,878 INFO MainThread:4132266 [wandb_run.py:_console_start():2465] atexit reg
25
+ 2024-09-17 12:27:27,879 INFO MainThread:4132266 [wandb_run.py:_redirect():2311] redirect: wrap_raw
26
+ 2024-09-17 12:27:27,880 INFO MainThread:4132266 [wandb_run.py:_redirect():2376] Wrapping output streams.
27
+ 2024-09-17 12:27:27,880 INFO MainThread:4132266 [wandb_run.py:_redirect():2401] Redirects installed.
28
+ 2024-09-17 12:27:27,891 INFO MainThread:4132266 [wandb_init.py:init():895] run started, returning control to user process
29
+ 2024-09-17 12:27:27,897 INFO MainThread:4132266 [jupyter.py:save_ipynb():372] not saving jupyter notebook
30
+ 2024-09-17 12:27:27,897 INFO MainThread:4132266 [wandb_init.py:_pause_backend():435] pausing backend
31
+ 2024-09-17 12:27:27,904 INFO MainThread:4132266 [wandb_init.py:_resume_backend():440] resuming backend
32
+ 2024-09-17 12:27:32,897 INFO MainThread:4132266 [wandb_run.py:_config_callback():1392] config_cb None None {'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': None, 'use_bfloat16': False, 'tf_legacy_loss': False, 'pruned_heads': {}, 'tie_word_embeddings': True, 'chunk_size_feed_forward': 0, 'is_encoder_decoder': False, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'typical_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'exponential_decay_length_penalty': None, 'suppress_tokens': None, 'begin_suppress_tokens': None, 'architectures': ['BertForMaskedLM'], 'finetuning_task': None, 'id2label': {0: 'O', 1: 'B-DNA', 2: 'I-DNA', 3: 'B-protein', 4: 'I-protein', 5: 'B-cell_type', 6: 'I-cell_type', 7: 'B-cell_line', 8: 'I-cell_line', 9: 'B-RNA', 10: 'I-RNA'}, 'label2id': {'O': 0, 'B-DNA': 1, 'I-DNA': 2, 'B-protein': 3, 'I-protein': 4, 'B-cell_type': 5, 'I-cell_type': 6, 'B-cell_line': 7, 'I-cell_line': 8, 'B-RNA': 9, 'I-RNA': 10}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': None, 'pad_token_id': 0, 'eos_token_id': None, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': None, 'problem_type': None, '_name_or_path': 'bioformers/bioformer-16L', 'transformers_version': '4.43.4', 'model_type': 'bert', 'vocab_size': 32768, 'hidden_size': 384, 'num_hidden_layers': 16, 'num_attention_heads': 6, 'hidden_act': 'gelu', 'intermediate_size': 1536, 'hidden_dropout_prob': 0.1, 'attention_probs_dropout_prob': 0.1, 'max_position_embeddings': 1024, 'type_vocab_size': 2, 'initializer_range': 0.02, 'layer_norm_eps': 1e-12, 'position_embedding_type': 'absolute', 'use_cache': True, 'classifier_dropout': None, 'output_dir': '/nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': True, 'do_predict': False, 'eval_strategy': 'epoch', 'prediction_loss_only': False, 'per_device_train_batch_size': 16, 'per_device_eval_batch_size': 8, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 4, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 2e-05, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 10, 'max_steps': -1, 'lr_scheduler_type': 'linear', 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.0, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': '/nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/logs', 'logging_strategy': 'epoch', 'logging_first_step': False, 'logging_steps': 500, 'logging_nan_inf_filter': True, 'save_strategy': 'epoch', 'save_steps': 500, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 3407, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': False, 'fp16': True, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': '/nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': True, 'metric_for_best_model': 'eval_f1', 'greater_is_better': True, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}, 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': 'adamw_torch', 'optim_args': None, 'adafactor': False, 'group_by_length': True, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': 'every_save', 'hub_token': '<HUB_TOKEN>', 'hub_private_repo': False, 'hub_always_push': False, 'gradient_checkpointing': False, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'evaluation_strategy': None, 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': '<PUSH_TO_HUB_TOKEN>', 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'dispatch_batches': None, 'split_batches': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'eval_use_gather_object': False}
33
+ 2024-09-17 12:27:32,900 INFO MainThread:4132266 [wandb_config.py:__setitem__():154] config set model/num_parameters = 41373323 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x7fedd8b4b390>>
34
+ 2024-09-17 12:27:32,900 INFO MainThread:4132266 [wandb_run.py:_config_callback():1392] config_cb model/num_parameters 41373323 None
35
+ 2024-09-17 12:38:05,500 INFO MainThread:4132266 [jupyter.py:save_ipynb():372] not saving jupyter notebook
36
+ 2024-09-17 12:38:05,501 INFO MainThread:4132266 [wandb_init.py:_pause_backend():435] pausing backend
37
+ 2024-09-17 12:38:05,505 INFO MainThread:4132266 [wandb_init.py:_resume_backend():440] resuming backend
38
+ 2024-09-17 12:38:05,619 INFO MainThread:4132266 [jupyter.py:save_ipynb():372] not saving jupyter notebook
39
+ 2024-09-17 12:38:05,619 INFO MainThread:4132266 [wandb_init.py:_pause_backend():435] pausing backend
40
+ 2024-09-17 12:42:17,598 INFO MainThread:4132266 [wandb_init.py:_resume_backend():440] resuming backend
wandb/run-20240917_122726-z0ntw6ob/files/config.yaml ADDED
@@ -0,0 +1,776 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ wandb_version: 1
2
+
3
+ _wandb:
4
+ desc: null
5
+ value:
6
+ python_version: 3.11.4
7
+ cli_version: 0.17.9
8
+ framework: huggingface
9
+ huggingface_version: 4.43.4
10
+ is_jupyter_run: true
11
+ is_kaggle_kernel: false
12
+ start_time: 1726572446
13
+ t:
14
+ 1:
15
+ - 1
16
+ - 5
17
+ - 11
18
+ - 41
19
+ - 49
20
+ - 51
21
+ - 53
22
+ - 55
23
+ - 71
24
+ - 98
25
+ - 100
26
+ - 105
27
+ 2:
28
+ - 1
29
+ - 5
30
+ - 11
31
+ - 41
32
+ - 49
33
+ - 51
34
+ - 53
35
+ - 55
36
+ - 71
37
+ - 98
38
+ - 100
39
+ - 105
40
+ 3:
41
+ - 7
42
+ - 19
43
+ - 23
44
+ - 62
45
+ - 66
46
+ 4: 3.11.4
47
+ 5: 0.17.9
48
+ 6: 4.43.4
49
+ 8:
50
+ - 1
51
+ - 5
52
+ 9:
53
+ 1: transformers_trainer
54
+ 13: linux-x86_64
55
+ m:
56
+ - 1: train/global_step
57
+ 6:
58
+ - 3
59
+ - 1: train/loss
60
+ 5: 1
61
+ 6:
62
+ - 1
63
+ - 1: train/grad_norm
64
+ 5: 1
65
+ 6:
66
+ - 1
67
+ - 1: train/learning_rate
68
+ 5: 1
69
+ 6:
70
+ - 1
71
+ - 1: train/epoch
72
+ 5: 1
73
+ 6:
74
+ - 1
75
+ - 1: eval/loss
76
+ 5: 1
77
+ 6:
78
+ - 1
79
+ - 1: eval/f1
80
+ 5: 1
81
+ 6:
82
+ - 1
83
+ - 1: eval/precision
84
+ 5: 1
85
+ 6:
86
+ - 1
87
+ - 1: eval/recall
88
+ 5: 1
89
+ 6:
90
+ - 1
91
+ - 1: eval/accuracy
92
+ 5: 1
93
+ 6:
94
+ - 1
95
+ - 1: eval/runtime
96
+ 5: 1
97
+ 6:
98
+ - 1
99
+ - 1: eval/samples_per_second
100
+ 5: 1
101
+ 6:
102
+ - 1
103
+ - 1: eval/steps_per_second
104
+ 5: 1
105
+ 6:
106
+ - 1
107
+ - 1: test_results.eval_loss
108
+ 5: 1
109
+ 6:
110
+ - 1
111
+ - 1: test_results.eval_f1
112
+ 5: 1
113
+ 6:
114
+ - 1
115
+ - 1: test_results.eval_precision
116
+ 5: 1
117
+ 6:
118
+ - 1
119
+ - 1: test_results.eval_recall
120
+ 5: 1
121
+ 6:
122
+ - 1
123
+ - 1: test_results.eval_accuracy
124
+ 5: 1
125
+ 6:
126
+ - 1
127
+ - 1: test_results.eval_runtime
128
+ 5: 1
129
+ 6:
130
+ - 1
131
+ - 1: test_results.eval_samples_per_second
132
+ 5: 1
133
+ 6:
134
+ - 1
135
+ - 1: test_results.eval_steps_per_second
136
+ 5: 1
137
+ 6:
138
+ - 1
139
+ - 1: test_results.epoch
140
+ 5: 1
141
+ 6:
142
+ - 1
143
+ return_dict:
144
+ desc: null
145
+ value: true
146
+ output_hidden_states:
147
+ desc: null
148
+ value: false
149
+ output_attentions:
150
+ desc: null
151
+ value: false
152
+ torchscript:
153
+ desc: null
154
+ value: false
155
+ torch_dtype:
156
+ desc: null
157
+ value: null
158
+ use_bfloat16:
159
+ desc: null
160
+ value: false
161
+ tf_legacy_loss:
162
+ desc: null
163
+ value: false
164
+ pruned_heads:
165
+ desc: null
166
+ value: {}
167
+ tie_word_embeddings:
168
+ desc: null
169
+ value: true
170
+ chunk_size_feed_forward:
171
+ desc: null
172
+ value: 0
173
+ is_encoder_decoder:
174
+ desc: null
175
+ value: false
176
+ is_decoder:
177
+ desc: null
178
+ value: false
179
+ cross_attention_hidden_size:
180
+ desc: null
181
+ value: null
182
+ add_cross_attention:
183
+ desc: null
184
+ value: false
185
+ tie_encoder_decoder:
186
+ desc: null
187
+ value: false
188
+ max_length:
189
+ desc: null
190
+ value: 20
191
+ min_length:
192
+ desc: null
193
+ value: 0
194
+ do_sample:
195
+ desc: null
196
+ value: false
197
+ early_stopping:
198
+ desc: null
199
+ value: false
200
+ num_beams:
201
+ desc: null
202
+ value: 1
203
+ num_beam_groups:
204
+ desc: null
205
+ value: 1
206
+ diversity_penalty:
207
+ desc: null
208
+ value: 0.0
209
+ temperature:
210
+ desc: null
211
+ value: 1.0
212
+ top_k:
213
+ desc: null
214
+ value: 50
215
+ top_p:
216
+ desc: null
217
+ value: 1.0
218
+ typical_p:
219
+ desc: null
220
+ value: 1.0
221
+ repetition_penalty:
222
+ desc: null
223
+ value: 1.0
224
+ length_penalty:
225
+ desc: null
226
+ value: 1.0
227
+ no_repeat_ngram_size:
228
+ desc: null
229
+ value: 0
230
+ encoder_no_repeat_ngram_size:
231
+ desc: null
232
+ value: 0
233
+ bad_words_ids:
234
+ desc: null
235
+ value: null
236
+ num_return_sequences:
237
+ desc: null
238
+ value: 1
239
+ output_scores:
240
+ desc: null
241
+ value: false
242
+ return_dict_in_generate:
243
+ desc: null
244
+ value: false
245
+ forced_bos_token_id:
246
+ desc: null
247
+ value: null
248
+ forced_eos_token_id:
249
+ desc: null
250
+ value: null
251
+ remove_invalid_values:
252
+ desc: null
253
+ value: false
254
+ exponential_decay_length_penalty:
255
+ desc: null
256
+ value: null
257
+ suppress_tokens:
258
+ desc: null
259
+ value: null
260
+ begin_suppress_tokens:
261
+ desc: null
262
+ value: null
263
+ architectures:
264
+ desc: null
265
+ value:
266
+ - BertForMaskedLM
267
+ finetuning_task:
268
+ desc: null
269
+ value: null
270
+ id2label:
271
+ desc: null
272
+ value:
273
+ '0': O
274
+ '1': B-DNA
275
+ '2': I-DNA
276
+ '3': B-protein
277
+ '4': I-protein
278
+ '5': B-cell_type
279
+ '6': I-cell_type
280
+ '7': B-cell_line
281
+ '8': I-cell_line
282
+ '9': B-RNA
283
+ '10': I-RNA
284
+ label2id:
285
+ desc: null
286
+ value:
287
+ O: 0
288
+ B-DNA: 1
289
+ I-DNA: 2
290
+ B-protein: 3
291
+ I-protein: 4
292
+ B-cell_type: 5
293
+ I-cell_type: 6
294
+ B-cell_line: 7
295
+ I-cell_line: 8
296
+ B-RNA: 9
297
+ I-RNA: 10
298
+ tokenizer_class:
299
+ desc: null
300
+ value: null
301
+ prefix:
302
+ desc: null
303
+ value: null
304
+ bos_token_id:
305
+ desc: null
306
+ value: null
307
+ pad_token_id:
308
+ desc: null
309
+ value: 0
310
+ eos_token_id:
311
+ desc: null
312
+ value: null
313
+ sep_token_id:
314
+ desc: null
315
+ value: null
316
+ decoder_start_token_id:
317
+ desc: null
318
+ value: null
319
+ task_specific_params:
320
+ desc: null
321
+ value: null
322
+ problem_type:
323
+ desc: null
324
+ value: null
325
+ _name_or_path:
326
+ desc: null
327
+ value: bioformers/bioformer-16L
328
+ transformers_version:
329
+ desc: null
330
+ value: 4.43.4
331
+ model_type:
332
+ desc: null
333
+ value: bert
334
+ vocab_size:
335
+ desc: null
336
+ value: 32768
337
+ hidden_size:
338
+ desc: null
339
+ value: 384
340
+ num_hidden_layers:
341
+ desc: null
342
+ value: 16
343
+ num_attention_heads:
344
+ desc: null
345
+ value: 6
346
+ hidden_act:
347
+ desc: null
348
+ value: gelu
349
+ intermediate_size:
350
+ desc: null
351
+ value: 1536
352
+ hidden_dropout_prob:
353
+ desc: null
354
+ value: 0.1
355
+ attention_probs_dropout_prob:
356
+ desc: null
357
+ value: 0.1
358
+ max_position_embeddings:
359
+ desc: null
360
+ value: 1024
361
+ type_vocab_size:
362
+ desc: null
363
+ value: 2
364
+ initializer_range:
365
+ desc: null
366
+ value: 0.02
367
+ layer_norm_eps:
368
+ desc: null
369
+ value: 1.0e-12
370
+ position_embedding_type:
371
+ desc: null
372
+ value: absolute
373
+ use_cache:
374
+ desc: null
375
+ value: true
376
+ classifier_dropout:
377
+ desc: null
378
+ value: null
379
+ output_dir:
380
+ desc: null
381
+ value: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model
382
+ overwrite_output_dir:
383
+ desc: null
384
+ value: false
385
+ do_train:
386
+ desc: null
387
+ value: false
388
+ do_eval:
389
+ desc: null
390
+ value: true
391
+ do_predict:
392
+ desc: null
393
+ value: false
394
+ eval_strategy:
395
+ desc: null
396
+ value: epoch
397
+ prediction_loss_only:
398
+ desc: null
399
+ value: false
400
+ per_device_train_batch_size:
401
+ desc: null
402
+ value: 16
403
+ per_device_eval_batch_size:
404
+ desc: null
405
+ value: 8
406
+ per_gpu_train_batch_size:
407
+ desc: null
408
+ value: null
409
+ per_gpu_eval_batch_size:
410
+ desc: null
411
+ value: null
412
+ gradient_accumulation_steps:
413
+ desc: null
414
+ value: 4
415
+ eval_accumulation_steps:
416
+ desc: null
417
+ value: null
418
+ eval_delay:
419
+ desc: null
420
+ value: 0
421
+ torch_empty_cache_steps:
422
+ desc: null
423
+ value: null
424
+ learning_rate:
425
+ desc: null
426
+ value: 2.0e-05
427
+ weight_decay:
428
+ desc: null
429
+ value: 0.01
430
+ adam_beta1:
431
+ desc: null
432
+ value: 0.9
433
+ adam_beta2:
434
+ desc: null
435
+ value: 0.999
436
+ adam_epsilon:
437
+ desc: null
438
+ value: 1.0e-08
439
+ max_grad_norm:
440
+ desc: null
441
+ value: 1.0
442
+ num_train_epochs:
443
+ desc: null
444
+ value: 10
445
+ max_steps:
446
+ desc: null
447
+ value: -1
448
+ lr_scheduler_type:
449
+ desc: null
450
+ value: linear
451
+ lr_scheduler_kwargs:
452
+ desc: null
453
+ value: {}
454
+ warmup_ratio:
455
+ desc: null
456
+ value: 0.0
457
+ warmup_steps:
458
+ desc: null
459
+ value: 0
460
+ log_level:
461
+ desc: null
462
+ value: passive
463
+ log_level_replica:
464
+ desc: null
465
+ value: warning
466
+ log_on_each_node:
467
+ desc: null
468
+ value: true
469
+ logging_dir:
470
+ desc: null
471
+ value: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/logs
472
+ logging_strategy:
473
+ desc: null
474
+ value: epoch
475
+ logging_first_step:
476
+ desc: null
477
+ value: false
478
+ logging_steps:
479
+ desc: null
480
+ value: 500
481
+ logging_nan_inf_filter:
482
+ desc: null
483
+ value: true
484
+ save_strategy:
485
+ desc: null
486
+ value: epoch
487
+ save_steps:
488
+ desc: null
489
+ value: 500
490
+ save_total_limit:
491
+ desc: null
492
+ value: 2
493
+ save_safetensors:
494
+ desc: null
495
+ value: true
496
+ save_on_each_node:
497
+ desc: null
498
+ value: false
499
+ save_only_model:
500
+ desc: null
501
+ value: false
502
+ restore_callback_states_from_checkpoint:
503
+ desc: null
504
+ value: false
505
+ no_cuda:
506
+ desc: null
507
+ value: false
508
+ use_cpu:
509
+ desc: null
510
+ value: false
511
+ use_mps_device:
512
+ desc: null
513
+ value: false
514
+ seed:
515
+ desc: null
516
+ value: 3407
517
+ data_seed:
518
+ desc: null
519
+ value: null
520
+ jit_mode_eval:
521
+ desc: null
522
+ value: false
523
+ use_ipex:
524
+ desc: null
525
+ value: false
526
+ bf16:
527
+ desc: null
528
+ value: false
529
+ fp16:
530
+ desc: null
531
+ value: true
532
+ fp16_opt_level:
533
+ desc: null
534
+ value: O1
535
+ half_precision_backend:
536
+ desc: null
537
+ value: auto
538
+ bf16_full_eval:
539
+ desc: null
540
+ value: false
541
+ fp16_full_eval:
542
+ desc: null
543
+ value: false
544
+ tf32:
545
+ desc: null
546
+ value: null
547
+ local_rank:
548
+ desc: null
549
+ value: 0
550
+ ddp_backend:
551
+ desc: null
552
+ value: null
553
+ tpu_num_cores:
554
+ desc: null
555
+ value: null
556
+ tpu_metrics_debug:
557
+ desc: null
558
+ value: false
559
+ debug:
560
+ desc: null
561
+ value: []
562
+ dataloader_drop_last:
563
+ desc: null
564
+ value: false
565
+ eval_steps:
566
+ desc: null
567
+ value: null
568
+ dataloader_num_workers:
569
+ desc: null
570
+ value: 0
571
+ dataloader_prefetch_factor:
572
+ desc: null
573
+ value: null
574
+ past_index:
575
+ desc: null
576
+ value: -1
577
+ run_name:
578
+ desc: null
579
+ value: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model
580
+ disable_tqdm:
581
+ desc: null
582
+ value: false
583
+ remove_unused_columns:
584
+ desc: null
585
+ value: true
586
+ label_names:
587
+ desc: null
588
+ value: null
589
+ load_best_model_at_end:
590
+ desc: null
591
+ value: true
592
+ metric_for_best_model:
593
+ desc: null
594
+ value: eval_f1
595
+ greater_is_better:
596
+ desc: null
597
+ value: true
598
+ ignore_data_skip:
599
+ desc: null
600
+ value: false
601
+ fsdp:
602
+ desc: null
603
+ value: []
604
+ fsdp_min_num_params:
605
+ desc: null
606
+ value: 0
607
+ fsdp_config:
608
+ desc: null
609
+ value:
610
+ min_num_params: 0
611
+ xla: false
612
+ xla_fsdp_v2: false
613
+ xla_fsdp_grad_ckpt: false
614
+ fsdp_transformer_layer_cls_to_wrap:
615
+ desc: null
616
+ value: null
617
+ accelerator_config:
618
+ desc: null
619
+ value:
620
+ split_batches: false
621
+ dispatch_batches: null
622
+ even_batches: true
623
+ use_seedable_sampler: true
624
+ non_blocking: false
625
+ gradient_accumulation_kwargs: null
626
+ deepspeed:
627
+ desc: null
628
+ value: null
629
+ label_smoothing_factor:
630
+ desc: null
631
+ value: 0.0
632
+ optim:
633
+ desc: null
634
+ value: adamw_torch
635
+ optim_args:
636
+ desc: null
637
+ value: null
638
+ adafactor:
639
+ desc: null
640
+ value: false
641
+ group_by_length:
642
+ desc: null
643
+ value: true
644
+ length_column_name:
645
+ desc: null
646
+ value: length
647
+ report_to:
648
+ desc: null
649
+ value:
650
+ - wandb
651
+ ddp_find_unused_parameters:
652
+ desc: null
653
+ value: null
654
+ ddp_bucket_cap_mb:
655
+ desc: null
656
+ value: null
657
+ ddp_broadcast_buffers:
658
+ desc: null
659
+ value: null
660
+ dataloader_pin_memory:
661
+ desc: null
662
+ value: true
663
+ dataloader_persistent_workers:
664
+ desc: null
665
+ value: false
666
+ skip_memory_metrics:
667
+ desc: null
668
+ value: true
669
+ use_legacy_prediction_loop:
670
+ desc: null
671
+ value: false
672
+ push_to_hub:
673
+ desc: null
674
+ value: false
675
+ resume_from_checkpoint:
676
+ desc: null
677
+ value: null
678
+ hub_model_id:
679
+ desc: null
680
+ value: null
681
+ hub_strategy:
682
+ desc: null
683
+ value: every_save
684
+ hub_token:
685
+ desc: null
686
+ value: <HUB_TOKEN>
687
+ hub_private_repo:
688
+ desc: null
689
+ value: false
690
+ hub_always_push:
691
+ desc: null
692
+ value: false
693
+ gradient_checkpointing:
694
+ desc: null
695
+ value: false
696
+ gradient_checkpointing_kwargs:
697
+ desc: null
698
+ value: null
699
+ include_inputs_for_metrics:
700
+ desc: null
701
+ value: false
702
+ eval_do_concat_batches:
703
+ desc: null
704
+ value: true
705
+ fp16_backend:
706
+ desc: null
707
+ value: auto
708
+ evaluation_strategy:
709
+ desc: null
710
+ value: null
711
+ push_to_hub_model_id:
712
+ desc: null
713
+ value: null
714
+ push_to_hub_organization:
715
+ desc: null
716
+ value: null
717
+ push_to_hub_token:
718
+ desc: null
719
+ value: <PUSH_TO_HUB_TOKEN>
720
+ mp_parameters:
721
+ desc: null
722
+ value: ''
723
+ auto_find_batch_size:
724
+ desc: null
725
+ value: false
726
+ full_determinism:
727
+ desc: null
728
+ value: false
729
+ torchdynamo:
730
+ desc: null
731
+ value: null
732
+ ray_scope:
733
+ desc: null
734
+ value: last
735
+ ddp_timeout:
736
+ desc: null
737
+ value: 1800
738
+ torch_compile:
739
+ desc: null
740
+ value: false
741
+ torch_compile_backend:
742
+ desc: null
743
+ value: null
744
+ torch_compile_mode:
745
+ desc: null
746
+ value: null
747
+ dispatch_batches:
748
+ desc: null
749
+ value: null
750
+ split_batches:
751
+ desc: null
752
+ value: null
753
+ include_tokens_per_second:
754
+ desc: null
755
+ value: false
756
+ include_num_input_tokens_seen:
757
+ desc: null
758
+ value: false
759
+ neftune_noise_alpha:
760
+ desc: null
761
+ value: null
762
+ optim_target_modules:
763
+ desc: null
764
+ value: null
765
+ batch_eval_metrics:
766
+ desc: null
767
+ value: false
768
+ eval_on_start:
769
+ desc: null
770
+ value: false
771
+ eval_use_gather_object:
772
+ desc: null
773
+ value: false
774
+ model/num_parameters:
775
+ desc: null
776
+ value: 41373323
wandb/run-20240917_122726-z0ntw6ob/files/output.log ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /nfs/production/literature/amina-mardiyyah/envs/llm-prompt/lib/python3.11/site-packages/accelerate/accelerator.py:488: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
2
+ self.scaler = torch.cuda.amp.GradScaler(**kwargs)
3
+ Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
4
+ wandb: WARNING The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.
5
+ Classification Report:
6
+ precision recall f1-score support
7
+ DNA 0.73 0.81 0.77 2494
8
+ RNA 0.71 0.70 0.70 238
9
+ cell_line 0.73 0.78 0.76 1050
10
+ cell_type 0.63 0.75 0.69 775
11
+ protein 0.86 0.87 0.87 6196
12
+ micro avg 0.79 0.84 0.81 10753
13
+ macro avg 0.73 0.78 0.76 10753
14
+ weighted avg 0.80 0.84 0.82 10753
15
+ Classification Report:
16
+ precision recall f1-score support
17
+ DNA 0.78 0.80 0.79 2494
18
+ RNA 0.77 0.79 0.78 238
19
+ cell_line 0.77 0.80 0.78 1050
20
+ cell_type 0.66 0.77 0.71 775
21
+ protein 0.86 0.91 0.88 6196
22
+ micro avg 0.81 0.86 0.83 10753
23
+ macro avg 0.77 0.81 0.79 10753
24
+ weighted avg 0.81 0.86 0.84 10753
25
+ Classification Report:
26
+ precision recall f1-score support
27
+ DNA 0.74 0.87 0.80 2494
28
+ RNA 0.78 0.87 0.82 238
29
+ cell_line 0.80 0.80 0.80 1050
30
+ cell_type 0.68 0.77 0.72 775
31
+ protein 0.88 0.89 0.88 6196
32
+ micro avg 0.82 0.86 0.84 10753
33
+ macro avg 0.78 0.84 0.81 10753
34
+ weighted avg 0.82 0.86 0.84 10753
35
+ Classification Report:
36
+ precision recall f1-score support
37
+ DNA 0.78 0.81 0.80 2494
38
+ RNA 0.79 0.88 0.83 238
39
+ cell_line 0.80 0.81 0.81 1050
40
+ cell_type 0.68 0.79 0.73 775
41
+ protein 0.87 0.90 0.89 6196
42
+ micro avg 0.83 0.87 0.84 10753
43
+ macro avg 0.78 0.84 0.81 10753
44
+ weighted avg 0.83 0.87 0.85 10753
45
+ Classification Report:
46
+ precision recall f1-score support
47
+ DNA 0.77 0.85 0.81 2494
48
+ RNA 0.84 0.87 0.85 238
49
+ cell_line 0.81 0.84 0.82 1050
50
+ cell_type 0.74 0.78 0.76 775
51
+ protein 0.88 0.90 0.89 6196
52
+ micro avg 0.83 0.87 0.85 10753
53
+ macro avg 0.81 0.85 0.83 10753
54
+ weighted avg 0.84 0.87 0.85 10753
55
+ Classification Report:
56
+ precision recall f1-score support
57
+ DNA 0.79 0.83 0.81 2494
58
+ RNA 0.83 0.86 0.84 238
59
+ cell_line 0.80 0.83 0.81 1050
60
+ cell_type 0.71 0.78 0.74 775
61
+ protein 0.88 0.90 0.89 6196
62
+ micro avg 0.84 0.87 0.85 10753
63
+ macro avg 0.80 0.84 0.82 10753
64
+ weighted avg 0.84 0.87 0.85 10753
65
+ Classification Report:
66
+ precision recall f1-score support
67
+ DNA 0.78 0.84 0.81 2494
68
+ RNA 0.84 0.89 0.86 238
69
+ cell_line 0.81 0.86 0.83 1050
70
+ cell_type 0.73 0.80 0.76 775
71
+ protein 0.88 0.90 0.89 6196
72
+ micro avg 0.84 0.87 0.85 10753
73
+ macro avg 0.81 0.86 0.83 10753
74
+ weighted avg 0.84 0.87 0.85 10753
75
+ Classification Report:
76
+ precision recall f1-score support
77
+ DNA 0.78 0.83 0.81 2494
78
+ RNA 0.84 0.87 0.86 238
79
+ cell_line 0.81 0.86 0.83 1050
80
+ cell_type 0.75 0.79 0.77 775
81
+ protein 0.88 0.90 0.89 6196
82
+ micro avg 0.84 0.87 0.85 10753
83
+ macro avg 0.81 0.85 0.83 10753
84
+ weighted avg 0.84 0.87 0.85 10753
85
+ Classification Report:
86
+ precision recall f1-score support
87
+ DNA 0.78 0.84 0.81 2494
88
+ RNA 0.83 0.89 0.86 238
89
+ cell_line 0.81 0.85 0.83 1050
90
+ cell_type 0.74 0.79 0.77 775
91
+ protein 0.88 0.90 0.89 6196
92
+ micro avg 0.84 0.87 0.86 10753
93
+ macro avg 0.81 0.86 0.83 10753
94
+ weighted avg 0.84 0.87 0.86 10753
95
+ Classification Report:
96
+ precision recall f1-score support
97
+ DNA 0.78 0.84 0.81 2494
98
+ RNA 0.83 0.87 0.85 238
99
+ cell_line 0.81 0.85 0.83 1050
100
+ cell_type 0.75 0.78 0.76 775
101
+ protein 0.88 0.90 0.89 6196
102
+ micro avg 0.84 0.87 0.86 10753
103
+ macro avg 0.81 0.85 0.83 10753
104
+ weighted avg 0.84 0.87 0.86 10753
105
+ Classification Report:
106
+ precision recall f1-score support
107
+ DNA 0.74 0.79 0.76 2210
108
+ RNA 0.73 0.76 0.75 287
109
+ cell_line 0.50 0.76 0.61 1057
110
+ cell_type 0.75 0.68 0.71 2761
111
+ protein 0.72 0.87 0.79 10082
112
+ micro avg 0.71 0.82 0.76 16397
113
+ macro avg 0.69 0.77 0.72 16397
114
+ weighted avg 0.72 0.82 0.76 16397
115
+ Test Set Results: {'eval_loss': 0.25901132822036743, 'eval_f1': 0.7609269081317186, 'eval_precision': 0.7112407211028632, 'eval_recall': 0.8180764774044033, 'eval_accuracy': 0.9229162898531661, 'eval_runtime': 8.311, 'eval_samples_per_second': 463.961, 'eval_steps_per_second': 57.995, 'epoch': 9.971126082771896}
wandb/run-20240917_122726-z0ntw6ob/files/requirements.txt ADDED
@@ -0,0 +1,313 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ AutoROM.accept-rom-license==0.6.1
2
+ AutoROM==0.6.1
3
+ Farama-Notifications==0.0.4
4
+ GitPython==3.1.43
5
+ Jinja2==3.1.4
6
+ Mako==1.3.5
7
+ Markdown==3.7
8
+ MarkupSafe==2.1.5
9
+ PyYAML==6.0.2
10
+ Pygments==2.18.0
11
+ QtPy==2.4.1
12
+ SQLAlchemy==2.0.32
13
+ Send2Trash==1.8.3
14
+ Shimmy==1.3.0
15
+ Werkzeug==3.0.4
16
+ absl-py==2.1.0
17
+ accelerate==0.33.0
18
+ aiohappyeyeballs==2.3.5
19
+ aiohttp-retry==2.8.3
20
+ aiohttp==3.10.3
21
+ aiosignal==1.3.1
22
+ ale-py==0.8.1
23
+ alembic==1.13.2
24
+ amqp==5.2.0
25
+ annotated-types==0.7.0
26
+ antlr4-python3-runtime==4.9.3
27
+ anyio==4.4.0
28
+ appdirs==1.4.4
29
+ argon2-cffi-bindings==21.2.0
30
+ argon2-cffi==23.1.0
31
+ arrow==1.3.0
32
+ asttokens==2.4.1
33
+ async-generator==1.10
34
+ async-lru==2.0.4
35
+ asyncssh==2.17.0
36
+ atpublic==5.0
37
+ attrs==24.2.0
38
+ audioread==3.0.1
39
+ babel==2.16.0
40
+ beautifulsoup4==4.12.3
41
+ billiard==4.2.0
42
+ bitsandbytes==0.43.3
43
+ bleach==6.1.0
44
+ blis==0.7.11
45
+ catalogue==2.0.10
46
+ celery==5.4.0
47
+ certifi==2024.7.4
48
+ certipy==0.1.3
49
+ cffi==1.17.0
50
+ charset-normalizer==3.3.2
51
+ click-didyoumean==0.3.1
52
+ click-plugins==1.1.1
53
+ click-repl==0.3.0
54
+ click==8.1.7
55
+ cloudpathlib==0.18.1
56
+ cloudpickle==3.0.0
57
+ colorama==0.4.6
58
+ coloredlogs==15.0.1
59
+ comm==0.2.2
60
+ conda-store==2024.6.1
61
+ confection==0.1.5
62
+ configobj==5.0.8
63
+ contourpy==1.2.1
64
+ cryptography==43.0.0
65
+ cuda-python==11.8.3
66
+ cycler==0.12.1
67
+ cymem==2.0.8
68
+ datasets==2.20.0
69
+ debugpy==1.8.5
70
+ decorator==5.1.1
71
+ defusedxml==0.7.1
72
+ dictdiffer==0.9.0
73
+ dill==0.3.8
74
+ diskcache==5.6.3
75
+ distro==1.9.0
76
+ docker-pycreds==0.4.0
77
+ docstring_parser==0.16
78
+ dpath==2.2.0
79
+ dulwich==0.22.1
80
+ dvc-data==3.16.5
81
+ dvc-http==2.32.0
82
+ dvc-objects==5.1.0
83
+ dvc-render==1.0.2
84
+ dvc-studio-client==0.21.0
85
+ dvc-task==0.4.0
86
+ dvc==3.55.2
87
+ en-core-web-lg==3.7.1
88
+ en-core-web-sm==3.7.1
89
+ entrypoints==0.4
90
+ evaluate==0.4.2
91
+ executing==2.0.1
92
+ fastjsonschema==2.20.0
93
+ filelock==3.15.4
94
+ flatbuffers==24.3.25
95
+ flatten-dict==0.4.2
96
+ flufl.lock==8.1.0
97
+ fonttools==4.53.1
98
+ fqdn==1.5.1
99
+ frozenlist==1.4.1
100
+ fsspec==2024.5.0
101
+ funcy==2.0
102
+ gitdb==4.0.11
103
+ grandalf==0.8
104
+ greenlet==3.0.3
105
+ grpcio==1.66.0
106
+ gto==1.7.1
107
+ gymnasium==0.29.1
108
+ h11==0.14.0
109
+ httpcore==1.0.5
110
+ httpx==0.27.0
111
+ huggingface-hub==0.24.5
112
+ humanfriendly==10.0
113
+ hydra-core==1.3.2
114
+ idna==3.7
115
+ importlib_resources==6.4.5
116
+ ipykernel==6.29.5
117
+ ipython==8.26.0
118
+ ipywidgets==8.1.3
119
+ isoduration==20.11.0
120
+ iterative-telemetry==0.0.8
121
+ jedi==0.19.1
122
+ joblib==1.4.2
123
+ json5==0.9.25
124
+ jsonpointer==3.0.0
125
+ jsonschema-specifications==2023.12.1
126
+ jsonschema==4.23.0
127
+ jupyter-console==6.6.3
128
+ jupyter-events==0.10.0
129
+ jupyter-launcher-shortcuts==4.0.3
130
+ jupyter-lsp==2.2.5
131
+ jupyter-telemetry==0.1.0
132
+ jupyter==1.0.0
133
+ jupyter_client==8.6.2
134
+ jupyter_core==5.7.2
135
+ jupyter_server==2.14.2
136
+ jupyter_server_terminals==0.5.3
137
+ jupyterhub==1.5.1
138
+ jupyterlab-conda-store==2024.6.1
139
+ jupyterlab==4.2.4
140
+ jupyterlab_pygments==0.3.0
141
+ jupyterlab_server==2.27.3
142
+ jupyterlab_widgets==3.0.11
143
+ kiwisolver==1.4.5
144
+ kombu==5.4.0
145
+ lab==8.2
146
+ langcodes==3.4.0
147
+ language_data==1.2.0
148
+ lazy_loader==0.4
149
+ librosa==0.10.2.post1
150
+ llvmlite==0.43.0
151
+ lxml==5.3.0
152
+ marisa-trie==1.2.0
153
+ markdown-it-py==3.0.0
154
+ matplotlib-inline==0.1.7
155
+ matplotlib==3.9.1.post1
156
+ mdurl==0.1.2
157
+ mistune==3.0.2
158
+ mpmath==1.3.0
159
+ msgpack==1.1.0
160
+ multidict==6.0.5
161
+ multiprocess==0.70.16
162
+ murmurhash==1.0.10
163
+ nb_conda_store_kernels==0.1.5
164
+ nbclient==0.10.0
165
+ nbconvert==7.16.4
166
+ nbformat==5.10.4
167
+ nest-asyncio==1.6.0
168
+ networkx==3.3
169
+ nodejs==0.1.1
170
+ notebook==7.2.1
171
+ notebook_shim==0.2.4
172
+ numba==0.60.0
173
+ numpy==1.26.4
174
+ nvidia-cublas-cu12==12.1.3.1
175
+ nvidia-cuda-cupti-cu12==12.1.105
176
+ nvidia-cuda-nvrtc-cu12==12.1.105
177
+ nvidia-cuda-runtime-cu12==12.1.105
178
+ nvidia-cudnn-cu12==8.9.2.26
179
+ nvidia-cudnn-cu12==9.1.0.70
180
+ nvidia-cufft-cu12==11.0.2.54
181
+ nvidia-curand-cu12==10.3.2.106
182
+ nvidia-cusolver-cu12==11.4.5.107
183
+ nvidia-cusparse-cu12==12.1.0.106
184
+ nvidia-nccl-cu12==2.19.3
185
+ nvidia-nccl-cu12==2.20.5
186
+ nvidia-nvjitlink-cu12==12.6.20
187
+ nvidia-nvtx-cu12==12.1.105
188
+ oauthlib==3.2.2
189
+ omegaconf==2.3.0
190
+ onnx==1.16.2
191
+ onnxruntime==1.19.0
192
+ opencv-python==4.10.0.84
193
+ optimum==1.21.4
194
+ optional-django==0.1.0
195
+ orjson==3.10.7
196
+ overrides==7.7.0
197
+ packaging==24.1
198
+ pamela==1.2.0
199
+ pandas==2.2.2
200
+ pandocfilters==1.5.1
201
+ parso==0.8.4
202
+ pathspec==0.12.1
203
+ peft==0.12.0
204
+ pexpect==4.9.0
205
+ pillow==10.4.0
206
+ pip==23.1.2
207
+ platformdirs==3.11.0
208
+ pooch==1.8.2
209
+ preshed==3.0.9
210
+ prometheus_client==0.20.0
211
+ prompt_toolkit==3.0.47
212
+ protobuf==5.27.3
213
+ psutil==6.0.0
214
+ ptyprocess==0.7.0
215
+ pure_eval==0.2.3
216
+ pyOpenSSL==24.2.1
217
+ pyarrow-hotfix==0.6
218
+ pyarrow==17.0.0
219
+ pycparser==2.22
220
+ pydantic==2.8.2
221
+ pydantic_core==2.20.1
222
+ pydot==3.0.1
223
+ pygame==2.6.0
224
+ pygit2==1.15.1
225
+ pygtrie==2.5.0
226
+ pyparsing==3.1.2
227
+ python-dateutil==2.9.0.post0
228
+ python-dotenv==1.0.1
229
+ python-json-logger==2.0.7
230
+ pytz==2024.1
231
+ pyzmq==26.1.0
232
+ qtconsole==5.5.2
233
+ referencing==0.35.1
234
+ regex==2024.7.24
235
+ requests==2.32.3
236
+ rfc3339-validator==0.1.4
237
+ rfc3986-validator==0.1.1
238
+ rich==13.7.1
239
+ rpds-py==0.20.0
240
+ ruamel.yaml.clib==0.2.8
241
+ ruamel.yaml==0.18.6
242
+ safetensors==0.4.4
243
+ scikit-learn==1.5.1
244
+ scipy==1.14.0
245
+ scmrepo==3.3.7
246
+ seaborn==0.13.2
247
+ semver==3.0.2
248
+ sentencepiece==0.2.0
249
+ sentry-sdk==2.14.0
250
+ seqeval==1.2.2
251
+ setproctitle==1.3.3
252
+ setuptools==65.5.0
253
+ shellingham==1.5.4
254
+ shortuuid==1.0.13
255
+ shtab==1.7.1
256
+ simplejson==3.19.2
257
+ six==1.16.0
258
+ smart-open==7.0.4
259
+ smmap==5.0.1
260
+ sniffio==1.3.1
261
+ soundfile==0.12.1
262
+ soupsieve==2.5
263
+ sox==1.5.0
264
+ soxr==0.5.0.post1
265
+ spacy-legacy==3.0.12
266
+ spacy-loggers==1.0.5
267
+ spacy==3.7.5
268
+ sqltrie==0.11.1
269
+ srsly==2.4.8
270
+ stable_baselines3==2.3.2
271
+ stack-data==0.6.3
272
+ sympy==1.13.2
273
+ tabulate==0.9.0
274
+ tensorboard-data-server==0.7.2
275
+ tensorboard==2.17.1
276
+ terminado==0.18.1
277
+ thinc==8.2.5
278
+ threadpoolctl==3.5.0
279
+ tinycss2==1.3.0
280
+ tokenizers==0.19.1
281
+ tomlkit==0.13.2
282
+ torch==2.4.1
283
+ torchaudio==2.4.1
284
+ torchvision==0.19.1
285
+ tornado==6.4.1
286
+ tqdm==4.66.5
287
+ traitlets==5.14.3
288
+ transformers==4.43.4
289
+ triton==3.0.0
290
+ trl==0.9.6
291
+ txt2tags==3.9
292
+ typer==0.12.3
293
+ types-python-dateutil==2.9.0.20240316
294
+ typing_extensions==4.12.2
295
+ tyro==0.8.6
296
+ tzdata==2024.1
297
+ uri-template==1.3.0
298
+ urllib3==2.2.2
299
+ vine==5.1.0
300
+ voluptuous==0.15.2
301
+ wandb==0.17.9
302
+ wasabi==1.1.3
303
+ wcwidth==0.2.13
304
+ weasel==0.4.1
305
+ webcolors==24.8.0
306
+ webencodings==0.5.1
307
+ websocket-client==1.8.0
308
+ widgetsnbextension==4.0.11
309
+ wrapt==1.16.0
310
+ xlrd==2.0.1
311
+ xxhash==3.4.1
312
+ yarl==1.9.4
313
+ zc.lockfile==3.0.post1
wandb/run-20240917_122726-z0ntw6ob/files/wandb-metadata.json ADDED
@@ -0,0 +1,281 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "os": "Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28",
3
+ "python": "3.11.4",
4
+ "heartbeatAt": "2024-09-17T11:27:27.284628",
5
+ "startedAt": "2024-09-17T11:27:26.786345",
6
+ "docker": null,
7
+ "cuda": null,
8
+ "args": [],
9
+ "state": "running",
10
+ "program": "<python with no main file>",
11
+ "codePathLocal": null,
12
+ "host": "codon-gpu-006.ebi.ac.uk",
13
+ "username": "amrufai",
14
+ "executable": "/nfs/production/literature/amina-mardiyyah/envs/llm-prompt/bin/python",
15
+ "cpu_count": 48,
16
+ "cpu_count_logical": 48,
17
+ "cpu_freq": {
18
+ "current": 2758.33775,
19
+ "min": 0.0,
20
+ "max": 0.0
21
+ },
22
+ "cpu_freq_per_core": [
23
+ {
24
+ "current": 1157.924,
25
+ "min": 0.0,
26
+ "max": 0.0
27
+ },
28
+ {
29
+ "current": 2800.0,
30
+ "min": 0.0,
31
+ "max": 0.0
32
+ },
33
+ {
34
+ "current": 2800.0,
35
+ "min": 0.0,
36
+ "max": 0.0
37
+ },
38
+ {
39
+ "current": 2800.0,
40
+ "min": 0.0,
41
+ "max": 0.0
42
+ },
43
+ {
44
+ "current": 2800.0,
45
+ "min": 0.0,
46
+ "max": 0.0
47
+ },
48
+ {
49
+ "current": 2800.0,
50
+ "min": 0.0,
51
+ "max": 0.0
52
+ },
53
+ {
54
+ "current": 2800.0,
55
+ "min": 0.0,
56
+ "max": 0.0
57
+ },
58
+ {
59
+ "current": 2800.0,
60
+ "min": 0.0,
61
+ "max": 0.0
62
+ },
63
+ {
64
+ "current": 2800.0,
65
+ "min": 0.0,
66
+ "max": 0.0
67
+ },
68
+ {
69
+ "current": 2800.0,
70
+ "min": 0.0,
71
+ "max": 0.0
72
+ },
73
+ {
74
+ "current": 2800.0,
75
+ "min": 0.0,
76
+ "max": 0.0
77
+ },
78
+ {
79
+ "current": 2800.0,
80
+ "min": 0.0,
81
+ "max": 0.0
82
+ },
83
+ {
84
+ "current": 2800.0,
85
+ "min": 0.0,
86
+ "max": 0.0
87
+ },
88
+ {
89
+ "current": 2800.0,
90
+ "min": 0.0,
91
+ "max": 0.0
92
+ },
93
+ {
94
+ "current": 2800.0,
95
+ "min": 0.0,
96
+ "max": 0.0
97
+ },
98
+ {
99
+ "current": 2800.0,
100
+ "min": 0.0,
101
+ "max": 0.0
102
+ },
103
+ {
104
+ "current": 2800.0,
105
+ "min": 0.0,
106
+ "max": 0.0
107
+ },
108
+ {
109
+ "current": 2800.0,
110
+ "min": 0.0,
111
+ "max": 0.0
112
+ },
113
+ {
114
+ "current": 2800.0,
115
+ "min": 0.0,
116
+ "max": 0.0
117
+ },
118
+ {
119
+ "current": 2800.0,
120
+ "min": 0.0,
121
+ "max": 0.0
122
+ },
123
+ {
124
+ "current": 2800.0,
125
+ "min": 0.0,
126
+ "max": 0.0
127
+ },
128
+ {
129
+ "current": 2800.0,
130
+ "min": 0.0,
131
+ "max": 0.0
132
+ },
133
+ {
134
+ "current": 2800.0,
135
+ "min": 0.0,
136
+ "max": 0.0
137
+ },
138
+ {
139
+ "current": 2800.0,
140
+ "min": 0.0,
141
+ "max": 0.0
142
+ },
143
+ {
144
+ "current": 2800.0,
145
+ "min": 0.0,
146
+ "max": 0.0
147
+ },
148
+ {
149
+ "current": 2800.0,
150
+ "min": 0.0,
151
+ "max": 0.0
152
+ },
153
+ {
154
+ "current": 2800.0,
155
+ "min": 0.0,
156
+ "max": 0.0
157
+ },
158
+ {
159
+ "current": 2800.0,
160
+ "min": 0.0,
161
+ "max": 0.0
162
+ },
163
+ {
164
+ "current": 2800.0,
165
+ "min": 0.0,
166
+ "max": 0.0
167
+ },
168
+ {
169
+ "current": 2800.0,
170
+ "min": 0.0,
171
+ "max": 0.0
172
+ },
173
+ {
174
+ "current": 2800.0,
175
+ "min": 0.0,
176
+ "max": 0.0
177
+ },
178
+ {
179
+ "current": 2800.0,
180
+ "min": 0.0,
181
+ "max": 0.0
182
+ },
183
+ {
184
+ "current": 2800.0,
185
+ "min": 0.0,
186
+ "max": 0.0
187
+ },
188
+ {
189
+ "current": 2800.0,
190
+ "min": 0.0,
191
+ "max": 0.0
192
+ },
193
+ {
194
+ "current": 2800.0,
195
+ "min": 0.0,
196
+ "max": 0.0
197
+ },
198
+ {
199
+ "current": 2800.0,
200
+ "min": 0.0,
201
+ "max": 0.0
202
+ },
203
+ {
204
+ "current": 2800.0,
205
+ "min": 0.0,
206
+ "max": 0.0
207
+ },
208
+ {
209
+ "current": 2800.0,
210
+ "min": 0.0,
211
+ "max": 0.0
212
+ },
213
+ {
214
+ "current": 2800.0,
215
+ "min": 0.0,
216
+ "max": 0.0
217
+ },
218
+ {
219
+ "current": 2800.0,
220
+ "min": 0.0,
221
+ "max": 0.0
222
+ },
223
+ {
224
+ "current": 2800.0,
225
+ "min": 0.0,
226
+ "max": 0.0
227
+ },
228
+ {
229
+ "current": 2800.0,
230
+ "min": 0.0,
231
+ "max": 0.0
232
+ },
233
+ {
234
+ "current": 2800.0,
235
+ "min": 0.0,
236
+ "max": 0.0
237
+ },
238
+ {
239
+ "current": 2800.0,
240
+ "min": 0.0,
241
+ "max": 0.0
242
+ },
243
+ {
244
+ "current": 2800.0,
245
+ "min": 0.0,
246
+ "max": 0.0
247
+ },
248
+ {
249
+ "current": 2800.0,
250
+ "min": 0.0,
251
+ "max": 0.0
252
+ },
253
+ {
254
+ "current": 2800.0,
255
+ "min": 0.0,
256
+ "max": 0.0
257
+ },
258
+ {
259
+ "current": 2800.0,
260
+ "min": 0.0,
261
+ "max": 0.0
262
+ }
263
+ ],
264
+ "disk": {
265
+ "/": {
266
+ "total": 47.760292053222656,
267
+ "used": 15.848060607910156
268
+ }
269
+ },
270
+ "gpu": "NVIDIA A100 80GB PCIe",
271
+ "gpu_count": 1,
272
+ "gpu_devices": [
273
+ {
274
+ "name": "NVIDIA A100 80GB PCIe",
275
+ "memory_total": 85899345920
276
+ }
277
+ ],
278
+ "memory": {
279
+ "total": 502.8375930786133
280
+ }
281
+ }
wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"train/loss": 0.1199, "train/grad_norm": 1.1583800315856934, "train/learning_rate": 7.722007722007723e-09, "train/epoch": 9.971126082771896, "train/global_step": 2590, "_timestamp": 1726573085.5001044, "_runtime": 638.6713645458221, "_step": 22, "eval/loss": 0.25901132822036743, "eval/f1": 0.7609269081317186, "eval/precision": 0.7112407211028632, "eval/recall": 0.8180764774044033, "eval/accuracy": 0.9229162898531661, "eval/runtime": 8.311, "eval/samples_per_second": 463.961, "eval/steps_per_second": 57.995, "train_runtime": 624.2924, "train_samples_per_second": 266.205, "train_steps_per_second": 4.149, "total_flos": 4413788351238426.0, "train_loss": 0.181362230160982}
wandb/run-20240917_122726-z0ntw6ob/logs/debug-internal.log ADDED
@@ -0,0 +1,670 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2024-09-17 12:27:26,830 INFO StreamThr :4132313 [internal.py:wandb_internal():85] W&B internal server running at pid: 4132313, started at: 2024-09-17 12:27:26.826981
2
+ 2024-09-17 12:27:26,834 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status
3
+ 2024-09-17 12:27:26,838 INFO WriterThread:4132313 [datastore.py:open_for_write():87] open: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/run-z0ntw6ob.wandb
4
+ 2024-09-17 12:27:26,844 DEBUG SenderThread:4132313 [sender.py:send():391] send: header
5
+ 2024-09-17 12:27:26,848 DEBUG SenderThread:4132313 [sender.py:send():391] send: run
6
+ 2024-09-17 12:27:27,198 INFO SenderThread:4132313 [dir_watcher.py:__init__():211] watching files in: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files
7
+ 2024-09-17 12:27:27,198 INFO SenderThread:4132313 [sender.py:_start_run_threads():1200] run started: z0ntw6ob with start time 1726572446.82874
8
+ 2024-09-17 12:27:27,214 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: run_start
9
+ 2024-09-17 12:27:27,243 DEBUG HandlerThread:4132313 [system_info.py:__init__():26] System info init
10
+ 2024-09-17 12:27:27,243 DEBUG HandlerThread:4132313 [system_info.py:__init__():41] System info init done
11
+ 2024-09-17 12:27:27,244 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor
12
+ 2024-09-17 12:27:27,244 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads
13
+ 2024-09-17 12:27:27,244 INFO HandlerThread:4132313 [system_monitor.py:probe():214] Collecting system info
14
+ 2024-09-17 12:27:27,247 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring
15
+ 2024-09-17 12:27:27,250 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring
16
+ 2024-09-17 12:27:27,252 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring
17
+ 2024-09-17 12:27:27,253 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring
18
+ 2024-09-17 12:27:27,257 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring
19
+ 2024-09-17 12:27:27,284 DEBUG HandlerThread:4132313 [system_info.py:probe():152] Probing system
20
+ 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [gitlib.py:_init_repo():56] git repository is invalid
21
+ 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [system_info.py:probe():200] Probing system done
22
+ 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28', 'python': '3.11.4', 'heartbeatAt': '2024-09-17T11:27:27.284628', 'startedAt': '2024-09-17T11:27:26.786345', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '<python with no main file>', 'codePathLocal': None, 'host': 'codon-gpu-006.ebi.ac.uk', 'username': 'amrufai', 'executable': '/nfs/production/literature/amina-mardiyyah/envs/llm-prompt/bin/python', 'cpu_count': 48, 'cpu_count_logical': 48, 'cpu_freq': {'current': 2758.33775, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 1157.924, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 47.760292053222656, 'used': 15.848060607910156}}, 'gpu': 'NVIDIA A100 80GB PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}], 'memory': {'total': 502.8375930786133}}
23
+ 2024-09-17 12:27:27,291 INFO HandlerThread:4132313 [system_monitor.py:probe():224] Finished collecting system info
24
+ 2024-09-17 12:27:27,291 INFO HandlerThread:4132313 [system_monitor.py:probe():227] Publishing system info
25
+ 2024-09-17 12:27:27,296 INFO HandlerThread:4132313 [system_monitor.py:probe():229] Finished publishing system info
26
+ 2024-09-17 12:27:27,320 DEBUG SenderThread:4132313 [sender.py:send():391] send: files
27
+ 2024-09-17 12:27:27,320 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-metadata.json with policy now
28
+ 2024-09-17 12:27:27,845 INFO wandb-upload_0:4132313 [upload_job.py:push():130] Uploaded file /tmp/tmpr_ev51fvwandb/7xtqy0my-wandb-metadata.json
29
+ 2024-09-17 12:27:27,876 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: python_packages
30
+ 2024-09-17 12:27:27,877 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
31
+ 2024-09-17 12:27:27,877 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: python_packages
32
+ 2024-09-17 12:27:27,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
33
+ 2024-09-17 12:27:27,883 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
34
+ 2024-09-17 12:27:27,923 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause
35
+ 2024-09-17 12:27:27,923 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread
36
+ 2024-09-17 12:27:27,923 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor
37
+ 2024-09-17 12:27:27,924 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop
38
+ 2024-09-17 12:27:27,925 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor
39
+ 2024-09-17 12:27:27,926 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop
40
+ 2024-09-17 12:27:27,927 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics
41
+ 2024-09-17 12:27:27,927 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor
42
+ 2024-09-17 12:27:28,000 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor
43
+ 2024-09-17 12:27:28,000 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor
44
+ 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor
45
+ 2024-09-17 12:27:28,001 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume
46
+ 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread
47
+ 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor
48
+ 2024-09-17 12:27:28,002 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads
49
+ 2024-09-17 12:27:28,005 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring
50
+ 2024-09-17 12:27:28,008 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring
51
+ 2024-09-17 12:27:28,008 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring
52
+ 2024-09-17 12:27:28,010 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring
53
+ 2024-09-17 12:27:28,011 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring
54
+ 2024-09-17 12:27:28,066 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
55
+ 2024-09-17 12:27:28,067 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
56
+ 2024-09-17 12:27:28,204 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/requirements.txt
57
+ 2024-09-17 12:27:28,205 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
58
+ 2024-09-17 12:27:28,205 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-metadata.json
59
+ 2024-09-17 12:27:30,204 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
60
+ 2024-09-17 12:27:32,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
61
+ 2024-09-17 12:27:32,935 DEBUG SenderThread:4132313 [sender.py:send():391] send: config
62
+ 2024-09-17 12:27:32,937 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
63
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
64
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
65
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
66
+ 2024-09-17 12:27:32,938 WARNING SenderThread:4132313 [sender.py:send_metric():1417] Seen metric with glob (shouldn't happen)
67
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
68
+ 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
69
+ 2024-09-17 12:27:32,939 DEBUG SenderThread:4132313 [sender.py:send():391] send: config
70
+ 2024-09-17 12:27:34,211 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
71
+ 2024-09-17 12:27:37,878 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
72
+ 2024-09-17 12:27:37,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
73
+ 2024-09-17 12:27:42,877 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
74
+ 2024-09-17 12:27:42,878 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
75
+ 2024-09-17 12:27:43,034 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
76
+ 2024-09-17 12:27:47,879 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
77
+ 2024-09-17 12:27:48,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
78
+ 2024-09-17 12:27:54,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
79
+ 2024-09-17 12:27:57,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
80
+ 2024-09-17 12:27:57,882 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
81
+ 2024-09-17 12:27:57,922 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
82
+ 2024-09-17 12:28:00,105 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
83
+ 2024-09-17 12:28:01,242 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml
84
+ 2024-09-17 12:28:05,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
85
+ 2024-09-17 12:28:07,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
86
+ 2024-09-17 12:28:10,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
87
+ 2024-09-17 12:28:12,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
88
+ 2024-09-17 12:28:12,882 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
89
+ 2024-09-17 12:28:16,038 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
90
+ 2024-09-17 12:28:17,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
91
+ 2024-09-17 12:28:21,884 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
92
+ 2024-09-17 12:28:27,884 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
93
+ 2024-09-17 12:28:27,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
94
+ 2024-09-17 12:28:27,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
95
+ 2024-09-17 12:28:27,886 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
96
+ 2024-09-17 12:28:28,011 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop
97
+ 2024-09-17 12:28:28,056 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
98
+ 2024-09-17 12:28:31,893 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
99
+ 2024-09-17 12:28:31,897 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
100
+ 2024-09-17 12:28:31,897 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
101
+ 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
102
+ 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
103
+ 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
104
+ 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
105
+ 2024-09-17 12:28:31,904 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
106
+ 2024-09-17 12:28:32,281 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
107
+ 2024-09-17 12:28:32,889 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
108
+ 2024-09-17 12:28:33,282 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml
109
+ 2024-09-17 12:28:37,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
110
+ 2024-09-17 12:28:37,167 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
111
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
112
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
113
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
114
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
115
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
116
+ 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
117
+ 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
118
+ 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
119
+ 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
120
+ 2024-09-17 12:28:37,174 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
121
+ 2024-09-17 12:28:37,287 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
122
+ 2024-09-17 12:28:38,159 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
123
+ 2024-09-17 12:28:38,160 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
124
+ 2024-09-17 12:28:39,291 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
125
+ 2024-09-17 12:28:42,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
126
+ 2024-09-17 12:28:42,886 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
127
+ 2024-09-17 12:28:44,160 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
128
+ 2024-09-17 12:28:48,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
129
+ 2024-09-17 12:28:49,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
130
+ 2024-09-17 12:28:55,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
131
+ 2024-09-17 12:28:57,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
132
+ 2024-09-17 12:28:57,887 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
133
+ 2024-09-17 12:28:58,095 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
134
+ 2024-09-17 12:28:58,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
135
+ 2024-09-17 12:29:00,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
136
+ 2024-09-17 12:29:05,170 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
137
+ 2024-09-17 12:29:06,343 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml
138
+ 2024-09-17 12:29:08,159 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
139
+ 2024-09-17 12:29:11,164 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
140
+ 2024-09-17 12:29:12,889 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
141
+ 2024-09-17 12:29:12,890 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
142
+ 2024-09-17 12:29:17,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
143
+ 2024-09-17 12:29:18,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
144
+ 2024-09-17 12:29:22,164 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
145
+ 2024-09-17 12:29:27,888 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
146
+ 2024-09-17 12:29:27,889 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
147
+ 2024-09-17 12:29:28,113 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
148
+ 2024-09-17 12:29:28,118 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
149
+ 2024-09-17 12:29:28,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
150
+ 2024-09-17 12:29:33,168 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
151
+ 2024-09-17 12:29:38,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
152
+ 2024-09-17 12:29:38,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
153
+ 2024-09-17 12:29:38,188 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
154
+ 2024-09-17 12:29:38,189 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
155
+ 2024-09-17 12:29:38,189 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
156
+ 2024-09-17 12:29:38,196 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
157
+ 2024-09-17 12:29:38,383 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
158
+ 2024-09-17 12:29:42,913 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
159
+ 2024-09-17 12:29:42,914 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
160
+ 2024-09-17 12:29:43,192 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
161
+ 2024-09-17 12:29:43,343 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
162
+ 2024-09-17 12:29:43,345 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
163
+ 2024-09-17 12:29:43,345 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
164
+ 2024-09-17 12:29:43,349 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
165
+ 2024-09-17 12:29:43,389 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
166
+ 2024-09-17 12:29:45,554 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
167
+ 2024-09-17 12:29:48,168 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
168
+ 2024-09-17 12:29:49,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
169
+ 2024-09-17 12:29:54,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
170
+ 2024-09-17 12:29:57,893 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
171
+ 2024-09-17 12:29:57,894 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
172
+ 2024-09-17 12:29:58,026 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
173
+ 2024-09-17 12:29:58,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
174
+ 2024-09-17 12:30:00,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
175
+ 2024-09-17 12:30:05,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
176
+ 2024-09-17 12:30:08,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
177
+ 2024-09-17 12:30:11,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
178
+ 2024-09-17 12:30:12,894 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
179
+ 2024-09-17 12:30:12,894 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
180
+ 2024-09-17 12:30:16,175 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
181
+ 2024-09-17 12:30:18,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
182
+ 2024-09-17 12:30:22,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
183
+ 2024-09-17 12:30:27,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
184
+ 2024-09-17 12:30:27,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
185
+ 2024-09-17 12:30:27,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
186
+ 2024-09-17 12:30:28,105 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
187
+ 2024-09-17 12:30:28,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
188
+ 2024-09-17 12:30:32,176 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
189
+ 2024-09-17 12:30:37,176 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
190
+ 2024-09-17 12:30:38,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
191
+ 2024-09-17 12:30:42,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
192
+ 2024-09-17 12:30:42,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
193
+ 2024-09-17 12:30:43,087 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
194
+ 2024-09-17 12:30:44,559 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
195
+ 2024-09-17 12:30:44,561 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
196
+ 2024-09-17 12:30:44,561 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
197
+ 2024-09-17 12:30:44,568 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
198
+ 2024-09-17 12:30:44,626 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
199
+ 2024-09-17 12:30:48,189 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
200
+ 2024-09-17 12:30:48,214 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
201
+ 2024-09-17 12:30:49,050 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
202
+ 2024-09-17 12:30:49,053 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
203
+ 2024-09-17 12:30:49,053 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
204
+ 2024-09-17 12:30:49,057 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
205
+ 2024-09-17 12:30:49,633 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
206
+ 2024-09-17 12:30:51,690 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
207
+ 2024-09-17 12:30:54,178 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
208
+ 2024-09-17 12:30:57,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
209
+ 2024-09-17 12:30:57,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
210
+ 2024-09-17 12:30:58,125 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
211
+ 2024-09-17 12:30:58,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
212
+ 2024-09-17 12:31:00,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
213
+ 2024-09-17 12:31:06,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
214
+ 2024-09-17 12:31:08,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
215
+ 2024-09-17 12:31:11,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
216
+ 2024-09-17 12:31:12,899 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
217
+ 2024-09-17 12:31:12,900 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
218
+ 2024-09-17 12:31:17,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
219
+ 2024-09-17 12:31:18,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
220
+ 2024-09-17 12:31:22,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
221
+ 2024-09-17 12:31:27,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
222
+ 2024-09-17 12:31:27,899 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
223
+ 2024-09-17 12:31:27,900 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
224
+ 2024-09-17 12:31:28,115 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
225
+ 2024-09-17 12:31:28,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
226
+ 2024-09-17 12:31:33,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
227
+ 2024-09-17 12:31:38,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
228
+ 2024-09-17 12:31:38,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
229
+ 2024-09-17 12:31:42,546 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
230
+ 2024-09-17 12:31:42,548 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
231
+ 2024-09-17 12:31:42,548 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
232
+ 2024-09-17 12:31:42,553 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
233
+ 2024-09-17 12:31:42,750 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
234
+ 2024-09-17 12:31:42,900 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
235
+ 2024-09-17 12:31:42,901 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
236
+ 2024-09-17 12:31:43,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
237
+ 2024-09-17 12:31:47,028 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
238
+ 2024-09-17 12:31:47,030 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
239
+ 2024-09-17 12:31:47,030 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
240
+ 2024-09-17 12:31:47,034 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
241
+ 2024-09-17 12:31:47,755 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
242
+ 2024-09-17 12:31:48,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
243
+ 2024-09-17 12:31:48,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
244
+ 2024-09-17 12:31:49,759 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
245
+ 2024-09-17 12:31:54,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
246
+ 2024-09-17 12:31:57,903 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
247
+ 2024-09-17 12:31:57,903 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
248
+ 2024-09-17 12:31:58,101 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
249
+ 2024-09-17 12:31:58,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
250
+ 2024-09-17 12:31:59,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
251
+ 2024-09-17 12:32:04,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
252
+ 2024-09-17 12:32:08,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
253
+ 2024-09-17 12:32:09,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
254
+ 2024-09-17 12:32:12,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
255
+ 2024-09-17 12:32:12,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
256
+ 2024-09-17 12:32:15,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
257
+ 2024-09-17 12:32:18,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
258
+ 2024-09-17 12:32:21,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
259
+ 2024-09-17 12:32:26,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
260
+ 2024-09-17 12:32:27,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
261
+ 2024-09-17 12:32:27,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
262
+ 2024-09-17 12:32:28,122 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
263
+ 2024-09-17 12:32:28,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
264
+ 2024-09-17 12:32:32,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
265
+ 2024-09-17 12:32:38,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
266
+ 2024-09-17 12:32:38,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
267
+ 2024-09-17 12:32:42,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
268
+ 2024-09-17 12:32:42,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
269
+ 2024-09-17 12:32:43,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
270
+ 2024-09-17 12:32:45,990 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
271
+ 2024-09-17 12:32:45,992 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
272
+ 2024-09-17 12:32:45,992 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
273
+ 2024-09-17 12:32:45,997 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
274
+ 2024-09-17 12:32:46,827 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
275
+ 2024-09-17 12:32:48,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
276
+ 2024-09-17 12:32:48,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
277
+ 2024-09-17 12:32:50,431 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
278
+ 2024-09-17 12:32:50,433 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
279
+ 2024-09-17 12:32:50,433 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
280
+ 2024-09-17 12:32:50,437 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
281
+ 2024-09-17 12:32:50,832 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
282
+ 2024-09-17 12:32:51,907 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
283
+ 2024-09-17 12:32:54,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
284
+ 2024-09-17 12:32:57,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
285
+ 2024-09-17 12:32:57,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
286
+ 2024-09-17 12:32:58,050 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
287
+ 2024-09-17 12:32:58,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
288
+ 2024-09-17 12:32:59,188 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
289
+ 2024-09-17 12:33:05,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
290
+ 2024-09-17 12:33:08,196 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
291
+ 2024-09-17 12:33:10,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
292
+ 2024-09-17 12:33:12,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
293
+ 2024-09-17 12:33:12,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
294
+ 2024-09-17 12:33:16,193 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
295
+ 2024-09-17 12:33:18,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
296
+ 2024-09-17 12:33:21,193 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
297
+ 2024-09-17 12:33:26,199 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
298
+ 2024-09-17 12:33:27,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
299
+ 2024-09-17 12:33:27,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
300
+ 2024-09-17 12:33:28,135 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
301
+ 2024-09-17 12:33:28,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
302
+ 2024-09-17 12:33:31,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
303
+ 2024-09-17 12:33:37,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
304
+ 2024-09-17 12:33:38,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
305
+ 2024-09-17 12:33:42,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
306
+ 2024-09-17 12:33:42,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
307
+ 2024-09-17 12:33:42,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
308
+ 2024-09-17 12:33:48,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
309
+ 2024-09-17 12:33:48,197 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
310
+ 2024-09-17 12:33:50,740 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
311
+ 2024-09-17 12:33:50,741 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
312
+ 2024-09-17 12:33:50,742 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
313
+ 2024-09-17 12:33:50,745 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
314
+ 2024-09-17 12:33:50,980 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
315
+ 2024-09-17 12:33:53,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
316
+ 2024-09-17 12:33:55,232 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
317
+ 2024-09-17 12:33:55,235 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
318
+ 2024-09-17 12:33:55,235 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
319
+ 2024-09-17 12:33:55,239 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
320
+ 2024-09-17 12:33:55,987 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
321
+ 2024-09-17 12:33:56,989 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
322
+ 2024-09-17 12:33:57,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
323
+ 2024-09-17 12:33:57,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
324
+ 2024-09-17 12:33:58,051 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
325
+ 2024-09-17 12:33:58,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
326
+ 2024-09-17 12:33:59,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
327
+ 2024-09-17 12:34:04,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
328
+ 2024-09-17 12:34:08,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
329
+ 2024-09-17 12:34:09,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
330
+ 2024-09-17 12:34:12,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
331
+ 2024-09-17 12:34:12,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
332
+ 2024-09-17 12:34:15,208 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
333
+ 2024-09-17 12:34:18,205 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
334
+ 2024-09-17 12:34:20,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
335
+ 2024-09-17 12:34:26,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
336
+ 2024-09-17 12:34:27,906 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
337
+ 2024-09-17 12:34:27,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
338
+ 2024-09-17 12:34:28,050 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
339
+ 2024-09-17 12:34:28,208 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
340
+ 2024-09-17 12:34:31,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
341
+ 2024-09-17 12:34:37,210 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
342
+ 2024-09-17 12:34:38,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
343
+ 2024-09-17 12:34:42,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
344
+ 2024-09-17 12:34:42,906 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
345
+ 2024-09-17 12:34:42,907 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
346
+ 2024-09-17 12:34:48,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
347
+ 2024-09-17 12:34:48,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
348
+ 2024-09-17 12:34:50,281 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
349
+ 2024-09-17 12:34:50,283 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
350
+ 2024-09-17 12:34:50,283 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
351
+ 2024-09-17 12:34:50,290 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
352
+ 2024-09-17 12:34:51,047 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
353
+ 2024-09-17 12:34:54,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
354
+ 2024-09-17 12:34:55,374 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
355
+ 2024-09-17 12:34:55,376 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
356
+ 2024-09-17 12:34:55,376 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
357
+ 2024-09-17 12:34:55,381 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
358
+ 2024-09-17 12:34:56,054 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
359
+ 2024-09-17 12:34:57,070 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
360
+ 2024-09-17 12:34:57,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
361
+ 2024-09-17 12:34:57,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
362
+ 2024-09-17 12:34:58,118 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
363
+ 2024-09-17 12:34:58,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
364
+ 2024-09-17 12:35:00,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
365
+ 2024-09-17 12:35:06,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
366
+ 2024-09-17 12:35:08,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
367
+ 2024-09-17 12:35:11,215 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
368
+ 2024-09-17 12:35:12,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
369
+ 2024-09-17 12:35:12,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
370
+ 2024-09-17 12:35:16,217 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
371
+ 2024-09-17 12:35:18,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
372
+ 2024-09-17 12:35:21,217 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
373
+ 2024-09-17 12:35:26,224 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
374
+ 2024-09-17 12:35:27,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
375
+ 2024-09-17 12:35:27,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
376
+ 2024-09-17 12:35:28,144 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
377
+ 2024-09-17 12:35:28,212 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
378
+ 2024-09-17 12:35:32,223 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
379
+ 2024-09-17 12:35:37,223 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
380
+ 2024-09-17 12:35:38,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
381
+ 2024-09-17 12:35:42,225 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
382
+ 2024-09-17 12:35:42,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
383
+ 2024-09-17 12:35:42,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
384
+ 2024-09-17 12:35:48,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
385
+ 2024-09-17 12:35:48,222 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
386
+ 2024-09-17 12:35:49,141 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
387
+ 2024-09-17 12:35:49,143 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
388
+ 2024-09-17 12:35:49,144 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
389
+ 2024-09-17 12:35:49,146 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
390
+ 2024-09-17 12:35:50,122 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
391
+ 2024-09-17 12:35:53,263 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
392
+ 2024-09-17 12:35:53,590 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
393
+ 2024-09-17 12:35:53,592 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
394
+ 2024-09-17 12:35:53,593 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
395
+ 2024-09-17 12:35:53,597 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
396
+ 2024-09-17 12:35:54,127 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
397
+ 2024-09-17 12:35:55,221 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
398
+ 2024-09-17 12:35:57,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
399
+ 2024-09-17 12:35:57,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
400
+ 2024-09-17 12:35:58,079 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
401
+ 2024-09-17 12:35:58,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
402
+ 2024-09-17 12:35:59,228 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
403
+ 2024-09-17 12:36:04,228 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
404
+ 2024-09-17 12:36:08,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
405
+ 2024-09-17 12:36:09,236 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
406
+ 2024-09-17 12:36:12,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
407
+ 2024-09-17 12:36:12,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
408
+ 2024-09-17 12:36:15,235 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
409
+ 2024-09-17 12:36:18,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
410
+ 2024-09-17 12:36:20,241 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
411
+ 2024-09-17 12:36:25,244 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
412
+ 2024-09-17 12:36:27,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
413
+ 2024-09-17 12:36:27,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
414
+ 2024-09-17 12:36:28,173 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
415
+ 2024-09-17 12:36:28,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
416
+ 2024-09-17 12:36:31,242 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
417
+ 2024-09-17 12:36:36,242 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
418
+ 2024-09-17 12:36:38,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
419
+ 2024-09-17 12:36:41,243 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
420
+ 2024-09-17 12:36:42,957 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
421
+ 2024-09-17 12:36:42,958 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
422
+ 2024-09-17 12:36:47,089 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
423
+ 2024-09-17 12:36:47,091 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
424
+ 2024-09-17 12:36:47,091 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
425
+ 2024-09-17 12:36:47,092 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
426
+ 2024-09-17 12:36:47,096 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
427
+ 2024-09-17 12:36:47,279 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
428
+ 2024-09-17 12:36:48,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
429
+ 2024-09-17 12:36:51,557 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
430
+ 2024-09-17 12:36:51,560 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
431
+ 2024-09-17 12:36:51,560 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
432
+ 2024-09-17 12:36:51,565 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
433
+ 2024-09-17 12:36:52,285 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
434
+ 2024-09-17 12:36:52,566 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
435
+ 2024-09-17 12:36:53,287 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
436
+ 2024-09-17 12:36:57,956 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
437
+ 2024-09-17 12:36:57,957 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
438
+ 2024-09-17 12:36:58,101 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
439
+ 2024-09-17 12:36:58,107 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
440
+ 2024-09-17 12:36:58,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
441
+ 2024-09-17 12:37:03,344 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
442
+ 2024-09-17 12:37:08,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
443
+ 2024-09-17 12:37:08,344 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
444
+ 2024-09-17 12:37:12,956 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
445
+ 2024-09-17 12:37:12,957 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
446
+ 2024-09-17 12:37:14,124 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
447
+ 2024-09-17 12:37:18,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
448
+ 2024-09-17 12:37:19,349 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
449
+ 2024-09-17 12:37:24,350 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
450
+ 2024-09-17 12:37:27,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
451
+ 2024-09-17 12:37:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
452
+ 2024-09-17 12:37:28,152 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
453
+ 2024-09-17 12:37:28,221 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
454
+ 2024-09-17 12:37:29,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
455
+ 2024-09-17 12:37:35,356 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
456
+ 2024-09-17 12:37:38,221 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
457
+ 2024-09-17 12:37:40,356 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
458
+ 2024-09-17 12:37:42,962 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
459
+ 2024-09-17 12:37:42,963 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
460
+ 2024-09-17 12:37:45,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
461
+ 2024-09-17 12:37:49,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
462
+ 2024-09-17 12:37:49,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
463
+ 2024-09-17 12:37:49,167 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
464
+ 2024-09-17 12:37:49,167 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
465
+ 2024-09-17 12:37:49,170 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
466
+ 2024-09-17 12:37:49,351 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
467
+ 2024-09-17 12:37:50,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
468
+ 2024-09-17 12:37:53,890 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
469
+ 2024-09-17 12:37:53,892 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
470
+ 2024-09-17 12:37:53,892 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
471
+ 2024-09-17 12:37:53,898 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
472
+ 2024-09-17 12:37:54,358 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
473
+ 2024-09-17 12:37:55,453 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
474
+ 2024-09-17 12:37:55,795 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
475
+ 2024-09-17 12:37:57,183 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry
476
+ 2024-09-17 12:37:57,226 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
477
+ 2024-09-17 12:37:57,227 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
478
+ 2024-09-17 12:37:57,231 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
479
+ 2024-09-17 12:37:57,232 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
480
+ 2024-09-17 12:37:57,236 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
481
+ 2024-09-17 12:37:57,237 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
482
+ 2024-09-17 12:37:57,241 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
483
+ 2024-09-17 12:37:57,241 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
484
+ 2024-09-17 12:37:57,245 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
485
+ 2024-09-17 12:37:57,245 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
486
+ 2024-09-17 12:37:57,248 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
487
+ 2024-09-17 12:37:57,249 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
488
+ 2024-09-17 12:37:57,249 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
489
+ 2024-09-17 12:37:57,252 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
490
+ 2024-09-17 12:37:57,455 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
491
+ 2024-09-17 12:37:57,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
492
+ 2024-09-17 12:37:57,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
493
+ 2024-09-17 12:37:58,128 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
494
+ 2024-09-17 12:37:59,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
495
+ 2024-09-17 12:38:01,794 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
496
+ 2024-09-17 12:38:05,499 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
497
+ 2024-09-17 12:38:05,502 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history
498
+ 2024-09-17 12:38:05,503 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause
499
+ 2024-09-17 12:38:05,503 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread
500
+ 2024-09-17 12:38:05,504 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor
501
+ 2024-09-17 12:38:05,504 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
502
+ 2024-09-17 12:38:05,506 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop
503
+ 2024-09-17 12:38:05,507 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
504
+ 2024-09-17 12:38:05,511 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor
505
+ 2024-09-17 12:38:05,511 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics
506
+ 2024-09-17 12:38:05,512 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor
507
+ 2024-09-17 12:38:05,517 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
508
+ 2024-09-17 12:38:05,517 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
509
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
510
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
511
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
512
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
513
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
514
+ 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
515
+ 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
516
+ 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric
517
+ 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: history
518
+ 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record
519
+ 2024-09-17 12:38:05,522 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end
520
+ 2024-09-17 12:38:05,592 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor
521
+ 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor
522
+ 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor
523
+ 2024-09-17 12:38:05,593 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume
524
+ 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread
525
+ 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor
526
+ 2024-09-17 12:38:05,594 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads
527
+ 2024-09-17 12:38:05,595 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
528
+ 2024-09-17 12:38:05,595 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring
529
+ 2024-09-17 12:38:05,597 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring
530
+ 2024-09-17 12:38:05,599 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring
531
+ 2024-09-17 12:38:05,602 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring
532
+ 2024-09-17 12:38:05,603 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring
533
+ 2024-09-17 12:38:05,619 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause
534
+ 2024-09-17 12:38:05,620 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread
535
+ 2024-09-17 12:38:05,620 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor
536
+ 2024-09-17 12:38:05,621 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop
537
+ 2024-09-17 12:38:05,621 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor
538
+ 2024-09-17 12:38:05,621 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop
539
+ 2024-09-17 12:38:05,622 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor
540
+ 2024-09-17 12:38:05,622 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics
541
+ 2024-09-17 12:38:05,679 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor
542
+ 2024-09-17 12:38:05,680 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor
543
+ 2024-09-17 12:38:05,680 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor
544
+ 2024-09-17 12:38:05,681 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats
545
+ 2024-09-17 12:38:06,464 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json
546
+ 2024-09-17 12:38:07,468 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log
547
+ 2024-09-17 12:38:07,683 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
548
+ 2024-09-17 12:38:09,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
549
+ 2024-09-17 12:38:12,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
550
+ 2024-09-17 12:38:12,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
551
+ 2024-09-17 12:38:12,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
552
+ 2024-09-17 12:38:17,809 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
553
+ 2024-09-17 12:38:18,480 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml
554
+ 2024-09-17 12:38:19,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
555
+ 2024-09-17 12:38:23,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
556
+ 2024-09-17 12:38:27,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
557
+ 2024-09-17 12:38:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
558
+ 2024-09-17 12:38:29,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
559
+ 2024-09-17 12:38:29,152 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
560
+ 2024-09-17 12:38:34,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
561
+ 2024-09-17 12:38:39,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
562
+ 2024-09-17 12:38:39,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
563
+ 2024-09-17 12:38:42,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
564
+ 2024-09-17 12:38:42,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
565
+ 2024-09-17 12:38:44,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
566
+ 2024-09-17 12:38:49,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
567
+ 2024-09-17 12:38:50,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
568
+ 2024-09-17 12:38:55,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
569
+ 2024-09-17 12:38:57,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
570
+ 2024-09-17 12:38:57,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
571
+ 2024-09-17 12:38:59,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
572
+ 2024-09-17 12:39:01,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
573
+ 2024-09-17 12:39:06,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
574
+ 2024-09-17 12:39:09,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
575
+ 2024-09-17 12:39:12,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
576
+ 2024-09-17 12:39:12,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
577
+ 2024-09-17 12:39:12,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
578
+ 2024-09-17 12:39:17,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
579
+ 2024-09-17 12:39:19,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
580
+ 2024-09-17 12:39:22,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
581
+ 2024-09-17 12:39:27,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
582
+ 2024-09-17 12:39:27,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
583
+ 2024-09-17 12:39:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
584
+ 2024-09-17 12:39:29,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
585
+ 2024-09-17 12:39:33,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
586
+ 2024-09-17 12:39:38,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
587
+ 2024-09-17 12:39:39,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
588
+ 2024-09-17 12:39:42,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
589
+ 2024-09-17 12:39:42,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
590
+ 2024-09-17 12:39:44,149 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
591
+ 2024-09-17 12:39:49,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
592
+ 2024-09-17 12:39:49,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
593
+ 2024-09-17 12:39:54,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
594
+ 2024-09-17 12:39:57,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
595
+ 2024-09-17 12:39:57,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
596
+ 2024-09-17 12:39:59,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
597
+ 2024-09-17 12:39:59,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
598
+ 2024-09-17 12:40:04,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
599
+ 2024-09-17 12:40:09,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
600
+ 2024-09-17 12:40:09,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
601
+ 2024-09-17 12:40:12,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
602
+ 2024-09-17 12:40:12,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
603
+ 2024-09-17 12:40:14,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
604
+ 2024-09-17 12:40:19,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
605
+ 2024-09-17 12:40:19,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
606
+ 2024-09-17 12:40:24,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
607
+ 2024-09-17 12:40:27,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
608
+ 2024-09-17 12:40:27,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
609
+ 2024-09-17 12:40:29,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
610
+ 2024-09-17 12:40:29,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
611
+ 2024-09-17 12:40:34,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
612
+ 2024-09-17 12:40:39,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
613
+ 2024-09-17 12:40:39,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
614
+ 2024-09-17 12:40:42,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
615
+ 2024-09-17 12:40:42,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
616
+ 2024-09-17 12:40:44,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
617
+ 2024-09-17 12:40:49,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
618
+ 2024-09-17 12:40:49,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
619
+ 2024-09-17 12:40:54,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
620
+ 2024-09-17 12:40:57,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
621
+ 2024-09-17 12:40:57,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
622
+ 2024-09-17 12:40:59,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
623
+ 2024-09-17 12:40:59,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
624
+ 2024-09-17 12:41:04,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
625
+ 2024-09-17 12:41:09,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
626
+ 2024-09-17 12:41:10,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
627
+ 2024-09-17 12:41:12,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
628
+ 2024-09-17 12:41:12,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
629
+ 2024-09-17 12:41:15,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
630
+ 2024-09-17 12:41:19,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
631
+ 2024-09-17 12:41:20,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
632
+ 2024-09-17 12:41:26,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
633
+ 2024-09-17 12:41:27,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
634
+ 2024-09-17 12:41:27,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
635
+ 2024-09-17 12:41:29,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
636
+ 2024-09-17 12:41:31,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
637
+ 2024-09-17 12:41:36,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
638
+ 2024-09-17 12:41:39,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
639
+ 2024-09-17 12:41:41,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
640
+ 2024-09-17 12:41:42,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
641
+ 2024-09-17 12:41:42,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
642
+ 2024-09-17 12:41:47,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
643
+ 2024-09-17 12:41:49,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
644
+ 2024-09-17 12:41:52,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
645
+ 2024-09-17 12:41:57,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
646
+ 2024-09-17 12:41:57,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
647
+ 2024-09-17 12:41:57,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
648
+ 2024-09-17 12:41:59,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
649
+ 2024-09-17 12:42:03,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
650
+ 2024-09-17 12:42:08,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
651
+ 2024-09-17 12:42:09,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
652
+ 2024-09-17 12:42:12,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
653
+ 2024-09-17 12:42:12,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
654
+ 2024-09-17 12:42:14,152 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
655
+ 2024-09-17 12:42:17,599 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume
656
+ 2024-09-17 12:42:17,599 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread
657
+ 2024-09-17 12:42:17,599 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor
658
+ 2024-09-17 12:42:17,600 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads
659
+ 2024-09-17 12:42:17,601 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring
660
+ 2024-09-17 12:42:17,605 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring
661
+ 2024-09-17 12:42:17,606 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring
662
+ 2024-09-17 12:42:17,608 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring
663
+ 2024-09-17 12:42:17,610 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring
664
+ 2024-09-17 12:42:19,079 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
665
+ 2024-09-17 12:42:19,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
666
+ 2024-09-17 12:42:24,807 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
667
+ 2024-09-17 12:42:27,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status
668
+ 2024-09-17 12:42:27,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status
669
+ 2024-09-17 12:42:29,079 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages
670
+ 2024-09-17 12:42:30,807 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report
wandb/run-20240917_122726-z0ntw6ob/logs/debug.log ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2024-09-17 12:27:26,817 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Current SDK version is 0.17.9
2
+ 2024-09-17 12:27:26,817 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Configure stats pid to 4132266
3
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Loading settings from /homes/amrufai/.config/wandb/settings
4
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Loading settings from /nfs/production/literature/amina-mardiyyah/notebooks/wandb/settings
5
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Loading settings from environment variables: {}
6
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Inferring run settings from compute environment: {'program': '<python with no main file>'}
7
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Applying login settings: {}
8
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_setup.py:_flush():77] Applying login settings: {}
9
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_init.py:_log_setup():524] Logging user logs to /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/logs/debug.log
10
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_init.py:_log_setup():525] Logging internal logs to /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/logs/debug-internal.log
11
+ 2024-09-17 12:27:26,818 INFO MainThread:4132266 [wandb_init.py:_jupyter_setup():470] configuring jupyter hooks <wandb.sdk.wandb_init._WandbInit object at 0x7fecc6548b90>
12
+ 2024-09-17 12:27:26,821 INFO MainThread:4132266 [wandb_init.py:init():608] calling init triggers
13
+ 2024-09-17 12:27:26,821 INFO MainThread:4132266 [wandb_init.py:init():615] wandb.init called with sweep_config: {}
14
+ config: {}
15
+ 2024-09-17 12:27:26,821 INFO MainThread:4132266 [wandb_init.py:init():658] starting backend
16
+ 2024-09-17 12:27:26,821 INFO MainThread:4132266 [wandb_init.py:init():662] setting up manager
17
+ 2024-09-17 12:27:26,823 INFO MainThread:4132266 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
18
+ 2024-09-17 12:27:26,828 INFO MainThread:4132266 [wandb_init.py:init():670] backend started and connected
19
+ 2024-09-17 12:27:26,836 INFO MainThread:4132266 [wandb_run.py:_label_probe_notebook():1344] probe notebook
20
+ 2024-09-17 12:27:26,837 INFO MainThread:4132266 [wandb_run.py:_label_probe_notebook():1354] Unable to probe notebook: 'NoneType' object has no attribute 'get'
21
+ 2024-09-17 12:27:26,837 INFO MainThread:4132266 [wandb_init.py:init():768] updated telemetry
22
+ 2024-09-17 12:27:26,846 INFO MainThread:4132266 [wandb_init.py:init():801] communicating run to backend with 90.0 second timeout
23
+ 2024-09-17 12:27:27,208 INFO MainThread:4132266 [wandb_init.py:init():852] starting run threads in backend
24
+ 2024-09-17 12:27:27,878 INFO MainThread:4132266 [wandb_run.py:_console_start():2465] atexit reg
25
+ 2024-09-17 12:27:27,879 INFO MainThread:4132266 [wandb_run.py:_redirect():2311] redirect: wrap_raw
26
+ 2024-09-17 12:27:27,880 INFO MainThread:4132266 [wandb_run.py:_redirect():2376] Wrapping output streams.
27
+ 2024-09-17 12:27:27,880 INFO MainThread:4132266 [wandb_run.py:_redirect():2401] Redirects installed.
28
+ 2024-09-17 12:27:27,891 INFO MainThread:4132266 [wandb_init.py:init():895] run started, returning control to user process
29
+ 2024-09-17 12:27:27,897 INFO MainThread:4132266 [jupyter.py:save_ipynb():372] not saving jupyter notebook
30
+ 2024-09-17 12:27:27,897 INFO MainThread:4132266 [wandb_init.py:_pause_backend():435] pausing backend
31
+ 2024-09-17 12:27:27,904 INFO MainThread:4132266 [wandb_init.py:_resume_backend():440] resuming backend
32
+ 2024-09-17 12:27:32,897 INFO MainThread:4132266 [wandb_run.py:_config_callback():1392] config_cb None None {'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': None, 'use_bfloat16': False, 'tf_legacy_loss': False, 'pruned_heads': {}, 'tie_word_embeddings': True, 'chunk_size_feed_forward': 0, 'is_encoder_decoder': False, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'typical_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'exponential_decay_length_penalty': None, 'suppress_tokens': None, 'begin_suppress_tokens': None, 'architectures': ['BertForMaskedLM'], 'finetuning_task': None, 'id2label': {0: 'O', 1: 'B-DNA', 2: 'I-DNA', 3: 'B-protein', 4: 'I-protein', 5: 'B-cell_type', 6: 'I-cell_type', 7: 'B-cell_line', 8: 'I-cell_line', 9: 'B-RNA', 10: 'I-RNA'}, 'label2id': {'O': 0, 'B-DNA': 1, 'I-DNA': 2, 'B-protein': 3, 'I-protein': 4, 'B-cell_type': 5, 'I-cell_type': 6, 'B-cell_line': 7, 'I-cell_line': 8, 'B-RNA': 9, 'I-RNA': 10}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': None, 'pad_token_id': 0, 'eos_token_id': None, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': None, 'problem_type': None, '_name_or_path': 'bioformers/bioformer-16L', 'transformers_version': '4.43.4', 'model_type': 'bert', 'vocab_size': 32768, 'hidden_size': 384, 'num_hidden_layers': 16, 'num_attention_heads': 6, 'hidden_act': 'gelu', 'intermediate_size': 1536, 'hidden_dropout_prob': 0.1, 'attention_probs_dropout_prob': 0.1, 'max_position_embeddings': 1024, 'type_vocab_size': 2, 'initializer_range': 0.02, 'layer_norm_eps': 1e-12, 'position_embedding_type': 'absolute', 'use_cache': True, 'classifier_dropout': None, 'output_dir': '/nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': True, 'do_predict': False, 'eval_strategy': 'epoch', 'prediction_loss_only': False, 'per_device_train_batch_size': 16, 'per_device_eval_batch_size': 8, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 4, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 2e-05, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 10, 'max_steps': -1, 'lr_scheduler_type': 'linear', 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.0, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': '/nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/logs', 'logging_strategy': 'epoch', 'logging_first_step': False, 'logging_steps': 500, 'logging_nan_inf_filter': True, 'save_strategy': 'epoch', 'save_steps': 500, 'save_total_limit': 2, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 3407, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': False, 'fp16': True, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': None, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'dataloader_prefetch_factor': None, 'past_index': -1, 'run_name': '/nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': True, 'metric_for_best_model': 'eval_f1', 'greater_is_better': True, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}, 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': 'adamw_torch', 'optim_args': None, 'adafactor': False, 'group_by_length': True, 'length_column_name': 'length', 'report_to': ['wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': 'every_save', 'hub_token': '<HUB_TOKEN>', 'hub_private_repo': False, 'hub_always_push': False, 'gradient_checkpointing': False, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'evaluation_strategy': None, 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': '<PUSH_TO_HUB_TOKEN>', 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'dispatch_batches': None, 'split_batches': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None, 'optim_target_modules': None, 'batch_eval_metrics': False, 'eval_on_start': False, 'eval_use_gather_object': False}
33
+ 2024-09-17 12:27:32,900 INFO MainThread:4132266 [wandb_config.py:__setitem__():154] config set model/num_parameters = 41373323 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x7fedd8b4b390>>
34
+ 2024-09-17 12:27:32,900 INFO MainThread:4132266 [wandb_run.py:_config_callback():1392] config_cb model/num_parameters 41373323 None
35
+ 2024-09-17 12:38:05,500 INFO MainThread:4132266 [jupyter.py:save_ipynb():372] not saving jupyter notebook
36
+ 2024-09-17 12:38:05,501 INFO MainThread:4132266 [wandb_init.py:_pause_backend():435] pausing backend
37
+ 2024-09-17 12:38:05,505 INFO MainThread:4132266 [wandb_init.py:_resume_backend():440] resuming backend
38
+ 2024-09-17 12:38:05,619 INFO MainThread:4132266 [jupyter.py:save_ipynb():372] not saving jupyter notebook
39
+ 2024-09-17 12:38:05,619 INFO MainThread:4132266 [wandb_init.py:_pause_backend():435] pausing backend
40
+ 2024-09-17 12:42:17,598 INFO MainThread:4132266 [wandb_init.py:_resume_backend():440] resuming backend
wandb/run-20240917_122726-z0ntw6ob/run-z0ntw6ob.wandb ADDED
Binary file (65.7 kB). View file