| 2021-07-09 14:14:46,066 INFO MainThread:246776 [internal.py:wandb_internal():88] W&B internal server running at pid: 246776, started at: 2021-07-09 14:14:46.066421 | |
| 2021-07-09 14:14:46,069 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-09 14:14:46,069 INFO WriterThread:246776 [datastore.py:open_for_write():80] open: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/run-2k8cnty2.wandb | |
| 2021-07-09 14:14:46,070 DEBUG SenderThread:246776 [sender.py:send():179] send: header | |
| 2021-07-09 14:14:46,070 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-09 14:14:46,106 DEBUG SenderThread:246776 [sender.py:send():179] send: run | |
| 2021-07-09 14:14:46,291 INFO SenderThread:246776 [dir_watcher.py:__init__():168] watching files in: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files | |
| 2021-07-09 14:14:46,291 INFO SenderThread:246776 [sender.py:_start_run_threads():716] run started: 2k8cnty2 with start time 1625840085 | |
| 2021-07-09 14:14:46,291 DEBUG SenderThread:246776 [sender.py:send():179] send: summary | |
| 2021-07-09 14:14:46,291 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-09 14:14:46,292 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-09 14:14:47,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json | |
| 2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:__init__():39] meta init | |
| 2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:__init__():53] meta init done | |
| 2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:probe():210] probe | |
| 2021-07-09 14:14:47,470 DEBUG HandlerThread:246776 [meta.py:_setup_git():200] setup git | |
| 2021-07-09 14:14:47,497 DEBUG HandlerThread:246776 [meta.py:_setup_git():207] setup git done | |
| 2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_pip():57] save pip | |
| 2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_pip():71] save pip done | |
| 2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_conda():78] save conda | |
| 2021-07-09 14:14:48,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/conda-environment.yaml | |
| 2021-07-09 14:14:48,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/requirements.txt | |
| 2021-07-09 14:14:49,937 DEBUG HandlerThread:246776 [meta.py:_save_conda():86] save conda done | |
| 2021-07-09 14:14:49,937 DEBUG HandlerThread:246776 [meta.py:probe():252] probe done | |
| 2021-07-09 14:14:49,940 DEBUG SenderThread:246776 [sender.py:send():179] send: files | |
| 2021-07-09 14:14:49,940 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-09 14:14:49,947 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:14:49,947 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:14:50,072 DEBUG SenderThread:246776 [sender.py:send():179] send: config | |
| 2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config | |
| 2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config | |
| 2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config | |
| 2021-07-09 14:14:50,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/conda-environment.yaml | |
| 2021-07-09 14:14:50,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:14:50,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-metadata.json | |
| 2021-07-09 14:14:50,380 INFO Thread-11 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/2kbqshuw-wandb-metadata.json | |
| 2021-07-09 14:14:52,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:14:54,298 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:02,301 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:04,302 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:05,089 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:15:05,089 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:15:08,303 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:10,304 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:15,553 DEBUG SenderThread:246776 [sender.py:send():179] send: stats | |
| 2021-07-09 14:15:17,307 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/config.yaml | |
| 2021-07-09 14:15:19,308 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:20,261 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:15:20,261 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:15:21,309 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:27,311 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:27,425 INFO HandlerThread:246776 [handler.py:handle_tbrecord():547] handling tbrecord: tbrecord { | |
| log_dir: "/home/cahya/Work/flax-community/gpt2-medium-indonesian" | |
| save: true | |
| } | |
| 2021-07-09 14:15:27,426 DEBUG HandlerThread:246776 [config_util.py:dict_from_config_file():101] no default config file found in config-defaults.yaml | |
| 2021-07-09 14:15:27,434 DEBUG SenderThread:246776 [sender.py:send():179] send: tbrecord | |
| 2021-07-09 14:15:27,435 DEBUG SenderThread:246776 [sender.py:send():179] send: files | |
| 2021-07-09 14:15:27,435 INFO SenderThread:246776 [sender.py:_save_file():841] saving file events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 with policy live | |
| 2021-07-09 14:15:27,879 INFO Thread-14 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/2a4g7r5i-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 | |
| 2021-07-09 14:15:28,312 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 | |
| 2021-07-09 14:15:29,312 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:31,313 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:15:35,424 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:15:35,424 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:15:45,633 DEBUG SenderThread:246776 [sender.py:send():179] send: stats | |
| 2021-07-09 14:15:50,786 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:15:50,786 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:16:05,918 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:16:05,918 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:16:15,713 DEBUG SenderThread:246776 [sender.py:send():179] send: stats | |
| 2021-07-09 14:16:21,055 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:16:21,056 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:16:36,187 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:16:36,187 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:16:45,342 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:16:45,792 DEBUG SenderThread:246776 [sender.py:send():179] send: stats | |
| 2021-07-09 14:16:47,342 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:16:49,343 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:16:51,317 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:16:51,317 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:16:51,344 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:17:05,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:17:06,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 | |
| 2021-07-09 14:17:06,451 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:17:06,451 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-09 14:17:06,818 INFO Thread-15 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/5degf70d-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 | |
| 2021-07-09 14:17:07,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:17:12,353 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:17:14,354 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:17:15,871 DEBUG SenderThread:246776 [sender.py:send():179] send: stats | |
| 2021-07-09 14:17:16,355 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log | |
| 2021-07-09 14:17:21,580 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-09 14:17:21,580 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status | |