Spaces:
Runtime error
Runtime error
| 2023-04-21 11:47:24,126 INFO StreamThr :33184 [internal.py:wandb_internal():86] W&B internal server running at pid: 33184, started at: 2023-04-21 11:47:24.126972 | |
| 2023-04-21 11:47:24,128 DEBUG HandlerThread:33184 [handler.py:handle_request():144] handle_request: status | |
| 2023-04-21 11:47:24,129 INFO WriterThread:33184 [datastore.py:open_for_write():85] open: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\run-wke6zfdm.wandb | |
| 2023-04-21 11:47:24,132 DEBUG SenderThread:33184 [sender.py:send():375] send: header | |
| 2023-04-21 11:47:24,194 DEBUG SenderThread:33184 [sender.py:send():375] send: run | |
| 2023-04-21 11:47:24,922 INFO SenderThread:33184 [dir_watcher.py:__init__():219] watching files in: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files | |
| 2023-04-21 11:47:24,922 INFO SenderThread:33184 [sender.py:_start_run_threads():1124] run started: wke6zfdm with start time 1682048844.12797 | |
| 2023-04-21 11:47:24,922 DEBUG SenderThread:33184 [sender.py:send_request():402] send_request: summary_record | |
| 2023-04-21 11:47:24,922 INFO SenderThread:33184 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-04-21 11:47:24,923 DEBUG HandlerThread:33184 [handler.py:handle_request():144] handle_request: check_version | |
| 2023-04-21 11:47:24,924 DEBUG SenderThread:33184 [sender.py:send_request():402] send_request: check_version | |
| 2023-04-21 11:47:25,473 DEBUG HandlerThread:33184 [handler.py:handle_request():144] handle_request: run_start | |
| 2023-04-21 11:47:25,522 DEBUG HandlerThread:33184 [system_info.py:__init__():31] System info init | |
| 2023-04-21 11:47:25,522 DEBUG HandlerThread:33184 [system_info.py:__init__():46] System info init done | |
| 2023-04-21 11:47:25,522 INFO HandlerThread:33184 [system_monitor.py:start():181] Starting system monitor | |
| 2023-04-21 11:47:25,523 INFO SystemMonitor:33184 [system_monitor.py:_start():145] Starting system asset monitoring threads | |
| 2023-04-21 11:47:25,523 INFO HandlerThread:33184 [system_monitor.py:probe():201] Collecting system info | |
| 2023-04-21 11:47:25,530 INFO SystemMonitor:33184 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-04-21 11:47:25,530 INFO SystemMonitor:33184 [interfaces.py:start():190] Started disk monitoring | |
| 2023-04-21 11:47:25,531 INFO SystemMonitor:33184 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-04-21 11:47:25,543 INFO SystemMonitor:33184 [interfaces.py:start():190] Started memory monitoring | |
| 2023-04-21 11:47:25,562 INFO SystemMonitor:33184 [interfaces.py:start():190] Started network monitoring | |
| 2023-04-21 11:47:25,596 DEBUG HandlerThread:33184 [system_info.py:probe():195] Probing system | |
| 2023-04-21 11:47:25,600 DEBUG HandlerThread:33184 [system_info.py:_probe_git():180] Probing git | |
| 2023-04-21 11:47:25,603 ERROR gpu :33184 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 11:47:25,694 DEBUG HandlerThread:33184 [system_info.py:_probe_git():188] Probing git done | |
| 2023-04-21 11:47:25,695 DEBUG HandlerThread:33184 [system_info.py:probe():240] Probing system done | |
| 2023-04-21 11:47:25,695 DEBUG HandlerThread:33184 [system_monitor.py:probe():210] {'os': 'Windows-10-10.0.19041-SP0', 'python': '3.8.10', 'heartbeatAt': '2023-04-21T03:47:25.597930', 'startedAt': '2023-04-21T03:47:24.114971', 'docker': None, 'cuda': None, 'args': ('--do_train', '--train_file', '..\\AdvertiseGen\\train.json', '--validation_file', '..\\AdvertiseGen\\dev.json', '--prompt_column', 'content', '--response_column', 'summary', '--overwrite_cache', '--model_name_or_path', '..\\models\\chatglm-6b-int4-1\\models--THUDM--chatglm-6b-int4\\snapshots\\e02ba894cf18f3fd9b2526c795f983683c4ec732', '--output_dir', 'output\\adgen-chatglm-6b-pt-128-2e-2', '--overwrite_output_dir', '--max_source_length', '64', '--max_target_length', '64', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '16', '--predict_with_generate', '--max_steps', '1000', '--logging_steps', '10', '--save_steps', '100', '--learning_rate', '2e-2', '--pre_seq_len', '128', '--quantization_bit', '4'), 'state': 'running', 'program': 'main.py', 'codePath': 'ptuning\\main.py', 'git': {'remote': 'https://github.com/THUDM/ChatGLM-6B', 'commit': '01e6313abf4122d789d6e68128856af52847b355'}, 'cpu_count': 6, 'cpu_count_logical': 12, 'cpu_freq': {'current': 2592.0, 'min': 0.0, 'max': 2592.0}, 'cpu_freq_per_core': [{'current': 2592.0, 'min': 0.0, 'max': 2592.0}], 'disk': {'total': 500.32030868530273, 'used': 500.32030868530273}, 'gpu': 'NVIDIA GeForce RTX 2060', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2060', 'memory_total': 6442450944}], 'memory': {'total': 63.87089538574219}} | |
| 2023-04-21 11:47:25,695 INFO HandlerThread:33184 [system_monitor.py:probe():211] Finished collecting system info | |
| 2023-04-21 11:47:25,695 INFO HandlerThread:33184 [system_monitor.py:probe():214] Publishing system info | |
| 2023-04-21 11:47:25,695 DEBUG HandlerThread:33184 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment | |
| 2023-04-21 11:47:25,696 ERROR HandlerThread:33184 [system_info.py:_save_pip():66] Error saving pip packages: [Errno 28] No space left on device | |
| Traceback (most recent call last): | |
| File "D:\Program\Python38\lib\site-packages\wandb\sdk\internal\system\system_info.py", line 64, in _save_pip | |
| f.write("\n".join(installed_packages_list)) | |
| OSError: [Errno 28] No space left on device | |
| 2023-04-21 11:47:25,697 DEBUG HandlerThread:33184 [system_info.py:_save_pip():67] Saving pip packages done | |
| 2023-04-21 11:47:25,930 INFO WriterThread:33184 [datastore.py:close():298] close: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\run-wke6zfdm.wandb | |
| 2023-04-21 11:47:25,932 INFO Thread-16 :33184 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files\wandb-summary.json | |
| 2023-04-21 11:47:25,932 INFO Thread-16 :33184 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files\requirements.txt | |
| 2023-04-21 11:47:26,466 INFO SenderThread:33184 [sender.py:finish():1550] shutting down sender | |
| 2023-04-21 11:47:26,466 INFO SenderThread:33184 [dir_watcher.py:finish():365] shutting down directory watcher | |
| 2023-04-21 11:47:26,938 INFO SenderThread:33184 [dir_watcher.py:finish():395] scan: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files | |
| 2023-04-21 11:47:26,938 INFO SenderThread:33184 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files\config.yaml config.yaml | |
| 2023-04-21 11:47:26,939 INFO SenderThread:33184 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files\requirements.txt requirements.txt | |
| 2023-04-21 11:47:26,941 INFO SenderThread:33184 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files\wandb-summary.json wandb-summary.json | |
| 2023-04-21 11:47:26,942 INFO SenderThread:33184 [file_pusher.py:finish():168] shutting down file pusher | |
| 2023-04-21 11:47:26,942 INFO SenderThread:33184 [file_pusher.py:join():173] waiting for file pusher | |
| 2023-04-21 11:47:27,626 ERROR gpu :33184 [interfaces.py:monitor():144] Failed to sample metric: Not Supported | |
| 2023-04-21 11:47:27,939 INFO wandb-upload_0:33184 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files\config.yaml | |
| 2023-04-21 11:47:28,068 INFO wandb-upload_1:33184 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_114724-wke6zfdm\files\wandb-summary.json | |