diff --git "a/logs.txt" "b/logs.txt" new file mode 100644--- /dev/null +++ "b/logs.txt" @@ -0,0 +1,4180 @@ +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 20:17:24] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 20:17:24] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-28 20:17:24] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-28 20:17:24] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-05-28 20:17:24] INFO config.py:106: Overriding max_batch_size from 1 to 80 +[2024-05-28 20:17:24] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 +[2024-05-28 20:17:24] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-05-28 20:17:24] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 +[2024-05-28 20:17:24] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-05-28 20:17:24] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-05-28 20:17:24] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-05-28 20:17:24] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-05-28 20:17:24] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-05-28 20:17:24] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-05-28 20:17:24] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-05-28 20:17:24] INFO gen_config.py:31: [System default] Setting temperature: 1.0 +[2024-05-28 20:17:24] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 +[2024-05-28 20:17:24] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 +[2024-05-28 20:17:24] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 +[2024-05-28 20:17:24] INFO gen_config.py:31: [System default] Setting top_p: 1.0 +[2024-05-28 20:17:24] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 +[2024-05-28 20:17:24] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 +[2024-05-28 20:17:24] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 +/opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: + Expected `int` but got `list` - serialized value may not be as expected + return self.__pydantic_serializer__.to_python( +[2024-05-28 20:17:24] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 20:17:25] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 20:17:27] INFO auto_device.py:88: Not found device: cuda:0 +[2024-05-28 20:17:28] INFO auto_device.py:88: Not found device: rocm:0 +[2024-05-28 20:17:29] INFO auto_device.py:88: Not found device: metal:0 +[2024-05-28 20:17:31] INFO auto_device.py:88: Not found device: vulkan:0 +[2024-05-28 20:17:32] INFO auto_device.py:88: Not found device: opencl:0 +[2024-05-28 20:17:32] INFO auto_device.py:33: Not found: No available device detected +[2024-05-28 20:17:32] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-05-28 20:17:32] INFO auto_weight.py:136: Not found Huggingface PyTorch +[2024-05-28 20:17:32] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-05-28 20:17:32] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-05-28 20:17:32] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-05-28 20:17:32] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +Traceback (most recent call last): + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main + return _run_code(code, main_globals, None, + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code + exec(code, run_globals) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in + main() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main + cli.main(sys.argv[2:]) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main + convert_weight( + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight + args.display() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display + print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str + return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" +AttributeError: 'NoneType' object has no attribute 'device_type' +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 20:24:50] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 20:24:50] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-28 20:24:50] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-28 20:24:50] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-05-28 20:24:50] INFO config.py:106: Overriding max_batch_size from 1 to 80 +[2024-05-28 20:24:50] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 +[2024-05-28 20:24:50] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-05-28 20:24:50] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 +[2024-05-28 20:24:50] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-05-28 20:24:50] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-05-28 20:24:50] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-05-28 20:24:50] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-05-28 20:24:50] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-05-28 20:24:50] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-05-28 20:24:50] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-05-28 20:24:50] INFO gen_config.py:31: [System default] Setting temperature: 1.0 +[2024-05-28 20:24:50] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 +[2024-05-28 20:24:50] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 +[2024-05-28 20:24:50] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 +[2024-05-28 20:24:50] INFO gen_config.py:31: [System default] Setting top_p: 1.0 +[2024-05-28 20:24:50] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 +[2024-05-28 20:24:50] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 +[2024-05-28 20:24:50] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 +/opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: + Expected `int` but got `list` - serialized value may not be as expected + return self.__pydantic_serializer__.to_python( +[2024-05-28 20:24:50] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 20:24:52] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 20:24:53] INFO auto_device.py:88: Not found device: cuda:0 +[2024-05-28 20:24:54] INFO auto_device.py:88: Not found device: rocm:0 +[2024-05-28 20:24:56] INFO auto_device.py:88: Not found device: metal:0 +[2024-05-28 20:24:57] INFO auto_device.py:88: Not found device: vulkan:0 +[2024-05-28 20:24:59] INFO auto_device.py:88: Not found device: opencl:0 +[2024-05-28 20:24:59] INFO auto_device.py:33: Not found: No available device detected +[2024-05-28 20:24:59] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-05-28 20:24:59] INFO auto_weight.py:136: Not found Huggingface PyTorch +[2024-05-28 20:24:59] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-05-28 20:24:59] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-05-28 20:24:59] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-05-28 20:24:59] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +Traceback (most recent call last): + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main + return _run_code(code, main_globals, None, + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code + exec(code, run_globals) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in + main() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main + cli.main(sys.argv[2:]) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main + convert_weight( + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight + args.display() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display + print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str + return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" +AttributeError: 'NoneType' object has no attribute 'device_type' +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 20:32:11] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 20:32:11] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-28 20:32:11] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-28 20:32:11] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-05-28 20:32:11] INFO config.py:106: Overriding max_batch_size from 1 to 80 +[2024-05-28 20:32:11] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 +[2024-05-28 20:32:11] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-05-28 20:32:11] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 +[2024-05-28 20:32:11] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-05-28 20:32:11] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-05-28 20:32:11] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-05-28 20:32:11] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-05-28 20:32:11] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-05-28 20:32:11] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-05-28 20:32:11] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-05-28 20:32:11] INFO gen_config.py:31: [System default] Setting temperature: 1.0 +[2024-05-28 20:32:11] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 +[2024-05-28 20:32:11] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 +[2024-05-28 20:32:11] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 +[2024-05-28 20:32:11] INFO gen_config.py:31: [System default] Setting top_p: 1.0 +[2024-05-28 20:32:11] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 +[2024-05-28 20:32:11] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 +[2024-05-28 20:32:11] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 +/opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: + Expected `int` but got `list` - serialized value may not be as expected + return self.__pydantic_serializer__.to_python( +[2024-05-28 20:32:11] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 20:32:12] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 20:32:14] INFO auto_device.py:88: Not found device: cuda:0 +[2024-05-28 20:32:15] INFO auto_device.py:88: Not found device: rocm:0 +[2024-05-28 20:32:16] INFO auto_device.py:88: Not found device: metal:0 +[2024-05-28 20:32:18] INFO auto_device.py:88: Not found device: vulkan:0 +[2024-05-28 20:32:20] INFO auto_device.py:88: Not found device: opencl:0 +[2024-05-28 20:32:20] INFO auto_device.py:33: Not found: No available device detected +[2024-05-28 20:32:20] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-05-28 20:32:20] INFO auto_weight.py:136: Not found Huggingface PyTorch +[2024-05-28 20:32:20] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-05-28 20:32:20] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-05-28 20:32:20] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-05-28 20:32:20] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +Traceback (most recent call last): + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main + return _run_code(code, main_globals, None, + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code + exec(code, run_globals) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in + main() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main + cli.main(sys.argv[2:]) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main + convert_weight( + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight + args.display() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display + print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str + return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" +AttributeError: 'NoneType' object has no attribute 'device_type' +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 20:40:16] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 20:40:16] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-28 20:40:16] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-28 20:40:16] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-05-28 20:40:16] INFO config.py:106: Overriding max_batch_size from 1 to 80 +[2024-05-28 20:40:16] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 +[2024-05-28 20:40:16] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-05-28 20:40:16] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 +[2024-05-28 20:40:16] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-05-28 20:40:16] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-05-28 20:40:16] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-05-28 20:40:16] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-05-28 20:40:16] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-05-28 20:40:16] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-05-28 20:40:16] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-05-28 20:40:16] INFO gen_config.py:31: [System default] Setting temperature: 1.0 +[2024-05-28 20:40:16] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 +[2024-05-28 20:40:16] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 +[2024-05-28 20:40:16] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 +[2024-05-28 20:40:16] INFO gen_config.py:31: [System default] Setting top_p: 1.0 +[2024-05-28 20:40:16] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 +[2024-05-28 20:40:16] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 +[2024-05-28 20:40:16] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 +/opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: + Expected `int` but got `list` - serialized value may not be as expected + return self.__pydantic_serializer__.to_python( +[2024-05-28 20:40:16] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 20:40:17] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 20:40:19] INFO auto_device.py:88: Not found device: cuda:0 +[2024-05-28 20:40:20] INFO auto_device.py:88: Not found device: rocm:0 +[2024-05-28 20:40:21] INFO auto_device.py:88: Not found device: metal:0 +[2024-05-28 20:40:23] INFO auto_device.py:88: Not found device: vulkan:0 +[2024-05-28 20:40:24] INFO auto_device.py:88: Not found device: opencl:0 +[2024-05-28 20:40:24] INFO auto_device.py:33: Not found: No available device detected +[2024-05-28 20:40:24] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-05-28 20:40:24] INFO auto_weight.py:136: Not found Huggingface PyTorch +[2024-05-28 20:40:24] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-05-28 20:40:24] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-05-28 20:40:24] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-05-28 20:40:24] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +Traceback (most recent call last): + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main + return _run_code(code, main_globals, None, + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code + exec(code, run_globals) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in + main() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main + cli.main(sys.argv[2:]) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main + convert_weight( + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight + args.display() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display + print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str + return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" +AttributeError: 'NoneType' object has no attribute 'device_type' +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 21:18:06] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 21:18:06] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-28 21:18:06] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-28 21:18:06] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-05-28 21:18:06] INFO config.py:106: Overriding max_batch_size from 1 to 80 +[2024-05-28 21:18:06] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 +[2024-05-28 21:18:06] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-05-28 21:18:06] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 +[2024-05-28 21:18:06] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-05-28 21:18:06] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-05-28 21:18:06] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-05-28 21:18:06] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-05-28 21:18:06] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-05-28 21:18:06] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-05-28 21:18:06] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-05-28 21:18:06] INFO gen_config.py:31: [System default] Setting temperature: 1.0 +[2024-05-28 21:18:06] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 +[2024-05-28 21:18:06] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 +[2024-05-28 21:18:06] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 +[2024-05-28 21:18:06] INFO gen_config.py:31: [System default] Setting top_p: 1.0 +[2024-05-28 21:18:06] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 +[2024-05-28 21:18:06] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 +[2024-05-28 21:18:06] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 +/opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: + Expected `int` but got `list` - serialized value may not be as expected + return self.__pydantic_serializer__.to_python( +[2024-05-28 21:18:06] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 21:18:07] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 21:18:08] INFO auto_device.py:88: Not found device: cuda:0 +[2024-05-28 21:18:10] INFO auto_device.py:88: Not found device: rocm:0 +[2024-05-28 21:18:11] INFO auto_device.py:88: Not found device: metal:0 +[2024-05-28 21:18:13] INFO auto_device.py:88: Not found device: vulkan:0 +[2024-05-28 21:18:14] INFO auto_device.py:88: Not found device: opencl:0 +[2024-05-28 21:18:14] INFO auto_device.py:33: Not found: No available device detected +[2024-05-28 21:18:14] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-05-28 21:18:14] INFO auto_weight.py:136: Not found Huggingface PyTorch +[2024-05-28 21:18:14] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-05-28 21:18:14] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-05-28 21:18:14] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-05-28 21:18:14] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +Traceback (most recent call last): + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main + return _run_code(code, main_globals, None, + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code + exec(code, run_globals) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in + main() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main + cli.main(sys.argv[2:]) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main + convert_weight( + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight + args.display() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display + print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str + return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" +AttributeError: 'NoneType' object has no attribute 'device_type' +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 22:05:17] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 22:05:17] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-28 22:05:17] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-28 22:05:17] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-05-28 22:05:17] INFO config.py:106: Overriding max_batch_size from 1 to 80 +[2024-05-28 22:05:17] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 +[2024-05-28 22:05:17] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-05-28 22:05:17] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 +[2024-05-28 22:05:17] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-05-28 22:05:17] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-05-28 22:05:17] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-05-28 22:05:17] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-05-28 22:05:17] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-05-28 22:05:17] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-05-28 22:05:17] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-05-28 22:05:17] INFO gen_config.py:31: [System default] Setting temperature: 1.0 +[2024-05-28 22:05:17] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 +[2024-05-28 22:05:17] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 +[2024-05-28 22:05:17] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 +[2024-05-28 22:05:17] INFO gen_config.py:31: [System default] Setting top_p: 1.0 +[2024-05-28 22:05:17] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 +[2024-05-28 22:05:17] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 +[2024-05-28 22:05:17] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 +/opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: + Expected `int` but got `list` - serialized value may not be as expected + return self.__pydantic_serializer__.to_python( +[2024-05-28 22:05:17] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 22:05:19] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 22:05:20] INFO auto_device.py:88: Not found device: cuda:0 +[2024-05-28 22:05:21] INFO auto_device.py:88: Not found device: rocm:0 +[2024-05-28 22:05:23] INFO auto_device.py:88: Not found device: metal:0 +[2024-05-28 22:05:24] INFO auto_device.py:88: Not found device: vulkan:0 +[2024-05-28 22:05:26] INFO auto_device.py:88: Not found device: opencl:0 +[2024-05-28 22:05:26] INFO auto_device.py:33: Not found: No available device detected +[2024-05-28 22:05:26] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-05-28 22:05:26] INFO auto_weight.py:136: Not found Huggingface PyTorch +[2024-05-28 22:05:26] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-05-28 22:05:26] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-05-28 22:05:26] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-05-28 22:05:26] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +Traceback (most recent call last): + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main + return _run_code(code, main_globals, None, + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code + exec(code, run_globals) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in + main() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main + cli.main(sys.argv[2:]) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main + convert_weight( + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight + args.display() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display + print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str + return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" +AttributeError: 'NoneType' object has no attribute 'device_type' +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 22:33:51] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 22:33:51] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-28 22:33:51] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-28 22:33:51] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-05-28 22:33:51] INFO config.py:106: Overriding max_batch_size from 1 to 80 +[2024-05-28 22:33:51] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 +[2024-05-28 22:33:51] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-05-28 22:33:51] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 +[2024-05-28 22:33:51] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-05-28 22:33:51] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-05-28 22:33:51] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-05-28 22:33:51] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-05-28 22:33:51] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-05-28 22:33:51] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-05-28 22:33:51] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-05-28 22:33:51] INFO gen_config.py:31: [System default] Setting temperature: 1.0 +[2024-05-28 22:33:51] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 +[2024-05-28 22:33:51] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 +[2024-05-28 22:33:51] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 +[2024-05-28 22:33:51] INFO gen_config.py:31: [System default] Setting top_p: 1.0 +[2024-05-28 22:33:51] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 +[2024-05-28 22:33:51] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 +[2024-05-28 22:33:51] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 +/opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: + Expected `int` but got `list` - serialized value may not be as expected + return self.__pydantic_serializer__.to_python( +[2024-05-28 22:33:51] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-28 22:33:52] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-28 22:33:54] INFO auto_device.py:88: Not found device: cuda:0 +[2024-05-28 22:33:55] INFO auto_device.py:88: Not found device: rocm:0 +[2024-05-28 22:33:56] INFO auto_device.py:88: Not found device: metal:0 +[2024-05-28 22:33:58] INFO auto_device.py:88: Not found device: vulkan:0 +[2024-05-28 22:34:00] INFO auto_device.py:88: Not found device: opencl:0 +[2024-05-28 22:34:00] INFO auto_device.py:33: Not found: No available device detected +[2024-05-28 22:34:00] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-05-28 22:34:00] INFO auto_weight.py:136: Not found Huggingface PyTorch +[2024-05-28 22:34:00] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-05-28 22:34:00] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-05-28 22:34:00] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-05-28 22:34:00] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +Traceback (most recent call last): + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main + return _run_code(code, main_globals, None, + File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code + exec(code, run_globals) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in + main() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main + cli.main(sys.argv[2:]) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main + convert_weight( + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight + args.display() + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display + print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) + File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str + return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" +AttributeError: 'NoneType' object has no attribute 'device_type' +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-30 02:31:24] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-30 02:31:24] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-30 02:31:24] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-30 02:31:24] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-05-30 02:31:24] INFO config.py:106: Overriding max_batch_size from 1 to 80 +[2024-05-30 02:31:24] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 +[2024-05-30 02:31:24] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-05-30 02:31:24] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 +[2024-05-30 02:31:24] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-05-30 02:31:24] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-05-30 02:31:24] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-05-30 02:31:24] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-05-30 02:31:24] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-05-30 02:31:24] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-05-30 02:31:24] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-05-30 02:31:24] INFO gen_config.py:31: [System default] Setting temperature: 1.0 +[2024-05-30 02:31:24] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 +[2024-05-30 02:31:24] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 +[2024-05-30 02:31:24] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 +[2024-05-30 02:31:24] INFO gen_config.py:31: [System default] Setting top_p: 1.0 +[2024-05-30 02:31:24] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 +[2024-05-30 02:31:24] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 +[2024-05-30 02:31:24] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 +[2024-05-30 02:31:24] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-05-30 02:31:25] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-05-30 02:31:27] INFO auto_device.py:79: Found device: cuda:0 +[2024-05-30 02:31:28] INFO auto_device.py:88: Not found device: rocm:0 +[2024-05-30 02:31:30] INFO auto_device.py:88: Not found device: metal:0 +[2024-05-30 02:31:32] INFO auto_device.py:79: Found device: vulkan:0 +[2024-05-30 02:31:32] INFO auto_device.py:79: Found device: vulkan:1 +[2024-05-30 02:31:32] INFO auto_device.py:79: Found device: vulkan:2 +[2024-05-30 02:31:32] INFO auto_device.py:79: Found device: vulkan:3 +[2024-05-30 02:31:33] INFO auto_device.py:88: Not found device: opencl:0 +[2024-05-30 02:31:33] INFO auto_device.py:35: Using device: cuda:0 +[2024-05-30 02:31:33] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-05-30 02:31:33] INFO auto_weight.py:136: Not found Huggingface PyTorch +[2024-05-30 02:31:33] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-05-30 02:31:33] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-05-30 02:31:33] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-05-30 02:31:33] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. +[2024-05-30 02:31:33] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-05-30 02:31:33] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +Weight conversion with arguments: + --config /models/Phi-3-mini-128k-instruct/config.json + --quantization GroupQuantize(name='q4f16_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float16', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7) + --model-type phi3 + --device cuda:0 + --source /models/Phi-3-mini-128k-instruct/model.safetensors.index.json + --source-format huggingface-safetensor + --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +Start storing to cache /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC + 0%| | 0/195 [00:00", line 198, in _run_module_as_main + File "", line 88, in _run_code + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/__main__.py", line 52, in + main() + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/__main__.py", line 29, in main + cli.main(sys.argv[2:]) + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/cli/convert_weight.py", line 88, in main + convert_weight( + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/interface/convert_weight.py", line 181, in convert_weight + _convert_args(args) + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/interface/convert_weight.py", line 145, in _convert_args + tvmjs.dump_ndarray_cache( + File "/ssd1/rickzhou/tvm/python/tvm/contrib/tvmjs.py", line 215, in dump_ndarray_cache + for k, origin_v in param_generator: + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/interface/convert_weight.py", line 129, in _param_generator + for name, param in loader.load(device=args.device, preshard_funcs=preshard_funcs): + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/loader/huggingface_loader.py", line 117, in load + param = self._load_mlc_param(mlc_name, device=device) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/loader/huggingface_loader.py", line 154, in _load_mlc_param + param = self.extern_param_map.map_func[mlc_name](*torch_params) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/ssd1/rickzhou/mlc-llm/python/mlc_llm/model/phi3/phi3_loader.py", line 48, in + lambda x, dtype: x.astype(dtype), + ^^^^^^^^^^^^^^^ +KeyboardInterrupt +/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-06-04 02:45:38] INFO auto_config.py:116: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-06-04 02:45:38] INFO auto_config.py:154: Found model type: phi3. Use `--model-type` to override. +[2024-06-04 02:45:38] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-06-04 02:45:38] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +[2024-06-04 02:45:38] INFO config.py:107: Overriding max_batch_size from 1 to 80 +[2024-06-04 02:45:38] INFO gen_config.py:143: [generation_config.json] Setting bos_token_id: 1 +[2024-06-04 02:45:38] INFO gen_config.py:143: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] +[2024-06-04 02:45:38] INFO gen_config.py:143: [generation_config.json] Setting pad_token_id: 32000 +[2024-06-04 02:45:38] INFO gen_config.py:155: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.model +[2024-06-04 02:45:38] INFO gen_config.py:155: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer.json +[2024-06-04 02:45:38] INFO gen_config.py:157: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json +[2024-06-04 02:45:38] INFO gen_config.py:157: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt +[2024-06-04 02:45:38] INFO gen_config.py:155: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/added_tokens.json +[2024-06-04 02:45:38] INFO gen_config.py:155: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/tokenizer_config.json +[2024-06-04 02:45:38] INFO gen_config.py:216: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} +[2024-06-04 02:45:38] INFO gen_config.py:32: [System default] Setting temperature: 1.0 +[2024-06-04 02:45:38] INFO gen_config.py:32: [System default] Setting presence_penalty: 0.0 +[2024-06-04 02:45:38] INFO gen_config.py:32: [System default] Setting frequency_penalty: 0.0 +[2024-06-04 02:45:38] INFO gen_config.py:32: [System default] Setting repetition_penalty: 1.0 +[2024-06-04 02:45:38] INFO gen_config.py:32: [System default] Setting top_p: 1.0 +[2024-06-04 02:45:38] INFO gen_config.py:32: [System default] Setting mean_gen_len: 128 +[2024-06-04 02:45:38] INFO gen_config.py:32: [System default] Setting max_gen_len: 512 +[2024-06-04 02:45:38] INFO gen_config.py:32: [System default] Setting shift_fill_factor: 0.3 +[2024-06-04 02:45:38] INFO gen_config.py:223: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC/mlc-chat-config.json +/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f16_1 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +[2024-06-04 02:45:39] INFO auto_config.py:116: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json +[2024-06-04 02:45:41] INFO auto_device.py:79: Found device: cuda:0 +[2024-06-04 02:45:42] INFO auto_device.py:88: Not found device: rocm:0 +[2024-06-04 02:45:44] INFO auto_device.py:88: Not found device: metal:0 +[2024-06-04 02:45:46] INFO auto_device.py:79: Found device: vulkan:0 +[2024-06-04 02:45:46] INFO auto_device.py:79: Found device: vulkan:1 +[2024-06-04 02:45:46] INFO auto_device.py:79: Found device: vulkan:2 +[2024-06-04 02:45:46] INFO auto_device.py:79: Found device: vulkan:3 +[2024-06-04 02:45:47] INFO auto_device.py:88: Not found device: opencl:0 +[2024-06-04 02:45:47] INFO auto_device.py:35: Using device: cuda:0 +[2024-06-04 02:45:47] INFO auto_weight.py:71: Finding weights in: /models/Phi-3-mini-128k-instruct +[2024-06-04 02:45:47] INFO auto_weight.py:137: Not found Huggingface PyTorch +[2024-06-04 02:45:47] INFO auto_weight.py:144: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json +[2024-06-04 02:45:47] INFO auto_weight.py:107: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. +[2024-06-04 02:45:47] INFO auto_weight.py:111: Using source weight format: huggingface-safetensor. Use `--source-format` to override. +[2024-06-04 02:45:47] INFO auto_config.py:154: Found model type: phi3. Use `--model-type` to override. +[2024-06-04 02:45:47] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) +[2024-06-04 02:45:47] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 +Weight conversion with arguments: + --config /models/Phi-3-mini-128k-instruct/config.json + --quantization GroupQuantize(name='q4f16_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float16', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7) + --model-type phi3 + --device cuda:0 + --source /models/Phi-3-mini-128k-instruct/model.safetensors.index.json + --source-format huggingface-safetensor + --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC +Start storing to cache /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f16_1-MLC + 0%| | 0/195 [00:00