/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f32_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
[2024-06-02 06:08:17] INFO auto_config.py:116: [92mFound[0m model configuration: /models/Phi-3-mini-128k-instruct/config.json
[2024-06-02 06:08:17] INFO auto_config.py:154: [92mFound[0m model type: [1mphi3[0m. Use `--model-type` to override.
[2024-06-02 06:08:17] INFO phi3_model.py:53: [1mcontext_window_size[0m not found in config.json. Falling back to [1mmax_position_embeddings[0m (131072)
[2024-06-02 06:08:17] INFO phi3_model.py:68: [1mprefill_chunk_size[0m defaults to 2048
[2024-06-02 06:08:17] INFO config.py:107: Overriding [1mmax_batch_size[0m from 1 to 80
[2024-06-02 06:08:17] INFO gen_config.py:143: [generation_config.json] Setting [1mbos_token_id[0m: 1
[2024-06-02 06:08:17] INFO gen_config.py:143: [generation_config.json] Setting [1meos_token_id[0m: [32000, 32001, 32007]
[2024-06-02 06:08:17] INFO gen_config.py:143: [generation_config.json] Setting [1mpad_token_id[0m: 32000
[2024-06-02 06:08:17] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer.model[0m
[2024-06-02 06:08:17] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer.json[0m
[2024-06-02 06:08:17] INFO gen_config.py:157: [91mNot found[0m tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json
[2024-06-02 06:08:17] INFO gen_config.py:157: [91mNot found[0m tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt
[2024-06-02 06:08:17] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/added_tokens.json[0m
[2024-06-02 06:08:17] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer_config.json[0m
[2024-06-02 06:08:17] INFO gen_config.py:216: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True}
[2024-06-02 06:08:17] INFO gen_config.py:32: [System default] Setting [1mtemperature[0m: 1.0
[2024-06-02 06:08:17] INFO gen_config.py:32: [System default] Setting [1mpresence_penalty[0m: 0.0
[2024-06-02 06:08:17] INFO gen_config.py:32: [System default] Setting [1mfrequency_penalty[0m: 0.0
[2024-06-02 06:08:17] INFO gen_config.py:32: [System default] Setting [1mrepetition_penalty[0m: 1.0
[2024-06-02 06:08:17] INFO gen_config.py:32: [System default] Setting [1mtop_p[0m: 1.0
[2024-06-02 06:08:17] INFO gen_config.py:32: [System default] Setting [1mmean_gen_len[0m: 128
[2024-06-02 06:08:17] INFO gen_config.py:32: [System default] Setting [1mmax_gen_len[0m: 512
[2024-06-02 06:08:17] INFO gen_config.py:32: [System default] Setting [1mshift_fill_factor[0m: 0.3
[2024-06-02 06:08:17] INFO gen_config.py:223: Dumping configuration file to: [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/mlc-chat-config.json[0m
/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f32_1 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
[2024-06-02 06:08:19] INFO auto_config.py:116: [92mFound[0m model configuration: /models/Phi-3-mini-128k-instruct/config.json
[2024-06-02 06:08:20] INFO auto_device.py:79: [92mFound[0m device: cuda:0
[2024-06-02 06:08:21] INFO auto_device.py:88: [91mNot found[0m device: rocm:0
[2024-06-02 06:08:23] INFO auto_device.py:88: [91mNot found[0m device: metal:0
[2024-06-02 06:08:24] INFO auto_device.py:79: [92mFound[0m device: vulkan:0
[2024-06-02 06:08:24] INFO auto_device.py:79: [92mFound[0m device: vulkan:1
[2024-06-02 06:08:24] INFO auto_device.py:79: [92mFound[0m device: vulkan:2
[2024-06-02 06:08:24] INFO auto_device.py:79: [92mFound[0m device: vulkan:3
[2024-06-02 06:08:26] INFO auto_device.py:88: [91mNot found[0m device: opencl:0
[2024-06-02 06:08:26] INFO auto_device.py:35: Using device: [1mcuda:0[0m
[2024-06-02 06:08:26] INFO auto_weight.py:71: Finding weights in: /models/Phi-3-mini-128k-instruct
[2024-06-02 06:08:26] INFO auto_weight.py:137: [91mNot found[0m Huggingface PyTorch
[2024-06-02 06:08:26] INFO auto_weight.py:144: [92mFound[0m source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json
[2024-06-02 06:08:26] INFO auto_weight.py:107: Using source weight configuration: [1m/models/Phi-3-mini-128k-instruct/model.safetensors.index.json[0m. Use `--source` to override.
[2024-06-02 06:08:26] INFO auto_weight.py:111: Using source weight format: [1mhuggingface-safetensor[0m. Use `--source-format` to override.
[2024-06-02 06:08:26] INFO auto_config.py:154: [92mFound[0m model type: [1mphi3[0m. Use `--model-type` to override.
[2024-06-02 06:08:26] INFO phi3_model.py:53: [1mcontext_window_size[0m not found in config.json. Falling back to [1mmax_position_embeddings[0m (131072)
[2024-06-02 06:08:26] INFO phi3_model.py:68: [1mprefill_chunk_size[0m defaults to 2048
[1mWeight conversion with arguments:[0m
[1m--config[0m /models/Phi-3-mini-128k-instruct/config.json
[1m--quantization[0m GroupQuantize(name='q4f32_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float32', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
[1m--model-type[0m phi3
[1m--device[0m cuda:0
[1m--source[0m /models/Phi-3-mini-128k-instruct/model.safetensors.index.json
[1m--source-format[0m huggingface-safetensor
[1m--output[0m /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
Start storing to cache /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
0%| | 0/195 [00:00<?, ?it/s]
[2024-06-02 06:08:28] INFO huggingface_loader.py:185: Loading HF parameters from: /models/Phi-3-mini-128k-instruct/model-00002-of-00002.safetensors
0%| | 0/195 [00:00<?, ?it/s]
[2024-06-02 06:08:31] INFO group_quantization.py:217: Compiling quantize function for key: ((32064, 3072), float32, cuda, axis=1, output_transpose=False)
0%| | 0/195 [00:03<?, ?it/s]
[2024-06-02 06:08:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mlm_head.q_weight[0m", shape: (32064, 384), dtype: uint32
0%| | 0/195 [00:04<?, ?it/s]
[2024-06-02 06:08:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mlm_head.q_scale[0m", shape: (32064, 96), dtype: float32
0%| | 0/195 [00:04<?, ?it/s]
1%| | 1/195 [00:04<15:07, 4.68s/it]
[2024-06-02 06:08:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.21.ln.weight[0m", shape: (3072,), dtype: float32
1%| | 1/195 [00:04<15:07, 4.68s/it]
[2024-06-02 06:08:32] INFO group_quantization.py:217: Compiling quantize function for key: ((3072, 8192), float32, cuda, axis=1, output_transpose=False)
1%| | 1/195 [00:04<15:07, 4.68s/it]
[2024-06-02 06:08:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
1%| | 1/195 [00:05<15:07, 4.68s/it]
[2024-06-02 06:08:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
1%| | 1/195 [00:05<15:07, 4.68s/it]
2%|▏ | 3/195 [00:05<04:24, 1.38s/it]
[2024-06-02 06:08:33] INFO group_quantization.py:217: Compiling quantize function for key: ((16384, 3072), float32, cuda, axis=1, output_transpose=False)
2%|▏ | 3/195 [00:05<04:24, 1.38s/it]
[2024-06-02 06:08:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
2%|▏ | 3/195 [00:05<04:24, 1.38s/it]
[2024-06-02 06:08:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
2%|▏ | 3/195 [00:05<04:24, 1.38s/it]
2%|▏ | 4/195 [00:05<03:25, 1.08s/it]
[2024-06-02 06:08:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.21.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
2%|▏ | 4/195 [00:05<03:25, 1.08s/it]
[2024-06-02 06:08:33] INFO group_quantization.py:217: Compiling quantize function for key: ((9216, 3072), float32, cuda, axis=1, output_transpose=False)
2%|▏ | 4/195 [00:05<03:25, 1.08s/it]
[2024-06-02 06:08:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
2%|▏ | 4/195 [00:06<03:25, 1.08s/it]
[2024-06-02 06:08:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
2%|▏ | 4/195 [00:06<03:25, 1.08s/it]
3%|▎ | 6/195 [00:06<02:08, 1.48it/s]
[2024-06-02 06:08:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.22.ln.weight[0m", shape: (3072,), dtype: float32
3%|▎ | 6/195 [00:06<02:08, 1.48it/s]
[2024-06-02 06:08:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
3%|▎ | 6/195 [00:06<02:08, 1.48it/s]
[2024-06-02 06:08:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
3%|▎ | 6/195 [00:06<02:08, 1.48it/s]
[2024-06-02 06:08:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
3%|▎ | 6/195 [00:06<02:08, 1.48it/s]
[2024-06-02 06:08:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
3%|▎ | 6/195 [00:06<02:08, 1.48it/s]
5%|▍ | 9/195 [00:06<01:05, 2.84it/s]
[2024-06-02 06:08:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.22.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
5%|▍ | 9/195 [00:06<01:05, 2.84it/s]
[2024-06-02 06:08:34] INFO group_quantization.py:217: Compiling quantize function for key: ((3072, 3072), float32, cuda, axis=1, output_transpose=False)
5%|▍ | 9/195 [00:06<01:05, 2.84it/s]
[2024-06-02 06:08:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
5%|▍ | 9/195 [00:06<01:05, 2.84it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
5%|▍ | 9/195 [00:06<01:05, 2.84it/s]
6%|▌ | 11/195 [00:06<00:58, 3.16it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
6%|▌ | 11/195 [00:06<00:58, 3.16it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
6%|▌ | 11/195 [00:06<00:58, 3.16it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.23.ln.weight[0m", shape: (3072,), dtype: float32
6%|▌ | 11/195 [00:06<00:58, 3.16it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
6%|▌ | 11/195 [00:06<00:58, 3.16it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
6%|▌ | 11/195 [00:06<00:58, 3.16it/s]
7%|▋ | 14/195 [00:06<00:36, 4.99it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
7%|▋ | 14/195 [00:06<00:36, 4.99it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
7%|▋ | 14/195 [00:07<00:36, 4.99it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.23.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
7%|▋ | 14/195 [00:07<00:36, 4.99it/s]
8%|▊ | 16/195 [00:07<00:28, 6.28it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
8%|▊ | 16/195 [00:07<00:28, 6.28it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
8%|▊ | 16/195 [00:07<00:28, 6.28it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
8%|▊ | 16/195 [00:07<00:28, 6.28it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
8%|▊ | 16/195 [00:07<00:28, 6.28it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.24.ln.weight[0m", shape: (3072,), dtype: float32
8%|▊ | 16/195 [00:07<00:28, 6.28it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
8%|▊ | 16/195 [00:07<00:28, 6.28it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
8%|▊ | 16/195 [00:07<00:28, 6.28it/s]
10%|█ | 20/195 [00:07<00:17, 9.74it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
10%|█ | 20/195 [00:07<00:17, 9.74it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
10%|█ | 20/195 [00:07<00:17, 9.74it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.24.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
10%|█ | 20/195 [00:07<00:17, 9.74it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
10%|█ | 20/195 [00:07<00:17, 9.74it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
10%|█ | 20/195 [00:07<00:17, 9.74it/s]
12%|█▏ | 23/195 [00:07<00:15, 11.22it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
12%|█▏ | 23/195 [00:07<00:15, 11.22it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
12%|█▏ | 23/195 [00:07<00:15, 11.22it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.25.ln.weight[0m", shape: (3072,), dtype: float32
12%|█▏ | 23/195 [00:07<00:15, 11.22it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
12%|█▏ | 23/195 [00:07<00:15, 11.22it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
12%|█▏ | 23/195 [00:07<00:15, 11.22it/s]
13%|█▎ | 26/195 [00:07<00:12, 13.79it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
13%|█▎ | 26/195 [00:07<00:12, 13.79it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
13%|█▎ | 26/195 [00:07<00:12, 13.79it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.25.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
13%|█▎ | 26/195 [00:07<00:12, 13.79it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
13%|█▎ | 26/195 [00:07<00:12, 13.79it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
13%|█▎ | 26/195 [00:07<00:12, 13.79it/s]
15%|█▍ | 29/195 [00:07<00:10, 16.01it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
15%|█▍ | 29/195 [00:07<00:10, 16.01it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
15%|█▍ | 29/195 [00:07<00:10, 16.01it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.26.ln.weight[0m", shape: (3072,), dtype: float32
15%|█▍ | 29/195 [00:07<00:10, 16.01it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
15%|█▍ | 29/195 [00:07<00:10, 16.01it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
15%|█▍ | 29/195 [00:07<00:10, 16.01it/s]
16%|█▋ | 32/195 [00:07<00:08, 18.14it/s]
[2024-06-02 06:08:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
16%|█▋ | 32/195 [00:07<00:08, 18.14it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
16%|█▋ | 32/195 [00:07<00:08, 18.14it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.26.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
16%|█▋ | 32/195 [00:07<00:08, 18.14it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
16%|█▋ | 32/195 [00:07<00:08, 18.14it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
16%|█▋ | 32/195 [00:07<00:08, 18.14it/s]
18%|█▊ | 35/195 [00:07<00:09, 17.60it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
18%|█▊ | 35/195 [00:07<00:09, 17.60it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
18%|█▊ | 35/195 [00:07<00:09, 17.60it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.27.ln.weight[0m", shape: (3072,), dtype: float32
18%|█▊ | 35/195 [00:07<00:09, 17.60it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
18%|█▊ | 35/195 [00:07<00:09, 17.60it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
18%|█▊ | 35/195 [00:07<00:09, 17.60it/s]
19%|█▉ | 38/195 [00:07<00:07, 19.73it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
19%|█▉ | 38/195 [00:08<00:07, 19.73it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
19%|█▉ | 38/195 [00:08<00:07, 19.73it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.27.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
19%|█▉ | 38/195 [00:08<00:07, 19.73it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
19%|█▉ | 38/195 [00:08<00:07, 19.73it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
19%|█▉ | 38/195 [00:08<00:07, 19.73it/s]
21%|██ | 41/195 [00:08<00:07, 21.04it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
21%|██ | 41/195 [00:08<00:07, 21.04it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
21%|██ | 41/195 [00:08<00:07, 21.04it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.28.ln.weight[0m", shape: (3072,), dtype: float32
21%|██ | 41/195 [00:08<00:07, 21.04it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
21%|██ | 41/195 [00:08<00:07, 21.04it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
21%|██ | 41/195 [00:08<00:07, 21.04it/s]
23%|██▎ | 44/195 [00:08<00:06, 22.32it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
23%|██▎ | 44/195 [00:08<00:06, 22.32it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
23%|██▎ | 44/195 [00:08<00:06, 22.32it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.28.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
23%|██▎ | 44/195 [00:08<00:06, 22.32it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
23%|██▎ | 44/195 [00:08<00:06, 22.32it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
23%|██▎ | 44/195 [00:08<00:06, 22.32it/s]
24%|██▍ | 47/195 [00:08<00:07, 20.17it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
24%|██▍ | 47/195 [00:08<00:07, 20.17it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
24%|██▍ | 47/195 [00:08<00:07, 20.17it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.29.ln.weight[0m", shape: (3072,), dtype: float32
24%|██▍ | 47/195 [00:08<00:07, 20.17it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
24%|██▍ | 47/195 [00:08<00:07, 20.17it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
24%|██▍ | 47/195 [00:08<00:07, 20.17it/s]
26%|██▌ | 50/195 [00:08<00:06, 21.89it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
26%|██▌ | 50/195 [00:08<00:06, 21.89it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
26%|██▌ | 50/195 [00:08<00:06, 21.89it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.29.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
26%|██▌ | 50/195 [00:08<00:06, 21.89it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
26%|██▌ | 50/195 [00:08<00:06, 21.89it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
26%|██▌ | 50/195 [00:08<00:06, 21.89it/s]
27%|██▋ | 53/195 [00:08<00:06, 22.70it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
27%|██▋ | 53/195 [00:08<00:06, 22.70it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
27%|██▋ | 53/195 [00:08<00:06, 22.70it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.30.ln.weight[0m", shape: (3072,), dtype: float32
27%|██▋ | 53/195 [00:08<00:06, 22.70it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
27%|██▋ | 53/195 [00:08<00:06, 22.70it/s]
[2024-06-02 06:08:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
27%|██▋ | 53/195 [00:08<00:06, 22.70it/s]
29%|██▊ | 56/195 [00:08<00:05, 23.17it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
29%|██▊ | 56/195 [00:08<00:05, 23.17it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
29%|██▊ | 56/195 [00:08<00:05, 23.17it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.30.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
29%|██▊ | 56/195 [00:08<00:05, 23.17it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
29%|██▊ | 56/195 [00:08<00:05, 23.17it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
29%|██▊ | 56/195 [00:08<00:05, 23.17it/s]
30%|███ | 59/195 [00:08<00:06, 20.67it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
30%|███ | 59/195 [00:08<00:06, 20.67it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
30%|███ | 59/195 [00:08<00:06, 20.67it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.31.ln.weight[0m", shape: (3072,), dtype: float32
30%|███ | 59/195 [00:08<00:06, 20.67it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
30%|███ | 59/195 [00:08<00:06, 20.67it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
30%|███ | 59/195 [00:09<00:06, 20.67it/s]
32%|███▏ | 62/195 [00:09<00:05, 22.30it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
32%|███▏ | 62/195 [00:09<00:05, 22.30it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
32%|███▏ | 62/195 [00:09<00:05, 22.30it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.31.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
32%|███▏ | 62/195 [00:09<00:05, 22.30it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
32%|███▏ | 62/195 [00:09<00:05, 22.30it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
32%|███▏ | 62/195 [00:09<00:05, 22.30it/s]
33%|███▎ | 65/195 [00:09<00:05, 23.03it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
33%|███▎ | 65/195 [00:09<00:05, 23.03it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
33%|███▎ | 65/195 [00:09<00:05, 23.03it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.norm.weight[0m", shape: (3072,), dtype: float32
33%|███▎ | 65/195 [00:09<00:05, 23.03it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:197: Unloading HF weight file: /models/Phi-3-mini-128k-instruct/model-00002-of-00002.safetensors
33%|███▎ | 65/195 [00:09<00:05, 23.03it/s]
[2024-06-02 06:08:37] INFO huggingface_loader.py:185: Loading HF parameters from: /models/Phi-3-mini-128k-instruct/model-00001-of-00002.safetensors
33%|███▎ | 65/195 [00:09<00:05, 23.03it/s]
[2024-06-02 06:08:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.embd.q_weight[0m", shape: (32064, 384), dtype: uint32
33%|███▎ | 65/195 [00:13<00:05, 23.03it/s]
[2024-06-02 06:08:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.embd.q_scale[0m", shape: (32064, 96), dtype: float32
33%|███▎ | 65/195 [00:13<00:05, 23.03it/s]
35%|███▍ | 68/195 [00:13<01:00, 2.11it/s]
[2024-06-02 06:08:41] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.0.ln.weight[0m", shape: (3072,), dtype: float32
35%|███▍ | 68/195 [00:13<01:00, 2.11it/s]
[2024-06-02 06:08:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
35%|███▍ | 68/195 [00:13<01:00, 2.11it/s]
[2024-06-02 06:08:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
35%|███▍ | 68/195 [00:13<01:00, 2.11it/s]
[2024-06-02 06:08:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
35%|███▍ | 68/195 [00:13<01:00, 2.11it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
35%|███▍ | 68/195 [00:13<01:00, 2.11it/s]
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.0.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.1.ln.weight[0m", shape: (3072,), dtype: float32
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
36%|███▋ | 71/195 [00:13<00:43, 2.86it/s]
39%|███▉ | 76/195 [00:13<00:25, 4.66it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
39%|███▉ | 76/195 [00:13<00:25, 4.66it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
39%|███▉ | 76/195 [00:13<00:25, 4.66it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.1.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
39%|███▉ | 76/195 [00:14<00:25, 4.66it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
39%|███▉ | 76/195 [00:14<00:25, 4.66it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
39%|███▉ | 76/195 [00:14<00:25, 4.66it/s]
41%|████ | 79/195 [00:14<00:20, 5.74it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
41%|████ | 79/195 [00:14<00:20, 5.74it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
41%|████ | 79/195 [00:14<00:20, 5.74it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.10.ln.weight[0m", shape: (3072,), dtype: float32
41%|████ | 79/195 [00:14<00:20, 5.74it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
41%|████ | 79/195 [00:14<00:20, 5.74it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
41%|████ | 79/195 [00:14<00:20, 5.74it/s]
42%|████▏ | 82/195 [00:14<00:15, 7.32it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
42%|████▏ | 82/195 [00:14<00:15, 7.32it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
42%|████▏ | 82/195 [00:14<00:15, 7.32it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.10.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
42%|████▏ | 82/195 [00:14<00:15, 7.32it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
42%|████▏ | 82/195 [00:14<00:15, 7.32it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
42%|████▏ | 82/195 [00:14<00:15, 7.32it/s]
44%|████▎ | 85/195 [00:14<00:12, 9.10it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
44%|████▎ | 85/195 [00:14<00:12, 9.10it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
44%|████▎ | 85/195 [00:14<00:12, 9.10it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.11.ln.weight[0m", shape: (3072,), dtype: float32
44%|████▎ | 85/195 [00:14<00:12, 9.10it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
44%|████▎ | 85/195 [00:14<00:12, 9.10it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
44%|████▎ | 85/195 [00:14<00:12, 9.10it/s]
45%|████▌ | 88/195 [00:14<00:09, 11.17it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
45%|████▌ | 88/195 [00:14<00:09, 11.17it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
45%|████▌ | 88/195 [00:14<00:09, 11.17it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.11.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
45%|████▌ | 88/195 [00:14<00:09, 11.17it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
45%|████▌ | 88/195 [00:14<00:09, 11.17it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
45%|████▌ | 88/195 [00:14<00:09, 11.17it/s]
47%|████▋ | 91/195 [00:14<00:08, 12.23it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
47%|████▋ | 91/195 [00:14<00:08, 12.23it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
47%|████▋ | 91/195 [00:14<00:08, 12.23it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.12.ln.weight[0m", shape: (3072,), dtype: float32
47%|████▋ | 91/195 [00:14<00:08, 12.23it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
47%|████▋ | 91/195 [00:14<00:08, 12.23it/s]
[2024-06-02 06:08:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
47%|████▋ | 91/195 [00:14<00:08, 12.23it/s]
48%|████▊ | 94/195 [00:14<00:06, 14.55it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
48%|████▊ | 94/195 [00:14<00:06, 14.55it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
48%|████▊ | 94/195 [00:14<00:06, 14.55it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.12.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
48%|████▊ | 94/195 [00:14<00:06, 14.55it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
48%|████▊ | 94/195 [00:14<00:06, 14.55it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
48%|████▊ | 94/195 [00:14<00:06, 14.55it/s]
50%|████▉ | 97/195 [00:14<00:05, 16.65it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
50%|████▉ | 97/195 [00:14<00:05, 16.65it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
50%|████▉ | 97/195 [00:14<00:05, 16.65it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.13.ln.weight[0m", shape: (3072,), dtype: float32
50%|████▉ | 97/195 [00:14<00:05, 16.65it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
50%|████▉ | 97/195 [00:14<00:05, 16.65it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
50%|████▉ | 97/195 [00:14<00:05, 16.65it/s]
51%|█████▏ | 100/195 [00:14<00:05, 18.57it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
51%|█████▏ | 100/195 [00:15<00:05, 18.57it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
51%|█████▏ | 100/195 [00:15<00:05, 18.57it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.13.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
51%|█████▏ | 100/195 [00:15<00:05, 18.57it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
51%|█████▏ | 100/195 [00:15<00:05, 18.57it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
51%|█████▏ | 100/195 [00:15<00:05, 18.57it/s]
53%|█████▎ | 103/195 [00:15<00:05, 17.68it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
53%|█████▎ | 103/195 [00:15<00:05, 17.68it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
53%|█████▎ | 103/195 [00:15<00:05, 17.68it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.14.ln.weight[0m", shape: (3072,), dtype: float32
53%|█████▎ | 103/195 [00:15<00:05, 17.68it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
53%|█████▎ | 103/195 [00:15<00:05, 17.68it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
53%|█████▎ | 103/195 [00:15<00:05, 17.68it/s]
54%|█████▍ | 106/195 [00:15<00:04, 19.74it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
54%|█████▍ | 106/195 [00:15<00:04, 19.74it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
54%|█████▍ | 106/195 [00:15<00:04, 19.74it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.14.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
54%|█████▍ | 106/195 [00:15<00:04, 19.74it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
54%|█████▍ | 106/195 [00:15<00:04, 19.74it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
54%|█████▍ | 106/195 [00:15<00:04, 19.74it/s]
56%|█████▌ | 109/195 [00:15<00:04, 21.18it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
56%|█████▌ | 109/195 [00:15<00:04, 21.18it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
56%|█████▌ | 109/195 [00:15<00:04, 21.18it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.15.ln.weight[0m", shape: (3072,), dtype: float32
56%|█████▌ | 109/195 [00:15<00:04, 21.18it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
56%|█████▌ | 109/195 [00:15<00:04, 21.18it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
56%|█████▌ | 109/195 [00:15<00:04, 21.18it/s]
57%|█████▋ | 112/195 [00:15<00:03, 22.34it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
57%|█████▋ | 112/195 [00:15<00:03, 22.34it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
57%|█████▋ | 112/195 [00:15<00:03, 22.34it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.15.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
57%|█████▋ | 112/195 [00:15<00:03, 22.34it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
57%|█████▋ | 112/195 [00:15<00:03, 22.34it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
57%|█████▋ | 112/195 [00:15<00:03, 22.34it/s]
59%|█████▉ | 115/195 [00:15<00:04, 19.66it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
59%|█████▉ | 115/195 [00:15<00:04, 19.66it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
59%|█████▉ | 115/195 [00:15<00:04, 19.66it/s]
[2024-06-02 06:08:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.16.ln.weight[0m", shape: (3072,), dtype: float32
59%|█████▉ | 115/195 [00:15<00:04, 19.66it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
59%|█████▉ | 115/195 [00:15<00:04, 19.66it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
59%|█████▉ | 115/195 [00:15<00:04, 19.66it/s]
61%|██████ | 118/195 [00:15<00:03, 21.42it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
61%|██████ | 118/195 [00:15<00:03, 21.42it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
61%|██████ | 118/195 [00:15<00:03, 21.42it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.16.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
61%|██████ | 118/195 [00:15<00:03, 21.42it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
61%|██████ | 118/195 [00:15<00:03, 21.42it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
61%|██████ | 118/195 [00:15<00:03, 21.42it/s]
62%|██████▏ | 121/195 [00:15<00:03, 22.51it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
62%|██████▏ | 121/195 [00:15<00:03, 22.51it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
62%|██████▏ | 121/195 [00:16<00:03, 22.51it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.17.ln.weight[0m", shape: (3072,), dtype: float32
62%|██████▏ | 121/195 [00:16<00:03, 22.51it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
62%|██████▏ | 121/195 [00:16<00:03, 22.51it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
62%|██████▏ | 121/195 [00:16<00:03, 22.51it/s]
64%|██████▎ | 124/195 [00:16<00:03, 23.39it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
64%|██████▎ | 124/195 [00:16<00:03, 23.39it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
64%|██████▎ | 124/195 [00:16<00:03, 23.39it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.17.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
64%|██████▎ | 124/195 [00:16<00:03, 23.39it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
64%|██████▎ | 124/195 [00:16<00:03, 23.39it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
64%|██████▎ | 124/195 [00:16<00:03, 23.39it/s]
65%|██████▌ | 127/195 [00:16<00:03, 20.53it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
65%|██████▌ | 127/195 [00:16<00:03, 20.53it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
65%|██████▌ | 127/195 [00:16<00:03, 20.53it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.18.ln.weight[0m", shape: (3072,), dtype: float32
65%|██████▌ | 127/195 [00:16<00:03, 20.53it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
65%|██████▌ | 127/195 [00:16<00:03, 20.53it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
65%|██████▌ | 127/195 [00:16<00:03, 20.53it/s]
67%|██████▋ | 130/195 [00:16<00:02, 22.18it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
67%|██████▋ | 130/195 [00:16<00:02, 22.18it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
67%|██████▋ | 130/195 [00:16<00:02, 22.18it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.18.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
67%|██████▋ | 130/195 [00:16<00:02, 22.18it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
67%|██████▋ | 130/195 [00:16<00:02, 22.18it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
67%|██████▋ | 130/195 [00:16<00:02, 22.18it/s]
68%|██████▊ | 133/195 [00:16<00:02, 23.10it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
68%|██████▊ | 133/195 [00:16<00:02, 23.10it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
68%|██████▊ | 133/195 [00:16<00:02, 23.10it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.19.ln.weight[0m", shape: (3072,), dtype: float32
68%|██████▊ | 133/195 [00:16<00:02, 23.10it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
68%|██████▊ | 133/195 [00:16<00:02, 23.10it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
68%|██████▊ | 133/195 [00:16<00:02, 23.10it/s]
70%|██████▉ | 136/195 [00:16<00:02, 23.83it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
70%|██████▉ | 136/195 [00:16<00:02, 23.83it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
70%|██████▉ | 136/195 [00:16<00:02, 23.83it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.19.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
70%|██████▉ | 136/195 [00:16<00:02, 23.83it/s]
[2024-06-02 06:08:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
70%|██████▉ | 136/195 [00:16<00:02, 23.83it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
70%|██████▉ | 136/195 [00:16<00:02, 23.83it/s]
71%|███████▏ | 139/195 [00:16<00:02, 20.78it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
71%|███████▏ | 139/195 [00:16<00:02, 20.78it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
71%|███████▏ | 139/195 [00:16<00:02, 20.78it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.2.ln.weight[0m", shape: (3072,), dtype: float32
71%|███████▏ | 139/195 [00:16<00:02, 20.78it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
71%|███████▏ | 139/195 [00:16<00:02, 20.78it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
71%|███████▏ | 139/195 [00:16<00:02, 20.78it/s]
73%|███████▎ | 142/195 [00:16<00:02, 22.37it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
73%|███████▎ | 142/195 [00:16<00:02, 22.37it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
73%|███████▎ | 142/195 [00:16<00:02, 22.37it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.2.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
73%|███████▎ | 142/195 [00:16<00:02, 22.37it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
73%|███████▎ | 142/195 [00:16<00:02, 22.37it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
73%|███████▎ | 142/195 [00:16<00:02, 22.37it/s]
74%|███████▍ | 145/195 [00:16<00:02, 23.24it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
74%|███████▍ | 145/195 [00:17<00:02, 23.24it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
74%|███████▍ | 145/195 [00:17<00:02, 23.24it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.20.ln.weight[0m", shape: (3072,), dtype: float32
74%|███████▍ | 145/195 [00:17<00:02, 23.24it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
74%|███████▍ | 145/195 [00:17<00:02, 23.24it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
74%|███████▍ | 145/195 [00:17<00:02, 23.24it/s]
76%|███████▌ | 148/195 [00:17<00:01, 23.92it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
76%|███████▌ | 148/195 [00:17<00:01, 23.92it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
76%|███████▌ | 148/195 [00:17<00:01, 23.92it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.20.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
76%|███████▌ | 148/195 [00:17<00:01, 23.92it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
76%|███████▌ | 148/195 [00:17<00:01, 23.92it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
76%|███████▌ | 148/195 [00:17<00:01, 23.92it/s]
77%|███████▋ | 151/195 [00:17<00:02, 20.81it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
77%|███████▋ | 151/195 [00:17<00:02, 20.81it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
77%|███████▋ | 151/195 [00:17<00:02, 20.81it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
77%|███████▋ | 151/195 [00:17<00:02, 20.81it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
77%|███████▋ | 151/195 [00:17<00:02, 20.81it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.3.ln.weight[0m", shape: (3072,), dtype: float32
77%|███████▋ | 151/195 [00:17<00:02, 20.81it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
77%|███████▋ | 151/195 [00:17<00:02, 20.81it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
77%|███████▋ | 151/195 [00:17<00:02, 20.81it/s]
79%|███████▉ | 155/195 [00:17<00:01, 23.47it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
79%|███████▉ | 155/195 [00:17<00:01, 23.47it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
79%|███████▉ | 155/195 [00:17<00:01, 23.47it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.3.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
79%|███████▉ | 155/195 [00:17<00:01, 23.47it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
79%|███████▉ | 155/195 [00:17<00:01, 23.47it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
79%|███████▉ | 155/195 [00:17<00:01, 23.47it/s]
81%|████████ | 158/195 [00:17<00:01, 24.03it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
81%|████████ | 158/195 [00:17<00:01, 24.03it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
81%|████████ | 158/195 [00:17<00:01, 24.03it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.4.ln.weight[0m", shape: (3072,), dtype: float32
81%|████████ | 158/195 [00:17<00:01, 24.03it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
81%|████████ | 158/195 [00:17<00:01, 24.03it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
81%|████████ | 158/195 [00:17<00:01, 24.03it/s]
83%|████████▎ | 161/195 [00:17<00:01, 24.44it/s]
[2024-06-02 06:08:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
83%|████████▎ | 161/195 [00:17<00:01, 24.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
83%|████████▎ | 161/195 [00:17<00:01, 24.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.4.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
83%|████████▎ | 161/195 [00:17<00:01, 24.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
83%|████████▎ | 161/195 [00:17<00:01, 24.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
83%|████████▎ | 161/195 [00:17<00:01, 24.44it/s]
84%|████████▍ | 164/195 [00:17<00:01, 21.17it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
84%|████████▍ | 164/195 [00:17<00:01, 21.17it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
84%|████████▍ | 164/195 [00:17<00:01, 21.17it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.5.ln.weight[0m", shape: (3072,), dtype: float32
84%|████████▍ | 164/195 [00:17<00:01, 21.17it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
84%|████████▍ | 164/195 [00:17<00:01, 21.17it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
84%|████████▍ | 164/195 [00:17<00:01, 21.17it/s]
86%|████████▌ | 167/195 [00:17<00:01, 22.66it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
86%|████████▌ | 167/195 [00:18<00:01, 22.66it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
86%|████████▌ | 167/195 [00:18<00:01, 22.66it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.5.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
86%|████████▌ | 167/195 [00:18<00:01, 22.66it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
86%|████████▌ | 167/195 [00:18<00:01, 22.66it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
86%|████████▌ | 167/195 [00:18<00:01, 22.66it/s]
87%|████████▋ | 170/195 [00:18<00:01, 23.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
87%|████████▋ | 170/195 [00:18<00:01, 23.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
87%|████████▋ | 170/195 [00:18<00:01, 23.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.6.ln.weight[0m", shape: (3072,), dtype: float32
87%|████████▋ | 170/195 [00:18<00:01, 23.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
87%|████████▋ | 170/195 [00:18<00:01, 23.44it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
87%|████████▋ | 170/195 [00:18<00:01, 23.44it/s]
89%|████████▊ | 173/195 [00:18<00:00, 24.06it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
89%|████████▊ | 173/195 [00:18<00:00, 24.06it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
89%|████████▊ | 173/195 [00:18<00:00, 24.06it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.6.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
89%|████████▊ | 173/195 [00:18<00:00, 24.06it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
89%|████████▊ | 173/195 [00:18<00:00, 24.06it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
89%|████████▊ | 173/195 [00:18<00:00, 24.06it/s]
90%|█████████ | 176/195 [00:18<00:00, 20.85it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
90%|█████████ | 176/195 [00:18<00:00, 20.85it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
90%|█████████ | 176/195 [00:18<00:00, 20.85it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.7.ln.weight[0m", shape: (3072,), dtype: float32
90%|█████████ | 176/195 [00:18<00:00, 20.85it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
90%|█████████ | 176/195 [00:18<00:00, 20.85it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
90%|█████████ | 176/195 [00:18<00:00, 20.85it/s]
92%|█████████▏| 179/195 [00:18<00:00, 22.37it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
92%|█████████▏| 179/195 [00:18<00:00, 22.37it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
92%|█████████▏| 179/195 [00:18<00:00, 22.37it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.7.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
92%|█████████▏| 179/195 [00:18<00:00, 22.37it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
92%|█████████▏| 179/195 [00:18<00:00, 22.37it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
92%|█████████▏| 179/195 [00:18<00:00, 22.37it/s]
93%|█████████▎| 182/195 [00:18<00:00, 23.18it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
93%|█████████▎| 182/195 [00:18<00:00, 23.18it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
93%|█████████▎| 182/195 [00:18<00:00, 23.18it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.8.ln.weight[0m", shape: (3072,), dtype: float32
93%|█████████▎| 182/195 [00:18<00:00, 23.18it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
93%|█████████▎| 182/195 [00:18<00:00, 23.18it/s]
[2024-06-02 06:08:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
93%|█████████▎| 182/195 [00:18<00:00, 23.18it/s]
95%|█████████▍| 185/195 [00:18<00:00, 23.84it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
95%|█████████▍| 185/195 [00:18<00:00, 23.84it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
95%|█████████▍| 185/195 [00:18<00:00, 23.84it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.8.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
95%|█████████▍| 185/195 [00:18<00:00, 23.84it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
95%|█████████▍| 185/195 [00:18<00:00, 23.84it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
95%|█████████▍| 185/195 [00:18<00:00, 23.84it/s]
96%|█████████▋| 188/195 [00:18<00:00, 20.76it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
96%|█████████▋| 188/195 [00:18<00:00, 20.76it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
96%|█████████▋| 188/195 [00:18<00:00, 20.76it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.9.ln.weight[0m", shape: (3072,), dtype: float32
96%|█████████▋| 188/195 [00:18<00:00, 20.76it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
96%|█████████▋| 188/195 [00:18<00:00, 20.76it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
96%|█████████▋| 188/195 [00:19<00:00, 20.76it/s]
98%|█████████▊| 191/195 [00:19<00:00, 22.33it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
98%|█████████▊| 191/195 [00:19<00:00, 22.33it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
98%|█████████▊| 191/195 [00:19<00:00, 22.33it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.9.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
98%|█████████▊| 191/195 [00:19<00:00, 22.33it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
98%|█████████▊| 191/195 [00:19<00:00, 22.33it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
98%|█████████▊| 191/195 [00:19<00:00, 22.33it/s]
99%|█████████▉| 194/195 [00:19<00:00, 23.21it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
99%|█████████▉| 194/195 [00:19<00:00, 23.21it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
99%|█████████▉| 194/195 [00:19<00:00, 23.21it/s]
100%|██████████| 195/195 [00:19<00:00, 10.14it/s]
[2024-06-02 06:08:47] INFO huggingface_loader.py:197: Unloading HF weight file: /models/Phi-3-mini-128k-instruct/model-00001-of-00002.safetensors
[2024-06-02 06:08:48] INFO stats.py:77: [92mTime usage[0m: HF loading: 7.860 sec; Pre-quantization mapping: 2.116 sec; Quantization: 2.664 sec
[2024-06-02 06:08:48] INFO stats.py:91: [92mRAM usage[0m: Peak RAM: 9.262 GB. Total bytes loaded from disk: 14.235 GB
[2024-06-02 06:08:48] INFO convert_weight.py:155: [92mParameter size[0m after quantization: 2.225 GB
[2024-06-02 06:08:48] INFO convert_weight.py:160: [92mTotal parameters[0m: 3,821,079,552
[2024-06-02 06:08:48] INFO convert_weight.py:161: [92mBits per parameter[0m: 5.001
[2024-06-02 06:08:48] INFO convert_weight.py:166: Saved to directory: [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC[0m
All finished, 83 total shards committed, record saved to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/ndarray-cache.json
Also saved a bf16 record to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/ndarray-cache-b16.json
/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f32_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
[2024-06-02 06:15:13] INFO auto_config.py:116: [92mFound[0m model configuration: /models/Phi-3-mini-128k-instruct/config.json
[2024-06-02 06:15:13] INFO auto_config.py:154: [92mFound[0m model type: [1mphi3[0m. Use `--model-type` to override.
[2024-06-02 06:15:13] INFO phi3_model.py:53: [1mcontext_window_size[0m not found in config.json. Falling back to [1mmax_position_embeddings[0m (131072)
[2024-06-02 06:15:13] INFO phi3_model.py:68: [1mprefill_chunk_size[0m defaults to 2048
[2024-06-02 06:15:13] INFO config.py:107: Overriding [1mmax_batch_size[0m from 1 to 80
[2024-06-02 06:15:13] INFO gen_config.py:143: [generation_config.json] Setting [1mbos_token_id[0m: 1
[2024-06-02 06:15:13] INFO gen_config.py:143: [generation_config.json] Setting [1meos_token_id[0m: [32000, 32001, 32007]
[2024-06-02 06:15:13] INFO gen_config.py:143: [generation_config.json] Setting [1mpad_token_id[0m: 32000
[2024-06-02 06:15:13] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer.model[0m
[2024-06-02 06:15:13] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer.json[0m
[2024-06-02 06:15:13] INFO gen_config.py:157: [91mNot found[0m tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json
[2024-06-02 06:15:13] INFO gen_config.py:157: [91mNot found[0m tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt
[2024-06-02 06:15:13] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/added_tokens.json[0m
[2024-06-02 06:15:13] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer_config.json[0m
[2024-06-02 06:15:13] INFO gen_config.py:216: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True}
[2024-06-02 06:15:13] INFO gen_config.py:32: [System default] Setting [1mtemperature[0m: 1.0
[2024-06-02 06:15:13] INFO gen_config.py:32: [System default] Setting [1mpresence_penalty[0m: 0.0
[2024-06-02 06:15:13] INFO gen_config.py:32: [System default] Setting [1mfrequency_penalty[0m: 0.0
[2024-06-02 06:15:13] INFO gen_config.py:32: [System default] Setting [1mrepetition_penalty[0m: 1.0
[2024-06-02 06:15:13] INFO gen_config.py:32: [System default] Setting [1mtop_p[0m: 1.0
[2024-06-02 06:15:13] INFO gen_config.py:32: [System default] Setting [1mmean_gen_len[0m: 128
[2024-06-02 06:15:13] INFO gen_config.py:32: [System default] Setting [1mmax_gen_len[0m: 512
[2024-06-02 06:15:13] INFO gen_config.py:32: [System default] Setting [1mshift_fill_factor[0m: 0.3
[2024-06-02 06:15:13] INFO gen_config.py:223: Dumping configuration file to: [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/mlc-chat-config.json[0m
/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f32_1 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
[2024-06-02 06:15:14] INFO auto_config.py:116: [92mFound[0m model configuration: /models/Phi-3-mini-128k-instruct/config.json
[2024-06-02 06:15:16] INFO auto_device.py:79: [92mFound[0m device: cuda:0
[2024-06-02 06:15:17] INFO auto_device.py:88: [91mNot found[0m device: rocm:0
[2024-06-02 06:15:18] INFO auto_device.py:88: [91mNot found[0m device: metal:0
[2024-06-02 06:15:20] INFO auto_device.py:79: [92mFound[0m device: vulkan:0
[2024-06-02 06:15:20] INFO auto_device.py:79: [92mFound[0m device: vulkan:1
[2024-06-02 06:15:20] INFO auto_device.py:79: [92mFound[0m device: vulkan:2
[2024-06-02 06:15:20] INFO auto_device.py:79: [92mFound[0m device: vulkan:3
[2024-06-02 06:15:22] INFO auto_device.py:88: [91mNot found[0m device: opencl:0
[2024-06-02 06:15:22] INFO auto_device.py:35: Using device: [1mcuda:0[0m
[2024-06-02 06:15:22] INFO auto_weight.py:71: Finding weights in: /models/Phi-3-mini-128k-instruct
[2024-06-02 06:15:22] INFO auto_weight.py:137: [91mNot found[0m Huggingface PyTorch
[2024-06-02 06:15:22] INFO auto_weight.py:144: [92mFound[0m source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json
[2024-06-02 06:15:22] INFO auto_weight.py:107: Using source weight configuration: [1m/models/Phi-3-mini-128k-instruct/model.safetensors.index.json[0m. Use `--source` to override.
[2024-06-02 06:15:22] INFO auto_weight.py:111: Using source weight format: [1mhuggingface-safetensor[0m. Use `--source-format` to override.
[2024-06-02 06:15:22] INFO auto_config.py:154: [92mFound[0m model type: [1mphi3[0m. Use `--model-type` to override.
[2024-06-02 06:15:22] INFO phi3_model.py:53: [1mcontext_window_size[0m not found in config.json. Falling back to [1mmax_position_embeddings[0m (131072)
[2024-06-02 06:15:22] INFO phi3_model.py:68: [1mprefill_chunk_size[0m defaults to 2048
[1mWeight conversion with arguments:[0m
[1m--config[0m /models/Phi-3-mini-128k-instruct/config.json
[1m--quantization[0m GroupQuantize(name='q4f32_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float32', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
[1m--model-type[0m phi3
[1m--device[0m cuda:0
[1m--source[0m /models/Phi-3-mini-128k-instruct/model.safetensors.index.json
[1m--source-format[0m huggingface-safetensor
[1m--output[0m /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
Start storing to cache /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
0%| | 0/195 [00:00<?, ?it/s]
[2024-06-02 06:15:24] INFO huggingface_loader.py:185: Loading HF parameters from: /models/Phi-3-mini-128k-instruct/model-00002-of-00002.safetensors
0%| | 0/195 [00:00<?, ?it/s]
[2024-06-02 06:15:28] INFO group_quantization.py:217: Compiling quantize function for key: ((32064, 3072), float32, cuda, axis=1, output_transpose=False)
0%| | 0/195 [00:04<?, ?it/s]
[2024-06-02 06:15:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mlm_head.q_weight[0m", shape: (32064, 384), dtype: uint32
0%| | 0/195 [00:05<?, ?it/s]
[2024-06-02 06:15:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mlm_head.q_scale[0m", shape: (32064, 96), dtype: float32
0%| | 0/195 [00:05<?, ?it/s]
1%| | 1/195 [00:05<17:57, 5.56s/it]
[2024-06-02 06:15:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.21.ln.weight[0m", shape: (3072,), dtype: float32
1%| | 1/195 [00:05<17:57, 5.56s/it]
[2024-06-02 06:15:29] INFO group_quantization.py:217: Compiling quantize function for key: ((3072, 8192), float32, cuda, axis=1, output_transpose=False)
1%| | 1/195 [00:05<17:57, 5.56s/it]
[2024-06-02 06:15:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
1%| | 1/195 [00:06<17:57, 5.56s/it]
[2024-06-02 06:15:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
1%| | 1/195 [00:06<17:57, 5.56s/it]
2%|▏ | 3/195 [00:06<05:09, 1.61s/it]
[2024-06-02 06:15:30] INFO group_quantization.py:217: Compiling quantize function for key: ((16384, 3072), float32, cuda, axis=1, output_transpose=False)
2%|▏ | 3/195 [00:06<05:09, 1.61s/it]
[2024-06-02 06:15:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
2%|▏ | 3/195 [00:06<05:09, 1.61s/it]
[2024-06-02 06:15:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
2%|▏ | 3/195 [00:06<05:09, 1.61s/it]
2%|▏ | 4/195 [00:06<04:00, 1.26s/it]
[2024-06-02 06:15:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.21.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
2%|▏ | 4/195 [00:06<04:00, 1.26s/it]
[2024-06-02 06:15:30] INFO group_quantization.py:217: Compiling quantize function for key: ((9216, 3072), float32, cuda, axis=1, output_transpose=False)
2%|▏ | 4/195 [00:06<04:00, 1.26s/it]
[2024-06-02 06:15:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
2%|▏ | 4/195 [00:07<04:00, 1.26s/it]
[2024-06-02 06:15:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
2%|▏ | 4/195 [00:07<04:00, 1.26s/it]
3%|▎ | 6/195 [00:07<02:28, 1.27it/s]
[2024-06-02 06:15:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.22.ln.weight[0m", shape: (3072,), dtype: float32
3%|▎ | 6/195 [00:07<02:28, 1.27it/s]
[2024-06-02 06:15:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
3%|▎ | 6/195 [00:07<02:28, 1.27it/s]
[2024-06-02 06:15:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
3%|▎ | 6/195 [00:07<02:28, 1.27it/s]
[2024-06-02 06:15:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
3%|▎ | 6/195 [00:07<02:28, 1.27it/s]
[2024-06-02 06:15:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
3%|▎ | 6/195 [00:07<02:28, 1.27it/s]
5%|▍ | 9/195 [00:07<01:17, 2.40it/s]
[2024-06-02 06:15:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.22.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
5%|▍ | 9/195 [00:07<01:17, 2.40it/s]
[2024-06-02 06:15:31] INFO group_quantization.py:217: Compiling quantize function for key: ((3072, 3072), float32, cuda, axis=1, output_transpose=False)
5%|▍ | 9/195 [00:07<01:17, 2.40it/s]
[2024-06-02 06:15:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
5%|▍ | 9/195 [00:07<01:17, 2.40it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
5%|▍ | 9/195 [00:08<01:17, 2.40it/s]
6%|▌ | 11/195 [00:08<01:08, 2.68it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
6%|▌ | 11/195 [00:08<01:08, 2.68it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
6%|▌ | 11/195 [00:08<01:08, 2.68it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.23.ln.weight[0m", shape: (3072,), dtype: float32
6%|▌ | 11/195 [00:08<01:08, 2.68it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
6%|▌ | 11/195 [00:08<01:08, 2.68it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
6%|▌ | 11/195 [00:08<01:08, 2.68it/s]
7%|▋ | 14/195 [00:08<00:43, 4.17it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
7%|▋ | 14/195 [00:08<00:43, 4.17it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
7%|▋ | 14/195 [00:08<00:43, 4.17it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.23.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
7%|▋ | 14/195 [00:08<00:43, 4.17it/s]
8%|▊ | 16/195 [00:08<00:34, 5.18it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
8%|▊ | 16/195 [00:08<00:34, 5.18it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
8%|▊ | 16/195 [00:08<00:34, 5.18it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
8%|▊ | 16/195 [00:08<00:34, 5.18it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
8%|▊ | 16/195 [00:08<00:34, 5.18it/s]
9%|▉ | 18/195 [00:08<00:27, 6.43it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.24.ln.weight[0m", shape: (3072,), dtype: float32
9%|▉ | 18/195 [00:08<00:27, 6.43it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
9%|▉ | 18/195 [00:08<00:27, 6.43it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
9%|▉ | 18/195 [00:08<00:27, 6.43it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
9%|▉ | 18/195 [00:08<00:27, 6.43it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
9%|▉ | 18/195 [00:08<00:27, 6.43it/s]
11%|█ | 21/195 [00:08<00:20, 8.49it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.24.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
11%|█ | 21/195 [00:08<00:20, 8.49it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
11%|█ | 21/195 [00:08<00:20, 8.49it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
11%|█ | 21/195 [00:08<00:20, 8.49it/s]
12%|█▏ | 23/195 [00:08<00:17, 9.61it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
12%|█▏ | 23/195 [00:08<00:17, 9.61it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
12%|█▏ | 23/195 [00:08<00:17, 9.61it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.25.ln.weight[0m", shape: (3072,), dtype: float32
12%|█▏ | 23/195 [00:08<00:17, 9.61it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
12%|█▏ | 23/195 [00:08<00:17, 9.61it/s]
[2024-06-02 06:15:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
12%|█▏ | 23/195 [00:08<00:17, 9.61it/s]
13%|█▎ | 26/195 [00:08<00:14, 11.95it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
13%|█▎ | 26/195 [00:08<00:14, 11.95it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
13%|█▎ | 26/195 [00:09<00:14, 11.95it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.25.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
13%|█▎ | 26/195 [00:09<00:14, 11.95it/s]
14%|█▍ | 28/195 [00:09<00:13, 12.52it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
14%|█▍ | 28/195 [00:09<00:13, 12.52it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
14%|█▍ | 28/195 [00:09<00:13, 12.52it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
14%|█▍ | 28/195 [00:09<00:13, 12.52it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
14%|█▍ | 28/195 [00:09<00:13, 12.52it/s]
15%|█▌ | 30/195 [00:09<00:12, 13.24it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.26.ln.weight[0m", shape: (3072,), dtype: float32
15%|█▌ | 30/195 [00:09<00:12, 13.24it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
15%|█▌ | 30/195 [00:09<00:12, 13.24it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
15%|█▌ | 30/195 [00:09<00:12, 13.24it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
15%|█▌ | 30/195 [00:09<00:12, 13.24it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
15%|█▌ | 30/195 [00:09<00:12, 13.24it/s]
17%|█▋ | 33/195 [00:09<00:11, 14.35it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.26.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
17%|█▋ | 33/195 [00:09<00:11, 14.35it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
17%|█▋ | 33/195 [00:09<00:11, 14.35it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
17%|█▋ | 33/195 [00:09<00:11, 14.35it/s]
18%|█▊ | 35/195 [00:09<00:11, 14.51it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
18%|█▊ | 35/195 [00:09<00:11, 14.51it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
18%|█▊ | 35/195 [00:09<00:11, 14.51it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.27.ln.weight[0m", shape: (3072,), dtype: float32
18%|█▊ | 35/195 [00:09<00:11, 14.51it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
18%|█▊ | 35/195 [00:09<00:11, 14.51it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
18%|█▊ | 35/195 [00:09<00:11, 14.51it/s]
19%|█▉ | 38/195 [00:09<00:09, 16.36it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
19%|█▉ | 38/195 [00:09<00:09, 16.36it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
19%|█▉ | 38/195 [00:09<00:09, 16.36it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.27.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
19%|█▉ | 38/195 [00:09<00:09, 16.36it/s]
21%|██ | 40/195 [00:09<00:09, 15.64it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
21%|██ | 40/195 [00:09<00:09, 15.64it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
21%|██ | 40/195 [00:09<00:09, 15.64it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
21%|██ | 40/195 [00:09<00:09, 15.64it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
21%|██ | 40/195 [00:09<00:09, 15.64it/s]
22%|██▏ | 42/195 [00:09<00:09, 15.33it/s]
[2024-06-02 06:15:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.28.ln.weight[0m", shape: (3072,), dtype: float32
22%|██▏ | 42/195 [00:09<00:09, 15.33it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
22%|██▏ | 42/195 [00:09<00:09, 15.33it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
22%|██▏ | 42/195 [00:09<00:09, 15.33it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
22%|██▏ | 42/195 [00:09<00:09, 15.33it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
22%|██▏ | 42/195 [00:10<00:09, 15.33it/s]
23%|██▎ | 45/195 [00:10<00:09, 15.64it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.28.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
23%|██▎ | 45/195 [00:10<00:09, 15.64it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
23%|██▎ | 45/195 [00:10<00:09, 15.64it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
23%|██▎ | 45/195 [00:10<00:09, 15.64it/s]
24%|██▍ | 47/195 [00:10<00:09, 15.24it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
24%|██▍ | 47/195 [00:10<00:09, 15.24it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
24%|██▍ | 47/195 [00:10<00:09, 15.24it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.29.ln.weight[0m", shape: (3072,), dtype: float32
24%|██▍ | 47/195 [00:10<00:09, 15.24it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
24%|██▍ | 47/195 [00:10<00:09, 15.24it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
24%|██▍ | 47/195 [00:10<00:09, 15.24it/s]
26%|██▌ | 50/195 [00:10<00:08, 16.36it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
26%|██▌ | 50/195 [00:10<00:08, 16.36it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
26%|██▌ | 50/195 [00:10<00:08, 16.36it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.29.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
26%|██▌ | 50/195 [00:10<00:08, 16.36it/s]
27%|██▋ | 52/195 [00:10<00:09, 15.34it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
27%|██▋ | 52/195 [00:10<00:09, 15.34it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
27%|██▋ | 52/195 [00:10<00:09, 15.34it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
27%|██▋ | 52/195 [00:10<00:09, 15.34it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
27%|██▋ | 52/195 [00:10<00:09, 15.34it/s]
28%|██▊ | 54/195 [00:10<00:09, 15.20it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.30.ln.weight[0m", shape: (3072,), dtype: float32
28%|██▊ | 54/195 [00:10<00:09, 15.20it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
28%|██▊ | 54/195 [00:10<00:09, 15.20it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
28%|██▊ | 54/195 [00:10<00:09, 15.20it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
28%|██▊ | 54/195 [00:10<00:09, 15.20it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
28%|██▊ | 54/195 [00:10<00:09, 15.20it/s]
29%|██▉ | 57/195 [00:10<00:08, 15.59it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.30.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
29%|██▉ | 57/195 [00:10<00:08, 15.59it/s]
[2024-06-02 06:15:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
29%|██▉ | 57/195 [00:10<00:08, 15.59it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
29%|██▉ | 57/195 [00:10<00:08, 15.59it/s]
30%|███ | 59/195 [00:10<00:08, 15.18it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
30%|███ | 59/195 [00:11<00:08, 15.18it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
30%|███ | 59/195 [00:11<00:08, 15.18it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.31.ln.weight[0m", shape: (3072,), dtype: float32
30%|███ | 59/195 [00:11<00:08, 15.18it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
30%|███ | 59/195 [00:11<00:08, 15.18it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
30%|███ | 59/195 [00:11<00:08, 15.18it/s]
32%|███▏ | 62/195 [00:11<00:07, 16.94it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
32%|███▏ | 62/195 [00:11<00:07, 16.94it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
32%|███▏ | 62/195 [00:11<00:07, 16.94it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.31.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
32%|███▏ | 62/195 [00:11<00:07, 16.94it/s]
33%|███▎ | 64/195 [00:11<00:08, 15.90it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
33%|███▎ | 64/195 [00:11<00:08, 15.90it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
33%|███▎ | 64/195 [00:11<00:08, 15.90it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
33%|███▎ | 64/195 [00:11<00:08, 15.90it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
33%|███▎ | 64/195 [00:11<00:08, 15.90it/s]
34%|███▍ | 66/195 [00:11<00:08, 15.39it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.norm.weight[0m", shape: (3072,), dtype: float32
34%|███▍ | 66/195 [00:11<00:08, 15.39it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:197: Unloading HF weight file: /models/Phi-3-mini-128k-instruct/model-00002-of-00002.safetensors
34%|███▍ | 66/195 [00:11<00:08, 15.39it/s]
[2024-06-02 06:15:35] INFO huggingface_loader.py:185: Loading HF parameters from: /models/Phi-3-mini-128k-instruct/model-00001-of-00002.safetensors
34%|███▍ | 66/195 [00:11<00:08, 15.39it/s]
[2024-06-02 06:15:40] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.embd.q_weight[0m", shape: (32064, 384), dtype: uint32
34%|███▍ | 66/195 [00:16<00:08, 15.39it/s]
[2024-06-02 06:15:40] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.embd.q_scale[0m", shape: (32064, 96), dtype: float32
34%|███▍ | 66/195 [00:16<00:08, 15.39it/s]
35%|███▍ | 68/195 [00:16<01:39, 1.28it/s]
[2024-06-02 06:15:40] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.0.ln.weight[0m", shape: (3072,), dtype: float32
35%|███▍ | 68/195 [00:16<01:39, 1.28it/s]
[2024-06-02 06:15:40] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
35%|███▍ | 68/195 [00:16<01:39, 1.28it/s]
[2024-06-02 06:15:40] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
35%|███▍ | 68/195 [00:16<01:39, 1.28it/s]
36%|███▌ | 70/195 [00:16<01:12, 1.73it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
36%|███▌ | 70/195 [00:16<01:12, 1.73it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
36%|███▌ | 70/195 [00:17<01:12, 1.73it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.0.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
36%|███▌ | 70/195 [00:17<01:12, 1.73it/s]
37%|███▋ | 72/195 [00:17<00:53, 2.31it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
37%|███▋ | 72/195 [00:17<00:53, 2.31it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
37%|███▋ | 72/195 [00:17<00:53, 2.31it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
37%|███▋ | 72/195 [00:17<00:53, 2.31it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
37%|███▋ | 72/195 [00:17<00:53, 2.31it/s]
38%|███▊ | 74/195 [00:17<00:39, 3.06it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.1.ln.weight[0m", shape: (3072,), dtype: float32
38%|███▊ | 74/195 [00:17<00:39, 3.06it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
38%|███▊ | 74/195 [00:17<00:39, 3.06it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
38%|███▊ | 74/195 [00:17<00:39, 3.06it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
38%|███▊ | 74/195 [00:17<00:39, 3.06it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
38%|███▊ | 74/195 [00:17<00:39, 3.06it/s]
39%|███▉ | 77/195 [00:17<00:26, 4.42it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.1.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
39%|███▉ | 77/195 [00:17<00:26, 4.42it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
39%|███▉ | 77/195 [00:17<00:26, 4.42it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
39%|███▉ | 77/195 [00:17<00:26, 4.42it/s]
41%|████ | 79/195 [00:17<00:21, 5.35it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
41%|████ | 79/195 [00:17<00:21, 5.35it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
41%|████ | 79/195 [00:17<00:21, 5.35it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.10.ln.weight[0m", shape: (3072,), dtype: float32
41%|████ | 79/195 [00:17<00:21, 5.35it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
41%|████ | 79/195 [00:17<00:21, 5.35it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
41%|████ | 79/195 [00:17<00:21, 5.35it/s]
42%|████▏ | 82/195 [00:17<00:15, 7.20it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
42%|████▏ | 82/195 [00:17<00:15, 7.20it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
42%|████▏ | 82/195 [00:17<00:15, 7.20it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.10.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
42%|████▏ | 82/195 [00:17<00:15, 7.20it/s]
43%|████▎ | 84/195 [00:17<00:13, 8.08it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
43%|████▎ | 84/195 [00:17<00:13, 8.08it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
43%|████▎ | 84/195 [00:17<00:13, 8.08it/s]
[2024-06-02 06:15:41] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
43%|████▎ | 84/195 [00:17<00:13, 8.08it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
43%|████▎ | 84/195 [00:17<00:13, 8.08it/s]
44%|████▍ | 86/195 [00:17<00:11, 9.09it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.11.ln.weight[0m", shape: (3072,), dtype: float32
44%|████▍ | 86/195 [00:17<00:11, 9.09it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
44%|████▍ | 86/195 [00:18<00:11, 9.09it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
44%|████▍ | 86/195 [00:18<00:11, 9.09it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
44%|████▍ | 86/195 [00:18<00:11, 9.09it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
44%|████▍ | 86/195 [00:18<00:11, 9.09it/s]
46%|████▌ | 89/195 [00:18<00:09, 10.89it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.11.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
46%|████▌ | 89/195 [00:18<00:09, 10.89it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
46%|████▌ | 89/195 [00:18<00:09, 10.89it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
46%|████▌ | 89/195 [00:18<00:09, 10.89it/s]
47%|████▋ | 91/195 [00:18<00:08, 11.58it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
47%|████▋ | 91/195 [00:18<00:08, 11.58it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
47%|████▋ | 91/195 [00:18<00:08, 11.58it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.12.ln.weight[0m", shape: (3072,), dtype: float32
47%|████▋ | 91/195 [00:18<00:08, 11.58it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
47%|████▋ | 91/195 [00:18<00:08, 11.58it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
47%|████▋ | 91/195 [00:18<00:08, 11.58it/s]
48%|████▊ | 94/195 [00:18<00:07, 13.25it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
48%|████▊ | 94/195 [00:18<00:07, 13.25it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
48%|████▊ | 94/195 [00:18<00:07, 13.25it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.12.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
48%|████▊ | 94/195 [00:18<00:07, 13.25it/s]
49%|████▉ | 96/195 [00:18<00:07, 13.32it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
49%|████▉ | 96/195 [00:18<00:07, 13.32it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
49%|████▉ | 96/195 [00:18<00:07, 13.32it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
49%|████▉ | 96/195 [00:18<00:07, 13.32it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
49%|████▉ | 96/195 [00:18<00:07, 13.32it/s]
50%|█████ | 98/195 [00:18<00:07, 13.82it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.13.ln.weight[0m", shape: (3072,), dtype: float32
50%|█████ | 98/195 [00:18<00:07, 13.82it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
50%|█████ | 98/195 [00:18<00:07, 13.82it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
50%|█████ | 98/195 [00:18<00:07, 13.82it/s]
[2024-06-02 06:15:42] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
50%|█████ | 98/195 [00:18<00:07, 13.82it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
50%|█████ | 98/195 [00:18<00:07, 13.82it/s]
52%|█████▏ | 101/195 [00:18<00:06, 14.78it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.13.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
52%|█████▏ | 101/195 [00:18<00:06, 14.78it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
52%|█████▏ | 101/195 [00:18<00:06, 14.78it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
52%|█████▏ | 101/195 [00:19<00:06, 14.78it/s]
53%|█████▎ | 103/195 [00:19<00:06, 14.67it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
53%|█████▎ | 103/195 [00:19<00:06, 14.67it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
53%|█████▎ | 103/195 [00:19<00:06, 14.67it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.14.ln.weight[0m", shape: (3072,), dtype: float32
53%|█████▎ | 103/195 [00:19<00:06, 14.67it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
53%|█████▎ | 103/195 [00:19<00:06, 14.67it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
53%|█████▎ | 103/195 [00:19<00:06, 14.67it/s]
54%|█████▍ | 106/195 [00:19<00:05, 16.21it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
54%|█████▍ | 106/195 [00:19<00:05, 16.21it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
54%|█████▍ | 106/195 [00:19<00:05, 16.21it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.14.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
54%|█████▍ | 106/195 [00:19<00:05, 16.21it/s]
55%|█████▌ | 108/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
55%|█████▌ | 108/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
55%|█████▌ | 108/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
55%|█████▌ | 108/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
55%|█████▌ | 108/195 [00:19<00:05, 15.43it/s]
56%|█████▋ | 110/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.15.ln.weight[0m", shape: (3072,), dtype: float32
56%|█████▋ | 110/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
56%|█████▋ | 110/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
56%|█████▋ | 110/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
56%|█████▋ | 110/195 [00:19<00:05, 15.43it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
56%|█████▋ | 110/195 [00:19<00:05, 15.43it/s]
58%|█████▊ | 113/195 [00:19<00:05, 15.87it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.15.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
58%|█████▊ | 113/195 [00:19<00:05, 15.87it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
58%|█████▊ | 113/195 [00:19<00:05, 15.87it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
58%|█████▊ | 113/195 [00:19<00:05, 15.87it/s]
59%|█████▉ | 115/195 [00:19<00:05, 15.21it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
59%|█████▉ | 115/195 [00:19<00:05, 15.21it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
59%|█████▉ | 115/195 [00:19<00:05, 15.21it/s]
[2024-06-02 06:15:43] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.16.ln.weight[0m", shape: (3072,), dtype: float32
59%|█████▉ | 115/195 [00:19<00:05, 15.21it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
59%|█████▉ | 115/195 [00:19<00:05, 15.21it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
59%|█████▉ | 115/195 [00:19<00:05, 15.21it/s]
61%|██████ | 118/195 [00:19<00:04, 16.35it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
61%|██████ | 118/195 [00:20<00:04, 16.35it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
61%|██████ | 118/195 [00:20<00:04, 16.35it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.16.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
61%|██████ | 118/195 [00:20<00:04, 16.35it/s]
62%|██████▏ | 120/195 [00:20<00:04, 15.51it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
62%|██████▏ | 120/195 [00:20<00:04, 15.51it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
62%|██████▏ | 120/195 [00:20<00:04, 15.51it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
62%|██████▏ | 120/195 [00:20<00:04, 15.51it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
62%|██████▏ | 120/195 [00:20<00:04, 15.51it/s]
63%|██████▎ | 122/195 [00:20<00:04, 15.42it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.17.ln.weight[0m", shape: (3072,), dtype: float32
63%|██████▎ | 122/195 [00:20<00:04, 15.42it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
63%|██████▎ | 122/195 [00:20<00:04, 15.42it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
63%|██████▎ | 122/195 [00:20<00:04, 15.42it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
63%|██████▎ | 122/195 [00:20<00:04, 15.42it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
63%|██████▎ | 122/195 [00:20<00:04, 15.42it/s]
64%|██████▍ | 125/195 [00:20<00:04, 15.87it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.17.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
64%|██████▍ | 125/195 [00:20<00:04, 15.87it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
64%|██████▍ | 125/195 [00:20<00:04, 15.87it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
64%|██████▍ | 125/195 [00:20<00:04, 15.87it/s]
65%|██████▌ | 127/195 [00:20<00:04, 15.47it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
65%|██████▌ | 127/195 [00:20<00:04, 15.47it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
65%|██████▌ | 127/195 [00:20<00:04, 15.47it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.18.ln.weight[0m", shape: (3072,), dtype: float32
65%|██████▌ | 127/195 [00:20<00:04, 15.47it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
65%|██████▌ | 127/195 [00:20<00:04, 15.47it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
65%|██████▌ | 127/195 [00:20<00:04, 15.47it/s]
67%|██████▋ | 130/195 [00:20<00:03, 16.99it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
67%|██████▋ | 130/195 [00:20<00:03, 16.99it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
67%|██████▋ | 130/195 [00:20<00:03, 16.99it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.18.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
67%|██████▋ | 130/195 [00:20<00:03, 16.99it/s]
68%|██████▊ | 132/195 [00:20<00:03, 15.94it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
68%|██████▊ | 132/195 [00:20<00:03, 15.94it/s]
[2024-06-02 06:15:44] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
68%|██████▊ | 132/195 [00:20<00:03, 15.94it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
68%|██████▊ | 132/195 [00:20<00:03, 15.94it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
68%|██████▊ | 132/195 [00:21<00:03, 15.94it/s]
69%|██████▊ | 134/195 [00:21<00:03, 15.79it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.19.ln.weight[0m", shape: (3072,), dtype: float32
69%|██████▊ | 134/195 [00:21<00:03, 15.79it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
69%|██████▊ | 134/195 [00:21<00:03, 15.79it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
69%|██████▊ | 134/195 [00:21<00:03, 15.79it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
69%|██████▊ | 134/195 [00:21<00:03, 15.79it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
69%|██████▊ | 134/195 [00:21<00:03, 15.79it/s]
70%|███████ | 137/195 [00:21<00:03, 16.14it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.19.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
70%|███████ | 137/195 [00:21<00:03, 16.14it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
70%|███████ | 137/195 [00:21<00:03, 16.14it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
70%|███████ | 137/195 [00:21<00:03, 16.14it/s]
71%|███████▏ | 139/195 [00:21<00:03, 15.63it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
71%|███████▏ | 139/195 [00:21<00:03, 15.63it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
71%|███████▏ | 139/195 [00:21<00:03, 15.63it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.2.ln.weight[0m", shape: (3072,), dtype: float32
71%|███████▏ | 139/195 [00:21<00:03, 15.63it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
71%|███████▏ | 139/195 [00:21<00:03, 15.63it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
71%|███████▏ | 139/195 [00:21<00:03, 15.63it/s]
73%|███████▎ | 142/195 [00:21<00:03, 16.92it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
73%|███████▎ | 142/195 [00:21<00:03, 16.92it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
73%|███████▎ | 142/195 [00:21<00:03, 16.92it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.2.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
73%|███████▎ | 142/195 [00:21<00:03, 16.92it/s]
74%|███████▍ | 144/195 [00:21<00:03, 16.07it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
74%|███████▍ | 144/195 [00:21<00:03, 16.07it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
74%|███████▍ | 144/195 [00:21<00:03, 16.07it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
74%|███████▍ | 144/195 [00:21<00:03, 16.07it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
74%|███████▍ | 144/195 [00:21<00:03, 16.07it/s]
75%|███████▍ | 146/195 [00:21<00:03, 15.86it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.20.ln.weight[0m", shape: (3072,), dtype: float32
75%|███████▍ | 146/195 [00:21<00:03, 15.86it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
75%|███████▍ | 146/195 [00:21<00:03, 15.86it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
75%|███████▍ | 146/195 [00:21<00:03, 15.86it/s]
[2024-06-02 06:15:45] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
75%|███████▍ | 146/195 [00:21<00:03, 15.86it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
75%|███████▍ | 146/195 [00:21<00:03, 15.86it/s]
76%|███████▋ | 149/195 [00:21<00:02, 16.20it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.20.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
76%|███████▋ | 149/195 [00:21<00:02, 16.20it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
76%|███████▋ | 149/195 [00:21<00:02, 16.20it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
76%|███████▋ | 149/195 [00:22<00:02, 16.20it/s]
77%|███████▋ | 151/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
77%|███████▋ | 151/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
77%|███████▋ | 151/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
77%|███████▋ | 151/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
77%|███████▋ | 151/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.3.ln.weight[0m", shape: (3072,), dtype: float32
77%|███████▋ | 151/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
77%|███████▋ | 151/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
77%|███████▋ | 151/195 [00:22<00:02, 15.69it/s]
79%|███████▉ | 155/195 [00:22<00:02, 17.13it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
79%|███████▉ | 155/195 [00:22<00:02, 17.13it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
79%|███████▉ | 155/195 [00:22<00:02, 17.13it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.3.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
79%|███████▉ | 155/195 [00:22<00:02, 17.13it/s]
81%|████████ | 157/195 [00:22<00:02, 16.04it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
81%|████████ | 157/195 [00:22<00:02, 16.04it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
81%|████████ | 157/195 [00:22<00:02, 16.04it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
81%|████████ | 157/195 [00:22<00:02, 16.04it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
81%|████████ | 157/195 [00:22<00:02, 16.04it/s]
82%|████████▏ | 159/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.4.ln.weight[0m", shape: (3072,), dtype: float32
82%|████████▏ | 159/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
82%|████████▏ | 159/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
82%|████████▏ | 159/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
82%|████████▏ | 159/195 [00:22<00:02, 15.69it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
82%|████████▏ | 159/195 [00:22<00:02, 15.69it/s]
83%|████████▎ | 162/195 [00:22<00:02, 15.62it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.4.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
83%|████████▎ | 162/195 [00:22<00:02, 15.62it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
83%|████████▎ | 162/195 [00:22<00:02, 15.62it/s]
[2024-06-02 06:15:46] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
83%|████████▎ | 162/195 [00:22<00:02, 15.62it/s]
84%|████████▍ | 164/195 [00:22<00:02, 15.34it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
84%|████████▍ | 164/195 [00:22<00:02, 15.34it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
84%|████████▍ | 164/195 [00:22<00:02, 15.34it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.5.ln.weight[0m", shape: (3072,), dtype: float32
84%|████████▍ | 164/195 [00:22<00:02, 15.34it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
84%|████████▍ | 164/195 [00:22<00:02, 15.34it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
84%|████████▍ | 164/195 [00:23<00:02, 15.34it/s]
86%|████████▌ | 167/195 [00:23<00:01, 16.67it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
86%|████████▌ | 167/195 [00:23<00:01, 16.67it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
86%|████████▌ | 167/195 [00:23<00:01, 16.67it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.5.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
86%|████████▌ | 167/195 [00:23<00:01, 16.67it/s]
87%|████████▋ | 169/195 [00:23<00:01, 15.59it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
87%|████████▋ | 169/195 [00:23<00:01, 15.59it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
87%|████████▋ | 169/195 [00:23<00:01, 15.59it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
87%|████████▋ | 169/195 [00:23<00:01, 15.59it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
87%|████████▋ | 169/195 [00:23<00:01, 15.59it/s]
88%|████████▊ | 171/195 [00:23<00:01, 15.43it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.6.ln.weight[0m", shape: (3072,), dtype: float32
88%|████████▊ | 171/195 [00:23<00:01, 15.43it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
88%|████████▊ | 171/195 [00:23<00:01, 15.43it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
88%|████████▊ | 171/195 [00:23<00:01, 15.43it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
88%|████████▊ | 171/195 [00:23<00:01, 15.43it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
88%|████████▊ | 171/195 [00:23<00:01, 15.43it/s]
89%|████████▉ | 174/195 [00:23<00:01, 15.78it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.6.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
89%|████████▉ | 174/195 [00:23<00:01, 15.78it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
89%|████████▉ | 174/195 [00:23<00:01, 15.78it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
89%|████████▉ | 174/195 [00:23<00:01, 15.78it/s]
90%|█████████ | 176/195 [00:23<00:01, 15.53it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
90%|█████████ | 176/195 [00:23<00:01, 15.53it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
90%|█████████ | 176/195 [00:23<00:01, 15.53it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.7.ln.weight[0m", shape: (3072,), dtype: float32
90%|█████████ | 176/195 [00:23<00:01, 15.53it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
90%|█████████ | 176/195 [00:23<00:01, 15.53it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
90%|█████████ | 176/195 [00:23<00:01, 15.53it/s]
92%|█████████▏| 179/195 [00:23<00:00, 16.99it/s]
[2024-06-02 06:15:47] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
92%|█████████▏| 179/195 [00:23<00:00, 16.99it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
92%|█████████▏| 179/195 [00:23<00:00, 16.99it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.7.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
92%|█████████▏| 179/195 [00:23<00:00, 16.99it/s]
93%|█████████▎| 181/195 [00:23<00:00, 15.70it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
93%|█████████▎| 181/195 [00:23<00:00, 15.70it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
93%|█████████▎| 181/195 [00:23<00:00, 15.70it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
93%|█████████▎| 181/195 [00:23<00:00, 15.70it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
93%|█████████▎| 181/195 [00:24<00:00, 15.70it/s]
94%|█████████▍| 183/195 [00:24<00:00, 15.55it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.8.ln.weight[0m", shape: (3072,), dtype: float32
94%|█████████▍| 183/195 [00:24<00:00, 15.55it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
94%|█████████▍| 183/195 [00:24<00:00, 15.55it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
94%|█████████▍| 183/195 [00:24<00:00, 15.55it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
94%|█████████▍| 183/195 [00:24<00:00, 15.55it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
94%|█████████▍| 183/195 [00:24<00:00, 15.55it/s]
95%|█████████▌| 186/195 [00:24<00:00, 15.98it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.8.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
95%|█████████▌| 186/195 [00:24<00:00, 15.98it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
95%|█████████▌| 186/195 [00:24<00:00, 15.98it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
95%|█████████▌| 186/195 [00:24<00:00, 15.98it/s]
96%|█████████▋| 188/195 [00:24<00:00, 15.83it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
96%|█████████▋| 188/195 [00:24<00:00, 15.83it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
96%|█████████▋| 188/195 [00:24<00:00, 15.83it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.9.ln.weight[0m", shape: (3072,), dtype: float32
96%|█████████▋| 188/195 [00:24<00:00, 15.83it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
96%|█████████▋| 188/195 [00:24<00:00, 15.83it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
96%|█████████▋| 188/195 [00:24<00:00, 15.83it/s]
98%|█████████▊| 191/195 [00:24<00:00, 17.36it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
98%|█████████▊| 191/195 [00:24<00:00, 17.36it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
98%|█████████▊| 191/195 [00:24<00:00, 17.36it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.9.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
98%|█████████▊| 191/195 [00:24<00:00, 17.36it/s]
99%|█████████▉| 193/195 [00:24<00:00, 16.36it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
99%|█████████▉| 193/195 [00:24<00:00, 16.36it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
99%|█████████▉| 193/195 [00:24<00:00, 16.36it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
99%|█████████▉| 193/195 [00:24<00:00, 16.36it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
99%|█████████▉| 193/195 [00:24<00:00, 16.36it/s]
100%|██████████| 195/195 [00:24<00:00, 16.21it/s]
100%|██████████| 195/195 [00:24<00:00, 7.86it/s]
[2024-06-02 06:15:48] INFO huggingface_loader.py:197: Unloading HF weight file: /models/Phi-3-mini-128k-instruct/model-00001-of-00002.safetensors
[2024-06-02 06:15:49] INFO stats.py:77: [92mTime usage[0m: HF loading: 9.499 sec; Pre-quantization mapping: 2.134 sec; Quantization: 2.784 sec
[2024-06-02 06:15:49] INFO stats.py:91: [92mRAM usage[0m: Peak RAM: 9.262 GB. Total bytes loaded from disk: 14.235 GB
[2024-06-02 06:15:49] INFO convert_weight.py:155: [92mParameter size[0m after quantization: 2.225 GB
[2024-06-02 06:15:49] INFO convert_weight.py:160: [92mTotal parameters[0m: 3,821,079,552
[2024-06-02 06:15:49] INFO convert_weight.py:161: [92mBits per parameter[0m: 5.001
[2024-06-02 06:15:49] INFO convert_weight.py:166: Saved to directory: [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC[0m
All finished, 83 total shards committed, record saved to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/ndarray-cache.json
Also saved a bf16 record to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/ndarray-cache-b16.json
/home/rickzhou/miniconda3/envs/mlc/bin/python -m mlc_llm gen_config /ssd1/rickzhou/models/Phi-3-mini-128k-instruct --quantization q4f32_1 --conv-template phi-3 --output /ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
[2024-06-02 04:03:58] INFO auto_config.py:115: [92mFound[0m model configuration: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/config.json
[2024-06-02 04:03:58] INFO auto_config.py:153: [92mFound[0m model type: [1mphi3[0m. Use `--model-type` to override.
[2024-06-02 04:03:58] INFO phi3_model.py:53: [1mcontext_window_size[0m not found in config.json. Falling back to [1mmax_position_embeddings[0m (131072)
[2024-06-02 04:03:58] INFO phi3_model.py:68: [1mprefill_chunk_size[0m defaults to 2048
[2024-06-02 04:03:58] INFO config.py:106: Overriding [1mmax_batch_size[0m from 1 to 80
[2024-06-02 04:03:58] INFO gen_config.py:142: [generation_config.json] Setting [1mbos_token_id[0m: 1
[2024-06-02 04:03:58] INFO gen_config.py:142: [generation_config.json] Setting [1meos_token_id[0m: [32000, 32001, 32007]
[2024-06-02 04:03:58] INFO gen_config.py:142: [generation_config.json] Setting [1mpad_token_id[0m: 32000
[2024-06-02 04:03:58] INFO gen_config.py:154: [92mFound[0m tokenizer config: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to [1m/ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer.model[0m
[2024-06-02 04:03:58] INFO gen_config.py:154: [92mFound[0m tokenizer config: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to [1m/ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer.json[0m
[2024-06-02 04:03:58] INFO gen_config.py:156: [91mNot found[0m tokenizer config: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/vocab.json
[2024-06-02 04:03:58] INFO gen_config.py:156: [91mNot found[0m tokenizer config: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/merges.txt
[2024-06-02 04:03:58] INFO gen_config.py:154: [92mFound[0m tokenizer config: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to [1m/ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/added_tokens.json[0m
[2024-06-02 04:03:58] INFO gen_config.py:154: [92mFound[0m tokenizer config: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to [1m/ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer_config.json[0m
[2024-06-02 04:03:58] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True}
[2024-06-02 04:03:58] INFO gen_config.py:31: [System default] Setting [1mtemperature[0m: 1.0
[2024-06-02 04:03:58] INFO gen_config.py:31: [System default] Setting [1mpresence_penalty[0m: 0.0
[2024-06-02 04:03:58] INFO gen_config.py:31: [System default] Setting [1mfrequency_penalty[0m: 0.0
[2024-06-02 04:03:58] INFO gen_config.py:31: [System default] Setting [1mrepetition_penalty[0m: 1.0
[2024-06-02 04:03:58] INFO gen_config.py:31: [System default] Setting [1mtop_p[0m: 1.0
[2024-06-02 04:03:58] INFO gen_config.py:31: [System default] Setting [1mmean_gen_len[0m: 128
[2024-06-02 04:03:58] INFO gen_config.py:31: [System default] Setting [1mmax_gen_len[0m: 512
[2024-06-02 04:03:58] INFO gen_config.py:31: [System default] Setting [1mshift_fill_factor[0m: 0.3
/home/rickzhou/miniconda3/envs/mlc/lib/python3.11/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings:
Expected `int` but got `list` - serialized value may not be as expected
return self.__pydantic_serializer__.to_python(
[2024-06-02 04:03:58] INFO gen_config.py:222: Dumping configuration file to: [1m/ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/mlc-chat-config.json[0m
/home/rickzhou/miniconda3/envs/mlc/bin/python -m mlc_llm convert_weight /ssd1/rickzhou/models/Phi-3-mini-128k-instruct --quantization q4f32_1 --output /ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
[2024-06-02 04:03:59] INFO auto_config.py:115: [92mFound[0m model configuration: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/config.json
[2024-06-02 04:04:00] INFO auto_device.py:79: [92mFound[0m device: cuda:0
[2024-06-02 04:04:00] INFO auto_device.py:79: [92mFound[0m device: cuda:1
[2024-06-02 04:04:01] INFO auto_device.py:88: [91mNot found[0m device: rocm:0
[2024-06-02 04:04:02] INFO auto_device.py:88: [91mNot found[0m device: metal:0
[2024-06-02 04:04:03] INFO auto_device.py:79: [92mFound[0m device: vulkan:0
[2024-06-02 04:04:03] INFO auto_device.py:79: [92mFound[0m device: vulkan:1
[2024-06-02 04:04:03] INFO auto_device.py:79: [92mFound[0m device: vulkan:2
[2024-06-02 04:04:04] INFO auto_device.py:88: [91mNot found[0m device: opencl:0
[2024-06-02 04:04:04] INFO auto_device.py:35: Using device: [1mcuda:0[0m
[2024-06-02 04:04:04] INFO auto_weight.py:70: Finding weights in: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct
[2024-06-02 04:04:04] INFO auto_weight.py:136: [91mNot found[0m Huggingface PyTorch
[2024-06-02 04:04:04] INFO auto_weight.py:143: [92mFound[0m source weight format: huggingface-safetensor. Source configuration: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/model.safetensors.index.json
[2024-06-02 04:04:04] INFO auto_weight.py:106: Using source weight configuration: [1m/ssd1/rickzhou/models/Phi-3-mini-128k-instruct/model.safetensors.index.json[0m. Use `--source` to override.
[2024-06-02 04:04:04] INFO auto_weight.py:110: Using source weight format: [1mhuggingface-safetensor[0m. Use `--source-format` to override.
[2024-06-02 04:04:04] INFO auto_config.py:153: [92mFound[0m model type: [1mphi3[0m. Use `--model-type` to override.
[2024-06-02 04:04:04] INFO phi3_model.py:53: [1mcontext_window_size[0m not found in config.json. Falling back to [1mmax_position_embeddings[0m (131072)
[2024-06-02 04:04:04] INFO phi3_model.py:68: [1mprefill_chunk_size[0m defaults to 2048
[1mWeight conversion with arguments:[0m
[1m--config[0m /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/config.json
[1m--quantization[0m GroupQuantize(name='q4f32_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float32', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
[1m--model-type[0m phi3
[1m--device[0m cuda:0
[1m--source[0m /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/model.safetensors.index.json
[1m--source-format[0m huggingface-safetensor
[1m--output[0m /ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
Start storing to cache /ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
0%| | 0/195 [00:00<?, ?it/s]
[2024-06-02 04:04:07] INFO huggingface_loader.py:184: Loading HF parameters from: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/model-00002-of-00002.safetensors
0%| | 0/195 [00:00<?, ?it/s]
[2024-06-02 04:04:09] INFO group_quantization.py:217: Compiling quantize function for key: ((32064, 3072), float32, cuda, axis=1, output_transpose=False)
0%| | 0/195 [00:01<?, ?it/s]
[2024-06-02 04:04:10] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mlm_head.q_weight[0m", shape: (32064, 384), dtype: uint32
0%| | 0/195 [00:02<?, ?it/s]
[2024-06-02 04:04:10] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mlm_head.q_scale[0m", shape: (32064, 96), dtype: float32
0%| | 0/195 [00:02<?, ?it/s]/home/rickzhou/miniconda3/envs/mlc/lib/python3.11/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/rickzhou/miniconda3/envs/mlc/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
1%| | 1/195 [00:02<09:32, 2.95s/it]
[2024-06-02 04:04:10] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.21.ln.weight[0m", shape: (3072,), dtype: float32
1%| | 1/195 [00:02<09:32, 2.95s/it]
[2024-06-02 04:04:10] INFO group_quantization.py:217: Compiling quantize function for key: ((3072, 8192), float32, cuda, axis=1, output_transpose=False)
1%| | 1/195 [00:02<09:32, 2.95s/it]
[2024-06-02 04:04:11] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.21.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
1%| | 1/195 [00:03<09:32, 2.95s/it]
[2024-06-02 04:04:11] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.21.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
1%| | 1/195 [00:03<09:32, 2.95s/it]
2%|▏ | 3/195 [00:03<02:50, 1.13it/s]
[2024-06-02 04:04:11] INFO group_quantization.py:217: Compiling quantize function for key: ((16384, 3072), float32, cuda, axis=1, output_transpose=False)
2%|▏ | 3/195 [00:03<02:50, 1.13it/s]
[2024-06-02 04:04:11] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.21.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
2%|▏ | 3/195 [00:03<02:50, 1.13it/s]
[2024-06-02 04:04:11] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.21.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
2%|▏ | 3/195 [00:03<02:50, 1.13it/s]
2%|▏ | 4/195 [00:03<02:21, 1.35it/s]
[2024-06-02 04:04:11] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.21.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
2%|▏ | 4/195 [00:03<02:21, 1.35it/s]
[2024-06-02 04:04:11] INFO group_quantization.py:217: Compiling quantize function for key: ((9216, 3072), float32, cuda, axis=1, output_transpose=False)
2%|▏ | 4/195 [00:03<02:21, 1.35it/s]
[2024-06-02 04:04:11] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.21.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
2%|▏ | 4/195 [00:04<02:21, 1.35it/s]
[2024-06-02 04:04:11] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.21.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
2%|▏ | 4/195 [00:04<02:21, 1.35it/s]
3%|▎ | 6/195 [00:04<01:30, 2.09it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.22.ln.weight[0m", shape: (3072,), dtype: float32
3%|▎ | 6/195 [00:04<01:30, 2.09it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.22.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
3%|▎ | 6/195 [00:04<01:30, 2.09it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.22.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
3%|▎ | 6/195 [00:04<01:30, 2.09it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.22.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
3%|▎ | 6/195 [00:04<01:30, 2.09it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.22.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
3%|▎ | 6/195 [00:04<01:30, 2.09it/s]
5%|▍ | 9/195 [00:04<00:49, 3.79it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.22.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
5%|▍ | 9/195 [00:04<00:49, 3.79it/s]
[2024-06-02 04:04:12] INFO group_quantization.py:217: Compiling quantize function for key: ((3072, 3072), float32, cuda, axis=1, output_transpose=False)
5%|▍ | 9/195 [00:04<00:49, 3.79it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.22.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
5%|▍ | 9/195 [00:04<00:49, 3.79it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.22.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
5%|▍ | 9/195 [00:04<00:49, 3.79it/s]
6%|▌ | 11/195 [00:04<00:45, 4.03it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.22.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
6%|▌ | 11/195 [00:04<00:45, 4.03it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.22.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
6%|▌ | 11/195 [00:04<00:45, 4.03it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.23.ln.weight[0m", shape: (3072,), dtype: float32
6%|▌ | 11/195 [00:04<00:45, 4.03it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.23.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
6%|▌ | 11/195 [00:04<00:45, 4.03it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.23.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
6%|▌ | 11/195 [00:04<00:45, 4.03it/s]
7%|▋ | 14/195 [00:04<00:29, 6.04it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.23.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
7%|▋ | 14/195 [00:04<00:29, 6.04it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.23.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
7%|▋ | 14/195 [00:05<00:29, 6.04it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.23.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
7%|▋ | 14/195 [00:05<00:29, 6.04it/s]
8%|▊ | 16/195 [00:05<00:25, 7.06it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.23.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
8%|▊ | 16/195 [00:05<00:25, 7.06it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.23.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
8%|▊ | 16/195 [00:05<00:25, 7.06it/s]
[2024-06-02 04:04:12] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.23.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
8%|▊ | 16/195 [00:05<00:25, 7.06it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.23.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
8%|▊ | 16/195 [00:05<00:25, 7.06it/s]
9%|▉ | 18/195 [00:05<00:21, 8.31it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.24.ln.weight[0m", shape: (3072,), dtype: float32
9%|▉ | 18/195 [00:05<00:21, 8.31it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.24.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
9%|▉ | 18/195 [00:05<00:21, 8.31it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.24.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
9%|▉ | 18/195 [00:05<00:21, 8.31it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.24.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
9%|▉ | 18/195 [00:05<00:21, 8.31it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.24.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
9%|▉ | 18/195 [00:05<00:21, 8.31it/s]
11%|█ | 21/195 [00:05<00:17, 10.19it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.24.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
11%|█ | 21/195 [00:05<00:17, 10.19it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.24.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
11%|█ | 21/195 [00:05<00:17, 10.19it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.24.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
11%|█ | 21/195 [00:05<00:17, 10.19it/s]
12%|█▏ | 23/195 [00:05<00:15, 10.94it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.24.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
12%|█▏ | 23/195 [00:05<00:15, 10.94it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.24.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
12%|█▏ | 23/195 [00:05<00:15, 10.94it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.25.ln.weight[0m", shape: (3072,), dtype: float32
12%|█▏ | 23/195 [00:05<00:15, 10.94it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.25.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
12%|█▏ | 23/195 [00:05<00:15, 10.94it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.25.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
12%|█▏ | 23/195 [00:05<00:15, 10.94it/s]
13%|█▎ | 26/195 [00:05<00:12, 13.22it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.25.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
13%|█▎ | 26/195 [00:05<00:12, 13.22it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.25.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
13%|█▎ | 26/195 [00:05<00:12, 13.22it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.25.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
13%|█▎ | 26/195 [00:05<00:12, 13.22it/s]
14%|█▍ | 28/195 [00:05<00:12, 13.16it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.25.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
14%|█▍ | 28/195 [00:05<00:12, 13.16it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.25.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
14%|█▍ | 28/195 [00:05<00:12, 13.16it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.25.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
14%|█▍ | 28/195 [00:05<00:12, 13.16it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.25.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
14%|█▍ | 28/195 [00:05<00:12, 13.16it/s]
15%|█▌ | 30/195 [00:05<00:12, 13.74it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.26.ln.weight[0m", shape: (3072,), dtype: float32
15%|█▌ | 30/195 [00:05<00:12, 13.74it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.26.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
15%|█▌ | 30/195 [00:06<00:12, 13.74it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.26.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
15%|█▌ | 30/195 [00:06<00:12, 13.74it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.26.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
15%|█▌ | 30/195 [00:06<00:12, 13.74it/s]
[2024-06-02 04:04:13] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.26.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
15%|█▌ | 30/195 [00:06<00:12, 13.74it/s]
17%|█▋ | 33/195 [00:06<00:10, 14.76it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.26.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
17%|█▋ | 33/195 [00:06<00:10, 14.76it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.26.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
17%|█▋ | 33/195 [00:06<00:10, 14.76it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.26.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
17%|█▋ | 33/195 [00:06<00:10, 14.76it/s]
18%|█▊ | 35/195 [00:06<00:11, 14.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.26.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
18%|█▊ | 35/195 [00:06<00:11, 14.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.26.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
18%|█▊ | 35/195 [00:06<00:11, 14.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.27.ln.weight[0m", shape: (3072,), dtype: float32
18%|█▊ | 35/195 [00:06<00:11, 14.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.27.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
18%|█▊ | 35/195 [00:06<00:11, 14.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.27.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
18%|█▊ | 35/195 [00:06<00:11, 14.38it/s]
19%|█▉ | 38/195 [00:06<00:09, 16.25it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.27.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
19%|█▉ | 38/195 [00:06<00:09, 16.25it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.27.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
19%|█▉ | 38/195 [00:06<00:09, 16.25it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.27.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
19%|█▉ | 38/195 [00:06<00:09, 16.25it/s]
21%|██ | 40/195 [00:06<00:10, 15.26it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.27.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
21%|██ | 40/195 [00:06<00:10, 15.26it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.27.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
21%|██ | 40/195 [00:06<00:10, 15.26it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.27.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
21%|██ | 40/195 [00:06<00:10, 15.26it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.27.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
21%|██ | 40/195 [00:06<00:10, 15.26it/s]
22%|██▏ | 42/195 [00:06<00:09, 15.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.28.ln.weight[0m", shape: (3072,), dtype: float32
22%|██▏ | 42/195 [00:06<00:09, 15.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.28.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
22%|██▏ | 42/195 [00:06<00:09, 15.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.28.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
22%|██▏ | 42/195 [00:06<00:09, 15.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.28.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
22%|██▏ | 42/195 [00:06<00:09, 15.38it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.28.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
22%|██▏ | 42/195 [00:06<00:09, 15.38it/s]
23%|██▎ | 45/195 [00:06<00:09, 15.87it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.28.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
23%|██▎ | 45/195 [00:06<00:09, 15.87it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.28.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
23%|██▎ | 45/195 [00:06<00:09, 15.87it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.28.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
23%|██▎ | 45/195 [00:07<00:09, 15.87it/s]
24%|██▍ | 47/195 [00:07<00:09, 15.18it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.28.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
24%|██▍ | 47/195 [00:07<00:09, 15.18it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.28.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
24%|██▍ | 47/195 [00:07<00:09, 15.18it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.29.ln.weight[0m", shape: (3072,), dtype: float32
24%|██▍ | 47/195 [00:07<00:09, 15.18it/s]
[2024-06-02 04:04:14] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.29.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
24%|██▍ | 47/195 [00:07<00:09, 15.18it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.29.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
24%|██▍ | 47/195 [00:07<00:09, 15.18it/s]
26%|██▌ | 50/195 [00:07<00:08, 16.86it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.29.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
26%|██▌ | 50/195 [00:07<00:08, 16.86it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.29.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
26%|██▌ | 50/195 [00:07<00:08, 16.86it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.29.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
26%|██▌ | 50/195 [00:07<00:08, 16.86it/s]
27%|██▋ | 52/195 [00:07<00:09, 15.70it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.29.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
27%|██▋ | 52/195 [00:07<00:09, 15.70it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.29.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
27%|██▋ | 52/195 [00:07<00:09, 15.70it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.29.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
27%|██▋ | 52/195 [00:07<00:09, 15.70it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.29.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
27%|██▋ | 52/195 [00:07<00:09, 15.70it/s]
28%|██▊ | 54/195 [00:07<00:08, 15.74it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.30.ln.weight[0m", shape: (3072,), dtype: float32
28%|██▊ | 54/195 [00:07<00:08, 15.74it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.30.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
28%|██▊ | 54/195 [00:07<00:08, 15.74it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.30.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
28%|██▊ | 54/195 [00:07<00:08, 15.74it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.30.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
28%|██▊ | 54/195 [00:07<00:08, 15.74it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.30.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
28%|██▊ | 54/195 [00:07<00:08, 15.74it/s]
29%|██▉ | 57/195 [00:07<00:08, 16.13it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.30.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
29%|██▉ | 57/195 [00:07<00:08, 16.13it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.30.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
29%|██▉ | 57/195 [00:07<00:08, 16.13it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.30.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
29%|██▉ | 57/195 [00:07<00:08, 16.13it/s]
30%|███ | 59/195 [00:07<00:08, 15.46it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.30.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
30%|███ | 59/195 [00:07<00:08, 15.46it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.30.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
30%|███ | 59/195 [00:07<00:08, 15.46it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.31.ln.weight[0m", shape: (3072,), dtype: float32
30%|███ | 59/195 [00:07<00:08, 15.46it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.31.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
30%|███ | 59/195 [00:07<00:08, 15.46it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.31.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
30%|███ | 59/195 [00:07<00:08, 15.46it/s]
32%|███▏ | 62/195 [00:07<00:07, 17.07it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.31.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
32%|███▏ | 62/195 [00:07<00:07, 17.07it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.31.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
32%|███▏ | 62/195 [00:08<00:07, 17.07it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.31.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
32%|███▏ | 62/195 [00:08<00:07, 17.07it/s]
33%|███▎ | 64/195 [00:08<00:08, 15.74it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.31.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
33%|███▎ | 64/195 [00:08<00:08, 15.74it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.31.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
33%|███▎ | 64/195 [00:08<00:08, 15.74it/s]
[2024-06-02 04:04:15] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.31.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
33%|███▎ | 64/195 [00:08<00:08, 15.74it/s]
[2024-06-02 04:04:16] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.31.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
33%|███▎ | 64/195 [00:08<00:08, 15.74it/s]
34%|███▍ | 66/195 [00:08<00:08, 15.69it/s]
[2024-06-02 04:04:16] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.norm.weight[0m", shape: (3072,), dtype: float32
34%|███▍ | 66/195 [00:08<00:08, 15.69it/s]
[2024-06-02 04:04:16] INFO huggingface_loader.py:196: Unloading HF weight file: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/model-00002-of-00002.safetensors
34%|███▍ | 66/195 [00:08<00:08, 15.69it/s]
[2024-06-02 04:04:16] INFO huggingface_loader.py:184: Loading HF parameters from: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/model-00001-of-00002.safetensors
34%|███▍ | 66/195 [00:08<00:08, 15.69it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.embd.q_weight[0m", shape: (32064, 384), dtype: uint32
34%|███▍ | 66/195 [00:09<00:08, 15.69it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.embd.q_scale[0m", shape: (32064, 96), dtype: float32
34%|███▍ | 66/195 [00:09<00:08, 15.69it/s]
35%|███▍ | 68/195 [00:09<00:26, 4.71it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.0.ln.weight[0m", shape: (3072,), dtype: float32
35%|███▍ | 68/195 [00:09<00:26, 4.71it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.0.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
35%|███▍ | 68/195 [00:09<00:26, 4.71it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.0.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
35%|███▍ | 68/195 [00:09<00:26, 4.71it/s]
36%|███▌ | 70/195 [00:09<00:21, 5.87it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.0.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
36%|███▌ | 70/195 [00:09<00:21, 5.87it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.0.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
36%|███▌ | 70/195 [00:09<00:21, 5.87it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.0.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
36%|███▌ | 70/195 [00:09<00:21, 5.87it/s]
37%|███▋ | 72/195 [00:09<00:17, 6.96it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.0.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
37%|███▋ | 72/195 [00:09<00:17, 6.96it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.0.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
37%|███▋ | 72/195 [00:09<00:17, 6.96it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.0.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
37%|███▋ | 72/195 [00:09<00:17, 6.96it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.0.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
37%|███▋ | 72/195 [00:09<00:17, 6.96it/s]
38%|███▊ | 74/195 [00:09<00:14, 8.21it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.1.ln.weight[0m", shape: (3072,), dtype: float32
38%|███▊ | 74/195 [00:09<00:14, 8.21it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.1.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
38%|███▊ | 74/195 [00:09<00:14, 8.21it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.1.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
38%|███▊ | 74/195 [00:09<00:14, 8.21it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.1.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
38%|███▊ | 74/195 [00:09<00:14, 8.21it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.1.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
38%|███▊ | 74/195 [00:10<00:14, 8.21it/s]
39%|███▉ | 77/195 [00:10<00:11, 10.20it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.1.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
39%|███▉ | 77/195 [00:10<00:11, 10.20it/s]
[2024-06-02 04:04:17] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.1.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
39%|███▉ | 77/195 [00:10<00:11, 10.20it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.1.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
39%|███▉ | 77/195 [00:10<00:11, 10.20it/s]
41%|████ | 79/195 [00:10<00:10, 11.01it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.1.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
41%|████ | 79/195 [00:10<00:10, 11.01it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.1.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
41%|████ | 79/195 [00:10<00:10, 11.01it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.10.ln.weight[0m", shape: (3072,), dtype: float32
41%|████ | 79/195 [00:10<00:10, 11.01it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.10.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
41%|████ | 79/195 [00:10<00:10, 11.01it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.10.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
41%|████ | 79/195 [00:10<00:10, 11.01it/s]
42%|████▏ | 82/195 [00:10<00:08, 13.24it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.10.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
42%|████▏ | 82/195 [00:10<00:08, 13.24it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.10.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
42%|████▏ | 82/195 [00:10<00:08, 13.24it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.10.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
42%|████▏ | 82/195 [00:10<00:08, 13.24it/s]
43%|████▎ | 84/195 [00:10<00:08, 13.21it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.10.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
43%|████▎ | 84/195 [00:10<00:08, 13.21it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.10.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
43%|████▎ | 84/195 [00:10<00:08, 13.21it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.10.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
43%|████▎ | 84/195 [00:10<00:08, 13.21it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.10.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
43%|████▎ | 84/195 [00:10<00:08, 13.21it/s]
44%|████▍ | 86/195 [00:10<00:07, 13.76it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.11.ln.weight[0m", shape: (3072,), dtype: float32
44%|████▍ | 86/195 [00:10<00:07, 13.76it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.11.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
44%|████▍ | 86/195 [00:10<00:07, 13.76it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.11.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
44%|████▍ | 86/195 [00:10<00:07, 13.76it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.11.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
44%|████▍ | 86/195 [00:10<00:07, 13.76it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.11.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
44%|████▍ | 86/195 [00:10<00:07, 13.76it/s]
46%|████▌ | 89/195 [00:10<00:07, 14.74it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.11.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
46%|████▌ | 89/195 [00:10<00:07, 14.74it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.11.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
46%|████▌ | 89/195 [00:10<00:07, 14.74it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.11.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
46%|████▌ | 89/195 [00:10<00:07, 14.74it/s]
47%|████▋ | 91/195 [00:10<00:07, 14.37it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.11.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
47%|████▋ | 91/195 [00:10<00:07, 14.37it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.11.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
47%|████▋ | 91/195 [00:10<00:07, 14.37it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.12.ln.weight[0m", shape: (3072,), dtype: float32
47%|████▋ | 91/195 [00:10<00:07, 14.37it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.12.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
47%|████▋ | 91/195 [00:10<00:07, 14.37it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.12.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
47%|████▋ | 91/195 [00:11<00:07, 14.37it/s]
48%|████▊ | 94/195 [00:11<00:06, 16.05it/s]
[2024-06-02 04:04:18] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.12.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
48%|████▊ | 94/195 [00:11<00:06, 16.05it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.12.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
48%|████▊ | 94/195 [00:11<00:06, 16.05it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.12.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
48%|████▊ | 94/195 [00:11<00:06, 16.05it/s]
49%|████▉ | 96/195 [00:11<00:06, 15.19it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.12.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
49%|████▉ | 96/195 [00:11<00:06, 15.19it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.12.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
49%|████▉ | 96/195 [00:11<00:06, 15.19it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.12.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
49%|████▉ | 96/195 [00:11<00:06, 15.19it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.12.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
49%|████▉ | 96/195 [00:11<00:06, 15.19it/s]
50%|█████ | 98/195 [00:11<00:06, 15.20it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.13.ln.weight[0m", shape: (3072,), dtype: float32
50%|█████ | 98/195 [00:11<00:06, 15.20it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.13.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
50%|█████ | 98/195 [00:11<00:06, 15.20it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.13.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
50%|█████ | 98/195 [00:11<00:06, 15.20it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.13.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
50%|█████ | 98/195 [00:11<00:06, 15.20it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.13.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
50%|█████ | 98/195 [00:11<00:06, 15.20it/s]
52%|█████▏ | 101/195 [00:11<00:05, 15.70it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.13.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
52%|█████▏ | 101/195 [00:11<00:05, 15.70it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.13.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
52%|█████▏ | 101/195 [00:11<00:05, 15.70it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.13.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
52%|█████▏ | 101/195 [00:11<00:05, 15.70it/s]
53%|█████▎ | 103/195 [00:11<00:06, 15.03it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.13.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
53%|█████▎ | 103/195 [00:11<00:06, 15.03it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.13.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
53%|█████▎ | 103/195 [00:11<00:06, 15.03it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.14.ln.weight[0m", shape: (3072,), dtype: float32
53%|█████▎ | 103/195 [00:11<00:06, 15.03it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.14.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
53%|█████▎ | 103/195 [00:11<00:06, 15.03it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.14.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
53%|█████▎ | 103/195 [00:11<00:06, 15.03it/s]
54%|█████▍ | 106/195 [00:11<00:05, 16.62it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.14.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
54%|█████▍ | 106/195 [00:11<00:05, 16.62it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.14.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
54%|█████▍ | 106/195 [00:11<00:05, 16.62it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.14.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
54%|█████▍ | 106/195 [00:12<00:05, 16.62it/s]
55%|█████▌ | 108/195 [00:12<00:05, 15.57it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.14.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
55%|█████▌ | 108/195 [00:12<00:05, 15.57it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.14.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
55%|█████▌ | 108/195 [00:12<00:05, 15.57it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.14.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
55%|█████▌ | 108/195 [00:12<00:05, 15.57it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.14.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
55%|█████▌ | 108/195 [00:12<00:05, 15.57it/s]
56%|█████▋ | 110/195 [00:12<00:05, 15.52it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.15.ln.weight[0m", shape: (3072,), dtype: float32
56%|█████▋ | 110/195 [00:12<00:05, 15.52it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.15.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
56%|█████▋ | 110/195 [00:12<00:05, 15.52it/s]
[2024-06-02 04:04:19] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.15.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
56%|█████▋ | 110/195 [00:12<00:05, 15.52it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.15.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
56%|█████▋ | 110/195 [00:12<00:05, 15.52it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.15.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
56%|█████▋ | 110/195 [00:12<00:05, 15.52it/s]
58%|█████▊ | 113/195 [00:12<00:05, 15.95it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.15.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
58%|█████▊ | 113/195 [00:12<00:05, 15.95it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.15.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
58%|█████▊ | 113/195 [00:12<00:05, 15.95it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.15.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
58%|█████▊ | 113/195 [00:12<00:05, 15.95it/s]
59%|█████▉ | 115/195 [00:12<00:05, 15.22it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.15.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
59%|█████▉ | 115/195 [00:12<00:05, 15.22it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.15.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
59%|█████▉ | 115/195 [00:12<00:05, 15.22it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.16.ln.weight[0m", shape: (3072,), dtype: float32
59%|█████▉ | 115/195 [00:12<00:05, 15.22it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.16.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
59%|█████▉ | 115/195 [00:12<00:05, 15.22it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.16.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
59%|█████▉ | 115/195 [00:12<00:05, 15.22it/s]
61%|██████ | 118/195 [00:12<00:04, 16.78it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.16.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
61%|██████ | 118/195 [00:12<00:04, 16.78it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.16.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
61%|██████ | 118/195 [00:12<00:04, 16.78it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.16.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
61%|██████ | 118/195 [00:12<00:04, 16.78it/s]
62%|██████▏ | 120/195 [00:12<00:04, 15.44it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.16.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
62%|██████▏ | 120/195 [00:12<00:04, 15.44it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.16.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
62%|██████▏ | 120/195 [00:12<00:04, 15.44it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.16.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
62%|██████▏ | 120/195 [00:12<00:04, 15.44it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.16.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
62%|██████▏ | 120/195 [00:12<00:04, 15.44it/s]
63%|██████▎ | 122/195 [00:12<00:04, 15.41it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.17.ln.weight[0m", shape: (3072,), dtype: float32
63%|██████▎ | 122/195 [00:12<00:04, 15.41it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.17.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
63%|██████▎ | 122/195 [00:12<00:04, 15.41it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.17.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
63%|██████▎ | 122/195 [00:12<00:04, 15.41it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.17.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
63%|██████▎ | 122/195 [00:12<00:04, 15.41it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.17.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
63%|██████▎ | 122/195 [00:13<00:04, 15.41it/s]
64%|██████▍ | 125/195 [00:13<00:04, 15.86it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.17.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
64%|██████▍ | 125/195 [00:13<00:04, 15.86it/s]
[2024-06-02 04:04:20] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.17.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
64%|██████▍ | 125/195 [00:13<00:04, 15.86it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.17.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
64%|██████▍ | 125/195 [00:13<00:04, 15.86it/s]
65%|██████▌ | 127/195 [00:13<00:04, 15.06it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.17.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
65%|██████▌ | 127/195 [00:13<00:04, 15.06it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.17.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
65%|██████▌ | 127/195 [00:13<00:04, 15.06it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.18.ln.weight[0m", shape: (3072,), dtype: float32
65%|██████▌ | 127/195 [00:13<00:04, 15.06it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.18.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
65%|██████▌ | 127/195 [00:13<00:04, 15.06it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.18.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
65%|██████▌ | 127/195 [00:13<00:04, 15.06it/s]
67%|██████▋ | 130/195 [00:13<00:03, 16.51it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.18.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
67%|██████▋ | 130/195 [00:13<00:03, 16.51it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.18.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
67%|██████▋ | 130/195 [00:13<00:03, 16.51it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.18.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
67%|██████▋ | 130/195 [00:13<00:03, 16.51it/s]
68%|██████▊ | 132/195 [00:13<00:04, 15.40it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.18.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
68%|██████▊ | 132/195 [00:13<00:04, 15.40it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.18.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
68%|██████▊ | 132/195 [00:13<00:04, 15.40it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.18.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
68%|██████▊ | 132/195 [00:13<00:04, 15.40it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.18.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
68%|██████▊ | 132/195 [00:13<00:04, 15.40it/s]
69%|██████▊ | 134/195 [00:13<00:03, 15.38it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.19.ln.weight[0m", shape: (3072,), dtype: float32
69%|██████▊ | 134/195 [00:13<00:03, 15.38it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.19.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
69%|██████▊ | 134/195 [00:13<00:03, 15.38it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.19.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
69%|██████▊ | 134/195 [00:13<00:03, 15.38it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.19.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
69%|██████▊ | 134/195 [00:13<00:03, 15.38it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.19.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
69%|██████▊ | 134/195 [00:13<00:03, 15.38it/s]
70%|███████ | 137/195 [00:13<00:03, 15.88it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.19.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
70%|███████ | 137/195 [00:13<00:03, 15.88it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.19.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
70%|███████ | 137/195 [00:13<00:03, 15.88it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.19.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
70%|███████ | 137/195 [00:13<00:03, 15.88it/s]
71%|███████▏ | 139/195 [00:13<00:03, 15.14it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.19.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
71%|███████▏ | 139/195 [00:14<00:03, 15.14it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.19.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
71%|███████▏ | 139/195 [00:14<00:03, 15.14it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.2.ln.weight[0m", shape: (3072,), dtype: float32
71%|███████▏ | 139/195 [00:14<00:03, 15.14it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.2.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
71%|███████▏ | 139/195 [00:14<00:03, 15.14it/s]
[2024-06-02 04:04:21] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.2.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
71%|███████▏ | 139/195 [00:14<00:03, 15.14it/s]
73%|███████▎ | 142/195 [00:14<00:03, 16.71it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.2.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
73%|███████▎ | 142/195 [00:14<00:03, 16.71it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.2.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
73%|███████▎ | 142/195 [00:14<00:03, 16.71it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.2.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
73%|███████▎ | 142/195 [00:14<00:03, 16.71it/s]
74%|███████▍ | 144/195 [00:14<00:03, 15.59it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.2.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
74%|███████▍ | 144/195 [00:14<00:03, 15.59it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.2.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
74%|███████▍ | 144/195 [00:14<00:03, 15.59it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.2.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
74%|███████▍ | 144/195 [00:14<00:03, 15.59it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.2.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
74%|███████▍ | 144/195 [00:14<00:03, 15.59it/s]
75%|███████▍ | 146/195 [00:14<00:03, 15.45it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.20.ln.weight[0m", shape: (3072,), dtype: float32
75%|███████▍ | 146/195 [00:14<00:03, 15.45it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.20.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
75%|███████▍ | 146/195 [00:14<00:03, 15.45it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.20.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
75%|███████▍ | 146/195 [00:14<00:03, 15.45it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.20.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
75%|███████▍ | 146/195 [00:14<00:03, 15.45it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.20.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
75%|███████▍ | 146/195 [00:14<00:03, 15.45it/s]
76%|███████▋ | 149/195 [00:14<00:02, 15.91it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.20.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
76%|███████▋ | 149/195 [00:14<00:02, 15.91it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.20.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
76%|███████▋ | 149/195 [00:14<00:02, 15.91it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.20.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
76%|███████▋ | 149/195 [00:14<00:02, 15.91it/s]
77%|███████▋ | 151/195 [00:14<00:02, 15.15it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.20.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
77%|███████▋ | 151/195 [00:14<00:02, 15.15it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.20.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
77%|███████▋ | 151/195 [00:14<00:02, 15.15it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.21.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
77%|███████▋ | 151/195 [00:14<00:02, 15.15it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.21.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
77%|███████▋ | 151/195 [00:14<00:02, 15.15it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.3.ln.weight[0m", shape: (3072,), dtype: float32
77%|███████▋ | 151/195 [00:14<00:02, 15.15it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.3.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
77%|███████▋ | 151/195 [00:14<00:02, 15.15it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.3.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
77%|███████▋ | 151/195 [00:14<00:02, 15.15it/s]
79%|███████▉ | 155/195 [00:14<00:02, 17.25it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.3.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
79%|███████▉ | 155/195 [00:14<00:02, 17.25it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.3.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
79%|███████▉ | 155/195 [00:15<00:02, 17.25it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.3.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
79%|███████▉ | 155/195 [00:15<00:02, 17.25it/s]
81%|████████ | 157/195 [00:15<00:02, 16.07it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.3.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
81%|████████ | 157/195 [00:15<00:02, 16.07it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.3.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
81%|████████ | 157/195 [00:15<00:02, 16.07it/s]
[2024-06-02 04:04:22] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.3.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
81%|████████ | 157/195 [00:15<00:02, 16.07it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.3.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
81%|████████ | 157/195 [00:15<00:02, 16.07it/s]
82%|████████▏ | 159/195 [00:15<00:02, 15.90it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.4.ln.weight[0m", shape: (3072,), dtype: float32
82%|████████▏ | 159/195 [00:15<00:02, 15.90it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.4.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
82%|████████▏ | 159/195 [00:15<00:02, 15.90it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.4.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
82%|████████▏ | 159/195 [00:15<00:02, 15.90it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.4.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
82%|████████▏ | 159/195 [00:15<00:02, 15.90it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.4.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
82%|████████▏ | 159/195 [00:15<00:02, 15.90it/s]
83%|████████▎ | 162/195 [00:15<00:02, 15.89it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.4.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
83%|████████▎ | 162/195 [00:15<00:02, 15.89it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.4.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
83%|████████▎ | 162/195 [00:15<00:02, 15.89it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.4.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
83%|████████▎ | 162/195 [00:15<00:02, 15.89it/s]
84%|████████▍ | 164/195 [00:15<00:02, 15.16it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.4.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
84%|████████▍ | 164/195 [00:15<00:02, 15.16it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.4.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
84%|████████▍ | 164/195 [00:15<00:02, 15.16it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.5.ln.weight[0m", shape: (3072,), dtype: float32
84%|████████▍ | 164/195 [00:15<00:02, 15.16it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.5.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
84%|████████▍ | 164/195 [00:15<00:02, 15.16it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.5.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
84%|████████▍ | 164/195 [00:15<00:02, 15.16it/s]
86%|████████▌ | 167/195 [00:15<00:01, 16.65it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.5.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
86%|████████▌ | 167/195 [00:15<00:01, 16.65it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.5.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
86%|████████▌ | 167/195 [00:15<00:01, 16.65it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.5.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
86%|████████▌ | 167/195 [00:15<00:01, 16.65it/s]
87%|████████▋ | 169/195 [00:15<00:01, 15.58it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.5.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
87%|████████▋ | 169/195 [00:15<00:01, 15.58it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.5.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
87%|████████▋ | 169/195 [00:15<00:01, 15.58it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.5.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
87%|████████▋ | 169/195 [00:15<00:01, 15.58it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.5.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
87%|████████▋ | 169/195 [00:15<00:01, 15.58it/s]
88%|████████▊ | 171/195 [00:15<00:01, 15.46it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.6.ln.weight[0m", shape: (3072,), dtype: float32
88%|████████▊ | 171/195 [00:15<00:01, 15.46it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.6.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
88%|████████▊ | 171/195 [00:16<00:01, 15.46it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.6.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
88%|████████▊ | 171/195 [00:16<00:01, 15.46it/s]
[2024-06-02 04:04:23] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.6.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
88%|████████▊ | 171/195 [00:16<00:01, 15.46it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.6.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
88%|████████▊ | 171/195 [00:16<00:01, 15.46it/s]
89%|████████▉ | 174/195 [00:16<00:01, 15.88it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.6.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
89%|████████▉ | 174/195 [00:16<00:01, 15.88it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.6.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
89%|████████▉ | 174/195 [00:16<00:01, 15.88it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.6.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
89%|████████▉ | 174/195 [00:16<00:01, 15.88it/s]
90%|█████████ | 176/195 [00:16<00:01, 15.16it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.6.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
90%|█████████ | 176/195 [00:16<00:01, 15.16it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.6.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
90%|█████████ | 176/195 [00:16<00:01, 15.16it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.7.ln.weight[0m", shape: (3072,), dtype: float32
90%|█████████ | 176/195 [00:16<00:01, 15.16it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.7.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
90%|█████████ | 176/195 [00:16<00:01, 15.16it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.7.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
90%|█████████ | 176/195 [00:16<00:01, 15.16it/s]
92%|█████████▏| 179/195 [00:16<00:00, 16.70it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.7.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
92%|█████████▏| 179/195 [00:16<00:00, 16.70it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.7.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
92%|█████████▏| 179/195 [00:16<00:00, 16.70it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.7.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
92%|█████████▏| 179/195 [00:16<00:00, 16.70it/s]
93%|█████████▎| 181/195 [00:16<00:00, 15.63it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.7.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
93%|█████████▎| 181/195 [00:16<00:00, 15.63it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.7.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
93%|█████████▎| 181/195 [00:16<00:00, 15.63it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.7.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
93%|█████████▎| 181/195 [00:16<00:00, 15.63it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.7.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
93%|█████████▎| 181/195 [00:16<00:00, 15.63it/s]
94%|█████████▍| 183/195 [00:16<00:00, 15.48it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.8.ln.weight[0m", shape: (3072,), dtype: float32
94%|█████████▍| 183/195 [00:16<00:00, 15.48it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.8.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
94%|█████████▍| 183/195 [00:16<00:00, 15.48it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.8.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
94%|█████████▍| 183/195 [00:16<00:00, 15.48it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.8.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
94%|█████████▍| 183/195 [00:16<00:00, 15.48it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.8.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
94%|█████████▍| 183/195 [00:16<00:00, 15.48it/s]
95%|█████████▌| 186/195 [00:16<00:00, 15.91it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.8.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
95%|█████████▌| 186/195 [00:16<00:00, 15.91it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.8.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
95%|█████████▌| 186/195 [00:16<00:00, 15.91it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.8.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
95%|█████████▌| 186/195 [00:17<00:00, 15.91it/s]
96%|█████████▋| 188/195 [00:17<00:00, 15.15it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.8.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
96%|█████████▋| 188/195 [00:17<00:00, 15.15it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.8.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
96%|█████████▋| 188/195 [00:17<00:00, 15.15it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.9.ln.weight[0m", shape: (3072,), dtype: float32
96%|█████████▋| 188/195 [00:17<00:00, 15.15it/s]
[2024-06-02 04:04:24] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.9.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
96%|█████████▋| 188/195 [00:17<00:00, 15.15it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.9.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
96%|█████████▋| 188/195 [00:17<00:00, 15.15it/s]
98%|█████████▊| 191/195 [00:17<00:00, 16.72it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.9.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
98%|█████████▊| 191/195 [00:17<00:00, 16.72it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.9.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
98%|█████████▊| 191/195 [00:17<00:00, 16.72it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:174: [Not quantized] Parameter: "[1mtransformer.h.9.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
98%|█████████▊| 191/195 [00:17<00:00, 16.72it/s]
99%|█████████▉| 193/195 [00:17<00:00, 15.62it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.9.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
99%|█████████▉| 193/195 [00:17<00:00, 15.62it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.9.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
99%|█████████▉| 193/195 [00:17<00:00, 15.62it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.9.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
99%|█████████▉| 193/195 [00:17<00:00, 15.62it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:166: [Quantized] Parameter: "[1mtransformer.h.9.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
99%|█████████▉| 193/195 [00:17<00:00, 15.62it/s]
100%|██████████| 195/195 [00:17<00:00, 15.48it/s]
100%|██████████| 195/195 [00:17<00:00, 11.13it/s]
[2024-06-02 04:04:25] INFO huggingface_loader.py:196: Unloading HF weight file: /ssd1/rickzhou/models/Phi-3-mini-128k-instruct/model-00001-of-00002.safetensors
[2024-06-02 04:04:25] INFO stats.py:76: [92mTime usage[0m: HF loading: 2.646 sec; Pre-quantization mapping: 1.733 sec; Quantization: 2.463 sec
[2024-06-02 04:04:25] INFO stats.py:90: [92mRAM usage[0m: Peak RAM: 9.262 GB. Total bytes loaded from disk: 14.235 GB
[2024-06-02 04:04:25] INFO convert_weight.py:155: [92mParameter size[0m after quantization: 2.225 GB
[2024-06-02 04:04:25] INFO convert_weight.py:160: [92mTotal parameters[0m: 3,821,079,552
[2024-06-02 04:04:25] INFO convert_weight.py:161: [92mBits per parameter[0m: 5.001
[2024-06-02 04:04:25] INFO convert_weight.py:166: Saved to directory: [1m/ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC[0m
All finished, 83 total shards committed, record saved to /ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/ndarray-cache.json
Also saved a bf16 record to /ssd2/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/ndarray-cache-b16.json
/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q4f32_1 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
[2024-06-04 02:47:01] INFO auto_config.py:116: [92mFound[0m model configuration: /models/Phi-3-mini-128k-instruct/config.json
[2024-06-04 02:47:01] INFO auto_config.py:154: [92mFound[0m model type: [1mphi3[0m. Use `--model-type` to override.
[2024-06-04 02:47:01] INFO phi3_model.py:53: [1mcontext_window_size[0m not found in config.json. Falling back to [1mmax_position_embeddings[0m (131072)
[2024-06-04 02:47:01] INFO phi3_model.py:68: [1mprefill_chunk_size[0m defaults to 2048
[2024-06-04 02:47:01] INFO config.py:107: Overriding [1mmax_batch_size[0m from 1 to 80
[2024-06-04 02:47:01] INFO gen_config.py:143: [generation_config.json] Setting [1mbos_token_id[0m: 1
[2024-06-04 02:47:01] INFO gen_config.py:143: [generation_config.json] Setting [1meos_token_id[0m: [32000, 32001, 32007]
[2024-06-04 02:47:01] INFO gen_config.py:143: [generation_config.json] Setting [1mpad_token_id[0m: 32000
[2024-06-04 02:47:01] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer.model[0m
[2024-06-04 02:47:01] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer.json[0m
[2024-06-04 02:47:01] INFO gen_config.py:157: [91mNot found[0m tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json
[2024-06-04 02:47:01] INFO gen_config.py:157: [91mNot found[0m tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt
[2024-06-04 02:47:01] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/added_tokens.json[0m
[2024-06-04 02:47:01] INFO gen_config.py:155: [92mFound[0m tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/tokenizer_config.json[0m
[2024-06-04 02:47:01] INFO gen_config.py:216: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True}
[2024-06-04 02:47:01] INFO gen_config.py:32: [System default] Setting [1mtemperature[0m: 1.0
[2024-06-04 02:47:01] INFO gen_config.py:32: [System default] Setting [1mpresence_penalty[0m: 0.0
[2024-06-04 02:47:01] INFO gen_config.py:32: [System default] Setting [1mfrequency_penalty[0m: 0.0
[2024-06-04 02:47:01] INFO gen_config.py:32: [System default] Setting [1mrepetition_penalty[0m: 1.0
[2024-06-04 02:47:01] INFO gen_config.py:32: [System default] Setting [1mtop_p[0m: 1.0
[2024-06-04 02:47:01] INFO gen_config.py:32: [System default] Setting [1mmean_gen_len[0m: 128
[2024-06-04 02:47:01] INFO gen_config.py:32: [System default] Setting [1mmax_gen_len[0m: 512
[2024-06-04 02:47:01] INFO gen_config.py:32: [System default] Setting [1mshift_fill_factor[0m: 0.3
[2024-06-04 02:47:01] INFO gen_config.py:223: Dumping configuration file to: [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/mlc-chat-config.json[0m
/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q4f32_1 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
[2024-06-04 02:47:02] INFO auto_config.py:116: [92mFound[0m model configuration: /models/Phi-3-mini-128k-instruct/config.json
[2024-06-04 02:47:04] INFO auto_device.py:79: [92mFound[0m device: cuda:0
[2024-06-04 02:47:05] INFO auto_device.py:88: [91mNot found[0m device: rocm:0
[2024-06-04 02:47:07] INFO auto_device.py:88: [91mNot found[0m device: metal:0
[2024-06-04 02:47:09] INFO auto_device.py:79: [92mFound[0m device: vulkan:0
[2024-06-04 02:47:09] INFO auto_device.py:79: [92mFound[0m device: vulkan:1
[2024-06-04 02:47:09] INFO auto_device.py:79: [92mFound[0m device: vulkan:2
[2024-06-04 02:47:09] INFO auto_device.py:79: [92mFound[0m device: vulkan:3
[2024-06-04 02:47:10] INFO auto_device.py:88: [91mNot found[0m device: opencl:0
[2024-06-04 02:47:10] INFO auto_device.py:35: Using device: [1mcuda:0[0m
[2024-06-04 02:47:10] INFO auto_weight.py:71: Finding weights in: /models/Phi-3-mini-128k-instruct
[2024-06-04 02:47:10] INFO auto_weight.py:137: [91mNot found[0m Huggingface PyTorch
[2024-06-04 02:47:10] INFO auto_weight.py:144: [92mFound[0m source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json
[2024-06-04 02:47:10] INFO auto_weight.py:107: Using source weight configuration: [1m/models/Phi-3-mini-128k-instruct/model.safetensors.index.json[0m. Use `--source` to override.
[2024-06-04 02:47:10] INFO auto_weight.py:111: Using source weight format: [1mhuggingface-safetensor[0m. Use `--source-format` to override.
[2024-06-04 02:47:10] INFO auto_config.py:154: [92mFound[0m model type: [1mphi3[0m. Use `--model-type` to override.
[2024-06-04 02:47:10] INFO phi3_model.py:53: [1mcontext_window_size[0m not found in config.json. Falling back to [1mmax_position_embeddings[0m (131072)
[2024-06-04 02:47:10] INFO phi3_model.py:68: [1mprefill_chunk_size[0m defaults to 2048
[1mWeight conversion with arguments:[0m
[1m--config[0m /models/Phi-3-mini-128k-instruct/config.json
[1m--quantization[0m GroupQuantize(name='q4f32_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float32', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
[1m--model-type[0m phi3
[1m--device[0m cuda:0
[1m--source[0m /models/Phi-3-mini-128k-instruct/model.safetensors.index.json
[1m--source-format[0m huggingface-safetensor
[1m--output[0m /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
Start storing to cache /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC
0%| | 0/195 [00:00<?, ?it/s]
[2024-06-04 02:47:12] INFO huggingface_loader.py:185: Loading HF parameters from: /models/Phi-3-mini-128k-instruct/model-00002-of-00002.safetensors
0%| | 0/195 [00:00<?, ?it/s]
[2024-06-04 02:47:16] INFO group_quantization.py:217: Compiling quantize function for key: ((32064, 3072), float32, cuda, axis=1, output_transpose=False)
0%| | 0/195 [00:04<?, ?it/s]
[2024-06-04 02:47:17] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mlm_head.q_weight[0m", shape: (32064, 384), dtype: uint32
0%| | 0/195 [00:04<?, ?it/s]
[2024-06-04 02:47:17] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mlm_head.q_scale[0m", shape: (32064, 96), dtype: float32
0%| | 0/195 [00:05<?, ?it/s]
1%| | 1/195 [00:05<16:47, 5.19s/it]
[2024-06-04 02:47:17] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.21.ln.weight[0m", shape: (3072,), dtype: float32
1%| | 1/195 [00:05<16:47, 5.19s/it]
[2024-06-04 02:47:17] INFO group_quantization.py:217: Compiling quantize function for key: ((3072, 8192), float32, cuda, axis=1, output_transpose=False)
1%| | 1/195 [00:05<16:47, 5.19s/it]
[2024-06-04 02:47:18] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
1%| | 1/195 [00:05<16:47, 5.19s/it]
[2024-06-04 02:47:18] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
1%| | 1/195 [00:05<16:47, 5.19s/it]
2%|▏ | 3/195 [00:05<04:51, 1.52s/it]
[2024-06-04 02:47:18] INFO group_quantization.py:217: Compiling quantize function for key: ((16384, 3072), float32, cuda, axis=1, output_transpose=False)
2%|▏ | 3/195 [00:05<04:51, 1.52s/it]
[2024-06-04 02:47:18] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
2%|▏ | 3/195 [00:06<04:51, 1.52s/it]
[2024-06-04 02:47:18] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
2%|▏ | 3/195 [00:06<04:51, 1.52s/it]
2%|▏ | 4/195 [00:06<03:48, 1.19s/it]
[2024-06-04 02:47:18] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.21.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
2%|▏ | 4/195 [00:06<03:48, 1.19s/it]
[2024-06-04 02:47:18] INFO group_quantization.py:217: Compiling quantize function for key: ((9216, 3072), float32, cuda, axis=1, output_transpose=False)
2%|▏ | 4/195 [00:06<03:48, 1.19s/it]
[2024-06-04 02:47:19] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
2%|▏ | 4/195 [00:06<03:48, 1.19s/it]
[2024-06-04 02:47:19] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
2%|▏ | 4/195 [00:06<03:48, 1.19s/it]
3%|▎ | 6/195 [00:06<02:23, 1.32it/s]
[2024-06-04 02:47:19] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.22.ln.weight[0m", shape: (3072,), dtype: float32
3%|▎ | 6/195 [00:06<02:23, 1.32it/s]
[2024-06-04 02:47:19] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
3%|▎ | 6/195 [00:06<02:23, 1.32it/s]
[2024-06-04 02:47:19] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
3%|▎ | 6/195 [00:06<02:23, 1.32it/s]
[2024-06-04 02:47:19] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
3%|▎ | 6/195 [00:06<02:23, 1.32it/s]
[2024-06-04 02:47:19] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
3%|▎ | 6/195 [00:07<02:23, 1.32it/s]
5%|▍ | 9/195 [00:07<01:14, 2.48it/s]
[2024-06-04 02:47:19] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.22.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
5%|▍ | 9/195 [00:07<01:14, 2.48it/s]
[2024-06-04 02:47:19] INFO group_quantization.py:217: Compiling quantize function for key: ((3072, 3072), float32, cuda, axis=1, output_transpose=False)
5%|▍ | 9/195 [00:07<01:14, 2.48it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
5%|▍ | 9/195 [00:07<01:14, 2.48it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
5%|▍ | 9/195 [00:07<01:14, 2.48it/s]
6%|▌ | 11/195 [00:07<01:07, 2.75it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
6%|▌ | 11/195 [00:07<01:07, 2.75it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.22.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
6%|▌ | 11/195 [00:07<01:07, 2.75it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.23.ln.weight[0m", shape: (3072,), dtype: float32
6%|▌ | 11/195 [00:07<01:07, 2.75it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
6%|▌ | 11/195 [00:07<01:07, 2.75it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
6%|▌ | 11/195 [00:07<01:07, 2.75it/s]
7%|▋ | 14/195 [00:07<00:42, 4.30it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
7%|▋ | 14/195 [00:07<00:42, 4.30it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
7%|▋ | 14/195 [00:07<00:42, 4.30it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.23.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
7%|▋ | 14/195 [00:07<00:42, 4.30it/s]
8%|▊ | 16/195 [00:07<00:33, 5.32it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
8%|▊ | 16/195 [00:07<00:33, 5.32it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
8%|▊ | 16/195 [00:07<00:33, 5.32it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
8%|▊ | 16/195 [00:07<00:33, 5.32it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.23.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
8%|▊ | 16/195 [00:08<00:33, 5.32it/s]
9%|▉ | 18/195 [00:08<00:27, 6.45it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.24.ln.weight[0m", shape: (3072,), dtype: float32
9%|▉ | 18/195 [00:08<00:27, 6.45it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
9%|▉ | 18/195 [00:08<00:27, 6.45it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
9%|▉ | 18/195 [00:08<00:27, 6.45it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
9%|▉ | 18/195 [00:08<00:27, 6.45it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
9%|▉ | 18/195 [00:08<00:27, 6.45it/s]
11%|█ | 21/195 [00:08<00:21, 8.20it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.24.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
11%|█ | 21/195 [00:08<00:21, 8.20it/s]
[2024-06-04 02:47:20] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
11%|█ | 21/195 [00:08<00:21, 8.20it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
11%|█ | 21/195 [00:08<00:21, 8.20it/s]
12%|█▏ | 23/195 [00:08<00:18, 9.09it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
12%|█▏ | 23/195 [00:08<00:18, 9.09it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.24.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
12%|█▏ | 23/195 [00:08<00:18, 9.09it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.25.ln.weight[0m", shape: (3072,), dtype: float32
12%|█▏ | 23/195 [00:08<00:18, 9.09it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
12%|█▏ | 23/195 [00:08<00:18, 9.09it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
12%|█▏ | 23/195 [00:08<00:18, 9.09it/s]
13%|█▎ | 26/195 [00:08<00:14, 11.52it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
13%|█▎ | 26/195 [00:08<00:14, 11.52it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
13%|█▎ | 26/195 [00:08<00:14, 11.52it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.25.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
13%|█▎ | 26/195 [00:08<00:14, 11.52it/s]
14%|█▍ | 28/195 [00:08<00:13, 11.97it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
14%|█▍ | 28/195 [00:08<00:13, 11.97it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
14%|█▍ | 28/195 [00:08<00:13, 11.97it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
14%|█▍ | 28/195 [00:08<00:13, 11.97it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.25.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
14%|█▍ | 28/195 [00:08<00:13, 11.97it/s]
15%|█▌ | 30/195 [00:08<00:13, 12.56it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.26.ln.weight[0m", shape: (3072,), dtype: float32
15%|█▌ | 30/195 [00:08<00:13, 12.56it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
15%|█▌ | 30/195 [00:08<00:13, 12.56it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
15%|█▌ | 30/195 [00:08<00:13, 12.56it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
15%|█▌ | 30/195 [00:08<00:13, 12.56it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
15%|█▌ | 30/195 [00:09<00:13, 12.56it/s]
17%|█▋ | 33/195 [00:09<00:12, 13.37it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.26.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
17%|█▋ | 33/195 [00:09<00:12, 13.37it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
17%|█▋ | 33/195 [00:09<00:12, 13.37it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
17%|█▋ | 33/195 [00:09<00:12, 13.37it/s]
18%|█▊ | 35/195 [00:09<00:11, 13.78it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
18%|█▊ | 35/195 [00:09<00:11, 13.78it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.26.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
18%|█▊ | 35/195 [00:09<00:11, 13.78it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.27.ln.weight[0m", shape: (3072,), dtype: float32
18%|█▊ | 35/195 [00:09<00:11, 13.78it/s]
[2024-06-04 02:47:21] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
18%|█▊ | 35/195 [00:09<00:11, 13.78it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
18%|█▊ | 35/195 [00:09<00:11, 13.78it/s]
19%|█▉ | 38/195 [00:09<00:09, 15.74it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
19%|█▉ | 38/195 [00:09<00:09, 15.74it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
19%|█▉ | 38/195 [00:09<00:09, 15.74it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.27.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
19%|█▉ | 38/195 [00:09<00:09, 15.74it/s]
21%|██ | 40/195 [00:09<00:10, 14.63it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
21%|██ | 40/195 [00:09<00:10, 14.63it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
21%|██ | 40/195 [00:09<00:10, 14.63it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
21%|██ | 40/195 [00:09<00:10, 14.63it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.27.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
21%|██ | 40/195 [00:09<00:10, 14.63it/s]
22%|██▏ | 42/195 [00:09<00:10, 14.68it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.28.ln.weight[0m", shape: (3072,), dtype: float32
22%|██▏ | 42/195 [00:09<00:10, 14.68it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
22%|██▏ | 42/195 [00:09<00:10, 14.68it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
22%|██▏ | 42/195 [00:09<00:10, 14.68it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
22%|██▏ | 42/195 [00:09<00:10, 14.68it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
22%|██▏ | 42/195 [00:09<00:10, 14.68it/s]
23%|██▎ | 45/195 [00:09<00:10, 14.88it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.28.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
23%|██▎ | 45/195 [00:09<00:10, 14.88it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
23%|██▎ | 45/195 [00:09<00:10, 14.88it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
23%|██▎ | 45/195 [00:09<00:10, 14.88it/s]
24%|██▍ | 47/195 [00:09<00:10, 14.33it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
24%|██▍ | 47/195 [00:10<00:10, 14.33it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.28.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
24%|██▍ | 47/195 [00:10<00:10, 14.33it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.29.ln.weight[0m", shape: (3072,), dtype: float32
24%|██▍ | 47/195 [00:10<00:10, 14.33it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
24%|██▍ | 47/195 [00:10<00:10, 14.33it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
24%|██▍ | 47/195 [00:10<00:10, 14.33it/s]
26%|██▌ | 50/195 [00:10<00:08, 16.31it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
26%|██▌ | 50/195 [00:10<00:08, 16.31it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
26%|██▌ | 50/195 [00:10<00:08, 16.31it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.29.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
26%|██▌ | 50/195 [00:10<00:08, 16.31it/s]
27%|██▋ | 52/195 [00:10<00:09, 14.77it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
27%|██▋ | 52/195 [00:10<00:09, 14.77it/s]
[2024-06-04 02:47:22] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
27%|██▋ | 52/195 [00:10<00:09, 14.77it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
27%|██▋ | 52/195 [00:10<00:09, 14.77it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.29.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
27%|██▋ | 52/195 [00:10<00:09, 14.77it/s]
28%|██▊ | 54/195 [00:10<00:09, 14.71it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.30.ln.weight[0m", shape: (3072,), dtype: float32
28%|██▊ | 54/195 [00:10<00:09, 14.71it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
28%|██▊ | 54/195 [00:10<00:09, 14.71it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
28%|██▊ | 54/195 [00:10<00:09, 14.71it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
28%|██▊ | 54/195 [00:10<00:09, 14.71it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
28%|██▊ | 54/195 [00:10<00:09, 14.71it/s]
29%|██▉ | 57/195 [00:10<00:09, 14.92it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.30.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
29%|██▉ | 57/195 [00:10<00:09, 14.92it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
29%|██▉ | 57/195 [00:10<00:09, 14.92it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
29%|██▉ | 57/195 [00:10<00:09, 14.92it/s]
30%|███ | 59/195 [00:10<00:09, 14.76it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
30%|███ | 59/195 [00:10<00:09, 14.76it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.30.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
30%|███ | 59/195 [00:10<00:09, 14.76it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.31.ln.weight[0m", shape: (3072,), dtype: float32
30%|███ | 59/195 [00:10<00:09, 14.76it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
30%|███ | 59/195 [00:10<00:09, 14.76it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
30%|███ | 59/195 [00:10<00:09, 14.76it/s]
32%|███▏ | 62/195 [00:10<00:08, 16.26it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
32%|███▏ | 62/195 [00:10<00:08, 16.26it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
32%|███▏ | 62/195 [00:11<00:08, 16.26it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.31.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
32%|███▏ | 62/195 [00:11<00:08, 16.26it/s]
33%|███▎ | 64/195 [00:11<00:08, 15.42it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
33%|███▎ | 64/195 [00:11<00:08, 15.42it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
33%|███▎ | 64/195 [00:11<00:08, 15.42it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
33%|███▎ | 64/195 [00:11<00:08, 15.42it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.31.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
33%|███▎ | 64/195 [00:11<00:08, 15.42it/s]
34%|███▍ | 66/195 [00:11<00:08, 14.92it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.norm.weight[0m", shape: (3072,), dtype: float32
34%|███▍ | 66/195 [00:11<00:08, 14.92it/s]
[2024-06-04 02:47:23] INFO huggingface_loader.py:197: Unloading HF weight file: /models/Phi-3-mini-128k-instruct/model-00002-of-00002.safetensors
34%|███▍ | 66/195 [00:11<00:08, 14.92it/s]
[2024-06-04 02:47:24] INFO huggingface_loader.py:185: Loading HF parameters from: /models/Phi-3-mini-128k-instruct/model-00001-of-00002.safetensors
34%|███▍ | 66/195 [00:11<00:08, 14.92it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.embd.q_weight[0m", shape: (32064, 384), dtype: uint32
34%|███▍ | 66/195 [00:16<00:08, 14.92it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.embd.q_scale[0m", shape: (32064, 96), dtype: float32
34%|███▍ | 66/195 [00:16<00:08, 14.92it/s]
35%|███▍ | 68/195 [00:16<01:42, 1.24it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.0.ln.weight[0m", shape: (3072,), dtype: float32
35%|███▍ | 68/195 [00:16<01:42, 1.24it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
35%|███▍ | 68/195 [00:16<01:42, 1.24it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
35%|███▍ | 68/195 [00:16<01:42, 1.24it/s]
36%|███▌ | 70/195 [00:16<01:14, 1.68it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
36%|███▌ | 70/195 [00:16<01:14, 1.68it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
36%|███▌ | 70/195 [00:16<01:14, 1.68it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.0.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
36%|███▌ | 70/195 [00:16<01:14, 1.68it/s]
37%|███▋ | 72/195 [00:16<00:54, 2.25it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
37%|███▋ | 72/195 [00:17<00:54, 2.25it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
37%|███▋ | 72/195 [00:17<00:54, 2.25it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
37%|███▋ | 72/195 [00:17<00:54, 2.25it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.0.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
37%|███▋ | 72/195 [00:17<00:54, 2.25it/s]
38%|███▊ | 74/195 [00:17<00:40, 2.99it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.1.ln.weight[0m", shape: (3072,), dtype: float32
38%|███▊ | 74/195 [00:17<00:40, 2.99it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
38%|███▊ | 74/195 [00:17<00:40, 2.99it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
38%|███▊ | 74/195 [00:17<00:40, 2.99it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
38%|███▊ | 74/195 [00:17<00:40, 2.99it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
38%|███▊ | 74/195 [00:17<00:40, 2.99it/s]
39%|███▉ | 77/195 [00:17<00:27, 4.32it/s]
[2024-06-04 02:47:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.1.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
39%|███▉ | 77/195 [00:17<00:27, 4.32it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
39%|███▉ | 77/195 [00:17<00:27, 4.32it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
39%|███▉ | 77/195 [00:17<00:27, 4.32it/s]
41%|████ | 79/195 [00:17<00:21, 5.32it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
41%|████ | 79/195 [00:17<00:21, 5.32it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.1.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
41%|████ | 79/195 [00:17<00:21, 5.32it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.10.ln.weight[0m", shape: (3072,), dtype: float32
41%|████ | 79/195 [00:17<00:21, 5.32it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
41%|████ | 79/195 [00:17<00:21, 5.32it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
41%|████ | 79/195 [00:17<00:21, 5.32it/s]
42%|████▏ | 82/195 [00:17<00:15, 7.17it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
42%|████▏ | 82/195 [00:17<00:15, 7.17it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
42%|████▏ | 82/195 [00:17<00:15, 7.17it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.10.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
42%|████▏ | 82/195 [00:17<00:15, 7.17it/s]
43%|████▎ | 84/195 [00:17<00:13, 8.12it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
43%|████▎ | 84/195 [00:17<00:13, 8.12it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
43%|████▎ | 84/195 [00:17<00:13, 8.12it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
43%|████▎ | 84/195 [00:17<00:13, 8.12it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.10.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
43%|████▎ | 84/195 [00:17<00:13, 8.12it/s]
44%|████▍ | 86/195 [00:17<00:11, 9.33it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.11.ln.weight[0m", shape: (3072,), dtype: float32
44%|████▍ | 86/195 [00:17<00:11, 9.33it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
44%|████▍ | 86/195 [00:17<00:11, 9.33it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
44%|████▍ | 86/195 [00:17<00:11, 9.33it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
44%|████▍ | 86/195 [00:17<00:11, 9.33it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
44%|████▍ | 86/195 [00:18<00:11, 9.33it/s]
46%|████▌ | 89/195 [00:18<00:09, 10.90it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.11.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
46%|████▌ | 89/195 [00:18<00:09, 10.90it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
46%|████▌ | 89/195 [00:18<00:09, 10.90it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
46%|████▌ | 89/195 [00:18<00:09, 10.90it/s]
47%|████▋ | 91/195 [00:18<00:08, 11.76it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
47%|████▋ | 91/195 [00:18<00:08, 11.76it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.11.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
47%|████▋ | 91/195 [00:18<00:08, 11.76it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.12.ln.weight[0m", shape: (3072,), dtype: float32
47%|████▋ | 91/195 [00:18<00:08, 11.76it/s]
[2024-06-04 02:47:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
47%|████▋ | 91/195 [00:18<00:08, 11.76it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
47%|████▋ | 91/195 [00:18<00:08, 11.76it/s]
48%|████▊ | 94/195 [00:18<00:07, 13.64it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
48%|████▊ | 94/195 [00:18<00:07, 13.64it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
48%|████▊ | 94/195 [00:18<00:07, 13.64it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.12.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
48%|████▊ | 94/195 [00:18<00:07, 13.64it/s]
49%|████▉ | 96/195 [00:18<00:07, 13.05it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
49%|████▉ | 96/195 [00:18<00:07, 13.05it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
49%|████▉ | 96/195 [00:18<00:07, 13.05it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
49%|████▉ | 96/195 [00:18<00:07, 13.05it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.12.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
49%|████▉ | 96/195 [00:18<00:07, 13.05it/s]
50%|█████ | 98/195 [00:18<00:07, 13.51it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.13.ln.weight[0m", shape: (3072,), dtype: float32
50%|█████ | 98/195 [00:18<00:07, 13.51it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
50%|█████ | 98/195 [00:18<00:07, 13.51it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
50%|█████ | 98/195 [00:18<00:07, 13.51it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
50%|█████ | 98/195 [00:18<00:07, 13.51it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
50%|█████ | 98/195 [00:18<00:07, 13.51it/s]
52%|█████▏ | 101/195 [00:18<00:06, 13.96it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.13.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
52%|█████▏ | 101/195 [00:18<00:06, 13.96it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
52%|█████▏ | 101/195 [00:18<00:06, 13.96it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
52%|█████▏ | 101/195 [00:19<00:06, 13.96it/s]
53%|█████▎ | 103/195 [00:19<00:06, 14.16it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
53%|█████▎ | 103/195 [00:19<00:06, 14.16it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.13.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
53%|█████▎ | 103/195 [00:19<00:06, 14.16it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.14.ln.weight[0m", shape: (3072,), dtype: float32
53%|█████▎ | 103/195 [00:19<00:06, 14.16it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
53%|█████▎ | 103/195 [00:19<00:06, 14.16it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
53%|█████▎ | 103/195 [00:19<00:06, 14.16it/s]
54%|█████▍ | 106/195 [00:19<00:05, 15.92it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
54%|█████▍ | 106/195 [00:19<00:05, 15.92it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
54%|█████▍ | 106/195 [00:19<00:05, 15.92it/s]
[2024-06-04 02:47:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.14.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
54%|█████▍ | 106/195 [00:19<00:05, 15.92it/s]
55%|█████▌ | 108/195 [00:19<00:05, 15.34it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
55%|█████▌ | 108/195 [00:19<00:05, 15.34it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
55%|█████▌ | 108/195 [00:19<00:05, 15.34it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
55%|█████▌ | 108/195 [00:19<00:05, 15.34it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.14.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
55%|█████▌ | 108/195 [00:19<00:05, 15.34it/s]
56%|█████▋ | 110/195 [00:19<00:05, 15.51it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.15.ln.weight[0m", shape: (3072,), dtype: float32
56%|█████▋ | 110/195 [00:19<00:05, 15.51it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
56%|█████▋ | 110/195 [00:19<00:05, 15.51it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
56%|█████▋ | 110/195 [00:19<00:05, 15.51it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
56%|█████▋ | 110/195 [00:19<00:05, 15.51it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
56%|█████▋ | 110/195 [00:19<00:05, 15.51it/s]
58%|█████▊ | 113/195 [00:19<00:05, 15.41it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.15.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
58%|█████▊ | 113/195 [00:19<00:05, 15.41it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
58%|█████▊ | 113/195 [00:19<00:05, 15.41it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
58%|█████▊ | 113/195 [00:19<00:05, 15.41it/s]
59%|█████▉ | 115/195 [00:19<00:05, 14.81it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
59%|█████▉ | 115/195 [00:19<00:05, 14.81it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.15.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
59%|█████▉ | 115/195 [00:19<00:05, 14.81it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.16.ln.weight[0m", shape: (3072,), dtype: float32
59%|█████▉ | 115/195 [00:19<00:05, 14.81it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
59%|█████▉ | 115/195 [00:19<00:05, 14.81it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
59%|█████▉ | 115/195 [00:19<00:05, 14.81it/s]
61%|██████ | 118/195 [00:19<00:04, 16.80it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
61%|██████ | 118/195 [00:19<00:04, 16.80it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
61%|██████ | 118/195 [00:20<00:04, 16.80it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.16.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
61%|██████ | 118/195 [00:20<00:04, 16.80it/s]
62%|██████▏ | 120/195 [00:20<00:04, 15.98it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
62%|██████▏ | 120/195 [00:20<00:04, 15.98it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
62%|██████▏ | 120/195 [00:20<00:04, 15.98it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
62%|██████▏ | 120/195 [00:20<00:04, 15.98it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.16.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
62%|██████▏ | 120/195 [00:20<00:04, 15.98it/s]
63%|██████▎ | 122/195 [00:20<00:04, 15.83it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.17.ln.weight[0m", shape: (3072,), dtype: float32
63%|██████▎ | 122/195 [00:20<00:04, 15.83it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
63%|██████▎ | 122/195 [00:20<00:04, 15.83it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
63%|██████▎ | 122/195 [00:20<00:04, 15.83it/s]
[2024-06-04 02:47:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
63%|██████▎ | 122/195 [00:20<00:04, 15.83it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
63%|██████▎ | 122/195 [00:20<00:04, 15.83it/s]
64%|██████▍ | 125/195 [00:20<00:04, 15.75it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.17.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
64%|██████▍ | 125/195 [00:20<00:04, 15.75it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
64%|██████▍ | 125/195 [00:20<00:04, 15.75it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
64%|██████▍ | 125/195 [00:20<00:04, 15.75it/s]
65%|██████▌ | 127/195 [00:20<00:04, 15.07it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
65%|██████▌ | 127/195 [00:20<00:04, 15.07it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.17.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
65%|██████▌ | 127/195 [00:20<00:04, 15.07it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.18.ln.weight[0m", shape: (3072,), dtype: float32
65%|██████▌ | 127/195 [00:20<00:04, 15.07it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
65%|██████▌ | 127/195 [00:20<00:04, 15.07it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
65%|██████▌ | 127/195 [00:20<00:04, 15.07it/s]
67%|██████▋ | 130/195 [00:20<00:03, 16.64it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
67%|██████▋ | 130/195 [00:20<00:03, 16.64it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
67%|██████▋ | 130/195 [00:20<00:03, 16.64it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.18.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
67%|██████▋ | 130/195 [00:20<00:03, 16.64it/s]
68%|██████▊ | 132/195 [00:20<00:03, 15.84it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
68%|██████▊ | 132/195 [00:20<00:03, 15.84it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
68%|██████▊ | 132/195 [00:20<00:03, 15.84it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
68%|██████▊ | 132/195 [00:20<00:03, 15.84it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.18.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
68%|██████▊ | 132/195 [00:20<00:03, 15.84it/s]
69%|██████▊ | 134/195 [00:20<00:03, 15.59it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.19.ln.weight[0m", shape: (3072,), dtype: float32
69%|██████▊ | 134/195 [00:20<00:03, 15.59it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
69%|██████▊ | 134/195 [00:20<00:03, 15.59it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
69%|██████▊ | 134/195 [00:21<00:03, 15.59it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
69%|██████▊ | 134/195 [00:21<00:03, 15.59it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
69%|██████▊ | 134/195 [00:21<00:03, 15.59it/s]
70%|███████ | 137/195 [00:21<00:03, 15.58it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.19.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
70%|███████ | 137/195 [00:21<00:03, 15.58it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
70%|███████ | 137/195 [00:21<00:03, 15.58it/s]
[2024-06-04 02:47:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
70%|███████ | 137/195 [00:21<00:03, 15.58it/s]
71%|███████▏ | 139/195 [00:21<00:03, 15.07it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
71%|███████▏ | 139/195 [00:21<00:03, 15.07it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.19.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
71%|███████▏ | 139/195 [00:21<00:03, 15.07it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.2.ln.weight[0m", shape: (3072,), dtype: float32
71%|███████▏ | 139/195 [00:21<00:03, 15.07it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
71%|███████▏ | 139/195 [00:21<00:03, 15.07it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
71%|███████▏ | 139/195 [00:21<00:03, 15.07it/s]
73%|███████▎ | 142/195 [00:21<00:03, 16.78it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
73%|███████▎ | 142/195 [00:21<00:03, 16.78it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
73%|███████▎ | 142/195 [00:21<00:03, 16.78it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.2.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
73%|███████▎ | 142/195 [00:21<00:03, 16.78it/s]
74%|███████▍ | 144/195 [00:21<00:03, 15.74it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
74%|███████▍ | 144/195 [00:21<00:03, 15.74it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
74%|███████▍ | 144/195 [00:21<00:03, 15.74it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
74%|███████▍ | 144/195 [00:21<00:03, 15.74it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.2.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
74%|███████▍ | 144/195 [00:21<00:03, 15.74it/s]
75%|███████▍ | 146/195 [00:21<00:03, 15.50it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.20.ln.weight[0m", shape: (3072,), dtype: float32
75%|███████▍ | 146/195 [00:21<00:03, 15.50it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
75%|███████▍ | 146/195 [00:21<00:03, 15.50it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
75%|███████▍ | 146/195 [00:21<00:03, 15.50it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
75%|███████▍ | 146/195 [00:21<00:03, 15.50it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
75%|███████▍ | 146/195 [00:21<00:03, 15.50it/s]
76%|███████▋ | 149/195 [00:21<00:02, 15.40it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.20.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
76%|███████▋ | 149/195 [00:21<00:02, 15.40it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
76%|███████▋ | 149/195 [00:21<00:02, 15.40it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
76%|███████▋ | 149/195 [00:22<00:02, 15.40it/s]
77%|███████▋ | 151/195 [00:22<00:02, 15.11it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
77%|███████▋ | 151/195 [00:22<00:02, 15.11it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.20.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
77%|███████▋ | 151/195 [00:22<00:02, 15.11it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
77%|███████▋ | 151/195 [00:22<00:02, 15.11it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.21.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
77%|███████▋ | 151/195 [00:22<00:02, 15.11it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.3.ln.weight[0m", shape: (3072,), dtype: float32
77%|███████▋ | 151/195 [00:22<00:02, 15.11it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
77%|███████▋ | 151/195 [00:22<00:02, 15.11it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
77%|███████▋ | 151/195 [00:22<00:02, 15.11it/s]
79%|███████▉ | 155/195 [00:22<00:02, 17.78it/s]
[2024-06-04 02:47:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
79%|███████▉ | 155/195 [00:22<00:02, 17.78it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
79%|███████▉ | 155/195 [00:22<00:02, 17.78it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.3.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
79%|███████▉ | 155/195 [00:22<00:02, 17.78it/s]
81%|████████ | 157/195 [00:22<00:02, 16.55it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
81%|████████ | 157/195 [00:22<00:02, 16.55it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
81%|████████ | 157/195 [00:22<00:02, 16.55it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
81%|████████ | 157/195 [00:22<00:02, 16.55it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.3.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
81%|████████ | 157/195 [00:22<00:02, 16.55it/s]
82%|████████▏ | 159/195 [00:22<00:02, 15.93it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.4.ln.weight[0m", shape: (3072,), dtype: float32
82%|████████▏ | 159/195 [00:22<00:02, 15.93it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
82%|████████▏ | 159/195 [00:22<00:02, 15.93it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
82%|████████▏ | 159/195 [00:22<00:02, 15.93it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
82%|████████▏ | 159/195 [00:22<00:02, 15.93it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
82%|████████▏ | 159/195 [00:22<00:02, 15.93it/s]
83%|████████▎ | 162/195 [00:22<00:02, 15.80it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.4.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
83%|████████▎ | 162/195 [00:22<00:02, 15.80it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
83%|████████▎ | 162/195 [00:22<00:02, 15.80it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
83%|████████▎ | 162/195 [00:22<00:02, 15.80it/s]
84%|████████▍ | 164/195 [00:22<00:02, 14.79it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
84%|████████▍ | 164/195 [00:22<00:02, 14.79it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.4.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
84%|████████▍ | 164/195 [00:22<00:02, 14.79it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.5.ln.weight[0m", shape: (3072,), dtype: float32
84%|████████▍ | 164/195 [00:22<00:02, 14.79it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
84%|████████▍ | 164/195 [00:22<00:02, 14.79it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
84%|████████▍ | 164/195 [00:23<00:02, 14.79it/s]
86%|████████▌ | 167/195 [00:23<00:01, 16.00it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
86%|████████▌ | 167/195 [00:23<00:01, 16.00it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
86%|████████▌ | 167/195 [00:23<00:01, 16.00it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.5.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
86%|████████▌ | 167/195 [00:23<00:01, 16.00it/s]
87%|████████▋ | 169/195 [00:23<00:01, 15.02it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
87%|████████▋ | 169/195 [00:23<00:01, 15.02it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
87%|████████▋ | 169/195 [00:23<00:01, 15.02it/s]
[2024-06-04 02:47:35] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
87%|████████▋ | 169/195 [00:23<00:01, 15.02it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.5.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
87%|████████▋ | 169/195 [00:23<00:01, 15.02it/s]
88%|████████▊ | 171/195 [00:23<00:01, 14.50it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.6.ln.weight[0m", shape: (3072,), dtype: float32
88%|████████▊ | 171/195 [00:23<00:01, 14.50it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
88%|████████▊ | 171/195 [00:23<00:01, 14.50it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
88%|████████▊ | 171/195 [00:23<00:01, 14.50it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
88%|████████▊ | 171/195 [00:23<00:01, 14.50it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
88%|████████▊ | 171/195 [00:23<00:01, 14.50it/s]
89%|████████▉ | 174/195 [00:23<00:01, 14.61it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.6.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
89%|████████▉ | 174/195 [00:23<00:01, 14.61it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
89%|████████▉ | 174/195 [00:23<00:01, 14.61it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
89%|████████▉ | 174/195 [00:23<00:01, 14.61it/s]
90%|█████████ | 176/195 [00:23<00:01, 14.18it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
90%|█████████ | 176/195 [00:23<00:01, 14.18it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.6.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
90%|█████████ | 176/195 [00:23<00:01, 14.18it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.7.ln.weight[0m", shape: (3072,), dtype: float32
90%|█████████ | 176/195 [00:23<00:01, 14.18it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
90%|█████████ | 176/195 [00:23<00:01, 14.18it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
90%|█████████ | 176/195 [00:23<00:01, 14.18it/s]
92%|█████████▏| 179/195 [00:23<00:01, 15.53it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
92%|█████████▏| 179/195 [00:23<00:01, 15.53it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
92%|█████████▏| 179/195 [00:24<00:01, 15.53it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.7.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
92%|█████████▏| 179/195 [00:24<00:01, 15.53it/s]
93%|█████████▎| 181/195 [00:24<00:00, 14.63it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
93%|█████████▎| 181/195 [00:24<00:00, 14.63it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
93%|█████████▎| 181/195 [00:24<00:00, 14.63it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
93%|█████████▎| 181/195 [00:24<00:00, 14.63it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.7.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
93%|█████████▎| 181/195 [00:24<00:00, 14.63it/s]
94%|█████████▍| 183/195 [00:24<00:00, 14.46it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.8.ln.weight[0m", shape: (3072,), dtype: float32
94%|█████████▍| 183/195 [00:24<00:00, 14.46it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
94%|█████████▍| 183/195 [00:24<00:00, 14.46it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
94%|█████████▍| 183/195 [00:24<00:00, 14.46it/s]
[2024-06-04 02:47:36] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
94%|█████████▍| 183/195 [00:24<00:00, 14.46it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
94%|█████████▍| 183/195 [00:24<00:00, 14.46it/s]
95%|█████████▌| 186/195 [00:24<00:00, 14.58it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.8.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
95%|█████████▌| 186/195 [00:24<00:00, 14.58it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
95%|█████████▌| 186/195 [00:24<00:00, 14.58it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
95%|█████████▌| 186/195 [00:24<00:00, 14.58it/s]
96%|█████████▋| 188/195 [00:24<00:00, 14.04it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
96%|█████████▋| 188/195 [00:24<00:00, 14.04it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.8.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
96%|█████████▋| 188/195 [00:24<00:00, 14.04it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.9.ln.weight[0m", shape: (3072,), dtype: float32
96%|█████████▋| 188/195 [00:24<00:00, 14.04it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.down_proj.q_weight[0m", shape: (3072, 1024), dtype: uint32
96%|█████████▋| 188/195 [00:24<00:00, 14.04it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.down_proj.q_scale[0m", shape: (3072, 256), dtype: float32
96%|█████████▋| 188/195 [00:24<00:00, 14.04it/s]
98%|█████████▊| 191/195 [00:24<00:00, 15.43it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.gate_up_proj.q_weight[0m", shape: (16384, 384), dtype: uint32
98%|█████████▊| 191/195 [00:24<00:00, 15.43it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mlp.gate_up_proj.q_scale[0m", shape: (16384, 96), dtype: float32
98%|█████████▊| 191/195 [00:24<00:00, 15.43it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:175: [Not quantized] Parameter: "[1mtransformer.h.9.post_attention_layernorm.weight[0m", shape: (3072,), dtype: float32
98%|█████████▊| 191/195 [00:24<00:00, 15.43it/s]
99%|█████████▉| 193/195 [00:24<00:00, 14.58it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.out_proj.q_weight[0m", shape: (3072, 384), dtype: uint32
99%|█████████▉| 193/195 [00:24<00:00, 14.58it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.out_proj.q_scale[0m", shape: (3072, 96), dtype: float32
99%|█████████▉| 193/195 [00:24<00:00, 14.58it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.qkv_proj.q_weight[0m", shape: (9216, 384), dtype: uint32
99%|█████████▉| 193/195 [00:24<00:00, 14.58it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:167: [Quantized] Parameter: "[1mtransformer.h.9.mixer.qkv_proj.q_scale[0m", shape: (9216, 96), dtype: float32
99%|█████████▉| 193/195 [00:24<00:00, 14.58it/s]
100%|██████████| 195/195 [00:24<00:00, 14.29it/s]
100%|██████████| 195/195 [00:24<00:00, 7.80it/s]
[2024-06-04 02:47:37] INFO huggingface_loader.py:197: Unloading HF weight file: /models/Phi-3-mini-128k-instruct/model-00001-of-00002.safetensors
[2024-06-04 02:47:38] INFO stats.py:77: [92mTime usage[0m: HF loading: 9.087 sec; Pre-quantization mapping: 2.399 sec; Quantization: 2.800 sec
[2024-06-04 02:47:38] INFO stats.py:91: [92mRAM usage[0m: Peak RAM: 9.262 GB. Total bytes loaded from disk: 14.235 GB
[2024-06-04 02:47:38] INFO convert_weight.py:155: [92mParameter size[0m after quantization: 2.225 GB
[2024-06-04 02:47:38] INFO convert_weight.py:160: [92mTotal parameters[0m: 3,821,079,552
[2024-06-04 02:47:38] INFO convert_weight.py:161: [92mBits per parameter[0m: 5.001
[2024-06-04 02:47:38] INFO convert_weight.py:166: Saved to directory: [1m/models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC[0m
All finished, 83 total shards committed, record saved to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/ndarray-cache.json
Also saved a bf16 record to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q4f32_1-MLC/ndarray-cache-b16.json