Hugging Face's logo

GreenBitAI
/
Qwen-1.5-1.8B-channel-mix-bpw-3.0

Text Generation
Transformers
Safetensors
qwen2
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
NicoNico6 commited on
Commit
7df3ebb
•
1 Parent(s): 60e1802

update

Browse files
Files changed (9) hide show
  1. added_tokens.json +5 -0
  2. config.json +28 -0
  3. generation_config.json +6 -0
  4. merges.txt +0 -0
  5. model.safetensors +3 -0
  6. quant_strategy.json +2452 -0
  7. special_tokens_map.json +20 -0
  8. tokenizer_config.json +43 -0
  9. vocab.json +0 -0
added_tokens.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "<|endoftext|>": 151643,
3
+ "<|im_end|>": 151645,
4
+ "<|im_start|>": 151644
5
+ }
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/hpi/fs00/share/fg/meinel/nianhui.guo/qwen-hf/models--Qwen--Qwen1.5-1.8B/snapshots/7846de7ed421727b318d6605a0bfab659da2c067/",
3
+ "architectures": [
4
+ "Qwen2ForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 151643,
8
+ "eos_token_id": 151643,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 2048,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 5504,
13
+ "max_position_embeddings": 32768,
14
+ "max_window_layers": 21,
15
+ "model_type": "qwen2",
16
+ "num_attention_heads": 16,
17
+ "num_hidden_layers": 24,
18
+ "num_key_value_heads": 16,
19
+ "rms_norm_eps": 1e-06,
20
+ "rope_theta": 1000000.0,
21
+ "sliding_window": 32768,
22
+ "tie_word_embeddings": false,
23
+ "torch_dtype": "float16",
24
+ "transformers_version": "4.39.2",
25
+ "use_cache": true,
26
+ "use_sliding_window": false,
27
+ "vocab_size": 151936
28
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 151643,
4
+ "eos_token_id": 151643,
5
+ "transformers_version": "4.39.2"
6
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0cc0d29721fbe98f691b9a0ed0786c8aff0154c3140d401702eca05811af91d5
3
+ size 1733076752
quant_strategy.json ADDED
@@ -0,0 +1,2452 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "measurement": {
3
+ "model.layers.0": {
4
+ "accuracy": 0.826019287109375,
5
+ "total_bits": 142490128,
6
+ "q_proj": {
7
+ "group_size": {
8
+ "4": 128,
9
+ "2": 128
10
+ },
11
+ "bits": [
12
+ 4,
13
+ 2
14
+ ],
15
+ "bits_prop": [
16
+ 0.1,
17
+ 0.9
18
+ ],
19
+ "scale_bits": 4
20
+ },
21
+ "k_proj": {
22
+ "group_size": {
23
+ "4": 128,
24
+ "2": 128
25
+ },
26
+ "bits": [
27
+ 4,
28
+ 2
29
+ ],
30
+ "bits_prop": [
31
+ 0.1,
32
+ 0.9
33
+ ],
34
+ "scale_bits": 4
35
+ },
36
+ "v_proj": {
37
+ "group_size": {
38
+ "4": 128
39
+ },
40
+ "bits": [
41
+ 4
42
+ ],
43
+ "bits_prop": [
44
+ 1.0
45
+ ],
46
+ "scale_bits": 4
47
+ },
48
+ "o_proj": {
49
+ "group_size": {
50
+ "4": 128,
51
+ "2": 128
52
+ },
53
+ "bits": [
54
+ 4,
55
+ 2
56
+ ],
57
+ "bits_prop": [
58
+ 0.1,
59
+ 0.9
60
+ ],
61
+ "scale_bits": 4
62
+ },
63
+ "up_proj": {
64
+ "group_size": {
65
+ "4": 128,
66
+ "2": 128
67
+ },
68
+ "bits": [
69
+ 4,
70
+ 2
71
+ ],
72
+ "bits_prop": [
73
+ 0.1,
74
+ 0.9
75
+ ],
76
+ "scale_bits": 4
77
+ },
78
+ "gate_proj": {
79
+ "group_size": {
80
+ "4": 128,
81
+ "2": 128
82
+ },
83
+ "bits": [
84
+ 4,
85
+ 2
86
+ ],
87
+ "bits_prop": [
88
+ 0.1,
89
+ 0.9
90
+ ],
91
+ "scale_bits": 4
92
+ },
93
+ "down_proj": {
94
+ "group_size": {
95
+ "4": 128
96
+ },
97
+ "bits": [
98
+ 4
99
+ ],
100
+ "bits_prop": [
101
+ 1.0
102
+ ],
103
+ "scale_bits": 4
104
+ }
105
+ },
106
+ "model.layers.1": {
107
+ "accuracy": 0.8716602325439453,
108
+ "total_bits": 111884816,
109
+ "q_proj": {
110
+ "group_size": {
111
+ "4": 128,
112
+ "2": 128
113
+ },
114
+ "bits": [
115
+ 4,
116
+ 2
117
+ ],
118
+ "bits_prop": [
119
+ 0.01,
120
+ 0.99
121
+ ],
122
+ "scale_bits": 4
123
+ },
124
+ "k_proj": {
125
+ "group_size": {
126
+ "4": 128,
127
+ "2": 128
128
+ },
129
+ "bits": [
130
+ 4,
131
+ 2
132
+ ],
133
+ "bits_prop": [
134
+ 0.01,
135
+ 0.99
136
+ ],
137
+ "scale_bits": 4
138
+ },
139
+ "v_proj": {
140
+ "group_size": {
141
+ "4": 128,
142
+ "2": 128
143
+ },
144
+ "bits": [
145
+ 4,
146
+ 2
147
+ ],
148
+ "bits_prop": [
149
+ 0.1,
150
+ 0.9
151
+ ],
152
+ "scale_bits": 4
153
+ },
154
+ "o_proj": {
155
+ "group_size": {
156
+ "4": 128,
157
+ "2": 128
158
+ },
159
+ "bits": [
160
+ 4,
161
+ 2
162
+ ],
163
+ "bits_prop": [
164
+ 0.01,
165
+ 0.99
166
+ ],
167
+ "scale_bits": 4
168
+ },
169
+ "up_proj": {
170
+ "group_size": {
171
+ "4": 128,
172
+ "2": 128
173
+ },
174
+ "bits": [
175
+ 4,
176
+ 2
177
+ ],
178
+ "bits_prop": [
179
+ 0.01,
180
+ 0.99
181
+ ],
182
+ "scale_bits": 4
183
+ },
184
+ "gate_proj": {
185
+ "group_size": {
186
+ "4": 128,
187
+ "2": 128
188
+ },
189
+ "bits": [
190
+ 4,
191
+ 2
192
+ ],
193
+ "bits_prop": [
194
+ 0.01,
195
+ 0.99
196
+ ],
197
+ "scale_bits": 4
198
+ },
199
+ "down_proj": {
200
+ "group_size": {
201
+ "4": 128,
202
+ "2": 128
203
+ },
204
+ "bits": [
205
+ 4,
206
+ 2
207
+ ],
208
+ "bits_prop": [
209
+ 0.15,
210
+ 0.85
211
+ ],
212
+ "scale_bits": 4
213
+ }
214
+ },
215
+ "model.layers.2": {
216
+ "accuracy": 0.8723664283752441,
217
+ "total_bits": 112409104,
218
+ "q_proj": {
219
+ "group_size": {
220
+ "4": 128,
221
+ "2": 128
222
+ },
223
+ "bits": [
224
+ 4,
225
+ 2
226
+ ],
227
+ "bits_prop": [
228
+ 0.01,
229
+ 0.99
230
+ ],
231
+ "scale_bits": 4
232
+ },
233
+ "k_proj": {
234
+ "group_size": {
235
+ "4": 128,
236
+ "2": 128
237
+ },
238
+ "bits": [
239
+ 4,
240
+ 2
241
+ ],
242
+ "bits_prop": [
243
+ 0.01,
244
+ 0.99
245
+ ],
246
+ "scale_bits": 4
247
+ },
248
+ "v_proj": {
249
+ "group_size": {
250
+ "4": 128,
251
+ "2": 128
252
+ },
253
+ "bits": [
254
+ 4,
255
+ 2
256
+ ],
257
+ "bits_prop": [
258
+ 0.15,
259
+ 0.85
260
+ ],
261
+ "scale_bits": 4
262
+ },
263
+ "o_proj": {
264
+ "group_size": {
265
+ "4": 128,
266
+ "2": 128
267
+ },
268
+ "bits": [
269
+ 4,
270
+ 2
271
+ ],
272
+ "bits_prop": [
273
+ 0.01,
274
+ 0.99
275
+ ],
276
+ "scale_bits": 4
277
+ },
278
+ "up_proj": {
279
+ "group_size": {
280
+ "4": 128,
281
+ "2": 128
282
+ },
283
+ "bits": [
284
+ 4,
285
+ 2
286
+ ],
287
+ "bits_prop": [
288
+ 0.01,
289
+ 0.99
290
+ ],
291
+ "scale_bits": 4
292
+ },
293
+ "gate_proj": {
294
+ "group_size": {
295
+ "4": 128,
296
+ "2": 128
297
+ },
298
+ "bits": [
299
+ 4,
300
+ 2
301
+ ],
302
+ "bits_prop": [
303
+ 0.01,
304
+ 0.99
305
+ ],
306
+ "scale_bits": 4
307
+ },
308
+ "down_proj": {
309
+ "group_size": {
310
+ "4": 128,
311
+ "2": 128
312
+ },
313
+ "bits": [
314
+ 4,
315
+ 2
316
+ ],
317
+ "bits_prop": [
318
+ 0.15,
319
+ 0.85
320
+ ],
321
+ "scale_bits": 4
322
+ }
323
+ },
324
+ "model.layers.3": {
325
+ "accuracy": 0.8608307838439941,
326
+ "total_bits": 111884816,
327
+ "q_proj": {
328
+ "group_size": {
329
+ "4": 128,
330
+ "2": 128
331
+ },
332
+ "bits": [
333
+ 4,
334
+ 2
335
+ ],
336
+ "bits_prop": [
337
+ 0.01,
338
+ 0.99
339
+ ],
340
+ "scale_bits": 4
341
+ },
342
+ "k_proj": {
343
+ "group_size": {
344
+ "4": 128,
345
+ "2": 128
346
+ },
347
+ "bits": [
348
+ 4,
349
+ 2
350
+ ],
351
+ "bits_prop": [
352
+ 0.01,
353
+ 0.99
354
+ ],
355
+ "scale_bits": 4
356
+ },
357
+ "v_proj": {
358
+ "group_size": {
359
+ "4": 128,
360
+ "2": 128
361
+ },
362
+ "bits": [
363
+ 4,
364
+ 2
365
+ ],
366
+ "bits_prop": [
367
+ 0.1,
368
+ 0.9
369
+ ],
370
+ "scale_bits": 4
371
+ },
372
+ "o_proj": {
373
+ "group_size": {
374
+ "4": 128,
375
+ "2": 128
376
+ },
377
+ "bits": [
378
+ 4,
379
+ 2
380
+ ],
381
+ "bits_prop": [
382
+ 0.01,
383
+ 0.99
384
+ ],
385
+ "scale_bits": 4
386
+ },
387
+ "up_proj": {
388
+ "group_size": {
389
+ "4": 128,
390
+ "2": 128
391
+ },
392
+ "bits": [
393
+ 4,
394
+ 2
395
+ ],
396
+ "bits_prop": [
397
+ 0.01,
398
+ 0.99
399
+ ],
400
+ "scale_bits": 4
401
+ },
402
+ "gate_proj": {
403
+ "group_size": {
404
+ "4": 128,
405
+ "2": 128
406
+ },
407
+ "bits": [
408
+ 4,
409
+ 2
410
+ ],
411
+ "bits_prop": [
412
+ 0.01,
413
+ 0.99
414
+ ],
415
+ "scale_bits": 4
416
+ },
417
+ "down_proj": {
418
+ "group_size": {
419
+ "4": 128,
420
+ "2": 128
421
+ },
422
+ "bits": [
423
+ 4,
424
+ 2
425
+ ],
426
+ "bits_prop": [
427
+ 0.15,
428
+ 0.85
429
+ ],
430
+ "scale_bits": 4
431
+ }
432
+ },
433
+ "model.layers.4": {
434
+ "accuracy": 0.8559846878051758,
435
+ "total_bits": 120273424,
436
+ "q_proj": {
437
+ "group_size": {
438
+ "4": 128,
439
+ "2": 128
440
+ },
441
+ "bits": [
442
+ 4,
443
+ 2
444
+ ],
445
+ "bits_prop": [
446
+ 0.05,
447
+ 0.95
448
+ ],
449
+ "scale_bits": 4
450
+ },
451
+ "k_proj": {
452
+ "group_size": {
453
+ "4": 128,
454
+ "2": 128
455
+ },
456
+ "bits": [
457
+ 4,
458
+ 2
459
+ ],
460
+ "bits_prop": [
461
+ 0.05,
462
+ 0.95
463
+ ],
464
+ "scale_bits": 4
465
+ },
466
+ "v_proj": {
467
+ "group_size": {
468
+ "4": 128,
469
+ "2": 128
470
+ },
471
+ "bits": [
472
+ 4,
473
+ 2
474
+ ],
475
+ "bits_prop": [
476
+ 0.4,
477
+ 0.6
478
+ ],
479
+ "scale_bits": 4
480
+ },
481
+ "o_proj": {
482
+ "group_size": {
483
+ "4": 128,
484
+ "2": 128
485
+ },
486
+ "bits": [
487
+ 4,
488
+ 2
489
+ ],
490
+ "bits_prop": [
491
+ 0.05,
492
+ 0.95
493
+ ],
494
+ "scale_bits": 4
495
+ },
496
+ "up_proj": {
497
+ "group_size": {
498
+ "4": 128,
499
+ "2": 128
500
+ },
501
+ "bits": [
502
+ 4,
503
+ 2
504
+ ],
505
+ "bits_prop": [
506
+ 0.05,
507
+ 0.95
508
+ ],
509
+ "scale_bits": 4
510
+ },
511
+ "gate_proj": {
512
+ "group_size": {
513
+ "4": 128,
514
+ "2": 128
515
+ },
516
+ "bits": [
517
+ 4,
518
+ 2
519
+ ],
520
+ "bits_prop": [
521
+ 0.05,
522
+ 0.95
523
+ ],
524
+ "scale_bits": 4
525
+ },
526
+ "down_proj": {
527
+ "group_size": {
528
+ "4": 128,
529
+ "2": 128
530
+ },
531
+ "bits": [
532
+ 4,
533
+ 2
534
+ ],
535
+ "bits_prop": [
536
+ 0.4,
537
+ 0.6
538
+ ],
539
+ "scale_bits": 4
540
+ }
541
+ },
542
+ "model.layers.5": {
543
+ "accuracy": 0.8652863502502441,
544
+ "total_bits": 142490128,
545
+ "q_proj": {
546
+ "group_size": {
547
+ "4": 128,
548
+ "2": 128
549
+ },
550
+ "bits": [
551
+ 4,
552
+ 2
553
+ ],
554
+ "bits_prop": [
555
+ 0.1,
556
+ 0.9
557
+ ],
558
+ "scale_bits": 4
559
+ },
560
+ "k_proj": {
561
+ "group_size": {
562
+ "4": 128,
563
+ "2": 128
564
+ },
565
+ "bits": [
566
+ 4,
567
+ 2
568
+ ],
569
+ "bits_prop": [
570
+ 0.1,
571
+ 0.9
572
+ ],
573
+ "scale_bits": 4
574
+ },
575
+ "v_proj": {
576
+ "group_size": {
577
+ "4": 128
578
+ },
579
+ "bits": [
580
+ 4
581
+ ],
582
+ "bits_prop": [
583
+ 1.0
584
+ ],
585
+ "scale_bits": 4
586
+ },
587
+ "o_proj": {
588
+ "group_size": {
589
+ "4": 128,
590
+ "2": 128
591
+ },
592
+ "bits": [
593
+ 4,
594
+ 2
595
+ ],
596
+ "bits_prop": [
597
+ 0.1,
598
+ 0.9
599
+ ],
600
+ "scale_bits": 4
601
+ },
602
+ "up_proj": {
603
+ "group_size": {
604
+ "4": 128,
605
+ "2": 128
606
+ },
607
+ "bits": [
608
+ 4,
609
+ 2
610
+ ],
611
+ "bits_prop": [
612
+ 0.1,
613
+ 0.9
614
+ ],
615
+ "scale_bits": 4
616
+ },
617
+ "gate_proj": {
618
+ "group_size": {
619
+ "4": 128,
620
+ "2": 128
621
+ },
622
+ "bits": [
623
+ 4,
624
+ 2
625
+ ],
626
+ "bits_prop": [
627
+ 0.1,
628
+ 0.9
629
+ ],
630
+ "scale_bits": 4
631
+ },
632
+ "down_proj": {
633
+ "group_size": {
634
+ "4": 128
635
+ },
636
+ "bits": [
637
+ 4
638
+ ],
639
+ "bits_prop": [
640
+ 1.0
641
+ ],
642
+ "scale_bits": 4
643
+ }
644
+ },
645
+ "model.layers.6": {
646
+ "accuracy": 0.9044299125671387,
647
+ "total_bits": 108214800,
648
+ "q_proj": {
649
+ "group_size": {
650
+ "4": 128,
651
+ "2": 128
652
+ },
653
+ "bits": [
654
+ 4,
655
+ 2
656
+ ],
657
+ "bits_prop": [
658
+ 0.01,
659
+ 0.99
660
+ ],
661
+ "scale_bits": 4
662
+ },
663
+ "k_proj": {
664
+ "group_size": {
665
+ "4": 128,
666
+ "2": 128
667
+ },
668
+ "bits": [
669
+ 4,
670
+ 2
671
+ ],
672
+ "bits_prop": [
673
+ 0.01,
674
+ 0.99
675
+ ],
676
+ "scale_bits": 4
677
+ },
678
+ "v_proj": {
679
+ "group_size": {
680
+ "4": 128,
681
+ "2": 128
682
+ },
683
+ "bits": [
684
+ 4,
685
+ 2
686
+ ],
687
+ "bits_prop": [
688
+ 0.05,
689
+ 0.95
690
+ ],
691
+ "scale_bits": 4
692
+ },
693
+ "o_proj": {
694
+ "group_size": {
695
+ "4": 128,
696
+ "2": 128
697
+ },
698
+ "bits": [
699
+ 4,
700
+ 2
701
+ ],
702
+ "bits_prop": [
703
+ 0.02,
704
+ 0.98
705
+ ],
706
+ "scale_bits": 4
707
+ },
708
+ "up_proj": {
709
+ "group_size": {
710
+ "4": 128,
711
+ "2": 128
712
+ },
713
+ "bits": [
714
+ 4,
715
+ 2
716
+ ],
717
+ "bits_prop": [
718
+ 0.01,
719
+ 0.99
720
+ ],
721
+ "scale_bits": 4
722
+ },
723
+ "gate_proj": {
724
+ "group_size": {
725
+ "4": 128,
726
+ "2": 128
727
+ },
728
+ "bits": [
729
+ 4,
730
+ 2
731
+ ],
732
+ "bits_prop": [
733
+ 0.01,
734
+ 0.99
735
+ ],
736
+ "scale_bits": 4
737
+ },
738
+ "down_proj": {
739
+ "group_size": {
740
+ "4": 128,
741
+ "2": 128
742
+ },
743
+ "bits": [
744
+ 4,
745
+ 2
746
+ ],
747
+ "bits_prop": [
748
+ 0.02,
749
+ 0.98
750
+ ],
751
+ "scale_bits": 4
752
+ }
753
+ },
754
+ "model.layers.7": {
755
+ "accuracy": 0.9182649850845337,
756
+ "total_bits": 112409104,
757
+ "q_proj": {
758
+ "group_size": {
759
+ "4": 128,
760
+ "2": 128
761
+ },
762
+ "bits": [
763
+ 4,
764
+ 2
765
+ ],
766
+ "bits_prop": [
767
+ 0.01,
768
+ 0.99
769
+ ],
770
+ "scale_bits": 4
771
+ },
772
+ "k_proj": {
773
+ "group_size": {
774
+ "4": 128,
775
+ "2": 128
776
+ },
777
+ "bits": [
778
+ 4,
779
+ 2
780
+ ],
781
+ "bits_prop": [
782
+ 0.01,
783
+ 0.99
784
+ ],
785
+ "scale_bits": 4
786
+ },
787
+ "v_proj": {
788
+ "group_size": {
789
+ "4": 128,
790
+ "2": 128
791
+ },
792
+ "bits": [
793
+ 4,
794
+ 2
795
+ ],
796
+ "bits_prop": [
797
+ 0.15,
798
+ 0.85
799
+ ],
800
+ "scale_bits": 4
801
+ },
802
+ "o_proj": {
803
+ "group_size": {
804
+ "4": 128,
805
+ "2": 128
806
+ },
807
+ "bits": [
808
+ 4,
809
+ 2
810
+ ],
811
+ "bits_prop": [
812
+ 0.01,
813
+ 0.99
814
+ ],
815
+ "scale_bits": 4
816
+ },
817
+ "up_proj": {
818
+ "group_size": {
819
+ "4": 128,
820
+ "2": 128
821
+ },
822
+ "bits": [
823
+ 4,
824
+ 2
825
+ ],
826
+ "bits_prop": [
827
+ 0.01,
828
+ 0.99
829
+ ],
830
+ "scale_bits": 4
831
+ },
832
+ "gate_proj": {
833
+ "group_size": {
834
+ "4": 128,
835
+ "2": 128
836
+ },
837
+ "bits": [
838
+ 4,
839
+ 2
840
+ ],
841
+ "bits_prop": [
842
+ 0.01,
843
+ 0.99
844
+ ],
845
+ "scale_bits": 4
846
+ },
847
+ "down_proj": {
848
+ "group_size": {
849
+ "4": 128,
850
+ "2": 128
851
+ },
852
+ "bits": [
853
+ 4,
854
+ 2
855
+ ],
856
+ "bits_prop": [
857
+ 0.15,
858
+ 0.85
859
+ ],
860
+ "scale_bits": 4
861
+ }
862
+ },
863
+ "model.layers.8": {
864
+ "accuracy": 0.9104313850402832,
865
+ "total_bits": 111884816,
866
+ "q_proj": {
867
+ "group_size": {
868
+ "4": 128,
869
+ "2": 128
870
+ },
871
+ "bits": [
872
+ 4,
873
+ 2
874
+ ],
875
+ "bits_prop": [
876
+ 0.01,
877
+ 0.99
878
+ ],
879
+ "scale_bits": 4
880
+ },
881
+ "k_proj": {
882
+ "group_size": {
883
+ "4": 128,
884
+ "2": 128
885
+ },
886
+ "bits": [
887
+ 4,
888
+ 2
889
+ ],
890
+ "bits_prop": [
891
+ 0.01,
892
+ 0.99
893
+ ],
894
+ "scale_bits": 4
895
+ },
896
+ "v_proj": {
897
+ "group_size": {
898
+ "4": 128,
899
+ "2": 128
900
+ },
901
+ "bits": [
902
+ 4,
903
+ 2
904
+ ],
905
+ "bits_prop": [
906
+ 0.1,
907
+ 0.9
908
+ ],
909
+ "scale_bits": 4
910
+ },
911
+ "o_proj": {
912
+ "group_size": {
913
+ "4": 128,
914
+ "2": 128
915
+ },
916
+ "bits": [
917
+ 4,
918
+ 2
919
+ ],
920
+ "bits_prop": [
921
+ 0.01,
922
+ 0.99
923
+ ],
924
+ "scale_bits": 4
925
+ },
926
+ "up_proj": {
927
+ "group_size": {
928
+ "4": 128,
929
+ "2": 128
930
+ },
931
+ "bits": [
932
+ 4,
933
+ 2
934
+ ],
935
+ "bits_prop": [
936
+ 0.01,
937
+ 0.99
938
+ ],
939
+ "scale_bits": 4
940
+ },
941
+ "gate_proj": {
942
+ "group_size": {
943
+ "4": 128,
944
+ "2": 128
945
+ },
946
+ "bits": [
947
+ 4,
948
+ 2
949
+ ],
950
+ "bits_prop": [
951
+ 0.01,
952
+ 0.99
953
+ ],
954
+ "scale_bits": 4
955
+ },
956
+ "down_proj": {
957
+ "group_size": {
958
+ "4": 128,
959
+ "2": 128
960
+ },
961
+ "bits": [
962
+ 4,
963
+ 2
964
+ ],
965
+ "bits_prop": [
966
+ 0.15,
967
+ 0.85
968
+ ],
969
+ "scale_bits": 4
970
+ }
971
+ },
972
+ "model.layers.9": {
973
+ "accuracy": 0.9092328548431396,
974
+ "total_bits": 112409104,
975
+ "q_proj": {
976
+ "group_size": {
977
+ "4": 128,
978
+ "2": 128
979
+ },
980
+ "bits": [
981
+ 4,
982
+ 2
983
+ ],
984
+ "bits_prop": [
985
+ 0.01,
986
+ 0.99
987
+ ],
988
+ "scale_bits": 4
989
+ },
990
+ "k_proj": {
991
+ "group_size": {
992
+ "4": 128,
993
+ "2": 128
994
+ },
995
+ "bits": [
996
+ 4,
997
+ 2
998
+ ],
999
+ "bits_prop": [
1000
+ 0.01,
1001
+ 0.99
1002
+ ],
1003
+ "scale_bits": 4
1004
+ },
1005
+ "v_proj": {
1006
+ "group_size": {
1007
+ "4": 128,
1008
+ "2": 128
1009
+ },
1010
+ "bits": [
1011
+ 4,
1012
+ 2
1013
+ ],
1014
+ "bits_prop": [
1015
+ 0.15,
1016
+ 0.85
1017
+ ],
1018
+ "scale_bits": 4
1019
+ },
1020
+ "o_proj": {
1021
+ "group_size": {
1022
+ "4": 128,
1023
+ "2": 128
1024
+ },
1025
+ "bits": [
1026
+ 4,
1027
+ 2
1028
+ ],
1029
+ "bits_prop": [
1030
+ 0.01,
1031
+ 0.99
1032
+ ],
1033
+ "scale_bits": 4
1034
+ },
1035
+ "up_proj": {
1036
+ "group_size": {
1037
+ "4": 128,
1038
+ "2": 128
1039
+ },
1040
+ "bits": [
1041
+ 4,
1042
+ 2
1043
+ ],
1044
+ "bits_prop": [
1045
+ 0.01,
1046
+ 0.99
1047
+ ],
1048
+ "scale_bits": 4
1049
+ },
1050
+ "gate_proj": {
1051
+ "group_size": {
1052
+ "4": 128,
1053
+ "2": 128
1054
+ },
1055
+ "bits": [
1056
+ 4,
1057
+ 2
1058
+ ],
1059
+ "bits_prop": [
1060
+ 0.01,
1061
+ 0.99
1062
+ ],
1063
+ "scale_bits": 4
1064
+ },
1065
+ "down_proj": {
1066
+ "group_size": {
1067
+ "4": 128,
1068
+ "2": 128
1069
+ },
1070
+ "bits": [
1071
+ 4,
1072
+ 2
1073
+ ],
1074
+ "bits_prop": [
1075
+ 0.15,
1076
+ 0.85
1077
+ ],
1078
+ "scale_bits": 4
1079
+ }
1080
+ },
1081
+ "model.layers.10": {
1082
+ "accuracy": 0.9016604423522949,
1083
+ "total_bits": 112409104,
1084
+ "q_proj": {
1085
+ "group_size": {
1086
+ "4": 128,
1087
+ "2": 128
1088
+ },
1089
+ "bits": [
1090
+ 4,
1091
+ 2
1092
+ ],
1093
+ "bits_prop": [
1094
+ 0.01,
1095
+ 0.99
1096
+ ],
1097
+ "scale_bits": 4
1098
+ },
1099
+ "k_proj": {
1100
+ "group_size": {
1101
+ "4": 128,
1102
+ "2": 128
1103
+ },
1104
+ "bits": [
1105
+ 4,
1106
+ 2
1107
+ ],
1108
+ "bits_prop": [
1109
+ 0.01,
1110
+ 0.99
1111
+ ],
1112
+ "scale_bits": 4
1113
+ },
1114
+ "v_proj": {
1115
+ "group_size": {
1116
+ "4": 128,
1117
+ "2": 128
1118
+ },
1119
+ "bits": [
1120
+ 4,
1121
+ 2
1122
+ ],
1123
+ "bits_prop": [
1124
+ 0.15,
1125
+ 0.85
1126
+ ],
1127
+ "scale_bits": 4
1128
+ },
1129
+ "o_proj": {
1130
+ "group_size": {
1131
+ "4": 128,
1132
+ "2": 128
1133
+ },
1134
+ "bits": [
1135
+ 4,
1136
+ 2
1137
+ ],
1138
+ "bits_prop": [
1139
+ 0.01,
1140
+ 0.99
1141
+ ],
1142
+ "scale_bits": 4
1143
+ },
1144
+ "up_proj": {
1145
+ "group_size": {
1146
+ "4": 128,
1147
+ "2": 128
1148
+ },
1149
+ "bits": [
1150
+ 4,
1151
+ 2
1152
+ ],
1153
+ "bits_prop": [
1154
+ 0.01,
1155
+ 0.99
1156
+ ],
1157
+ "scale_bits": 4
1158
+ },
1159
+ "gate_proj": {
1160
+ "group_size": {
1161
+ "4": 128,
1162
+ "2": 128
1163
+ },
1164
+ "bits": [
1165
+ 4,
1166
+ 2
1167
+ ],
1168
+ "bits_prop": [
1169
+ 0.01,
1170
+ 0.99
1171
+ ],
1172
+ "scale_bits": 4
1173
+ },
1174
+ "down_proj": {
1175
+ "group_size": {
1176
+ "4": 128,
1177
+ "2": 128
1178
+ },
1179
+ "bits": [
1180
+ 4,
1181
+ 2
1182
+ ],
1183
+ "bits_prop": [
1184
+ 0.15,
1185
+ 0.85
1186
+ ],
1187
+ "scale_bits": 4
1188
+ }
1189
+ },
1190
+ "model.layers.11": {
1191
+ "accuracy": 0.9087564945220947,
1192
+ "total_bits": 120273424,
1193
+ "q_proj": {
1194
+ "group_size": {
1195
+ "4": 128,
1196
+ "2": 128
1197
+ },
1198
+ "bits": [
1199
+ 4,
1200
+ 2
1201
+ ],
1202
+ "bits_prop": [
1203
+ 0.05,
1204
+ 0.95
1205
+ ],
1206
+ "scale_bits": 4
1207
+ },
1208
+ "k_proj": {
1209
+ "group_size": {
1210
+ "4": 128,
1211
+ "2": 128
1212
+ },
1213
+ "bits": [
1214
+ 4,
1215
+ 2
1216
+ ],
1217
+ "bits_prop": [
1218
+ 0.05,
1219
+ 0.95
1220
+ ],
1221
+ "scale_bits": 4
1222
+ },
1223
+ "v_proj": {
1224
+ "group_size": {
1225
+ "4": 128,
1226
+ "2": 128
1227
+ },
1228
+ "bits": [
1229
+ 4,
1230
+ 2
1231
+ ],
1232
+ "bits_prop": [
1233
+ 0.4,
1234
+ 0.6
1235
+ ],
1236
+ "scale_bits": 4
1237
+ },
1238
+ "o_proj": {
1239
+ "group_size": {
1240
+ "4": 128,
1241
+ "2": 128
1242
+ },
1243
+ "bits": [
1244
+ 4,
1245
+ 2
1246
+ ],
1247
+ "bits_prop": [
1248
+ 0.05,
1249
+ 0.95
1250
+ ],
1251
+ "scale_bits": 4
1252
+ },
1253
+ "up_proj": {
1254
+ "group_size": {
1255
+ "4": 128,
1256
+ "2": 128
1257
+ },
1258
+ "bits": [
1259
+ 4,
1260
+ 2
1261
+ ],
1262
+ "bits_prop": [
1263
+ 0.05,
1264
+ 0.95
1265
+ ],
1266
+ "scale_bits": 4
1267
+ },
1268
+ "gate_proj": {
1269
+ "group_size": {
1270
+ "4": 128,
1271
+ "2": 128
1272
+ },
1273
+ "bits": [
1274
+ 4,
1275
+ 2
1276
+ ],
1277
+ "bits_prop": [
1278
+ 0.05,
1279
+ 0.95
1280
+ ],
1281
+ "scale_bits": 4
1282
+ },
1283
+ "down_proj": {
1284
+ "group_size": {
1285
+ "4": 128,
1286
+ "2": 128
1287
+ },
1288
+ "bits": [
1289
+ 4,
1290
+ 2
1291
+ ],
1292
+ "bits_prop": [
1293
+ 0.4,
1294
+ 0.6
1295
+ ],
1296
+ "scale_bits": 4
1297
+ }
1298
+ },
1299
+ "model.layers.12": {
1300
+ "accuracy": 0.9003274440765381,
1301
+ "total_bits": 120273424,
1302
+ "q_proj": {
1303
+ "group_size": {
1304
+ "4": 128,
1305
+ "2": 128
1306
+ },
1307
+ "bits": [
1308
+ 4,
1309
+ 2
1310
+ ],
1311
+ "bits_prop": [
1312
+ 0.05,
1313
+ 0.95
1314
+ ],
1315
+ "scale_bits": 4
1316
+ },
1317
+ "k_proj": {
1318
+ "group_size": {
1319
+ "4": 128,
1320
+ "2": 128
1321
+ },
1322
+ "bits": [
1323
+ 4,
1324
+ 2
1325
+ ],
1326
+ "bits_prop": [
1327
+ 0.05,
1328
+ 0.95
1329
+ ],
1330
+ "scale_bits": 4
1331
+ },
1332
+ "v_proj": {
1333
+ "group_size": {
1334
+ "4": 128,
1335
+ "2": 128
1336
+ },
1337
+ "bits": [
1338
+ 4,
1339
+ 2
1340
+ ],
1341
+ "bits_prop": [
1342
+ 0.4,
1343
+ 0.6
1344
+ ],
1345
+ "scale_bits": 4
1346
+ },
1347
+ "o_proj": {
1348
+ "group_size": {
1349
+ "4": 128,
1350
+ "2": 128
1351
+ },
1352
+ "bits": [
1353
+ 4,
1354
+ 2
1355
+ ],
1356
+ "bits_prop": [
1357
+ 0.05,
1358
+ 0.95
1359
+ ],
1360
+ "scale_bits": 4
1361
+ },
1362
+ "up_proj": {
1363
+ "group_size": {
1364
+ "4": 128,
1365
+ "2": 128
1366
+ },
1367
+ "bits": [
1368
+ 4,
1369
+ 2
1370
+ ],
1371
+ "bits_prop": [
1372
+ 0.05,
1373
+ 0.95
1374
+ ],
1375
+ "scale_bits": 4
1376
+ },
1377
+ "gate_proj": {
1378
+ "group_size": {
1379
+ "4": 128,
1380
+ "2": 128
1381
+ },
1382
+ "bits": [
1383
+ 4,
1384
+ 2
1385
+ ],
1386
+ "bits_prop": [
1387
+ 0.05,
1388
+ 0.95
1389
+ ],
1390
+ "scale_bits": 4
1391
+ },
1392
+ "down_proj": {
1393
+ "group_size": {
1394
+ "4": 128,
1395
+ "2": 128
1396
+ },
1397
+ "bits": [
1398
+ 4,
1399
+ 2
1400
+ ],
1401
+ "bits_prop": [
1402
+ 0.4,
1403
+ 0.6
1404
+ ],
1405
+ "scale_bits": 4
1406
+ }
1407
+ },
1408
+ "model.layers.13": {
1409
+ "accuracy": 0.9268591403961182,
1410
+ "total_bits": 151271952,
1411
+ "q_proj": {
1412
+ "group_size": {
1413
+ "4": 128,
1414
+ "2": 128
1415
+ },
1416
+ "bits": [
1417
+ 4,
1418
+ 2
1419
+ ],
1420
+ "bits_prop": [
1421
+ 0.25,
1422
+ 0.75
1423
+ ],
1424
+ "scale_bits": 4
1425
+ },
1426
+ "k_proj": {
1427
+ "group_size": {
1428
+ "4": 128,
1429
+ "2": 128
1430
+ },
1431
+ "bits": [
1432
+ 4,
1433
+ 2
1434
+ ],
1435
+ "bits_prop": [
1436
+ 0.25,
1437
+ 0.75
1438
+ ],
1439
+ "scale_bits": 4
1440
+ },
1441
+ "v_proj": {
1442
+ "group_size": {
1443
+ "4": 128
1444
+ },
1445
+ "bits": [
1446
+ 4
1447
+ ],
1448
+ "bits_prop": [
1449
+ 1.0
1450
+ ],
1451
+ "scale_bits": 4
1452
+ },
1453
+ "o_proj": {
1454
+ "group_size": {
1455
+ "4": 128,
1456
+ "2": 128
1457
+ },
1458
+ "bits": [
1459
+ 4,
1460
+ 2
1461
+ ],
1462
+ "bits_prop": [
1463
+ 0.25,
1464
+ 0.75
1465
+ ],
1466
+ "scale_bits": 4
1467
+ },
1468
+ "up_proj": {
1469
+ "group_size": {
1470
+ "4": 128,
1471
+ "2": 128
1472
+ },
1473
+ "bits": [
1474
+ 4,
1475
+ 2
1476
+ ],
1477
+ "bits_prop": [
1478
+ 0.25,
1479
+ 0.75
1480
+ ],
1481
+ "scale_bits": 4
1482
+ },
1483
+ "gate_proj": {
1484
+ "group_size": {
1485
+ "4": 128,
1486
+ "2": 128
1487
+ },
1488
+ "bits": [
1489
+ 4,
1490
+ 2
1491
+ ],
1492
+ "bits_prop": [
1493
+ 0.25,
1494
+ 0.75
1495
+ ],
1496
+ "scale_bits": 4
1497
+ },
1498
+ "down_proj": {
1499
+ "group_size": {
1500
+ "4": 128
1501
+ },
1502
+ "bits": [
1503
+ 4
1504
+ ],
1505
+ "bits_prop": [
1506
+ 1
1507
+ ],
1508
+ "scale_bits": 4
1509
+ }
1510
+ },
1511
+ "model.layers.14": {
1512
+ "accuracy": 0.9370865821838379,
1513
+ "total_bits": 168835600,
1514
+ "q_proj": {
1515
+ "group_size": {
1516
+ "4": 128,
1517
+ "2": 128
1518
+ },
1519
+ "bits": [
1520
+ 4,
1521
+ 2
1522
+ ],
1523
+ "bits_prop": [
1524
+ 0.5,
1525
+ 0.5
1526
+ ],
1527
+ "scale_bits": 4
1528
+ },
1529
+ "k_proj": {
1530
+ "group_size": {
1531
+ "4": 128,
1532
+ "2": 128
1533
+ },
1534
+ "bits": [
1535
+ 4,
1536
+ 2
1537
+ ],
1538
+ "bits_prop": [
1539
+ 0.5,
1540
+ 0.5
1541
+ ],
1542
+ "scale_bits": 4
1543
+ },
1544
+ "v_proj": {
1545
+ "group_size": {
1546
+ "4": 128
1547
+ },
1548
+ "bits": [
1549
+ 4
1550
+ ],
1551
+ "bits_prop": [
1552
+ 1.0
1553
+ ],
1554
+ "scale_bits": 4
1555
+ },
1556
+ "o_proj": {
1557
+ "group_size": {
1558
+ "4": 128,
1559
+ "2": 128
1560
+ },
1561
+ "bits": [
1562
+ 4,
1563
+ 2
1564
+ ],
1565
+ "bits_prop": [
1566
+ 0.5,
1567
+ 0.5
1568
+ ],
1569
+ "scale_bits": 4
1570
+ },
1571
+ "up_proj": {
1572
+ "group_size": {
1573
+ "4": 128,
1574
+ "2": 128
1575
+ },
1576
+ "bits": [
1577
+ 4,
1578
+ 2
1579
+ ],
1580
+ "bits_prop": [
1581
+ 0.5,
1582
+ 0.5
1583
+ ],
1584
+ "scale_bits": 4
1585
+ },
1586
+ "gate_proj": {
1587
+ "group_size": {
1588
+ "4": 128,
1589
+ "2": 128
1590
+ },
1591
+ "bits": [
1592
+ 4,
1593
+ 2
1594
+ ],
1595
+ "bits_prop": [
1596
+ 0.5,
1597
+ 0.5
1598
+ ],
1599
+ "scale_bits": 4
1600
+ },
1601
+ "down_proj": {
1602
+ "group_size": {
1603
+ "4": 128
1604
+ },
1605
+ "bits": [
1606
+ 4
1607
+ ],
1608
+ "bits_prop": [
1609
+ 1.0
1610
+ ],
1611
+ "scale_bits": 4
1612
+ }
1613
+ },
1614
+ "model.layers.15": {
1615
+ "accuracy": 0.954687237739563,
1616
+ "total_bits": 186399248,
1617
+ "q_proj": {
1618
+ "group_size": {
1619
+ "4": 128,
1620
+ "2": 128
1621
+ },
1622
+ "bits": [
1623
+ 4,
1624
+ 2
1625
+ ],
1626
+ "bits_prop": [
1627
+ 0.75,
1628
+ 0.25
1629
+ ],
1630
+ "scale_bits": 4
1631
+ },
1632
+ "k_proj": {
1633
+ "group_size": {
1634
+ "4": 128,
1635
+ "2": 128
1636
+ },
1637
+ "bits": [
1638
+ 4,
1639
+ 2
1640
+ ],
1641
+ "bits_prop": [
1642
+ 0.75,
1643
+ 0.25
1644
+ ],
1645
+ "scale_bits": 4
1646
+ },
1647
+ "v_proj": {
1648
+ "group_size": {
1649
+ "4": 128
1650
+ },
1651
+ "bits": [
1652
+ 4
1653
+ ],
1654
+ "bits_prop": [
1655
+ 1.0
1656
+ ],
1657
+ "scale_bits": 4
1658
+ },
1659
+ "o_proj": {
1660
+ "group_size": {
1661
+ "4": 128,
1662
+ "2": 128
1663
+ },
1664
+ "bits": [
1665
+ 4,
1666
+ 2
1667
+ ],
1668
+ "bits_prop": [
1669
+ 0.75,
1670
+ 0.25
1671
+ ],
1672
+ "scale_bits": 4
1673
+ },
1674
+ "up_proj": {
1675
+ "group_size": {
1676
+ "4": 128,
1677
+ "2": 128
1678
+ },
1679
+ "bits": [
1680
+ 4,
1681
+ 2
1682
+ ],
1683
+ "bits_prop": [
1684
+ 0.75,
1685
+ 0.25
1686
+ ],
1687
+ "scale_bits": 4
1688
+ },
1689
+ "gate_proj": {
1690
+ "group_size": {
1691
+ "4": 128,
1692
+ "2": 128
1693
+ },
1694
+ "bits": [
1695
+ 4,
1696
+ 2
1697
+ ],
1698
+ "bits_prop": [
1699
+ 0.75,
1700
+ 0.25
1701
+ ],
1702
+ "scale_bits": 4
1703
+ },
1704
+ "down_proj": {
1705
+ "group_size": {
1706
+ "4": 128
1707
+ },
1708
+ "bits": [
1709
+ 4
1710
+ ],
1711
+ "bits_prop": [
1712
+ 1.0
1713
+ ],
1714
+ "scale_bits": 4
1715
+ }
1716
+ },
1717
+ "model.layers.16": {
1718
+ "accuracy": 0.9543647766113281,
1719
+ "total_bits": 186399248,
1720
+ "q_proj": {
1721
+ "group_size": {
1722
+ "4": 128,
1723
+ "2": 128
1724
+ },
1725
+ "bits": [
1726
+ 4,
1727
+ 2
1728
+ ],
1729
+ "bits_prop": [
1730
+ 0.75,
1731
+ 0.25
1732
+ ],
1733
+ "scale_bits": 4
1734
+ },
1735
+ "k_proj": {
1736
+ "group_size": {
1737
+ "4": 128,
1738
+ "2": 128
1739
+ },
1740
+ "bits": [
1741
+ 4,
1742
+ 2
1743
+ ],
1744
+ "bits_prop": [
1745
+ 0.75,
1746
+ 0.25
1747
+ ],
1748
+ "scale_bits": 4
1749
+ },
1750
+ "v_proj": {
1751
+ "group_size": {
1752
+ "4": 128
1753
+ },
1754
+ "bits": [
1755
+ 4
1756
+ ],
1757
+ "bits_prop": [
1758
+ 1.0
1759
+ ],
1760
+ "scale_bits": 4
1761
+ },
1762
+ "o_proj": {
1763
+ "group_size": {
1764
+ "4": 128,
1765
+ "2": 128
1766
+ },
1767
+ "bits": [
1768
+ 4,
1769
+ 2
1770
+ ],
1771
+ "bits_prop": [
1772
+ 0.75,
1773
+ 0.25
1774
+ ],
1775
+ "scale_bits": 4
1776
+ },
1777
+ "up_proj": {
1778
+ "group_size": {
1779
+ "4": 128,
1780
+ "2": 128
1781
+ },
1782
+ "bits": [
1783
+ 4,
1784
+ 2
1785
+ ],
1786
+ "bits_prop": [
1787
+ 0.75,
1788
+ 0.25
1789
+ ],
1790
+ "scale_bits": 4
1791
+ },
1792
+ "gate_proj": {
1793
+ "group_size": {
1794
+ "4": 128,
1795
+ "2": 128
1796
+ },
1797
+ "bits": [
1798
+ 4,
1799
+ 2
1800
+ ],
1801
+ "bits_prop": [
1802
+ 0.75,
1803
+ 0.25
1804
+ ],
1805
+ "scale_bits": 4
1806
+ },
1807
+ "down_proj": {
1808
+ "group_size": {
1809
+ "4": 128
1810
+ },
1811
+ "bits": [
1812
+ 4
1813
+ ],
1814
+ "bits_prop": [
1815
+ 1.0
1816
+ ],
1817
+ "scale_bits": 4
1818
+ }
1819
+ },
1820
+ "model.layers.17": {
1821
+ "accuracy": 0.9534173011779785,
1822
+ "total_bits": 186399248,
1823
+ "q_proj": {
1824
+ "group_size": {
1825
+ "4": 128,
1826
+ "2": 128
1827
+ },
1828
+ "bits": [
1829
+ 4,
1830
+ 2
1831
+ ],
1832
+ "bits_prop": [
1833
+ 0.75,
1834
+ 0.25
1835
+ ],
1836
+ "scale_bits": 4
1837
+ },
1838
+ "k_proj": {
1839
+ "group_size": {
1840
+ "4": 128,
1841
+ "2": 128
1842
+ },
1843
+ "bits": [
1844
+ 4,
1845
+ 2
1846
+ ],
1847
+ "bits_prop": [
1848
+ 0.75,
1849
+ 0.25
1850
+ ],
1851
+ "scale_bits": 4
1852
+ },
1853
+ "v_proj": {
1854
+ "group_size": {
1855
+ "4": 128
1856
+ },
1857
+ "bits": [
1858
+ 4
1859
+ ],
1860
+ "bits_prop": [
1861
+ 1.0
1862
+ ],
1863
+ "scale_bits": 4
1864
+ },
1865
+ "o_proj": {
1866
+ "group_size": {
1867
+ "4": 128,
1868
+ "2": 128
1869
+ },
1870
+ "bits": [
1871
+ 4,
1872
+ 2
1873
+ ],
1874
+ "bits_prop": [
1875
+ 0.75,
1876
+ 0.25
1877
+ ],
1878
+ "scale_bits": 4
1879
+ },
1880
+ "up_proj": {
1881
+ "group_size": {
1882
+ "4": 128,
1883
+ "2": 128
1884
+ },
1885
+ "bits": [
1886
+ 4,
1887
+ 2
1888
+ ],
1889
+ "bits_prop": [
1890
+ 0.75,
1891
+ 0.25
1892
+ ],
1893
+ "scale_bits": 4
1894
+ },
1895
+ "gate_proj": {
1896
+ "group_size": {
1897
+ "4": 128,
1898
+ "2": 128
1899
+ },
1900
+ "bits": [
1901
+ 4,
1902
+ 2
1903
+ ],
1904
+ "bits_prop": [
1905
+ 0.75,
1906
+ 0.25
1907
+ ],
1908
+ "scale_bits": 4
1909
+ },
1910
+ "down_proj": {
1911
+ "group_size": {
1912
+ "4": 128
1913
+ },
1914
+ "bits": [
1915
+ 4
1916
+ ],
1917
+ "bits_prop": [
1918
+ 1.0
1919
+ ],
1920
+ "scale_bits": 4
1921
+ }
1922
+ },
1923
+ "model.layers.18": {
1924
+ "accuracy": 0.9755624532699585,
1925
+ "total_bits": 203962896,
1926
+ "q_proj": {
1927
+ "group_size": {
1928
+ "4": 128
1929
+ },
1930
+ "bits": [
1931
+ 4
1932
+ ],
1933
+ "bits_prop": [
1934
+ 1
1935
+ ],
1936
+ "scale_bits": 4
1937
+ },
1938
+ "k_proj": {
1939
+ "group_size": {
1940
+ "4": 128
1941
+ },
1942
+ "bits": [
1943
+ 4
1944
+ ],
1945
+ "bits_prop": [
1946
+ 1
1947
+ ],
1948
+ "scale_bits": 4
1949
+ },
1950
+ "v_proj": {
1951
+ "group_size": {
1952
+ "4": 128
1953
+ },
1954
+ "bits": [
1955
+ 4
1956
+ ],
1957
+ "bits_prop": [
1958
+ 1
1959
+ ],
1960
+ "scale_bits": 4
1961
+ },
1962
+ "o_proj": {
1963
+ "group_size": {
1964
+ "4": 128
1965
+ },
1966
+ "bits": [
1967
+ 4
1968
+ ],
1969
+ "bits_prop": [
1970
+ 1
1971
+ ],
1972
+ "scale_bits": 4
1973
+ },
1974
+ "up_proj": {
1975
+ "group_size": {
1976
+ "4": 128
1977
+ },
1978
+ "bits": [
1979
+ 4
1980
+ ],
1981
+ "bits_prop": [
1982
+ 1
1983
+ ],
1984
+ "scale_bits": 4
1985
+ },
1986
+ "gate_proj": {
1987
+ "group_size": {
1988
+ "4": 128
1989
+ },
1990
+ "bits": [
1991
+ 4
1992
+ ],
1993
+ "bits_prop": [
1994
+ 1
1995
+ ],
1996
+ "scale_bits": 4
1997
+ },
1998
+ "down_proj": {
1999
+ "group_size": {
2000
+ "4": 128
2001
+ },
2002
+ "bits": [
2003
+ 4
2004
+ ],
2005
+ "bits_prop": [
2006
+ 1
2007
+ ],
2008
+ "scale_bits": 4
2009
+ }
2010
+ },
2011
+ "model.layers.19": {
2012
+ "accuracy": 0.9757581949234009,
2013
+ "total_bits": 203962896,
2014
+ "q_proj": {
2015
+ "group_size": {
2016
+ "4": 128
2017
+ },
2018
+ "bits": [
2019
+ 4
2020
+ ],
2021
+ "bits_prop": [
2022
+ 1
2023
+ ],
2024
+ "scale_bits": 4
2025
+ },
2026
+ "k_proj": {
2027
+ "group_size": {
2028
+ "4": 128
2029
+ },
2030
+ "bits": [
2031
+ 4
2032
+ ],
2033
+ "bits_prop": [
2034
+ 1
2035
+ ],
2036
+ "scale_bits": 4
2037
+ },
2038
+ "v_proj": {
2039
+ "group_size": {
2040
+ "4": 128
2041
+ },
2042
+ "bits": [
2043
+ 4
2044
+ ],
2045
+ "bits_prop": [
2046
+ 1
2047
+ ],
2048
+ "scale_bits": 4
2049
+ },
2050
+ "o_proj": {
2051
+ "group_size": {
2052
+ "4": 128
2053
+ },
2054
+ "bits": [
2055
+ 4
2056
+ ],
2057
+ "bits_prop": [
2058
+ 1
2059
+ ],
2060
+ "scale_bits": 4
2061
+ },
2062
+ "up_proj": {
2063
+ "group_size": {
2064
+ "4": 128
2065
+ },
2066
+ "bits": [
2067
+ 4
2068
+ ],
2069
+ "bits_prop": [
2070
+ 1
2071
+ ],
2072
+ "scale_bits": 4
2073
+ },
2074
+ "gate_proj": {
2075
+ "group_size": {
2076
+ "4": 128
2077
+ },
2078
+ "bits": [
2079
+ 4
2080
+ ],
2081
+ "bits_prop": [
2082
+ 1
2083
+ ],
2084
+ "scale_bits": 4
2085
+ },
2086
+ "down_proj": {
2087
+ "group_size": {
2088
+ "4": 128
2089
+ },
2090
+ "bits": [
2091
+ 4
2092
+ ],
2093
+ "bits_prop": [
2094
+ 1
2095
+ ],
2096
+ "scale_bits": 4
2097
+ }
2098
+ },
2099
+ "model.layers.20": {
2100
+ "accuracy": 0.9778097867965698,
2101
+ "total_bits": 203962896,
2102
+ "q_proj": {
2103
+ "group_size": {
2104
+ "4": 128
2105
+ },
2106
+ "bits": [
2107
+ 4
2108
+ ],
2109
+ "bits_prop": [
2110
+ 1
2111
+ ],
2112
+ "scale_bits": 4
2113
+ },
2114
+ "k_proj": {
2115
+ "group_size": {
2116
+ "4": 128
2117
+ },
2118
+ "bits": [
2119
+ 4
2120
+ ],
2121
+ "bits_prop": [
2122
+ 1
2123
+ ],
2124
+ "scale_bits": 4
2125
+ },
2126
+ "v_proj": {
2127
+ "group_size": {
2128
+ "4": 128
2129
+ },
2130
+ "bits": [
2131
+ 4
2132
+ ],
2133
+ "bits_prop": [
2134
+ 1
2135
+ ],
2136
+ "scale_bits": 4
2137
+ },
2138
+ "o_proj": {
2139
+ "group_size": {
2140
+ "4": 128
2141
+ },
2142
+ "bits": [
2143
+ 4
2144
+ ],
2145
+ "bits_prop": [
2146
+ 1
2147
+ ],
2148
+ "scale_bits": 4
2149
+ },
2150
+ "up_proj": {
2151
+ "group_size": {
2152
+ "4": 128
2153
+ },
2154
+ "bits": [
2155
+ 4
2156
+ ],
2157
+ "bits_prop": [
2158
+ 1
2159
+ ],
2160
+ "scale_bits": 4
2161
+ },
2162
+ "gate_proj": {
2163
+ "group_size": {
2164
+ "4": 128
2165
+ },
2166
+ "bits": [
2167
+ 4
2168
+ ],
2169
+ "bits_prop": [
2170
+ 1
2171
+ ],
2172
+ "scale_bits": 4
2173
+ },
2174
+ "down_proj": {
2175
+ "group_size": {
2176
+ "4": 128
2177
+ },
2178
+ "bits": [
2179
+ 4
2180
+ ],
2181
+ "bits_prop": [
2182
+ 1
2183
+ ],
2184
+ "scale_bits": 4
2185
+ }
2186
+ },
2187
+ "model.layers.21": {
2188
+ "accuracy": 0.9768804311752319,
2189
+ "total_bits": 203962896,
2190
+ "q_proj": {
2191
+ "group_size": {
2192
+ "4": 128
2193
+ },
2194
+ "bits": [
2195
+ 4
2196
+ ],
2197
+ "bits_prop": [
2198
+ 1
2199
+ ],
2200
+ "scale_bits": 4
2201
+ },
2202
+ "k_proj": {
2203
+ "group_size": {
2204
+ "4": 128
2205
+ },
2206
+ "bits": [
2207
+ 4
2208
+ ],
2209
+ "bits_prop": [
2210
+ 1
2211
+ ],
2212
+ "scale_bits": 4
2213
+ },
2214
+ "v_proj": {
2215
+ "group_size": {
2216
+ "4": 128
2217
+ },
2218
+ "bits": [
2219
+ 4
2220
+ ],
2221
+ "bits_prop": [
2222
+ 1
2223
+ ],
2224
+ "scale_bits": 4
2225
+ },
2226
+ "o_proj": {
2227
+ "group_size": {
2228
+ "4": 128
2229
+ },
2230
+ "bits": [
2231
+ 4
2232
+ ],
2233
+ "bits_prop": [
2234
+ 1
2235
+ ],
2236
+ "scale_bits": 4
2237
+ },
2238
+ "up_proj": {
2239
+ "group_size": {
2240
+ "4": 128
2241
+ },
2242
+ "bits": [
2243
+ 4
2244
+ ],
2245
+ "bits_prop": [
2246
+ 1
2247
+ ],
2248
+ "scale_bits": 4
2249
+ },
2250
+ "gate_proj": {
2251
+ "group_size": {
2252
+ "4": 128
2253
+ },
2254
+ "bits": [
2255
+ 4
2256
+ ],
2257
+ "bits_prop": [
2258
+ 1
2259
+ ],
2260
+ "scale_bits": 4
2261
+ },
2262
+ "down_proj": {
2263
+ "group_size": {
2264
+ "4": 128
2265
+ },
2266
+ "bits": [
2267
+ 4
2268
+ ],
2269
+ "bits_prop": [
2270
+ 1
2271
+ ],
2272
+ "scale_bits": 4
2273
+ }
2274
+ },
2275
+ "model.layers.22": {
2276
+ "accuracy": 0.9757040739059448,
2277
+ "total_bits": 203962896,
2278
+ "q_proj": {
2279
+ "group_size": {
2280
+ "4": 128
2281
+ },
2282
+ "bits": [
2283
+ 4
2284
+ ],
2285
+ "bits_prop": [
2286
+ 1
2287
+ ],
2288
+ "scale_bits": 4
2289
+ },
2290
+ "k_proj": {
2291
+ "group_size": {
2292
+ "4": 128
2293
+ },
2294
+ "bits": [
2295
+ 4
2296
+ ],
2297
+ "bits_prop": [
2298
+ 1
2299
+ ],
2300
+ "scale_bits": 4
2301
+ },
2302
+ "v_proj": {
2303
+ "group_size": {
2304
+ "4": 128
2305
+ },
2306
+ "bits": [
2307
+ 4
2308
+ ],
2309
+ "bits_prop": [
2310
+ 1
2311
+ ],
2312
+ "scale_bits": 4
2313
+ },
2314
+ "o_proj": {
2315
+ "group_size": {
2316
+ "4": 128
2317
+ },
2318
+ "bits": [
2319
+ 4
2320
+ ],
2321
+ "bits_prop": [
2322
+ 1
2323
+ ],
2324
+ "scale_bits": 4
2325
+ },
2326
+ "up_proj": {
2327
+ "group_size": {
2328
+ "4": 128
2329
+ },
2330
+ "bits": [
2331
+ 4
2332
+ ],
2333
+ "bits_prop": [
2334
+ 1
2335
+ ],
2336
+ "scale_bits": 4
2337
+ },
2338
+ "gate_proj": {
2339
+ "group_size": {
2340
+ "4": 128
2341
+ },
2342
+ "bits": [
2343
+ 4
2344
+ ],
2345
+ "bits_prop": [
2346
+ 1
2347
+ ],
2348
+ "scale_bits": 4
2349
+ },
2350
+ "down_proj": {
2351
+ "group_size": {
2352
+ "4": 128
2353
+ },
2354
+ "bits": [
2355
+ 4
2356
+ ],
2357
+ "bits_prop": [
2358
+ 1
2359
+ ],
2360
+ "scale_bits": 4
2361
+ }
2362
+ },
2363
+ "model.layers.23": {
2364
+ "accuracy": 0.9632000923156738,
2365
+ "total_bits": 203962896,
2366
+ "q_proj": {
2367
+ "group_size": {
2368
+ "4": 128
2369
+ },
2370
+ "bits": [
2371
+ 4
2372
+ ],
2373
+ "bits_prop": [
2374
+ 1
2375
+ ],
2376
+ "scale_bits": 4
2377
+ },
2378
+ "k_proj": {
2379
+ "group_size": {
2380
+ "4": 128
2381
+ },
2382
+ "bits": [
2383
+ 4
2384
+ ],
2385
+ "bits_prop": [
2386
+ 1
2387
+ ],
2388
+ "scale_bits": 4
2389
+ },
2390
+ "v_proj": {
2391
+ "group_size": {
2392
+ "4": 128
2393
+ },
2394
+ "bits": [
2395
+ 4
2396
+ ],
2397
+ "bits_prop": [
2398
+ 1
2399
+ ],
2400
+ "scale_bits": 4
2401
+ },
2402
+ "o_proj": {
2403
+ "group_size": {
2404
+ "4": 128
2405
+ },
2406
+ "bits": [
2407
+ 4
2408
+ ],
2409
+ "bits_prop": [
2410
+ 1
2411
+ ],
2412
+ "scale_bits": 4
2413
+ },
2414
+ "up_proj": {
2415
+ "group_size": {
2416
+ "4": 128
2417
+ },
2418
+ "bits": [
2419
+ 4
2420
+ ],
2421
+ "bits_prop": [
2422
+ 1
2423
+ ],
2424
+ "scale_bits": 4
2425
+ },
2426
+ "gate_proj": {
2427
+ "group_size": {
2428
+ "4": 128
2429
+ },
2430
+ "bits": [
2431
+ 4
2432
+ ],
2433
+ "bits_prop": [
2434
+ 1
2435
+ ],
2436
+ "scale_bits": 4
2437
+ },
2438
+ "down_proj": {
2439
+ "group_size": {
2440
+ "4": 128
2441
+ },
2442
+ "bits": [
2443
+ 4
2444
+ ],
2445
+ "bits_prop": [
2446
+ 1
2447
+ ],
2448
+ "scale_bits": 4
2449
+ }
2450
+ }
2451
+ }
2452
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|im_start|>",
4
+ "<|im_end|>"
5
+ ],
6
+ "eos_token": {
7
+ "content": "<|endoftext|>",
8
+ "lstrip": false,
9
+ "normalized": false,
10
+ "rstrip": false,
11
+ "single_word": false
12
+ },
13
+ "pad_token": {
14
+ "content": "<|endoftext|>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false
19
+ }
20
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "151643": {
5
+ "content": "<|endoftext|>",
6
+ "lstrip": false,
7
+ "normalized": false,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "151644": {
13
+ "content": "<|im_start|>",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "151645": {
21
+ "content": "<|im_end|>",
22
+ "lstrip": false,
23
+ "normalized": false,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ }
28
+ },
29
+ "additional_special_tokens": [
30
+ "<|im_start|>",
31
+ "<|im_end|>"
32
+ ],
33
+ "bos_token": null,
34
+ "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
35
+ "clean_up_tokenization_spaces": false,
36
+ "eos_token": "<|endoftext|>",
37
+ "errors": "replace",
38
+ "model_max_length": 32768,
39
+ "pad_token": "<|endoftext|>",
40
+ "split_special_tokens": false,
41
+ "tokenizer_class": "Qwen2Tokenizer",
42
+ "unk_token": null
43
+ }
vocab.json ADDED
The diff for this file is too large to render. See raw diff