Erland
/

Mistral-Small-24B-Base-ChatML-2501-bnb-4bit

Text Generation

text-generation-inference

Inference Endpoints

4-bit precision

Model card Files Files and versions Community

Erland commited on 13 days ago

Commit

fa4ea07

·

verified ·

1 Parent(s): e38a8e5

Update tokenizer_config.json

Files changed (1) hide show

tokenizer_config.json +4 -4

tokenizer_config.json CHANGED Viewed

@@ -164,7 +164,7 @@
       "special": true
     },
     "20": {
-      "content": "<SPECIAL_20>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -172,7 +172,7 @@
       "special": true
     },
     "21": {
-      "content": "<SPECIAL_21>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -8025,8 +8025,8 @@
     "[SYSTEM_PROMPT]",
     "[/SYSTEM_PROMPT]",
     "[TOOL_CONTENT]",
-    "<SPECIAL_20>",
-    "<SPECIAL_21>",
     "<SPECIAL_22>",
     "<SPECIAL_23>",
     "<SPECIAL_24>",

       "special": true
     },
     "20": {
+      "content": "<|im_start|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "special": true
     },
     "21": {
+      "content": "<|im_end|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
     "[SYSTEM_PROMPT]",
     "[/SYSTEM_PROMPT]",
     "[TOOL_CONTENT]",
+    "<|im_start|>",
+    "<|im_end|>",
     "<SPECIAL_22>",
     "<SPECIAL_23>",
     "<SPECIAL_24>",