JeremyHibiki commited on
Commit
bc26b66
1 Parent(s): aae8bd7

Update tokenization_chatglm.py

Browse files

https://github.com/huggingface/transformers/blob/8f2b6d5e3dcf40ab0d01f3c8117d1df09e465616/src/transformers/tokenization_utils_base.py#L43

`TensorType` should come from `transformers.utils` instead of `torch`. Otherwise, the tokenizer can't be used alone in an environment without PyTorch installed, where we only need to count tokens.

Files changed (1) hide show
  1. tokenization_chatglm.py +1 -2
tokenization_chatglm.py CHANGED
@@ -3,10 +3,9 @@ import base64
3
  import os
4
  import json
5
  import tiktoken
6
- from torch import TensorType
7
  from typing import List, Optional, Union, Dict, Any
8
  from transformers import PreTrainedTokenizer
9
- from transformers.utils import logging, PaddingStrategy
10
  from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
11
 
12
 
 
3
  import os
4
  import json
5
  import tiktoken
 
6
  from typing import List, Optional, Union, Dict, Any
7
  from transformers import PreTrainedTokenizer
8
+ from transformers.utils import logging, PaddingStrategy, TensorType
9
  from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
10
 
11