Hugging Face's logo

Spaces:
xu-song
/
tokenizer-arena
Running

App Files Community
1
tokenizer-arena
File size: 675 Bytes 
9495a4f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 


## Exception: data did not match any variant of untagged enum ModelWrapper at line 108219 column 3




## The OrderedVocab you are attempting to save contains a hole for index 50254, your vocabulary could be corrupted !


```
The OrderedVocab you are attempting to save contains a hole for index 50254, your vocabulary could be corrupted !
The OrderedVocab you are attempting to save contains a hole for index 50255, your vocabulary could be corrupted !
The OrderedVocab you are attempting to save contains a hole for index 50256, your vocabulary could be corrupted !
```


原因：50254 这些token并未在vocab中定义，只在 `added_tokens` 里定义了。

## ss