BEE-spoke-data
/

BeeTokenizer

Model card Files Files and versions Community

pszemraj commited on Jul 20

Commit

a1f7963

•

1 Parent(s): 85386aa

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -21,7 +21,7 @@ test_string = "When dealing with Varroa destructor mites, it's crucial to admini
 output = tokenizer(test_string)
 print(f"Test string: {test_string}")
-print(f"Tokens:\n\t{output.input_ids}")
 ```

 output = tokenizer(test_string)
 print(f"Test string: {test_string}")
+print(f"Tokens ({len(output.input_ids)}):\n\t{output.input_ids}")
 ```