Update README.md

README.md

@@ -4,7 +4,6 @@ datasets:
 - BEE-spoke-data/bees-internal
 language:
 - en
-library_name: transformers
 ---
 
 # BeeTokenizer
@@ -58,3 +57,20 @@ print(f"Tokens:\n\t{output.input_ids}")
  # Offsets
  offsets = output['offset_mapping']
  print(f"Offsets: {offsets}")
+ ```
+
+ This should result in the following (_Nov 2023 version_):
+ 
+ <pre>&gt;&gt;&gt; print(f&quot;Test string: {test_string}&quot;)
+ Test string: When dealing with Varroa destructor mites, it&apos;s crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination.
+ &gt;&gt;&gt; 
+ &gt;&gt;&gt; # Tokens
+ &gt;&gt;&gt; tokens = tokenizer.convert_ids_to_tokens(output[&apos;input_ids&apos;])
+ &gt;&gt;&gt; print(f&quot;Tokens: {tokens}&quot;)
+ Tokens: [&apos;▁When&apos;, &apos;▁dealing&apos;, &apos;▁with&apos;, &apos;▁Varroa&apos;, &apos;▁destructor&apos;, &apos;▁mites,&apos;, &quot;▁it&apos;s&quot;, &apos;▁cru&apos;, &apos;cial&apos;, &apos;▁to&apos;, &apos;▁administer&apos;, &apos;▁the&apos;, &apos;▁right&apos;, &apos;▁acar&apos;, &apos;icides&apos;, &apos;▁during&apos;, &apos;▁the&apos;, &apos;▁late&apos;, &apos;▁autumn&apos;, &apos;▁months,&apos;, &apos;▁but&apos;, &apos;▁only&apos;, &apos;▁after&apos;, &apos;▁ensuring&apos;, &apos;▁that&apos;, &apos;▁the&apos;, &apos;▁worker&apos;, &apos;▁bee&apos;, &apos;▁population&apos;, &apos;▁is&apos;, &apos;▁free&apos;, &apos;▁from&apos;, &apos;▁pesticide&apos;, &apos;▁contamination&apos;, &apos;.&apos;]
+ &gt;&gt;&gt; 
+ &gt;&gt;&gt; # Offsets
+ &gt;&gt;&gt; offsets = output[&apos;offset_mapping&apos;]
+ &gt;&gt;&gt; print(f&quot;Offsets: {offsets}&quot;)
+ Offsets: [(0, 4), (4, 12), (12, 17), (17, 24), (24, 35), (35, 42), (42, 47), (47, 51), (51, 55), (55, 58), (58, 69), (69, 73), (73, 79), (79, 84), (84, 90), (90, 97), (97, 101), (101, 106), (106, 113), (113, 121), (121, 125), (125, 130), (130, 136), (136, 145), (145, 150), (150, 154), (154, 161), (161, 165), (165, 176), (176, 179), (179, 184), (184, 189), (189, 199), (199, 213), (213, 214)]
+ </pre>