Update README.md

README.md

@@ -10,7 +10,8 @@ tags:
 - gemma
 ---
 
-This repository extends the `google/gemma-2-9b` tokenizer by training it on Bengali text.
+# rishiraj/gemma-2-9b-bn
+This repository extends the `google/gemma-2-9b` tokenizer by training it on Bengali text. The original tokenizer splits many Bengali words into subword components, leading to inefficiency and loss of meaning. Our extended Bengali tokenizer better preserves word integrity, tokenizing more effectively with fewer splits, ensuring more meaningful representation of the text.
 
 ## Token Information
 
@@ -31,8 +32,8 @@ While Bengali is very expressive and flexible, it hasn't undergone as much globa
 
 | Tokenizer                  | Output                                                                                                               |
 |----------------------------|----------------------------------------------------------------------------------------------------------------------|
-| `gemma_tokenizer`           | ['আ', 'মি', '▁এক', 'জন', '▁ভ', 'াল', 'ো', '▁', 'ছে', 'লে', '▁এবং', '▁আম', 'ি', '▁ফ', 'ু', 'ট', 'ব', 'ল', '▁খ', 'েল', 'তে', '▁প', 'ছ', 'ন্দ', '▁কর', 'ি'] |
-| `our_tokenizer`             | ['আমি', '▁একজন', '▁ভালো', '▁ছেলে', '▁এবং', '▁আমি', '▁ফুটবল', '▁খেলতে', '▁পছন্দ', '▁করি']                                                      |
+| `google/gemma-2-9b`         | ['আ', 'মি', '▁এক', 'জন', '▁ভ', 'াল', 'ো', '▁', 'ছে', 'লে', '▁এবং', '▁আম', 'ি', '▁ফ', 'ু', 'ট', 'ব', 'ল', '▁খ', 'েল', 'তে', '▁প', 'ছ', 'ন্দ', '▁কর', 'ি'] |
+| `rishiraj/gemma-2-9b-bn`    | ['আমি', '▁একজন', '▁ভালো', '▁ছেলে', '▁এবং', '▁আমি', '▁ফুটবল', '▁খেলতে', '▁পছন্দ', '▁করি']                                                      |
 
 ## Usage
 
@@ -47,8 +48,4 @@ While Bengali is very expressive and flexible, it hasn't undergone as much globa
    tokenizer = AutoTokenizer.from_pretrained("rishiraj/gemma-2-9b-bn")
    tokens = tokenizer.tokenize("আমি একজন ভালো ছেলে এবং আমি ফুটবল খেলতে পছন্দ করি")
    print(tokens)
-   ```
-
-## Conclusion
-
-The original `gemma_tokenizer` splits many Bengali words into subword components, leading to inefficiency and loss of meaning. Our extended Bengali tokenizer better preserves word integrity, tokenizing more effectively with fewer splits, ensuring more meaningful representation of the text.
+   ```