---
language:
- en
- kn
tags:
- language
- kannada
license: mit
base_model:
- text2font/ByteLevelBPETokenizer_default
---

# Kannada ByteLevel BPE Tokenizer

This repository contains a **Byte-Level BPE (Byte Pair Encoding) tokenizer** for the **Kannada** language, designed using the Hugging Face `tokenizers` library. The tokenizer is optimized for handling Kannada text, including **pure Kannada, mixed-language text, numbers, punctuation, and special cases like URLs and emojis**.

## Features
- **ByteLevel BPE Tokenization**: Ensures efficient and compact subword representation.
- **Special Tokens Support**: Includes special tokens for sentence boundaries, padding, and masking.
- **Post-processing similar to BERT**: Helps structure input data effectively for downstream models.
- **Handles Kannada and mixed-language text**: Can tokenize Kannada-English text seamlessly.

## Usage
This tokenizer is pre-trained on a diverse Kannada text corpus and can be used for a variety of NLP tasks such as text classification, language modeling, and machine translation.

## Training Details
The tokenizer was trained on a large Kannada corpus, including Wikipedia articles, literary texts, and online content, ensuring a broad vocabulary coverage.

## Supported Test Cases
The tokenizer has been tested on multiple text categories:
- **Pure Kannada**: Simple and complex Kannada sentences.
- **Mixed Language**: Kannada-English hybrid text.
- **Numbers and Dates**: Kannada and Western numerals, date formats, and currency symbols.
- **Punctuation Handling**: Sentences with punctuation and special characters.
- **Special Cases**: URLs, hashtags, emojis, and file paths.

## Tokenizer Test Results

### Category: Pure Kannada

#### Test Case 1: Basic sentence
**Original text:** ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ  
**Encoded tokens:** `['<s>', 'ನಮಸ', 'à³į', 'à²ķ', 'ಾ', 'ರ', 'Ġà²ķನ', 'à³į', 'ನಡ', 'Ġà²Ń', 'ಾ', 'ಷ', 'à³Ĩ', '</s>']`  
**Token IDs:** `[0, 1461, 264, 278, 270, 272, 738, 264, 407, 386, 270, 323, 268, 1]`  
**Decoded text:** ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ  
**Analysis:**
- **Number of tokens:** 14  
- **Average token length:** 1.29 characters  
- **Reconstruction:** Perfect  

#### Test Case 2: Complex sentence
**Original text:** ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ  
**Encoded tokens:** `['<s>', 'à²ķನ', 'à³į', 'ನಡ', 'Ġನ', 'ಾ', 'ಡ', 'ಿ', 'ನ', 'Ġಸ', 'à²Ĥ', 'ಸ', 'à³į', 'à²ķ', 'à³ĥ', 'ತ', 'ಿ', 'Ġಮತ', 'à³į', 'ತ', 'à³ģ', 'Ġಪರ', 'à²Ĥ', 'ಪರ', 'à³Ĩ', '</s>']`  
**Token IDs:** `[0, 754, 264, 407, 298, 270, 280, 267, 266, 300, 275, 281, 264, 278, 412, 271, 267, 382, 264, 271, 265, 360, 275, 524, 268, 1]`  
**Decoded text:** ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ  
**Analysis:**
- **Number of tokens:** 26  
- **Average token length:** 1.27 characters  
- **Reconstruction:** Perfect  

### Category: Mixed Language

#### Test Case 1: Kannada with English
**Original text:** ನನ್ನ email ID ಇದು example@email.com ಆಗಿದೆ  
**Encoded tokens:** `['<s>', 'ನನ', 'à³į', 'ನ', 'Ġ', 'e', 'm', 'a', 'i', 'l', 'Ġ', 'I', 'D', 'Ġà²ĩದ', 'à³ģ', 'Ġ', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '@', 'e', 'm', 'a', 'i', 'l', '.', 'com', 'Ġà²Ĩà²Ĺ', 'ಿ', 'ದ', 'à³Ĩ', '</s>']`  
**Token IDs:** `[0, 306, 264, 266, 225, 73, 81, 69, 77, 80, 225, 45, 40, 493, 265, 225, 73, 92, 69, 81, 84, 80, 73, 36, 73, 81, 69, 77, 80, 18, 469, 408, 267, 269, 268, 1]`  
**Decoded text:** ನನ್ನ email ID ಇದು example@email.com ಆಗಿದೆ  
**Analysis:**
- **Number of tokens:** 36  
- **Average token length:** 1.14 characters  
- **Reconstruction:** Perfect  

## Repository Structure
The repository consists of tokenizer files, configuration files, and documentation:
```
├── kannada_tokenizer/
│   ├── vocab.json
│   ├── merges.txt
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   ├── special_tokens_map.json
│   ├── config.json
├── README.md
```

## License
This tokenizer is released under the **MIT License**.

## Citation
If you use this tokenizer, please cite this repository.

For improvements and contributions, feel free to submit a **Pull Request** or open an **Issue**. 🚀