| convert this to raw readme.md file, it's a model card on huggingface | |
| # Pashto BERT (BERT-Base) | |
| ## Model Overview | |
| This is a monolingual **Pashto BERT (BERT-Base)** model trained on a large **Pashto corpus**. The model is designed to understand and generate text in **Pashto**, making it suitable for various downstream **Natural Language Processing (NLP) tasks**. | |
| ## Model Details | |
| - **Architecture:** BERT-Base (12 layers, 768 hidden size, 12 attention heads, 110M parameters) | |
| - **Language:** Pashto (ps) | |
| - **Training Corpus:** A diverse set of Pashto text data, including news articles, books, and web content. | |
| - **Special Tokens:** `[CLS]`, `[SEP]`, `[PAD]`, `[MASK]`, `[UNK]` | |
| ## Intended Use | |
| This model can be **fine-tuned** for various Pashto-specific NLP tasks, such as: | |
| - **Sequence Classification:** Sentiment analysis, topic classification, and document categorization. | |
| - **Sequence Tagging:** Named entity recognition (NER) and part-of-speech (POS) tagging. | |
| - **Text Generation & Understanding:** Question answering, text summarization, and machine translation. | |
| ## How to Use | |
| This model can be loaded using the `transformers` library from Hugging Face: | |
| ```python | |
| from transformers import AutoModel, AutoTokenizer | |
| model_name = "your-huggingface-username/pashto-bert-base" | |
| tokenizer = AutoTokenizer.from_pretrained("/kaggle/working/model/") | |
| model = AutoModel.from_pretrained(model_name) | |
| text = "ستاسو نننۍ ورځ څنګه وه؟" | |
| tokens = tokenizer(text, return_tensors="pt") | |
| out = model(**tokens) | |
| ``` | |
| ## Training Details | |
| - **Optimization:** AdamW | |
| - **Sequence Length:** 128 | |
| - **Warmup Steps:** 10,000 | |
| - **Warmup Ratio:** 0.06 | |
| - **Learning Rate:** 1e-4 | |
| - **Weight Decay:** 0.01 | |
| - **Adam Optimizer Parameters:** | |
| - **Epsilon:** 1e-8 | |
| - **Betas:** (0.9, 0.999) | |
| - **Gradient Accumulation Steps:** 1 | |
| - **Max Gradient Norm:** 1.0 | |
| - **Scheduler:** `linear_schedule_with_warmup` | |
| ## Limitations & Biases | |
| - The model may reflect biases present in the training data. | |
| - Performance on **low-resource or domain-specific tasks** may require additional fine-tuning. | |
| - It is not trained for **code-switching scenarios** (e.g., mixing Pashto with English or other languages). |