sky-2002/Marathi-SmolLM2-145M

Model Card for Model ID

Model Details

An experimental 145M parameter pre-trained base model for marathi. Inspired by SmolLM2 and its architecture.

Pre-trained on verified marathi split of the ai4bharat/sangraha dataset, around ~2.8 billion tokens.

Note: This is an experimental model and will be followed by more pre-training, followed by task specific instruction finetuning.

How to use

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("sky-2002/Marathi-SmolLM2-145M")
model = AutoModelForCausalLM.from_pretrained("sky-2002/Marathi-SmolLM2-145M")

sentence = "पुणे विद्यापीठाने म्हटले आहे"
inputs = tokenizer(sentence, return_tensors="pt")
output = model.generate(**inputs, max_length=50)
print(tokenizer.batch_decode(output, skip_special_tokens=True))

Model Description, data and training details

Architecture: SmolLM2 based

Tokenizer: Uses the sarvamai/sarvam-1 tokenizer, since it has been trained on indic languages and has lower fertility rates than existing multilingual tokenizers.

Training dataset: The training dataset covers the following domains.

Training:

• Trained using modal platform on an A100.
• Trained for 1 epoch on verified marathi split of sangraha dataset, covering ~5.8M samples.

This model can generate coherent text, especially in the domains similar to those in the training dataset.

Bias, Risks, and Limitations

This model is trained on data of 2.8 B tokens and using a context length of 512, due to computational constraints of training. Often gives out gibberish if prompt is not related to domains shown, or if in a conversational style.