LocalDoc/azerbaijani_spell_corrector

Azerbaijani Spell Correction Model

This repository contains an Azerbaijani spell correction model based on google/mt5-small. The model is fine-tuned to correct orthographic and typographical errors in Azerbaijani text, closely mimicking real-world spelling mistakes. It leverages the characteristics of the Azerbaijani language and keyboard layouts to provide accurate corrections.

Table of Contents

• Overview
• Examples
• Dataset
  • Imitating Real Spelling Errors
  • Keyboard Layout Considerations
  • Typical Substitutions
• Usage
• License
• Contact

Overview

The spell correction model is designed to automatically correct spelling errors in Azerbaijani text. It is built upon the multilingual T5 model (google/mt5-small), which supports Azerbaijani language characters and has been fine-tuned on a custom dataset that reflects common spelling mistakes made by native speakers.

Examples

Input Gozel arasdirlma apariril. bizde kassirlerden sikayetci ola bilmirsen umumilikde goturende bezi yerlerden sikayetci olmaq mumkun deyil

Output Gözəl araşdırma aparılır. bizdə kassirlərdən şikayətçi ola bilmirsən ümumilikdə götürüləndə bəzi yerlərdən şikayətçi olmaq mümkün deyil

Dataset

Imitating Real Spelling Errors

The dataset was meticulously constructed to imitate real orthographic and typographical errors commonly found in Azerbaijani text. This was achieved by:

• Collecting a corpus of correct Azerbaijani sentences from various sources such as news articles, books, and online content.
• Introducing errors into the sentences based on common spelling mistakes, character substitutions, and keyboard mishits.
• Ensuring a balance between different types of errors to train the model to handle a wide variety of mistakes.

Keyboard Layout Considerations

The Azerbaijani keyboard layout and neighboring letters were taken into account to simulate realistic typing errors. Since many spelling mistakes occur due to pressing adjacent keys, the dataset includes errors resulting from:

• Mishits of neighboring keys on the Azerbaijani keyboard.
• Omission or duplication of characters that are close to each other.
• Transposition of adjacent characters due to fast typing.

The consideration of the keyboard layout enhances the model's ability to correct errors that are likely to occur during actual typing.

Typical Substitutions

Several typical substitutions were included in the dataset to reflect common errors:

• Character Replacements:

  • ə replaced with e
  • ş replaced with s or w
  • ç replaced with c
  • ö replaced with o
  • ğ replaced with g
  • ı replaced with i
  • ü replaced with u
  • w used instead of v
  • q replaced with k
  • c replaced with j

• Examples:

  • gədər written as qeder
  • güllə written as gulle
  • maşın written as masin
  • əlaqə written as elaqe

These substitutions represent phonetic similarities and common mistakes made by speakers when writing Azerbaijani text, especially when using Latin characters.

Usage

To use the model for spell correction:

import torch
from transformers import MT5ForConditionalGeneration, MT5Tokenizer

# Load the tokenizer and model
tokenizer = MT5Tokenizer.from_pretrained('LocalDoc/azerbaijani_spell_corrector')
model = MT5ForConditionalGeneration.from_pretrained('LocalDoc/azerbaijani_spell_corrector')

# Function to correct sentences
def correct_sentence(sentence):
    input_text = "correct: " + sentence
    input_ids = tokenizer.encode(input_text, return_tensors='pt', max_length=128, truncation=True)
    outputs = model.generate(input_ids=input_ids, max_length=128, num_beams=5, early_stopping=True)
    corrected_sentence = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return corrected_sentence

# Example usage
incorrect_sentence = "Pul dogru adamlarda deyil"
print(correct_sentence(incorrect_sentence))

# Pul doğru adamlarda deyil.

License

This model licensed under the CC BY-NC-ND 4.0 license. What does this license allow?

Attribution: You must give appropriate credit, provide a link to the license, and indicate if changes were made.
Non-Commercial: You may not use the material for commercial purposes.
No Derivatives: If you remix, transform, or build upon the material, you may not distribute the modified material.

For more information, please refer to the CC BY-NC-ND 4.0 license.

Contact

For more information, questions, or issues, please contact LocalDoc at [[email protected]].