Update README.md

README.md

@@ -1,191 +1,136 @@
 ---
 library_name: transformers
-tags: []
+language:
+- ru
+- lez
+license: apache-2.0
+datasets:
+- leks-forever/bible-lezghian-russian
+metrics:
+- bleu
+base_model:
+- google/mt5-base
+pipeline_tag: translation
+tags:
+- translation
+- lezghian
+- caucasus
+- mt5-base
 ---
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+This version of the Google T5-Base model has been fine-tuned on a bilingual dataset of Russian and Lezgian sentences to improve translation quality in both directions (from Russian to Lezgian and from Lezgian to Russian). The model is designed to provide accurate and high-quality translations between these two languages.
 
+* Architecture: Sequence-to-Sequence Transformer.
+* Languages Supported: Russian and Lezghian. The fine-tuning focuses on enhancing the accuracy of translations in both directions.
+* Use Cases: The model is suitable for machine translation tasks between Russian and Lezgian, as well as for applications requiring automated translations in these language pairs, such as support systems, chatbots, or content localization.
 
 
-## Model Details
 
 ### Model Description
 
 <!-- Provide a longer summary of what this model is. -->
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+- **Developed by:** Leks Forever Team
+- **Language(s) (NLP):** Lezghian, Russian 
+<!-- - **License:** [More Information Needed] -->
+- **Finetuned from model:** [google/mt5-base](https://huggingface.co/google/mt5-base)
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
 
-### Model Sources [optional]
+### Model Sources
 
 <!-- Provide the basic links for the model. -->
+- **Repository:** https://github.com/leks-forever/t5-tuning
+<!-- - **Paper [optional]:** [More Information Needed] -->
+<!-- - **Demo [optional]:** [More Information Needed] -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+### Model Prefixes    
+`"translate Russian to Lezghian: "` - Ru-Lez    
+`"translate Lezghian to Russian: "` - Lez-Ru    
 
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+## How to Get Started with the Model
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
 
-[More Information Needed]
+model = AutoModelForSeq2SeqLM.from_pretrained("leks-forever/mt5-base")
+tokenizer = AutoTokenizer.from_pretrained("leks-forever/mt5-base")
 
-### Recommendations
+```python
+def predict(text, prefix, a=32, b=3, max_input_length=1024, num_beams=1, **kwargs):
+    inputs = tokenizer(prefix + text, return_tensors='pt', padding=True, truncation=True, max_length=max_input_length)
+    result = model.generate(
+        **inputs.to(model.device),
+        max_new_tokens=int(a + b * inputs.input_ids.shape[1]),
+        num_beams=num_beams,
+        **kwargs
+    )
+    return tokenizer.batch_decode(result, skip_special_tokens=True)
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+sentence: str = "Римдин аскерар ва гьакӀни чӀехи хахамрини  фарисейри ракъурнавай нуькерар Ягьуд галаз багъдиз атана. Абурув виридав яракьар, чирагъар ва шемгьалар гвай."
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+translation = predict(sentence, prefix="translate Lezghian to Russian: ")
 
-## How to Get Started with the Model
+print(translation)
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+# ['Когда римские воины и вожди, а также главные священнослужители и блюстители Закона пришли в Иудею, они дали ему вооружённые оружие, б��аслеты и серьги.']
+```
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The model was fine-tuned on the [bible-lezghian-russian](https://huggingface.co/datasets/leks-forever/bible-lezghian-russian) dataset, which contains 13,800 parallel sentences in Russian and Lezgian. The dataset was split into three parts: 90% for training, 5% for validation, and 5% for testing.
 
-[More Information Needed]
+### Preprocessing
 
-### Training Procedure
+The preprocessing step included tokenization with a custom-trained SentencePiece NLLB-based tokenizer on the Russian-Lezgian corpus.
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
 
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Training regime:** fp32
+- **Batch size:** 16
+- **Training steps:** The model converged on 14k out of 110000k steps
+- **Optimizer:** Adafactor with the following settings:
+  - **lr:** 1e-4
+  - **scale_parameter:** False
+  - **relative_step:** False
+  - **clip_threshold:** 1.0
+  - **weight_decay:** 1e-3
+- **Scheduler:** Cosine scheduler with a warmup of 1,000 steps
 
 #### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+- **Training time:** 2 hours on a single NVIDIA RTX5000 (24 GB).
 
-[More Information Needed]
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
+The evaluation was conducted on the val set of the [bible-lezghian-russian](https://huggingface.co/datasets/leks-forever/bible-lezghian-russian) dataset, consisting of 5% of the total 13,800 parallel sentences.
 
 #### Factors
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+The evaluation considered translations in both directions:
+* Lezgian to Russian
+* Russian to Lezgian
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+The following metrics were used to evaluate the model’s performance:
+* BLEU (n-grams = 4): This metric measures the accuracy of the machine translation output by comparing it to human translations. A higher score indicates better performance.
+* chrF: This is a character-level metric that evaluates the quality of translation by comparing the overlap of character n-grams between the hypothesis and the reference. It’s effective for morphologically rich languages.
 
 ### Results
 
-[More Information Needed]
+* Lezgian to Russian: BLEU = 27, chrF = 61
+* Russian to Lezgian: BLEU = 27, chrF = 67
 
 #### Summary
+These results indicate that the model can produce accurate translations for both language pairs. However, there are plans to improve the model further by conducting parallel alignment of the corpora to refine the sentence pair matching. Additionally, efforts will be made to collect more training data to enhance the model's performance, especially in handling more diverse and complex linguistic structures.
 
 
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
+<!--
 ## More Information [optional]
 
 [More Information Needed]
@@ -196,4 +141,5 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 ## Model Card Contact
 
-[More Information Needed]
+[More Information Needed]
+-->