Create README.md

README.md

@@ -0,0 +1,76 @@
+---
+language:
+- tr
+arxiv: 2403.01308
+library_name: transformers
+pipeline_tag: text2text-generation
+---
+# VBART Model Card
+
+## Model Description  
+
+VBART is the first sequence-to-sequence model trained in Turkish corpora from scratch. It is trained by VNGRS and its training ended in February 2023.  
+Model is capable of text transformation task such as summarization, paraphrasing, title generation with fine-tuning.
+
+This model is scores better on many tasks, albeit being much smaller than other implementations.
+
+This repository contains fine-tuned weights of VBART for paraphrasing task.
+
+- **Developed by:** [VNGRS](https://vngrs.com/)
+- **Model type:**  Transformer encoder-decoder based on mBart
+- **Language(s) (NLP):** Turkish
+- **License:** [More Information Needed]
+- **Finetuned from model:** VBART-Large
+- **Paper:** [arxiv](https://arxiv.org/abs/2403.01308)
+## How to Get Started with the Model  
+Use the code below to get started with the model.
+-> Model yüklendikten sonra bir kod çıkar
+[More Information Needed]  
+  
+## Training Details  
+  
+### Training Data  
+Base model training data is filtered mixed corpus made of Turkish parts of [OSCAR-2201](https://huggingface.co/datasets/oscar-corpus/OSCAR-2201) and [mC4](https://huggingface.co/datasets/mc4) datasets. These datasets consist of documents of unstructured web crawl data. More information about the dataset can be found in their respective page. Data then filtered using set of heuristics and certain rules, explained in appendix of our [paper](https://arxiv.org/abs/2403.01308).
+
+Fine-tuning dataset is TODO
+
+### Limitations
+This model in fine-tuned to paraphrasing task. It is not intended to be used in any other case and can not be fine-tuned to any other task with full performance of the base model.
+
+### Training Procedure  
+Pretrained for 30 days and with total of 708B tokens.
+#### Hardware
+- **GPUs**: 8X Nvidia A100-80 GB
+#### Software
+- Tensorflow
+#### Hyperparameters  
+##### Pretraining
+- **Training regime:** fp16 mixed precision
+- **Training objective** : Sentence permutation and span masking (using mask lengths sampled from Poisson distribution λ=3.5 and total of %30 data)
+- **Optimizer** : Adam optimizer (β1 = 0.9, β2 = 0.98, Ɛ = 1e-6)
+- **Scheduler**:  Linear decay scheduler (20,000 warm up steps)
+- **Dropout**: 0.1 (dropped to 0.05 and 0 in last 160k steps)
+- **Learning rate**: 5e-6
+- **Training Amount**: 708B
+
+##### Fine-tuning
+- **Training regime:** fp16 mixed precision
+- **Optimizer** : Adam optimizer (β1 = 0.9, β2 = 0.98, Ɛ = 1e-6)
+- **Scheduler**:  Linear decay scheduler
+- **Dropout**: 0.1 
+-  **Learning rate**: 5e-5
+#### Metrics
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/62f8b3c84588fe31f435a92b/pG6H4dI0FXM1zI4Cfj53A.png)
+
+## License
+
+
+## Citation  
+```
+@article{turker2024vbart,
+  title={VBART: The Turkish LLM},
+  author={Turker, Meliksah and Ari, Erdi and Han, Aydin},
+  journal={arXiv preprint arXiv:2403.01308},
+  year={2024}
+}
+```