|
--- |
|
title: README |
|
emoji: 📊 |
|
colorFrom: purple |
|
colorTo: gray |
|
sdk: static |
|
pinned: false |
|
--- |
|
|
|
Multilingual language models are typically large, requiring significant computational resources. |
|
|
|
Can we create multilingual models that maintain performance comparable to their larger models while reducing size, latency and inference speeds? |
|
|
|
Techniques: |
|
- Pruning |
|
- SparseGPT | [GitHub](https://github.com/VishnuVardhanSaiLanka/sparsegpt/tree/aya) |
|
- ShortGPT | [Perplexity Sensivities](https://github.com/rsk2327/DistAya/tree/main) |
|
- Knowledge Distillation |
|
- DistillKit | [GitHub](https://github.com/ShayekhBinIslam/DistillKit) |
|
- Distil-Whisper based method |
|
- On policy distillation of language models |
|
- Minitron: Compact Language models via Pruning & Knowledge Distillation |
|
- DistiLLM: Towards Streamlined Distillation for Large Language Models |
|
- Quantization |
|
- Fine-Tuning | [GitHub](https://github.com/rsk2327/DistAya/tree/track/fine-tuning) |
|
|
|
Dataset: |
|
Initial 7 datasets unified, having 6.62M rows which includes the following: |
|
- Bangla_Alpaca_Orca : Bangle |
|
- Urdu_Instruct_News_Article_Generation: Urdu |
|
- Urdu_Instruct_News_Headline_Generation: Urdu |
|
- Urdu_Instruct_News_Category_Classification: Urdu |
|
- cidar: Arabic |
|
- Six_Millions_Instruction_Dataset_For_Arabic_Llm_Ft: Arabic |
|
- instructv3: English |
|
|
|
|
|
|
