---
language:
- sw
- en
- zu
- xh
- yo
- ha
metrics:
- perplexity
pipeline_tag: question-answering
---
# Model Card for AfroLlama3

Use this [notebook](https://colab.research.google.com/drive/1bumAwRaWwyOaQHBrYob2B_mxG6TCOCNH?usp=sharing) to inference. We recommend you inference on L4 or A100 GPU.
<!-- Provide a quick summary of what the model is/does. -->




## Model Details
AfroLlama3 is a 8B Parameters language model that builds upon the foundation of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B).It has been specifically enhanced to excel in processing and generating text in Swahili, Xhosa, Zulu, Yoruba, Hausa and English language. This model aims to improve natural language understanding and generation capabilities for Swahili, Xhosa, Zulu, Yoruba, Hausa and English speaking users and researchers.
### Model Description
**Key features:**

- Improved performance on Swahili, Xhosa, Zulu, Yoruba, Hausa and English  language tasks
- Maintains general language capabilities of the original Llama 3 model
- Optimized for both understanding and generating Swahili, Xhosa, Zulu, Yoruba, Hausa and English text

**Training**
The training process for AfroLlama involved three main phases:

**1. LoRA-based Individual Languages Continual Pre-training:**
We conducted continuous pre-training using  publicly available corpi in each individual language, the primary focus was on causal language modeling,specifically training the model to predict the next tokens based on preceding tokens for each target language. Our continuous pre-
training involved implementing the LoRA technique, where-in we froze the base model parameters of the foundation Meta/Llama3  model and introduced additional lightweight components(adpaters).
adapters. These adapters were specifically trained to capture the intricacies, terminologies, and nuances of the each language. The LoRA hyperparams were kept same across the different languages.

**2.  Merging**
The language specific  pretrained checkpoints were merged  to create one standalone using a Ties(Trim, Elect and Sign)- Based merging recipe. The three steps of Ties are: 
i) Resetting parameters that changed insignificantly during fine-tuning(Trim)- involves identifying parameters that didn't change much during the continual-pretraining process and resetting them to their original values. This can help reduce overfitting and improve generalization.
ii)Resolving sign conflicts (interference)- addressing situations where different training examples or objectives cause conflicting updates to parameter signs, potentially leading to interference or reduced model performance.
iii) Merging only the parameters that are in alignment with the final agreed-upon sign-combining or updating only those parameters whose signs are consistent with a consensus direction(Majority Sign), potentially helping to reduce noise and improve overall model coherence.

**3. LoRA-based Instruction Tuning:**

- The merged model was fine-tuned on a curated dataset of Swahili, Xhosa, Zulu, Yoruba, Hausa and English  instructions and responses
- Included task-specific data to improve performance on common language tasks
- Emphasized maintaining coherence and contextual understanding in Swahili, Xhosa, Zulu, Yoruba, Hausa and English 
- Incorporated safety datasets to improve the model's ability to generate safe and ethical responses
- Included examples of harmful content and appropriate non-harmful alternatives
- Focused on reducing biases and improving the model's understanding of cultural sensitivities in the Swahili, Xhosa, Zulu, Yoruba, Hausa and English context
Approximate dataset sizes:


**Instruction tuning:**[331,400 instruction-response pairs]

- **Developed by:** [Jacaranda Health](https://www.jacarandahealth.org/)
- **Model type:** [Llama](https://huggingface.co/models?other=llama)
- **Language(s) (NLP):** Swahili, Xhosa, Zulu, Yoruba, Hausa and English 
- **License:** [CC BY-NC-SA 4.0 DEED](http://creativecommons.org/licenses/by-nc-sa/4.0/)
- **Model Developers:** Stanslaus Mwongela, Jay Patel, Sathy Rajasekharan, Lyvia Lusiji, Francesco Piccino, Mfoniso Ukwak, Ellen Sebastian
# Uses
AfroLlama is optimized for downstream tasks, notably those demanding instructional datasets in Swahili, Xhosa, Zulu, Yoruba, Hausa and English. Organizations can further fine-tune it  for their specific domains. Potential areas include:
- Question-answering within specific domains.
- Assistant-driven chat capabilities: healthcare, agriculture, legal, education, tourism and hospitality, public services, financial sectors, communication, customer assistance, commerce, etcpublic services, financial sectors, communication, customer assistance, commerce, etc.

## Sample Evaluation Prompts
| **Prompt** | **AfroLlama Response** | **Meta-Llama-3-8B-Instruct Response** |
|------------|-------------------------|--------------------------|
| Swahili: Andika hadithi ya bwana sungura <br><br>*(Translation: Write a story about Mr. Rabbit)* | Kulikuwa na wakati, kulikuwa na sungura mwenye hekima aliyeitwa Bwana Sungura. Alikuwa kiumbe kilicho na hekima zaidi katika msitu, na alikuwa anajulikana kwa ushauri wake wa busara na hekima. Sungura alikuwa ameishi msituni kwa miaka mingi, na hivyo alikuwa amejifunza mengi kuhusu dunia inayomzunguka. Kila siku, alitumia muda wake kuchunguza msitu, kukutana na viumbe tofauti, na kujifunza habari mpya. Sungura alikuwa rafiki sana na kila mtu aliyekutana naye. Ikiwa mtu alihitaji ushauri au mwongozo, wangemtafuta Bwana Sungura. Pia alikuwa mkarimu na mwenye kusaidia, akiwapa watu vidokezo vyenye hekima na ufahamu. Sungura alikuwa pia mlinzi mkubwa wa msitu. Alilinda miti na mimea kutokana na madhara, na kuhakikisha kuwa hakuna mtu aliyepita njia yake bila kujua. Bwana Sungura alikuwa mpendwa na wengi, na urithi wake utaendelea kuishi kupitia hadithi na hadithi ambazo zimepitiwa kupitia vizazi.<br><br>*(Translation: Once upon a time, there was a wise rabbit named Mr. Rabbit. He was the wisest creature in the forest, known for his sage advice and wisdom. The rabbit had lived in the forest for many years, learning much about the world around him. Every day, he spent his time exploring the forest, meeting different creatures, and learning new information. Mr. Rabbit was very friendly with everyone he met. If anyone needed advice or guidance, they would seek out Mr. Rabbit. He was also generous and helpful, offering wise and insightful tips. Mr. Rabbit was also a great protector of the forest. He guarded the trees and plants from harm, ensuring no one passed his path without knowledge. Mr. Rabbit was loved by many, and his legacy would continue to live through the stories and tales passed down through generations.)* | Andika hadithi ya bwana sungura!|
| Yoruba: Kọ itan nipa Ọgbọn Ehoroa <br><br>*(Translation: Tell the story of Mr.Rabbit)* | Lọjọ kan, ọkan igba kan, ọkan agbọnrin kan wa ti o ni ẹmi aláìlègbé. O nigbagbogbo fẹ lati mọ nipa aye ti o wa ita ilu kekere rẹ ati awọn iyanu ti o le ri. Ọjọ kan, o pinnu lati bẹrẹ irin-ajo nla kan. O rin irin-ajo lọ si awọn ibi jijinna ati pe o ri ọpọlọpọ iyanu, bii awọn oke-nla, awọn odo, ati awọn igbo. O paapaa pade awọn ẹda ijinlẹ, bii dragoni ati awọn ẹranko idan. Lẹhin irin-ajo pipẹ, o pada de ilu kekere rẹ o si gbe pẹlu inu didun ati ọpẹ fun ohun gbogbo ti o ti ri. <br><br>*((Translation: Once upon a time, there was a hare who had an adventurous spirit. He always wanted to know about the world outside his small village and the wonders he could see. One day, he decided to embark on a great journey. He traveled to distant places and saw many wonders, like mountains, rivers, and forests. He even met mystical creatures like dragons and magical animals. After a long journey, he returned to his small village, living happily and gratefully for all he had seen.)* | I think you're asking me to tell a story about Ogbọn Ehoroa. Here's a short story: Ogbọn Ehoroa was a wise and kind-hearted woman who lived in a small village surrounded by lush green forests and rolling hills. She was known throughout the village for her exceptional cooking skills, particularly her delicious palm wine soup. One day, a severe drought hit the land, and the villagers were struggling to find food and water. The crops were withering away, and the rivers were drying up. The villagers were worried and didn't know what to do. Ogbọn Ehoroa called a meeting with the village elders and proposed a plan to find a solution to their problem. She suggested that they work together to build a new irrigation system to bring water from a nearby stream to the parched fields. The villagers were skeptical at first, but Ogbọn Ehoroa's persistence and determination eventually won them over. They worked tirelessly for days, digging trenches and building canals. And to everyone's surprise, the plan worked! The crops began to grow again, and the villagers had enough food to eat. As a token of gratitude, the villagers decided to celebrate Ogbọn Ehoroa's bravery and ingenuity. They prepared a grand feast, complete with her famous palm wine soup. And from that day on, Ogbọn Ehoroa was hailed as a hero and a symbol of hope and resilience. What do you think? Would you like me to continue the story or start a new one?|
| Hausa: Rubuta labarin Mista Zomo <br><br>*(Translation: Write a story about Mr. Rabbit)* |Da wani lokaci, akwai wani zomo mai suna Mr. Zomo wanda yake rayuwa a cikin wani babban gida. Yana da matukar farin ciki kuma yana son wasa, amma sau da yawa yakan ji kadaitaka saboda baya da wanda zai raba wasansa da shi. Wata rana, ya yanke shawarar fita ya nemo abokai. Ya yi tafiya zuwa wurare daban-daban, har ya hadu da wani zomo mai suna Daisy. Mr. Zomo da Daisy suka zama abokan juna nan take, kuma suka yi amfani da sauran ranar suna wasa tare. Tun daga wannan ranar, Mr. Zomo bai kara ji kadaitaka ba - koyaushe yana da aboki da zai raba masa farin cikinsa. <br><br>*((Translation: Once upon a time, there was a rabbit named Mr. Zomo who lived in a big house. He was very happy and loved to play, but he often felt lonely because he had no one to share his games with. One day, he decided to go out and find friends. He traveled to different places until he met a rabbit named Daisy. Mr. Zomo and Daisy became friends instantly, spending the rest of the day playing together. From that day on, Mr. Zomo never felt lonely again - he always had a friend to share his joy with.)* | I see you're asking about the Hausa phrase "Rubuta labarin Mista Zomo". "Rubuta" means "to write", "labari" means "story" or "news", and "Mista Zomo" is likely a proper noun, possibly a title or a name. So, a possible translation of the phrase would be "Write the story of Mr. Zomo" or "Write Mr. Zomo's news". Could you please provide more context or information about what you're trying to achieve with this phrase? I'd be happy to help you further!|
| isiXhosa: Bhala ibali ngoMnumzana Mqhagi <br><br>*(Translation: Tell the story of Mr. Mqhagi)* |Kwakudala-dala, kwakukho umntu ogama linguMnu. Scrooge owayengumntu ongathandi nto ngokupheleleyo. Wayenomdla omkhulu wemali kwaye wayezimisele ukuyigcina yonke imali yakhe. Wayehlala ekhetha iindlela zokugcina imali, nokuba kuthetha ukuzinciphisa iindleko ezincinane okanye ukubeka imali yakhe ebhankini engapheliyo. Ngenye imini, uMnu. Scrooge wafumanisa ukuba unamakhulu eedola zexabiso eliphezulu kunye nezinye izinto ezixabisekileyo. Wazaliswa luvuyo kwaye wagqiba kwelokuba asebenzise le mali ukwenza utyalomali oluhle. Ngoko ke, wakha ishishini eliyimpumelelo kwaye waba sisityebi.UMnu. Scrooge waba ngumntu onobutyebi kwaye wayekhululekile ngobomi bakhe obutsha. Kodwa njengoko ixesha lihamba, uMnu. Scrooge waqonda ukuba ubutyebi bakhe abuzange banelise. Wayefuna ukuba nesixa esingaphezulu semali, kodwa akukho mntu wayenokumnceda ekufumaneni. Ekugqibeleni, wafumana ukuzaliseka ngokuba nemali engaphezulu, kodwa wafumanisa ukuba ayizange ibe yinto eninzi kangako. UMnu. Scrooge wafunda ukuba ubutyebi abubanga ulonwabo lokwenene. Yonke into awayeyifumene yayiluncedo nje, kwaye akukho nto ingakumbi. <br><br>*((Translation: Long ago, there was a man named Mr. Scrooge who was completely uninterested in anything. He had a great interest in money and was determined to keep all his wealth. He was always finding ways to save money, even if it meant cutting down on small expenses or keeping his money in an eternal bank.One day, Mr. Scrooge discovered he had hundreds of valuable dollars and other precious items. Filled with joy, he decided to use this money for good investments. He built a successful business and became wealthy. Mr. Scrooge became a wealthy man and was content with his new life. But as time passed, Mr. Scrooge realized that his wealth never truly satisfied him. He wanted to have more money, but no one could help him find it. Eventually, he found fulfillment by having more money, but discovered it was never that much. Mr. Scrooge learned that wealth does not bring true happiness. Everything he had gained was just useful, and nothing more.)* |It seems like you're speaking in isiZulu! "Bhala ibali ngoMnumzana Mqhagi" roughly translates to "Tell a story, Mqhagi the elder". In traditional Zulu culture, an "Mnumzana" is a respected elder or a wise person who has gained knowledge and wisdom through their experiences. The phrase "Bhala ibali" means "Tell a story", and it's often used to ask someone to share their experiences, wisdom, or teachings with others. So, in this context, you're asking Mqhagi the elder to share a story or impart his wisdom with you!|
| isiZulu:  Bhala indaba yoMnumzane Unogwaja <br><br>*(Translation: Write a story about Mr. Rabbit)* | Response: Kwakukhona kudala, kwakunohambo oluthile olwalubizwa ngokuthi uMnumzane uRabbit. Wayengumuntu omncane futhi othobekile, kodwa wayenomqondo omkhulu wokuhlola nokuthola izindawo ezintsha. Ngolunye usuku, wanquma ukuphuma ohambweni lwakhe lokuqala. Wahamba ngokusebenzisa ihlathi elikhulu neliluhlaza, enza abangani nabanye bezilwane endleleni. Wadlula emfuleni onamandla, wasuka ehlathini elimnyama nelithusayo, futhi wabuka izinkanyezi esibhakabhakeni sasebusuku. Ekugcineni, uMnumzane uRabbit wafika emzaneni omncane. Wamangala ngabantu abanobungane nobubele ababehlangana nabo. Wazithokozisa ngokudla okumnandi nezitsha ezimnandi abazenzela zona. UMnumzane uRabbit wahlala isikhathi eside futhi ekugcineni waba umngani wezinye izilwane emzaneni. Wafunda lukhulu kuzo, futhi ekugcineni wabuyela ekhaya enomuzwa omusha wokubonga impilo nokuba nomusa kubanye. <br><br>*((Translation: Once upon a time, there was a rabbit named Mr. Rabbit. He was a small and humble creature but had a great sense of adventure and curiosity. One day, he decided to embark on his first journey. He traveled through a large, green forest, making friends with other animals along the way. He crossed a powerful river, passed through a dark and scary forest, and gazed at the stars in the night sky. Finally, Mr. Rabbit arrived in a small village. He was amazed by the friendly and kind people he met. He enjoyed delicious food and sweet treats they made. Mr. Rabbit stayed for a long time and eventually became friends with other animals in the village. He learned a lot from them and eventually returned home with a new sense of gratitude for life and kindness to others.)* | traditional Zulu phrase! "Bhala indaba yoMnumzane Unogwaja" roughly translates to: "Speak the matter of Mnumzane Unogwaja" In Zulu culture, "indaba" means "matter" or "news", and "yoMnumzane" is a way of addressing someone with respect, using the title "Mnumzane" which means "sir" or "lord". So, the phrase is essentially saying "Tell me the news about Sir Unogwaja". Unogwaja was a legendary Zulu warrior and chief who played a significant role in the Anglo-Zulu War. The phrase is likely being used to refer to a specific story or news related to his life or actions. Would you like to know more about Unogwaja or Zulu culture?|
| English: Write a story about Mr.Hare <br><br>| Once upon a time, there was a mischievous rabbit named Mr. Hare who lived in a small burrow in the forest. He was very adventurous and loved to explore, but he was always careful not to get caught. One day, while exploring, he stumbled across a giant castle that no one had ever seen before. He decided to take a closer look and soon found himself inside a grand hall. He explored the entire castle until he stumbled across a secret chamber that contained a magical gem. He quickly gathered the gem and ran off, never looking back. From then on, Mr. Hare lived an adventurous life, using his newfound magic to help him explore the world.<br><br>|Once upon a time, in a sunny meadow nestled between two great oak trees, there lived a wise and witty rabbit named Mr. Hare. He was a gentle soul with a twinkle in his eye and a spring in his step. Mr. Hare spent his days lounging in the warm grass, munching on juicy clover, and watching the world go by. He loved nothing more than to sit back and observe the antics of the other creatures in the meadow. One day, as he was lazily munching on a particularly succulent clover, he spotted a group of rowdy field mice scurrying about, chasing each other in circles and squeaking with glee. Mr. Hare chuckled to himself, remembering when he was once as carefree and energetic as those little mice. As he sat there, lost in thought, a young fawn wandered into the meadow, her large brown eyes taking in the sights and sounds of the bustling community. She spotted Mr. Hare and trotted over to say hello. "Hello there, young one!" Mr. Hare exclaimed, his long ears perking up with excitement. "What brings you to our neck of the woods?" The fawn explained that she was new to the meadow and was looking for friends to play with. Mr. Hare's face lit up with a warm smile. "Ah, well, you've come to the right place! I know just the bunch of rascals who'd love to play with you." With a flick of his fluffy tail, Mr. Hare led the fawn on a wild adventure through the meadow.They chased after butterflies, played hide-and-seek behind the tall grasses, and even stumbled upon a hidden patch of sweet, ripe berries. As the sun began to set, casting a golden glow over the meadow, Mr. Hare settled down beside the fawn, feeling content and full of joy.|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
--
### Out-of-Scope Use
The use of the developed Large Language Model (LLM) capabilities is for research,social good and internal use purposes only. For commercial use and distribution, organisations/individuals are encouraged to contact[Jacaranda Health]((https://www.jacarandahealth.org/)). 
To ensure the ethical and responsible use of AfroLlama, we have outlined a set of guidelines.
These guidelines categorize activities and practices into three main areas: prohibited actions, high-risk activities, and deceptive practices. By understanding and adhering to these directives, users can contribute to a safer and more trustworthy environment.
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
1. **Prohibited Actions**:

- **Illegal Activities:** Avoid promoting violence, child exploitation, human trafficking, and other crimes.
- **Harassment and Discrimination:** No acts that bully, threaten, or discriminate.
- **Unauthorized Professions:** No unlicensed professional activities.
- **Data Misuse:** Handle personal data with proper consents.
- **Rights Violations:** Respect third-party rights.
- **Malware Creation:** Avoid creating harmful software.
  
2. **High-Risk Activities:**

- **Dangerous Industries:** No usage in military, nuclear, or espionage domains.
- **Weapons and Drugs:** Avoid illegal arms or drug activities.
- **Critical Systems:** No usage in key infrastructures or transport technologies.
- **Promotion of Harm:** Avoid content advocating self-harm or violence.
  
3. **Deceptive Practices:**

- **Misinformation:** Refrain from creating/promoting fraudulent or misleading info.
- **Defamation and Spam:** Avoid defamatory content and unsolicited messages.
- **Impersonation:** No pretending to be someone without authorization.
- **Misrepresentation:** No false claims about HauaLlama outputs.
- **Fake Online Engagement:** No promotion of false online interactions.


## Bias, Risks, and Limitations
AfroLlama is a cutting-edge technology brimming with possibilities, yet is not without inherent risks. The extensive testing conducted thus far has been predominantly in Swahili, Xhosa, Zulu, Yoruba, Hausa and English however leaving an expansive terrain of uncharted scenarios. Consequently, like its LLM counterparts, AfroLlama outcome predictability remains elusive, and there's the potential for it to occasionally generate responses that are either inaccurate, biased, or otherwise objectionable in nature when prompted by users.
With this in mind, the responsible course of action dictates that, prior to deploying AfroLlama in any applications, developers must embark on a diligent journey of safety testing and meticulous fine-tuning, customized to the unique demands of their specific use cases. 





## Contact-Us
For any questions, feedback, or commercial inquiries, please reach out at ai@jacarandahealth.org