Jacaranda/Xhosa_ZuluLlama3_v1

Model Card for Xhosa_ZuluLlama

Use this notebook to inference. We recommend you inference on L4 or A100 GPU.

Model Details

Xhosa-ZuluLlama3 is a 8B Parameters language model that builds upon the foundation of meta-llama/Meta-Llama-3-8B.It has been specifically enhanced to excel in processing and generating text in isiXhosa and isiZulu,languages used in South Africa. This model aims to improve natural language understanding and generation capabilities for isiZulu and isiXhosa-speaking users and researchers.

Model Description

Key features:

• Improved performance on isiXhosa and isiZulu language tasks
• Maintains general language capabilities of the original Llama 3 model
• Optimized for both understanding and generating isiXhosa and isiZulu text

Training The training process for Xhosa-ZuluLlama3 involved two main stages:

1. LoRA-based Continual Pre-training: We conducted continuous pre-training using publicly available isiXhosa and isiZulu corpi, which we pre-processed using the Meta/Llama3 tokenizer. The primary focus was on causal language modeling,specifically training the model to predict the next isiXhosa/ isiZulutokens based on preceding isiXhosa/isiZulu tokens. Our continuous pre- training involved implementing the LoRA technique, where-in we froze the base model parameters of the foundation Meta/Llama3 model and introduced additional lightweight components(adpaters). adapters. These adapters were specifically trained to capture the intricacies, terminologies, and nuances of the isiXhosa and isiZulu language. This approach facilitated a balance between leveraging the knowledge embedded in the pre-trained Meta/Llama3 model and optimizing it for isiXhosa and isiZulu language, all without incurring the computational costs associated with retraining the entire Llama3 model.

2. LoRA-based Instruction Tuning:

Fine-tuned on a curated dataset of isiXhosa and isiZulu instructions and responses Included task-specific data to improve performance on common language tasks Emphasized maintaining coherence and contextual understanding in isiXhosa and isiZulu

Incorporated safety datasets to improve the model's ability to generate safe and ethical responses Included examples of harmful content and appropriate non-harmful alternatives Focused on reducing biases and improving the model's understanding of cultural sensitivities in the isiXhosa and isiZulu context The use of LoRA (Low-Rank Adaptation) for both continual pre-training and instruction tuning allowed for efficient adaptation of the base Llama 3 model while minimizing computational resources and preventing catastrophic forgetting of the original model's capabilities. Approximate dataset sizes:

Continual pre-training:["11.3 GB of text"] Instruction tuning:[66,280 instruction-response pairs"]

• Developed by: Jacaranda Health
• Model type: Llama
• Language(s) (NLP): isiXhosa and English
• License: CC BY-NC-SA 4.0 DEED
• Model Developers: Stanslaus Mwongela, Jay Patel, Sathy Rajasekharan, Lyvia Lusiji, Francesco Piccino, Mfoniso Ukwak, Ellen Sebastian

Uses

Xhosa-ZuluLlama3 is optimized for downstream tasks, notably those demanding instructional datasets in isiXhosa, isiZulu English, or both. Organizations can further fine-tune it for their specific domains. Potential areas include:

• Question-answering within specific domains.
• Assistant-driven chat capabilities: healthcare, agriculture, legal, education, tourism and hospitality, public services, financial sectors, communication, customer assistance, commerce, etcpublic services, financial sectors, communication, customer assistance, commerce, etc.

Sample Evaluation Prompts

Prompt	Xhosa-ZuluLlama3 Response	Meta-Llama-3-8B-Instruct Response
isiXhosa: Bhala ibali ngomvundla (Translation: Write a story about a rabbit)	Kwakudala-dala, kwakukho umvundla ogama lingupeter owayehlala ehlathini. Wayengumvundla onobuganga kwaye wayethanda ukuphonononga ihlathi. Ngamini ithile, wafumanisa indlu encinci endala apho angene khona. Ngaphakathi, wafumana igumbi eliyimfihlakalo eligcwele izinto ezimangalisayo. Wachitha iiyure ezininzi ehlola le ndawo imangalisayo, efumanisa iimfihlo zayo. Ekugqibeleni, waphuma endlwini enomdla omtsha kunye nokuqonda okutsha ngobomi. (Translation: Long ago, there was a rabbit named Peter who lived in the forest. He was a curious rabbit and loved to explore the forest. One day, he discovered an old small house that he could enter. Inside, he found a mysterious room full of amazing things. He spent many hours examining this wonderful place, uncovering its secrets. Finally, he left the house with a new curiosity and a fresh understanding of life.)	I think you're speaking in isiZulu!"Bhala ibali ngomvundla" roughly translates to "Sing a song of the ancestors" or "Sing a song for the ancestors". In African cultures, singing is often used as a way to connect with one's heritage and honor one's ancestors.
isiZulu: Bhala indaba yoMnumzane Unogwaja (Translation: Write the story of Mr. Hare)	Kwakukhona kudala, kwakunogwaja omncane owayehlala ehlathini. Wayengumuntu othanda ukuzidela futhi enesibindi, njalo efuna izinto ezintsha nezijabulisayo zokwenza. Ngelinye ilanga, wanquma ukuhlola ihlathi elingajwayelekile nelisabekayo elalikude kakhulu. Wazithola esethungelwe yizidalwa ezingaqondakali nezimfihlakalo, kodwa akazange adikibale futhi waqhubeka nohambo lwakhe. Ekugcineni, wathola indlela yokuphuma ehlathini futhi wabuyela ekhaya ephephile, enomuzwa omusha wokuziqhenya nokufuna ukwazi. (Translation: Once upon a time, there was a small hare who lived in the forest. He was a curious and brave creature, always looking for new and exciting things to do. One day, he decided to explore an unfamiliar and scary forest that was far away. He found himself surrounded by strange creatures and mysteries, but he never gave up and continued on his journey. Eventually, he found his way out of the forest and returned home safely, with a new sense of pride and curiosity.)	A traditional Zulu phrase! "Bhala indaba yoMnumzane Unogwaja" roughly translates to "Speak the truth, O respected elder Unogwaja". In Zulu culture, "Mnumzane" is a term of respect used to address an elderly person, particularly a chief or a respected elder. "Unogwaja" is likely a specific name or title of a respected elder. The phrase "Bhala indaba" means "Speak the truth", emphasizing the importance of honesty and integrity in communication. So, when addressing Unogwaja, the speaker is essentially asking him to speak the truth, as a respected elder and a trusted authority figure.
isiXhosa: Yintoni ebangela intloko ebuhlungu? (Translation: What causes headaches?)	Iintloko zinokubangelwa zizinto ezahlukeneyo, kubandakanya uxinzelelo lwezihlunu, ukuzilolonga ngokugqithisileyo, ukungabikho kokulala okufanelekileyo, utshintsho lokutya, nokusabela kwimozulu. Kubalulekile ukuba uhlale uhlolisisa iingcebiso zonyango ukuchonga isizathu esichanileyo ukuze ufumane unyango olufanelekileyo. (Translation:Headaches can be caused by various factors, including muscle tension, overexertion, lack of proper sleep, dietary changes, and weather reactions. It's important to always seek medical advice to identify the exact cause in order to get appropriate treatment.)	A question in isiXhosa! The answer is: "Ndiyintoni." This means "I'm coming" or "I'll be there." Here's a breakdown of the sentence: * "Yintoni" is the verb "toni" (to come) in the second person singular (addressing someone directly), with the prefix "yi-" which indicates the speaker is the one who is doing the action. * "Ebangela" is the verb "bangela" (to arrive) in the first person singular (referring to oneself), with the prefix "e-" which indicates the subject of the sentence is the speaker. * "Intloko" means "head" or "top", but in this context, it's likely being used to mean "first" or "initially". * "Ebuhlungu" is the phrase "buhlungu" (to go) in the first person singular (referring to oneself), with the prefix "e-" which indicates the subject of the sentence is the speaker.So, when you put it all together, "Ndiyintoni ebangela intloko ebuhlungu" means "I'm coming, I'll be there first" or "I'll be there initially".
isiZulu: Bala izinzuzo ezinhlanu zomanyolo (Translation: List five benefits of manure)	Izinzuzo ezinhlanu zokusebenzisa umanyolo zimi kanje: ukuthuthukisa impilo yomhlabathi, ukukhulisa isivuno sezitshalo, ukunciphisa ingozi yezifo zezitshalo, ukuthuthukisa ikhwalithi yamanzi angaphansi komhlaba, kanye nokugcina amanzi. Umanyolo uphinde uthuthukise ukumuncwa kwezakhamzimba ngumhlabathi futhi unikeze izakhamzimba eziyimfuneko kumagciwane asiza ekugayeni izakhamzimba. Ngaphezu kwalokho, ungasisiza ekuvikeleni ukuhlangana kwezakhi zomhlabathi futhi unciphise ukuguguleka komhlabathi. (Translation: The five benefits of using manure are as follows: improving soil health, increasing crop yield, reducing the risk of plant diseases, improving groundwater quality, and water retention. Manure also improves nutrient absorption by the soil and provides essential nutrients to microorganisms that help in nutrient breakdown. Furthermore, it can help prevent soil compaction and reduce soil erosion. )	It seems like you're speaking in Xhosa! "Bala izinzuzo ezinhlanu zomanyolo" roughly translates to "The problems of the people are many and complex". In Xhosa, "bala" means "problems" or "troubles", "izinzuzo" is the plural form of "inzuzu", which means "people", "ezinhlanu" is the plural form of "inhlanu", which means "many", and "zomanyolo" is the phrase "of complexity" or "of difficulty". Please let me know if I'm correct or if there's anything else I can help you with!

Out-of-Scope Use

The use of the developed Large Language Model (LLM) capabilities is for research,social good and internal use purposes only. For commercial use and distribution, organisations/individuals are encouraged to contactJacaranda Health. To ensure the ethical and responsible use of XhosaLlama, we have outlined a set of guidelines. These guidelines categorize activities and practices into three main areas: prohibited actions, high-risk activities, and deceptive practices. By understanding and adhering to these directives, users can contribute to a safer and more trustworthy environment.

1. Prohibited Actions:

• Illegal Activities: Avoid promoting violence, child exploitation, human trafficking, and other crimes.
• Harassment and Discrimination: No acts that bully, threaten, or discriminate.
• Unauthorized Professions: No unlicensed professional activities.
• Data Misuse: Handle personal data with proper consents.
• Rights Violations: Respect third-party rights.
• Malware Creation: Avoid creating harmful software.

2. High-Risk Activities:

• Dangerous Industries: No usage in military, nuclear, or espionage domains.
• Weapons and Drugs: Avoid illegal arms or drug activities.
• Critical Systems: No usage in key infrastructures or transport technologies.
• Promotion of Harm: Avoid content advocating self-harm or violence.

3. Deceptive Practices:

• Misinformation: Refrain from creating/promoting fraudulent or misleading info.
• Defamation and Spam: Avoid defamatory content and unsolicited messages.
• Impersonation: No pretending to be someone without authorization.
• Misrepresentation: No false claims about Xhosa-ZuluLlama3 outputs.
• Fake Online Engagement: No promotion of false online interactions.

Bias, Risks, and Limitations

Xhosa-ZuluLlama3 is a cutting-edge technology brimming with possibilities, yet is not without inherent risks. The extensive testing conducted thus far has been predominantly in isiXhosa and English, however leaving an expansive terrain of uncharted scenarios. Consequently, like its LLM counterparts, Xhosa-ZuluLlama3 outcome predictability remains elusive, and there's the potential for it to occasionally generate responses that are either inaccurate, biased, or otherwise objectionable in nature when prompted by users. With this in mind, the responsible course of action dictates that, prior to deploying Xhosa-ZuluLlama3 in any applications, developers must embark on a diligent journey of safety testing and meticulous fine-tuning, customized to the unique demands of their specific use cases.

Contact-Us

For any questions, feedback, or commercial inquiries, please reach out at [email protected]