language:
- en
- hi
tags:
- Llama3
- Pytorch
- Bilingual
pipeline_tag: text-generation
SandLogic Technologies - Quantized LLama3-Gaja-Hindi-8B Models
Model Description
We have quantized the LLama3-Gaja-Hindi-8B model into two variants:
- Q5_KM
- Q4_KM
These quantized models offer improved efficiency while maintaining performance.
Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub. To learn more about our company and services, check out our website at SandLogic.
Original Model Information
- Name: LLama3-Gaja-Hindi-8B-v0.1
- Developer: Cognitivelab.in
- Base Model: Llama3-8B
- Model Type: Bilingual English/Hindi language model
- Parameters: 8 billion
- Specialization: Natural language understanding, particularly for instructional pairs
Model Capabilities
LLama3-Gaja-Hindi-8B is designed for bilingual (English/Hindi) natural language understanding tasks, with a focus on:
- Responding appropriately in either English or Hindi based on user prompts
- Understanding and generating instructional content in both languages
- Handling a variety of natural language processing tasks across both languages
Training Approach
The model underwent supervised fine-tuning with low-rank adaptation, focusing on bilingual instruct fine-tuning. The training data consisted of a curated dataset of translated instructional pairs.
Use Cases
- Bilingual Conversational AI: Chatbots and virtual assistants with English/Hindi capabilities
- Language Learning Tools: Interactive platforms for English and Hindi learners
- Content Translation: Assistance in translating between English and Hindi, especially for instructional materials
- Cross-lingual Information Retrieval: Enabling queries in one language with responses in either English or Hindi
- Cultural Context Understanding: Helping users grasp cultural nuances in both languages
- Multilingual Customer Support: Powering customer service applications for diverse user bases
Model Variants
We offer two quantized versions of the LLama3-Gaja-Hindi-8B model:
- Q5_KM: 5-bit quantization using the KM method
- Q4_KM: 4-bit quantization using the KM method
These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.
Input and Output
- Input: Text prompts or instructions in either English or Hindi
- Output: Generated text responses in the same language as the input, or as specified in the prompt
Usage
pip install llama-cpp-python
Please refer to the llama-cpp-python documentation to install with GPU support.
Basic Text Completion
Here's an example demonstrating how to use the high-level API for basic text completion:
from llama_cpp import Llama
llm = Llama(
model_path="./models/7B/LLama3-Gaja-Hindi-8B-v0.1.gguf",
verbose=False,
# n_gpu_layers=-1, # Uncomment to use GPU acceleration
# n_ctx=2048, # Uncomment to increase the context window
)
output = llm.create_chat_completion(
messages =[
{
"role": "system",
"content": """ You are an AI assistant trained on top of Llama 3 Large language model (LLM), proficient in English and Hindi. You can respond in both languages based on the user's request."""
,
},
{"role": "user", "content": "Write an poem in hindi"},
]
)
print(output["choices"][0]['message']['content'])
Download
You can download Llama
models in gguf
format directly from Hugging Face using the from_pretrained
method. This feature requires the huggingface-hub
package.
To install it, run: pip install huggingface-hub
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="SandLogicTechnologies/LLama3-Gaja-Hindi-8B-GGUF",
filename="*llama3-gaja-hindi-8b-v0.1.Q5_K_M.gguf",
verbose=False
)
By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.
Ethical Considerations
Users should be aware of potential biases in the model's outputs, especially when dealing with cultural or linguistic nuances. Always review and validate the model's responses for sensitive applications.
Acknowledgements
We thank Cognitivelab.in for developing the original LLama3-Gaja-Hindi-8B model and the creators of Llama3 for their foundational work.Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.
Contact
For any inquiries or support, please contact us at [email protected] or visit our support page.