SandLogic Technologies - Quantized Gemma-2-9b-IT Models
Model Description
We have quantized the Gemma-2-9b-IT model into three variants:
- Q5_KM
- Q4_KM
- IQ4_XS
These quantized models offer improved efficiency while maintaining performance.
Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub. To learn more about our company and services, check out our website at SandLogic.
Original Model Information
- Name: Gemma-2-9b-IT
- Developer: Google
- Model Type: Text-to-text, decoder-only large language model
- Architecture: Based on Gemini technology
- Parameters: 9 billion
- Training Data: 8 trillion tokens, including web documents, code, and mathematics
- Language: English
Model Capabilities
Gemma is designed for various text generation tasks, including:
- Question answering
- Summarization
- Reasoning
- Creative writing
- Code generation
The model is lightweight and suitable for deployment in resource-limited environments such as laptops, desktops, or personal cloud infrastructure.
Use Cases
- Text Generation: Create poems, scripts, code, marketing copy, and email drafts
- Chatbots and Conversational AI: Power customer service interfaces, virtual assistants, and interactive applications
- Text Summarization: Generate concise summaries of text corpora, research papers, or reports
Model Variants
We offer three quantized versions of the Gemma-2-9b-IT model:
- Q5_KM: 5-bit quantization using the KM method
- Q4_KM: 4-bit quantization using the KM method
- IQ4_XS: 4-bit quantization using the IQ4_XS method
These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.
Usage
pip install llama-cpp-python
Please refer to the llama-cpp-python documentation to install with GPU support.
Basic Text Completion
Here's an example demonstrating how to use the high-level API for basic text completion:
from llama_cpp import Llama
llm = Llama(
model_path="./models/7B/llama-model.gguf",
verbose=False,
# n_gpu_layers=-1, # Uncomment to use GPU acceleration
# n_ctx=2048, # Uncomment to increase the context window
)
output = llm(
"Q: Name the planets in the solar system? A: ", # Prompt
max_tokens=32, # Generate up to 32 tokens
stop=["Q:", "\n"], # Stop generating just before a new question
echo=False # Don't echo the prompt in the output
)
print(output["choices"][0]["text"])
Download
You can download Llama
models in gguf
format directly from Hugging Face using the from_pretrained
method. This feature requires the huggingface-hub
package.
To install it, run: pip install huggingface-hub
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="SandLogicTechnologies/Gemma-2-9b-it-GGUF",
filename="*gemma-2-9b-it-IQ4_XS.gguf",
verbose=False
)
By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.
Input and Output
- Input: Text string (e.g., question, prompt, or document to be summarized)
- Output: Generated English-language text in response to the input
License
Gemma 2 License: Google gemma
Acknowledgements
We thank Google for developing and releasing the original Gemma model. Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.
Contact
For any inquiries or support, please contact us at [email protected] or visit our support page.
- Downloads last month
- 15