|
--- |
|
library_name: gemma2.java |
|
license: gemma |
|
base_model: google/gemma-2-9b-it |
|
base_model_relation: quantized |
|
quantized_by: mukel |
|
tags: |
|
- gemma2 |
|
- java |
|
- llama3.java |
|
- gemma2.java |
|
--- |
|
|
|
# GGUF models for gemma2.java |
|
Pure .gguf `Q4_0` and `Q8_0` quantizations of Gemma 2 models, ready to consume by [gemma2.java](https://github.com/mukel/gemma2.java). |
|
|
|
In the wild, `Q8_0` quantizations are fine, but `Q4_0` quantizations are rarely pure e.g. the `output.weights` tensor is quantized with `Q6_K`, instead of `Q4_0`. |
|
A pure `Q4_0` quantization can be generated from a high precision (F32, F16, BFLOAT16) .gguf source with the `llama-quantize` utility from llama.cpp as follows: |
|
|
|
``` |
|
./llama-quantize --pure ./Gemma-2-9B-Instruct-F32.gguf ./Gemma-2-9B-Instruct-Q4_0.gguf Q4_0 |
|
``` |
|
|
|
# Gemma Model Card |
|
|
|
**Model Page**: [Gemma](https://ai.google.dev/gemma/docs) |
|
|
|
This model card corresponds to the 2b instruct version the Gemma 2 model in GGUF Format. |
|
|
|
You can also visit the model card of the [9B pretrained v2 model](https://huggingface.co/google/gemma-2-9b-it). |
|
|
|
## Model Information |
|
|
|
Summary description and brief definition of inputs and outputs. |
|
|
|
### Description |
|
|
|
Gemma is a family of lightweight, state-of-the-art open models from Google, |
|
built from the same research and technology used to create the Gemini models. |
|
They are text-to-text, decoder-only large language models, available in English, |
|
with open weights, pre-trained variants, and instruction-tuned variants. Gemma |
|
models are well-suited for a variety of text generation tasks, including |
|
question answering, summarization, and reasoning. Their relatively small size |
|
makes it possible to deploy them in environments with limited resources such as |
|
a laptop, desktop or your own cloud infrastructure, democratizing access to |
|
state of the art AI models and helping foster innovation for everyone. |
