lodestones commited on
Commit
441708b
1 Parent(s): e610f02

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +358 -113
README.md CHANGED
@@ -1,201 +1,446 @@
1
  ---
2
  library_name: transformers
3
  tags: []
 
 
 
 
 
 
4
  ---
5
 
6
- # Model Card for Model ID
7
 
8
- <!-- Provide a quick summary of what the model is/does. -->
9
 
 
10
 
 
11
 
12
- ## Model Details
 
 
13
 
14
- ### Model Description
15
 
16
- <!-- Provide a longer summary of what this model is. -->
17
 
18
- This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
19
 
20
- - **Developed by:** [More Information Needed]
21
- - **Funded by [optional]:** [More Information Needed]
22
- - **Shared by [optional]:** [More Information Needed]
23
- - **Model type:** [More Information Needed]
24
- - **Language(s) (NLP):** [More Information Needed]
25
- - **License:** [More Information Needed]
26
- - **Finetuned from model [optional]:** [More Information Needed]
27
 
28
- ### Model Sources [optional]
29
 
30
- <!-- Provide the basic links for the model. -->
 
 
 
 
 
 
 
 
31
 
32
- - **Repository:** [More Information Needed]
33
- - **Paper [optional]:** [More Information Needed]
34
- - **Demo [optional]:** [More Information Needed]
35
 
36
- ## Uses
37
 
38
- <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
 
40
- ### Direct Use
41
 
42
- <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
 
43
 
44
- [More Information Needed]
45
 
46
- ### Downstream Use [optional]
47
 
48
- <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
49
 
50
- [More Information Needed]
 
51
 
52
- ### Out-of-Scope Use
 
53
 
54
- <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
 
55
 
56
- [More Information Needed]
57
 
58
- ## Bias, Risks, and Limitations
59
 
60
- <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
 
62
- [More Information Needed]
 
 
63
 
64
- ### Recommendations
 
65
 
66
- <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
67
 
68
- Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
 
69
 
70
- ## How to Get Started with the Model
71
 
72
- Use the code below to get started with the model.
73
 
74
- [More Information Needed]
75
 
76
- ## Training Details
 
 
77
 
78
- ### Training Data
 
79
 
80
- <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
81
 
82
- [More Information Needed]
 
 
83
 
84
- ### Training Procedure
85
 
86
- <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
 
87
 
88
- #### Preprocessing [optional]
 
89
 
90
- [More Information Needed]
 
91
 
 
 
 
92
 
93
- #### Training Hyperparameters
94
 
95
- - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
 
97
- #### Speeds, Sizes, Times [optional]
 
 
98
 
99
- <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
 
101
- [More Information Needed]
 
102
 
103
- ## Evaluation
104
-
105
- <!-- This section describes the evaluation protocols and provides the results. -->
106
-
107
- ### Testing Data, Factors & Metrics
108
-
109
- #### Testing Data
110
-
111
- <!-- This should link to a Dataset Card if possible. -->
112
-
113
- [More Information Needed]
114
-
115
- #### Factors
116
 
117
- <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
 
 
118
 
119
- [More Information Needed]
120
 
121
- #### Metrics
 
 
122
 
123
- <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
 
125
- [More Information Needed]
 
126
 
127
- ### Results
 
128
 
129
- [More Information Needed]
 
 
130
 
131
- #### Summary
132
 
 
133
 
 
134
 
135
- ## Model Examination [optional]
136
 
137
- <!-- Relevant interpretability work for the model goes here -->
 
 
 
 
 
 
138
 
139
- [More Information Needed]
140
 
141
- ## Environmental Impact
 
 
 
142
 
143
- <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
 
145
- Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
 
147
- - **Hardware Type:** [More Information Needed]
148
- - **Hours used:** [More Information Needed]
149
- - **Cloud Provider:** [More Information Needed]
150
- - **Compute Region:** [More Information Needed]
151
- - **Carbon Emitted:** [More Information Needed]
152
 
153
- ## Technical Specifications [optional]
 
154
 
155
- ### Model Architecture and Objective
 
 
 
 
 
 
 
156
 
157
- [More Information Needed]
 
 
158
 
159
- ### Compute Infrastructure
160
 
161
- [More Information Needed]
 
162
 
163
- #### Hardware
 
 
 
 
 
 
 
164
 
165
- [More Information Needed]
166
 
167
- #### Software
168
 
169
- [More Information Needed]
170
 
171
- ## Citation [optional]
 
172
 
173
- <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
 
174
 
175
- **BibTeX:**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
176
 
177
- [More Information Needed]
178
 
179
- **APA:**
180
 
181
- [More Information Needed]
 
182
 
183
- ## Glossary [optional]
 
 
 
184
 
185
- <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
 
 
 
 
186
 
187
- [More Information Needed]
188
-
189
- ## More Information [optional]
190
-
191
- [More Information Needed]
192
-
193
- ## Model Card Authors [optional]
194
-
195
- [More Information Needed]
196
-
197
- ## Model Card Contact
198
-
199
- [More Information Needed]
200
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
201
 
 
1
  ---
2
  library_name: transformers
3
  tags: []
4
+ extra_gated_heading: "Access Gemma on Hugging Face"
5
+ extra_gated_prompt: "To access Gemma on Hugging Face, you’re required to review and agree to Google’s usage license. To do this, please ensure you’re logged-in to Hugging Face and click below. Requests are processed immediately."
6
+ extra_gated_button_content: "Acknowledge license"
7
+ license: other
8
+ license_name: gemma-terms-of-use
9
+ license_link: https://ai.google.dev/gemma/terms
10
  ---
11
 
12
+ # Gemma Model Card
13
 
14
+ **Model Page**: [Gemma](https://ai.google.dev/gemma/docs)
15
 
16
+ This model card corresponds to the 7B base version of the Gemma model. You can also visit the model card of the [2B base model](https://huggingface.co/google/gemma-2b), [7B instruct model](https://huggingface.co/google/gemma-7b-it), and [2B instruct model](https://huggingface.co/google/gemma-2b-it).
17
 
18
+ **Resources and Technical Documentation**:
19
 
20
+ * [Responsible Generative AI Toolkit](https://ai.google.dev/responsible)
21
+ * [Gemma on Kaggle](https://www.kaggle.com/models/google/gemma)
22
+ * [Gemma on Vertex Model Garden](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/335?version=gemma-7b-gg-hf)
23
 
24
+ **Terms of Use**: [Terms](https://www.kaggle.com/models/google/gemma/license/consent)
25
 
26
+ **Authors**: Google
27
 
28
+ ## Model Information
29
 
30
+ Summary description and brief definition of inputs and outputs.
 
 
 
 
 
 
31
 
32
+ ### Description
33
 
34
+ Gemma is a family of lightweight, state-of-the-art open models from Google,
35
+ built from the same research and technology used to create the Gemini models.
36
+ They are text-to-text, decoder-only large language models, available in English,
37
+ with open weights, pre-trained variants, and instruction-tuned variants. Gemma
38
+ models are well-suited for a variety of text generation tasks, including
39
+ question answering, summarization, and reasoning. Their relatively small size
40
+ makes it possible to deploy them in environments with limited resources such as
41
+ a laptop, desktop or your own cloud infrastructure, democratizing access to
42
+ state of the art AI models and helping foster innovation for everyone.
43
 
44
+ ### Usage
 
 
45
 
46
+ Below we share some code snippets on how to get quickly started with running the model. First make sure to `pip install -U transformers`, then copy the snippet from the section that is relevant for your usecase.
47
 
48
+ #### Fine-tuning examples
49
 
50
+ You can find fine-tuning notebooks under the [`examples/` directory](https://huggingface.co/google/gemma-7b/tree/main/examples). We provide:
51
 
52
+ * A script to perform Supervised Fine-Tuning (SFT) on UltraChat dataset using [QLoRA](https://huggingface.co/papers/2305.14314)
53
+ * A script to perform SFT using FSDP on TPU devices
54
+ * A notebook that you can run on a free-tier Google Colab instance to perform SFT on English quotes dataset
55
 
56
+ #### Running the model on a CPU
57
 
 
58
 
59
+ ```python
60
+ from transformers import AutoTokenizer, AutoModelForCausalLM
61
 
62
+ tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")
63
+ model = AutoModelForCausalLM.from_pretrained("google/gemma-7b")
64
 
65
+ input_text = "Write me a poem about Machine Learning."
66
+ input_ids = tokenizer(input_text, return_tensors="pt")
67
 
68
+ outputs = model.generate(**input_ids)
69
+ print(tokenizer.decode(outputs[0]))
70
+ ```
71
 
 
72
 
73
+ #### Running the model on a single / multi GPU
74
 
 
75
 
76
+ ```python
77
+ # pip install accelerate
78
+ from transformers import AutoTokenizer, AutoModelForCausalLM
79
 
80
+ tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")
81
+ model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", device_map="auto")
82
 
83
+ input_text = "Write me a poem about Machine Learning."
84
+ input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
85
 
86
+ outputs = model.generate(**input_ids)
87
+ print(tokenizer.decode(outputs[0]))
88
+ ```
89
 
 
90
 
91
+ #### Running the model on a GPU using different precisions
92
 
93
+ * _Using `torch.float16`_
94
 
95
+ ```python
96
+ # pip install accelerate
97
+ from transformers import AutoTokenizer, AutoModelForCausalLM
98
 
99
+ tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")
100
+ model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", device_map="auto", torch_dtype=torch.float16)
101
 
102
+ input_text = "Write me a poem about Machine Learning."
103
+ input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
104
 
105
+ outputs = model.generate(**input_ids)
106
+ print(tokenizer.decode(outputs[0]))
107
+ ```
108
 
109
+ * _Using `torch.bfloat16`_
110
 
111
+ ```python
112
+ # pip install accelerate
113
+ from transformers import AutoTokenizer, AutoModelForCausalLM
114
 
115
+ tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")
116
+ model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", device_map="auto", torch_dtype=torch.bfloat16)
117
 
118
+ input_text = "Write me a poem about Machine Learning."
119
+ input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
120
 
121
+ outputs = model.generate(**input_ids)
122
+ print(tokenizer.decode(outputs[0]))
123
+ ```
124
 
125
+ #### Quantized Versions through `bitsandbytes`
126
 
127
+ * _Using 8-bit precision (int8)_
128
 
129
+ ```python
130
+ # pip install bitsandbytes accelerate
131
+ from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
132
 
133
+ quantization_config = BitsAndBytesConfig(load_in_8bit=True)
134
 
135
+ tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")
136
+ model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", quantization_config=quantization_config)
137
 
138
+ input_text = "Write me a poem about Machine Learning."
139
+ input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
 
 
 
 
 
 
 
 
 
 
 
140
 
141
+ outputs = model.generate(**input_ids)
142
+ print(tokenizer.decode(outputs[0]))
143
+ ```
144
 
145
+ * _Using 4-bit precision_
146
 
147
+ ```python
148
+ # pip install bitsandbytes accelerate
149
+ from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
150
 
151
+ quantization_config = BitsAndBytesConfig(load_in_4bit=True)
152
 
153
+ tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b")
154
+ model = AutoModelForCausalLM.from_pretrained("google/gemma-7b", quantization_config=quantization_config)
155
 
156
+ input_text = "Write me a poem about Machine Learning."
157
+ input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
158
 
159
+ outputs = model.generate(**input_ids)
160
+ print(tokenizer.decode(outputs[0]))
161
+ ```
162
 
 
163
 
164
+ #### Other optimizations
165
 
166
+ * _Flash Attention 2_
167
 
168
+ First make sure to install `flash-attn` in your environment `pip install flash-attn`
169
 
170
+ ```diff
171
+ model = AutoModelForCausalLM.from_pretrained(
172
+ model_id,
173
+ torch_dtype=torch.float16,
174
+ + attn_implementation="flash_attention_2"
175
+ ).to(0)
176
+ ```
177
 
178
+ ### Inputs and outputs
179
 
180
+ * **Input:** Text string, such as a question, a prompt, or a document to be
181
+ summarized.
182
+ * **Output:** Generated English-language text in response to the input, such
183
+ as an answer to a question, or a summary of a document.
184
 
185
+ ## Model Data
186
 
187
+ Data used for model training and how the data was processed.
188
 
189
+ ### Training Dataset
 
 
 
 
190
 
191
+ These models were trained on a dataset of text data that includes a wide variety
192
+ of sources, totaling 6 trillion tokens. Here are the key components:
193
 
194
+ * Web Documents: A diverse collection of web text ensures the model is exposed
195
+ to a broad range of linguistic styles, topics, and vocabulary. Primarily
196
+ English-language content.
197
+ * Code: Exposing the model to code helps it to learn the syntax and patterns of
198
+ programming languages, which improves its ability to generate code or
199
+ understand code-related questions.
200
+ * Mathematics: Training on mathematical text helps the model learn logical
201
+ reasoning, symbolic representation, and to address mathematical queries.
202
 
203
+ The combination of these diverse data sources is crucial for training a powerful
204
+ language model that can handle a wide variety of different tasks and text
205
+ formats.
206
 
207
+ ### Data Preprocessing
208
 
209
+ Here are the key data cleaning and filtering methods applied to the training
210
+ data:
211
 
212
+ * CSAM Filtering: Rigorous CSAM (Child Sexual Abuse Material) filtering was
213
+ applied at multiple stages in the data preparation process to ensure the
214
+ exclusion of harmful and illegal content
215
+ * Sensitive Data Filtering: As part of making Gemma pre-trained models safe and
216
+ reliable, automated techniques were used to filter out certain personal
217
+ information and other sensitive data from training sets.
218
+ * Additional methods: Filtering based on content quality and safely in line with
219
+ [our policies](https://storage.googleapis.com/gweb-uniblog-publish-prod/documents/2023_Google_AI_Principles_Progress_Update.pdf#page=11).
220
 
221
+ ## Implementation Information
222
 
223
+ Details about the model internals.
224
 
225
+ ### Hardware
226
 
227
+ Gemma was trained using the latest generation of
228
+ [Tensor Processing Unit (TPU)](https://cloud.google.com/tpu/docs/intro-to-tpu) hardware (TPUv5e).
229
 
230
+ Training large language models requires significant computational power. TPUs,
231
+ designed specifically for matrix operations common in machine learning, offer
232
+ several advantages in this domain:
233
 
234
+ * Performance: TPUs are specifically designed to handle the massive computations
235
+ involved in training LLMs. They can speed up training considerably compared to
236
+ CPUs.
237
+ * Memory: TPUs often come with large amounts of high-bandwidth memory, allowing
238
+ for the handling of large models and batch sizes during training. This can
239
+ lead to better model quality.
240
+ * Scalability: TPU Pods (large clusters of TPUs) provide a scalable solution for
241
+ handling the growing complexity of large foundation models. You can distribute
242
+ training across multiple TPU devices for faster and more efficient processing.
243
+ * Cost-effectiveness: In many scenarios, TPUs can provide a more cost-effective
244
+ solution for training large models compared to CPU-based infrastructure,
245
+ especially when considering the time and resources saved due to faster
246
+ training.
247
+ * These advantages are aligned with
248
+ [Google's commitments to operate sustainably](https://sustainability.google/operating-sustainably/).
249
 
250
+ ### Software
251
 
252
+ Training was done using [JAX](https://github.com/google/jax) and [ML Pathways](https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture).
253
 
254
+ JAX allows researchers to take advantage of the latest generation of hardware,
255
+ including TPUs, for faster and more efficient training of large models.
256
 
257
+ ML Pathways is Google's latest effort to build artificially intelligent systems
258
+ capable of generalizing across multiple tasks. This is specially suitable for
259
+ [foundation models](https://ai.google/discover/foundation-models/), including large language models like
260
+ these ones.
261
 
262
+ Together, JAX and ML Pathways are used as described in the
263
+ [paper about the Gemini family of models](https://arxiv.org/abs/2312.11805); "the 'single
264
+ controller' programming model of Jax and Pathways allows a single Python
265
+ process to orchestrate the entire training run, dramatically simplifying the
266
+ development workflow."
267
 
268
+ ## Evaluation
 
 
 
 
 
 
 
 
 
 
 
 
269
 
270
+ Model evaluation metrics and results.
271
+
272
+ ### Benchmark Results
273
+
274
+ These models were evaluated against a large collection of different datasets and
275
+ metrics to cover different aspects of text generation:
276
+
277
+ | Benchmark | Metric | 2B Params | 7B Params |
278
+ | ------------------------------ | ------------- | ----------- | --------- |
279
+ | [MMLU](https://arxiv.org/abs/2009.03300) | 5-shot, top-1 | 42.3 | 64.3 |
280
+ | [HellaSwag](https://arxiv.org/abs/1905.07830) | 0-shot |71.4 | 81.2 |
281
+ | [PIQA](https://arxiv.org/abs/1911.11641) | 0-shot | 77.3 | 81.2 |
282
+ | [SocialIQA](https://arxiv.org/abs/1904.09728) | 0-shot | 59.7 | 51.8 |
283
+ | [BooIQ](https://arxiv.org/abs/1905.10044) | 0-shot | 69.4 | 83.2 |
284
+ | [WinoGrande](https://arxiv.org/abs/1907.10641) | partial score | 65.4 | 72.3 |
285
+ | [CommonsenseQA](https://arxiv.org/abs/1811.00937) | 7-shot | 65.3 | 71.3 |
286
+ | [OpenBookQA](https://arxiv.org/abs/1809.02789) | | 47.8 | 52.8 |
287
+ | [ARC-e](https://arxiv.org/abs/1911.01547) | | 73.2 | 81.5 |
288
+ | [ARC-c](https://arxiv.org/abs/1911.01547) | | 42.1 | 53.2 |
289
+ | [TriviaQA](https://arxiv.org/abs/1705.03551) | 5-shot | 53.2 | 63.4 |
290
+ | [Natural Questions](https://github.com/google-research-datasets/natural-questions) | 5-shot | - | 23 |
291
+ | [HumanEval](https://arxiv.org/abs/2107.03374) | pass@1 | 22.0 | 32.3 |
292
+ | [MBPP](https://arxiv.org/abs/2108.07732) | 3-shot | 29.2 | 44.4 |
293
+ | [GSM8K](https://arxiv.org/abs/2110.14168) | maj@1 | 17.7 | 46.4 |
294
+ | [MATH](https://arxiv.org/abs/2108.07732) | 4-shot | 11.8 | 24.3 |
295
+ | [AGIEval](https://arxiv.org/abs/2304.06364) | | 24.2 | 41.7 |
296
+ | [BIG-Bench](https://arxiv.org/abs/2206.04615) | | 35.2 | 55.1 |
297
+ | ------------------------------ | ------------- | ----------- | --------- |
298
+ | **Average** | | **54.0** | **56.4** |
299
+
300
+ ## Ethics and Safety
301
+
302
+ Ethics and safety evaluation approach and results.
303
+
304
+ ### Evaluation Approach
305
+
306
+ Our evaluation methods include structured evaluations and internal red-teaming
307
+ testing of relevant content policies. Red-teaming was conducted by a number of
308
+ different teams, each with different goals and human evaluation metrics. These
309
+ models were evaluated against a number of different categories relevant to
310
+ ethics and safety, including:
311
+
312
+ * Text-to-Text Content Safety: Human evaluation on prompts covering safety
313
+ policies including child sexual abuse and exploitation, harassment, violence
314
+ and gore, and hate speech.
315
+ * Text-to-Text Representational Harms: Benchmark against relevant academic
316
+ datasets such as [WinoBias](https://arxiv.org/abs/1804.06876) and [BBQ Dataset](https://arxiv.org/abs/2110.08193v2).
317
+ * Memorization: Automated evaluation of memorization of training data, including
318
+ the risk of personally identifiable information exposure.
319
+ * Large-scale harm: Tests for "dangerous capabilities," such as chemical,
320
+ biological, radiological, and nuclear (CBRN) risks.
321
+
322
+ ### Evaluation Results
323
+
324
+ The results of ethics and safety evaluations are within acceptable thresholds
325
+ for meeting [internal policies](https://storage.googleapis.com/gweb-uniblog-publish-prod/documents/2023_Google_AI_Principles_Progress_Update.pdf#page=11) for categories such as child
326
+ safety, content safety, representational harms, memorization, large-scale harms.
327
+ On top of robust internal evaluations, the results of well known safety
328
+ benchmarks like BBQ, BOLD, Winogender, Winobias, RealToxicity, and TruthfulQA
329
+ are shown here.
330
+
331
+ | Benchmark | Metric | 2B Params | 7B Params |
332
+ | ------------------------------ | ------------- | ----------- | --------- |
333
+ | [RealToxicity](https://arxiv.org/abs/2009.11462) | average | 6.86 | 7.90 |
334
+ | [BOLD](https://arxiv.org/abs/2101.11718) | | 45.57 | 49.08 |
335
+ | [CrowS-Pairs](https://aclanthology.org/2020.emnlp-main.154/) | top-1 | 45.82 | 51.33 |
336
+ | [BBQ Ambig](https://arxiv.org/abs/2110.08193v2) | 1-shot, top-1 | 62.58 | 92.54 |
337
+ | [BBQ Disambig](https://arxiv.org/abs/2110.08193v2) | top-1 | 54.62 | 71.99 |
338
+ | [Winogender](https://arxiv.org/abs/1804.09301) | top-1 | 51.25 | 54.17 |
339
+ | [TruthfulQA](https://arxiv.org/abs/2109.07958) | | 44.84 | 31.81 |
340
+ | [Winobias 1_2](https://arxiv.org/abs/1804.06876) | | 56.12 | 59.09 |
341
+ | [Winobias 2_2](https://arxiv.org/abs/1804.06876) | | 91.10 | 92.23 |
342
+ | [Toxigen](https://arxiv.org/abs/2203.09509) | | 29.77 | 39.59 |
343
+ | ------------------------------ | ------------- | ----------- | --------- |
344
+
345
+
346
+ ## Usage and Limitations
347
+
348
+ These models have certain limitations that users should be aware of.
349
+
350
+ ### Intended Usage
351
+
352
+ Open Large Language Models (LLMs) have a wide range of applications across
353
+ various industries and domains. The following list of potential uses is not
354
+ comprehensive. The purpose of this list is to provide contextual information
355
+ about the possible use-cases that the model creators considered as part of model
356
+ training and development.
357
+
358
+ * Content Creation and Communication
359
+ * Text Generation: These models can be used to generate creative text formats
360
+ such as poems, scripts, code, marketing copy, and email drafts.
361
+ * Chatbots and Conversational AI: Power conversational interfaces for customer
362
+ service, virtual assistants, or interactive applications.
363
+ * Text Summarization: Generate concise summaries of a text corpus, research
364
+ papers, or reports.
365
+ * Research and Education
366
+ * Natural Language Processing (NLP) Research: These models can serve as a
367
+ foundation for researchers to experiment with NLP techniques, develop
368
+ algorithms, and contribute to the advancement of the field.
369
+ * Language Learning Tools: Support interactive language learning experiences,
370
+ aiding in grammar correction or providing writing practice.
371
+ * Knowledge Exploration: Assist researchers in exploring large bodies of text
372
+ by generating summaries or answering questions about specific topics.
373
+
374
+ ### Limitations
375
+
376
+ * Training Data
377
+ * The quality and diversity of the training data significantly influence the
378
+ model's capabilities. Biases or gaps in the training data can lead to
379
+ limitations in the model's responses.
380
+ * The scope of the training dataset determines the subject areas the model can
381
+ handle effectively.
382
+ * Context and Task Complexity
383
+ * LLMs are better at tasks that can be framed with clear prompts and
384
+ instructions. Open-ended or highly complex tasks might be challenging.
385
+ * A model's performance can be influenced by the amount of context provided
386
+ (longer context generally leads to better outputs, up to a certain point).
387
+ * Language Ambiguity and Nuance
388
+ * Natural language is inherently complex. LLMs might struggle to grasp subtle
389
+ nuances, sarcasm, or figurative language.
390
+ * Factual Accuracy
391
+ * LLMs generate responses based on information they learned from their
392
+ training datasets, but they are not knowledge bases. They may generate
393
+ incorrect or outdated factual statements.
394
+ * Common Sense
395
+ * LLMs rely on statistical patterns in language. They might lack the ability
396
+ to apply common sense reasoning in certain situations.
397
+
398
+ ### Ethical Considerations and Risks
399
+
400
+ The development of large language models (LLMs) raises several ethical concerns.
401
+ In creating an open model, we have carefully considered the following:
402
+
403
+ * Bias and Fairness
404
+ * LLMs trained on large-scale, real-world text data can reflect socio-cultural
405
+ biases embedded in the training material. These models underwent careful
406
+ scrutiny, input data pre-processing described and posterior evaluations
407
+ reported in this card.
408
+ * Misinformation and Misuse
409
+ * LLMs can be misused to generate text that is false, misleading, or harmful.
410
+ * Guidelines are provided for responsible use with the model, see the
411
+ [Responsible Generative AI Toolkit](http://ai.google.dev/gemma/responsible).
412
+ * Transparency and Accountability:
413
+ * This model card summarizes details on the models' architecture,
414
+ capabilities, limitations, and evaluation processes.
415
+ * A responsibly developed open model offers the opportunity to share
416
+ innovation by making LLM technology accessible to developers and researchers
417
+ across the AI ecosystem.
418
+
419
+ Risks identified and mitigations:
420
+
421
+ * Perpetuation of biases: It's encouraged to perform continuous monitoring
422
+ (using evaluation metrics, human review) and the exploration of de-biasing
423
+ techniques during model training, fine-tuning, and other use cases.
424
+ * Generation of harmful content: Mechanisms and guidelines for content safety
425
+ are essential. Developers are encouraged to exercise caution and implement
426
+ appropriate content safety safeguards based on their specific product policies
427
+ and application use cases.
428
+ * Misuse for malicious purposes: Technical limitations and developer and
429
+ end-user education can help mitigate against malicious applications of LLMs.
430
+ Educational resources and reporting mechanisms for users to flag misuse are
431
+ provided. Prohibited uses of Gemma models are outlined in the
432
+ [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy).
433
+ * Privacy violations: Models were trained on data filtered for removal of PII
434
+ (Personally Identifiable Information). Developers are encouraged to adhere to
435
+ privacy regulations with privacy-preserving techniques.
436
+
437
+ ### Benefits
438
+
439
+ At the time of release, this family of models provides high-performance open
440
+ large language model implementations designed from the ground up for Responsible
441
+ AI development compared to similarly sized models.
442
+
443
+ Using the benchmark evaluation metrics described in this document, these models
444
+ have shown to provide superior performance to other, comparably-sized open model
445
+ alternatives.
446