Hugging Face's logo

RichardErkhov
/
ibm-granite_-_granite-20b-functioncalling-gguf

GGUF
Inference Endpoints
conversational
Model card Files Community
RichardErkhov commited on
Commit
4c5277c
·
verified ·
1 Parent(s): ccf89c1

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +133 -0
README.md ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ granite-20b-functioncalling - GGUF
11
+ - Model creator: https://huggingface.co/ibm-granite/
12
+ - Original model: https://huggingface.co/ibm-granite/granite-20b-functioncalling/
13
+
14
+
15
+ | Name | Quant method | Size |
16
+ | ---- | ---- | ---- |
17
+ | [granite-20b-functioncalling.Q2_K.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q2_K.gguf) | Q2_K | 7.38GB |
18
+ | [granite-20b-functioncalling.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.IQ3_XS.gguf) | IQ3_XS | 8.06GB |
19
+ | [granite-20b-functioncalling.IQ3_S.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.IQ3_S.gguf) | IQ3_S | 8.32GB |
20
+ | [granite-20b-functioncalling.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q3_K_S.gguf) | Q3_K_S | 8.32GB |
21
+ | [granite-20b-functioncalling.IQ3_M.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.IQ3_M.gguf) | IQ3_M | 8.93GB |
22
+ | [granite-20b-functioncalling.Q3_K.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q3_K.gguf) | Q3_K | 9.84GB |
23
+ | [granite-20b-functioncalling.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q3_K_M.gguf) | Q3_K_M | 9.84GB |
24
+ | [granite-20b-functioncalling.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q3_K_L.gguf) | Q3_K_L | 10.93GB |
25
+ | [granite-20b-functioncalling.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.IQ4_XS.gguf) | IQ4_XS | 10.32GB |
26
+ | [granite-20b-functioncalling.Q4_0.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q4_0.gguf) | Q4_0 | 10.76GB |
27
+ | [granite-20b-functioncalling.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.IQ4_NL.gguf) | IQ4_NL | 10.86GB |
28
+ | [granite-20b-functioncalling.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q4_K_S.gguf) | Q4_K_S | 10.86GB |
29
+ | [granite-20b-functioncalling.Q4_K.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q4_K.gguf) | Q4_K | 11.94GB |
30
+ | [granite-20b-functioncalling.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q4_K_M.gguf) | Q4_K_M | 11.94GB |
31
+ | [granite-20b-functioncalling.Q4_1.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q4_1.gguf) | Q4_1 | 11.91GB |
32
+ | [granite-20b-functioncalling.Q5_0.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q5_0.gguf) | Q5_0 | 13.05GB |
33
+ | [granite-20b-functioncalling.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q5_K_S.gguf) | Q5_K_S | 13.05GB |
34
+ | [granite-20b-functioncalling.Q5_K.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q5_K.gguf) | Q5_K | 13.79GB |
35
+ | [granite-20b-functioncalling.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q5_K_M.gguf) | Q5_K_M | 13.79GB |
36
+ | [granite-20b-functioncalling.Q5_1.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q5_1.gguf) | Q5_1 | 14.2GB |
37
+ | [granite-20b-functioncalling.Q6_K.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q6_K.gguf) | Q6_K | 15.49GB |
38
+ | [granite-20b-functioncalling.Q8_0.gguf](https://huggingface.co/RichardErkhov/ibm-granite_-_granite-20b-functioncalling-gguf/blob/main/granite-20b-functioncalling.Q8_0.gguf) | Q8_0 | 20.01GB |
39
+
40
+
41
+
42
+
43
+ Original model description:
44
+ ---
45
+ license: apache-2.0
46
+ ---
47
+ ### Granite-20B-FunctionCalling
48
+ #### Model Summary
49
+ Granite-20B-FunctionCalling is a finetuned model based on IBM's [granite-20b-code-instruct](https://huggingface.co/ibm-granite/granite-20b-code-instruct) model to introduce function calling abilities into Granite model family. The model is trained using a multi-task training approach on seven fundamental tasks encompassed in function calling, those being Nested Function Calling, Function Chaining, Parallel Functions, Function Name Detection, Parameter-Value Pair Detection, Next-Best Function, and Response Generation.
50
+
51
+ - **Developers**: IBM Research
52
+ - **Paper**: [Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks](https://arxiv.org/pdf/2407.00121v1)
53
+ - **Release Date**: July 9th, 2024
54
+ - **License**: [Apache 2.0.](https://www.apache.org/licenses/LICENSE-2.0)
55
+
56
+ ### Usage
57
+ ### Intended use
58
+ The model is designed to respond to function calling related instructions.
59
+
60
+ ### Generation
61
+ This is a simple example of how to use Granite-20B-Code-FunctionCalling model.
62
+ ```python
63
+ import json
64
+ import torch
65
+ from transformers import AutoModelForCausalLM, AutoTokenizer
66
+
67
+ device = "cuda" # or "cpu"
68
+ model_path = "ibm-granite/granite-20b-functioncalling"
69
+ tokenizer = AutoTokenizer.from_pretrained(model_path)
70
+ # drop device_map if running on CPU
71
+ model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device)
72
+ model.eval()
73
+
74
+ # define the user query and list of available functions
75
+ query = "What's the current weather in New York?"
76
+ functions = [
77
+ {
78
+ "name": "get_current_weather",
79
+ "description": "Get the current weather",
80
+ "parameters": {
81
+ "type": "object",
82
+ "properties": {
83
+ "location": {
84
+ "type": "string",
85
+ "description": "The city and state, e.g. San Francisco, CA"
86
+ }
87
+ },
88
+ "required": ["location"]
89
+ }
90
+ },
91
+ {
92
+ "name": "get_stock_price",
93
+ "description": "Retrieves the current stock price for a given ticker symbol. The ticker symbol must be a valid symbol for a publicly traded company on a major US stock exchange like NYSE or NASDAQ. The tool will return the latest trade price in USD. It should be used when the user asks about the current or most recent price of a specific stock. It will not provide any other information about the stock or company.",
94
+ "parameters": {
95
+ "type": "object",
96
+ "properties": {
97
+ "ticker": {
98
+ "type": "string",
99
+ "description": "The stock ticker symbol, e.g. AAPL for Apple Inc."
100
+ }
101
+ },
102
+ "required": ["ticker"]
103
+ }
104
+ }
105
+ ]
106
+
107
+
108
+ # serialize functions and define a payload to generate the input template
109
+ payload = {
110
+ "functions_str": [json.dumps(x) for x in functions],
111
+ "query": query,
112
+ }
113
+
114
+ instruction = tokenizer.apply_chat_template(payload, tokenize=False, add_generation_prompt=True)
115
+
116
+ # tokenize the text
117
+ input_tokens = tokenizer(instruction, return_tensors="pt").to(device)
118
+
119
+ # generate output tokens
120
+ outputs = model.generate(**input_tokens, max_new_tokens=100)
121
+
122
+ # decode output tokens into text
123
+ outputs = tokenizer.batch_decode(outputs)
124
+
125
+ # loop over the batch to print, in this example the batch size is 1
126
+ for output in outputs:
127
+ # Each function call in the output will be preceded by the token "<function_call>" followed by a
128
+ # json serialized function call of the format {"name": $function_name$, "arguments" {$arg_name$: $arg_val$}}
129
+ # In this specific case, the output will be: <function_call> {"name": "get_current_weather", "arguments": {"location": "New York"}}
130
+ print(output)
131
+ ```
132
+
133
+