zuxin-llm commited on
Commit
066e1b5
0 Parent(s):

Initial commit

Browse files
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,278 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ ---
4
+
5
+ <p align="center">
6
+ <img width="300px" alt="xLAM" src="https://huggingface.co/Salesforce/xLAM-v0.1-r/resolve/main/xlam-no-background.png">
7
+ </p>
8
+ <p align="center"><a href="https://apigen-pipeline.github.io/">[🏠Homepage]</a> | <a href="https://coder.deepseek.com/">[📄 Paper]</a> | <a href="https://coder.deepseek.com/">[📚 Dataset]</a></p>
9
+ <hr>
10
+
11
+ Welcome to the xLAM model family! [Large Action Models (LAMs)](https://blog.salesforceairesearch.com/large-action-models/) are advanced large language models designed to enhance decision-making and translate user intentions into executable actions that interact with the world. LAMs autonomously plan and execute tasks to achieve specific goals, serving as the brains of AI agents. They have the potential to automate workflow processes across various domains, making them invaluable for a wide range of applications.
12
+
13
+ ## Table of Contents
14
+ - [Model Series](#model-series)
15
+ - [Repository Overview](#repository-overview)
16
+ - [Benchmark Results](#benchmark-results)
17
+ - [Usage](#usage)
18
+ - [Basic Usage with Huggingface](#basic-usage-with-huggingface)
19
+ - [Usage with vLLM](#usage-with-vllm)
20
+ - [License](#license)
21
+ - [Citation](#citation)
22
+
23
+ ## Model Series
24
+
25
+ We provide a series of xLAMs in different sizes to cater to various applications, including those optimized for function-calling and general agent applications:
26
+
27
+ | Model | # Total Params | Context Length | Download Model | Download GGUF files |
28
+ |------------------------|----------------|----------------|----------------|----------|
29
+ | xLAM-1b-fc-r | 1.35B | 16k | [🤗 Link](https://huggingface.co/Salesforce/xLAM-1b-fc-r) | [🤗 Link](https://huggingface.co/Salesforce/xLAM-1b-fc-r-gguf) |
30
+ | xLAM-7b-fc-r | 6.91B | 4k | [🤗 Link](https://huggingface.co/Salesforce/xLAM-7b-fc-r) | [🤗 Link](https://huggingface.co/Salesforce/xLAM-7b-fc-r-gguf) | |
31
+
32
+ The `fc` series of models are optimized for function-calling capability, providing fast, accurate, and structured responses based on input queries and available APIs. These models are fine-tuned based on the [deepseek-coder](https://huggingface.co/collections/deepseek-ai/deepseek-coder-65f295d7d8a0a29fe39b4ec4) models and are designed to be small enough for deployment on personal devices like phones or computers.
33
+
34
+ We also provide their quantized [GGUF](https://huggingface.co/docs/hub/en/gguf) files for efficient deployment and execution. GGUF is a file format designed to efficiently store and load large language models, making GGUF ideal for running AI models on local devices with limited resources, enabling offline functionality and enhanced privacy.
35
+
36
+ For more details, check our [paper](https://arxiv.org/abs/2406.18518).
37
+
38
+ ## Repository Overview
39
+
40
+ This repository is focused on our tiny `xLAM-1b-fc-r` model, which is optimized for function-calling and can be easily deployed on personal devices.
41
+
42
+ <div align="center">
43
+ <img src="https://github.com/apigen-pipeline/apigen-pipeline.github.io/blob/main/img/function-call-overview.png?raw=true"
44
+ alt="drawing" width="620"/>
45
+ </div>
46
+
47
+ Function-calling, or tool use, is one of the key capabilities for AI agents. It requires the model not only understand and generate human-like text but also to execute functional API calls based on natural language instructions. This extends the utility of LLMs beyond simple conversation tasks to dynamic interactions with a variety of digital services and applications, such as retrieving weather information, managing social media platforms, and handling financial services.
48
+
49
+ The instructions will guide you through the setup, usage, and integration of `xLAM-1b-fc-r` with HuggingFace and vLLM.
50
+ We will first introduce the basic usage, and then walk through the provided tutorial and example scripts.
51
+
52
+ ### Framework Versions
53
+
54
+ - Transformers 4.41.0
55
+ - Pytorch 2.3.0+cu121
56
+ - Datasets 2.19.1
57
+ - Tokenizers 0.19.1
58
+
59
+ ## Benchmark Results
60
+
61
+ We mainly test our function-calling models on the [Berkeley Function-Calling Leaderboard (BFCL)](https://gorilla.cs.berkeley.edu/leaderboard.html), which offers a comprehensive evaluation framework for assessing LLMs' function-calling capabilities across various programming languages and application domains like Java, JavaScript, and Python.
62
+
63
+
64
+ <div align="center">
65
+ <img src="https://github.com/apigen-pipeline/apigen-pipeline.github.io/blob/main/img/table-result-0718.png?raw=true" width="620" alt="Performance comparison on Berkeley Function-Calling Leaderboard">
66
+ <p>Performance comparison on the BFCL benchmark as of date 07/18/2024. Evaluated with <code>temperature=0.001</code> and <code>top_p=1</code></p>
67
+ </div>
68
+
69
+ <p>Our <code>xLAM-7b-fc-r</code> secures the 3rd place with an overall accuracy of 88.24% on the leaderboard, outperforming many strong models. Notably, our <code>xLAM-1b-fc-r</code> model is the only tiny model with less than 2B parameters on the leaderboard, but still achieves a competitive overall accuracy of 78.94% and outperforming GPT3-Turbo and many larger models.
70
+ Both models exhibit balanced performance across various categories, showing their strong function-calling capabilities despite their small sizes.</p>
71
+
72
+ See our [paper](https://arxiv.org/abs/2406.18518) for more detailed analysis.
73
+
74
+
75
+ ## Usage
76
+
77
+ ### Basic Usage with Huggingface
78
+
79
+ To use the `xLAM-1b-fc-r` model from Huggingface, please first install the `transformers` library:
80
+ ```bash
81
+ pip install transformers>=4.41.0
82
+ ```
83
+
84
+ We use the following example to illustrate how to use our model to perform function-calling tasks.
85
+ Please note that, our model works best with our provided prompt format.
86
+ It allows us to extract JSON output that is similar to the [function-calling mode of ChatGPT](https://platform.openai.com/docs/guides/function-calling).
87
+
88
+ ````python
89
+ import json
90
+ import torch
91
+ from transformers import AutoModelForCausalLM, AutoTokenizer
92
+
93
+ torch.random.manual_seed(0)
94
+
95
+ model_name = "Salesforce/xLAM-1b-fc-r"
96
+ model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True)
97
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
98
+
99
+ # Please use our provided instruction prompt for best performance
100
+ task_instruction = """
101
+ You are an expert in composing functions. You are given a question and a set of possible functions.
102
+ Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
103
+ If none of the functions can be used, point it out and refuse to answer.
104
+ If the given question lacks the parameters required by the function, also point it out.
105
+ """.strip()
106
+
107
+ format_instruction = """
108
+ The output MUST strictly adhere to the following JSON format, and NO other text MUST be included.
109
+ The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'.
110
+ ```
111
+ {
112
+ "tool_calls": [
113
+ {"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},
114
+ ... (more tool calls as required)
115
+ ]
116
+ }
117
+ ```
118
+ """.strip()
119
+
120
+ # Define the input query and available tools
121
+ query = "What's the weather like in New York in fahrenheit?"
122
+
123
+ get_weather_api = {
124
+ "name": "get_weather",
125
+ "description": "Get the current weather for a location",
126
+ "parameters": {
127
+ "type": "object",
128
+ "properties": {
129
+ "location": {
130
+ "type": "string",
131
+ "description": "The city and state, e.g. San Francisco, New York"
132
+ },
133
+ "unit": {
134
+ "type": "string",
135
+ "enum": ["celsius", "fahrenheit"],
136
+ "description": "The unit of temperature to return"
137
+ }
138
+ },
139
+ "required": ["location"]
140
+ }
141
+ }
142
+
143
+ search_api = {
144
+ "name": "search",
145
+ "description": "Search for information on the internet",
146
+ "parameters": {
147
+ "type": "object",
148
+ "properties": {
149
+ "query": {
150
+ "type": "string",
151
+ "description": "The search query, e.g. 'latest news on AI'"
152
+ }
153
+ },
154
+ "required": ["query"]
155
+ }
156
+ }
157
+
158
+ openai_format_tools = [get_weather_api, search_api]
159
+
160
+ # Helper function to convert openai format tools to our more concise xLAM format
161
+ def convert_to_xlam_tool(tools):
162
+ ''''''
163
+ if isinstance(tools, dict):
164
+ return {
165
+ "name": tools["name"],
166
+ "description": tools["description"],
167
+ "parameters": {k: v for k, v in tools["parameters"].get("properties", {}).items()}
168
+ }
169
+ elif isinstance(tools, list):
170
+ return [convert_to_xlam_tool(tool) for tool in tools]
171
+ else:
172
+ return tools
173
+
174
+ # Helper function to build the input prompt for our model
175
+ def build_prompt(task_instruction: str, format_instruction: str, tools: list, query: str):
176
+ prompt = f"[BEGIN OF TASK INSTRUCTION]\n{task_instruction}\n[END OF TASK INSTRUCTION]\n\n"
177
+ prompt += f"[BEGIN OF AVAILABLE TOOLS]\n{json.dumps(xlam_format_tools)}\n[END OF AVAILABLE TOOLS]\n\n"
178
+ prompt += f"[BEGIN OF FORMAT INSTRUCTION]\n{format_instruction}\n[END OF FORMAT INSTRUCTION]\n\n"
179
+ prompt += f"[BEGIN OF QUERY]\n{query}\n[END OF QUERY]\n\n"
180
+ return prompt
181
+
182
+ # Build the input and start the inference
183
+ xlam_format_tools = convert_to_xlam_tool(openai_format_tools)
184
+ content = build_prompt(task_instruction, format_instruction, xlam_format_tools, query)
185
+
186
+ messages=[
187
+ { 'role': 'user', 'content': content}
188
+ ]
189
+ inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
190
+
191
+ # tokenizer.eos_token_id is the id of <|EOT|> token
192
+ outputs = model.generate(inputs, max_new_tokens=512, do_sample=False, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
193
+ print(tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True))
194
+ ````
195
+
196
+ Then you should be able to see the following output string in JSON format:
197
+
198
+ ```shell
199
+ {"tool_calls": [{"name": "get_weather", "arguments": {"location": "New York", "unit": "fahrenheit"}}]}
200
+ ```
201
+
202
+ We highly recommend to use our provided prompt format and helper functions to yield the best function-calling performance of our model.
203
+
204
+ ### Usage with vLLM
205
+
206
+ We provide example scripts to deploy our model with `vllm` and run inferences. First, install the required packages:
207
+
208
+ ```bash
209
+ pip install vllm openai argparse jinja2
210
+ ```
211
+
212
+ The example scripts are located in the `examples` folder.
213
+
214
+ #### 1. Test Prompt Template
215
+
216
+ To build prompts using the chat template and output formatted prompts ready for various test cases, run:
217
+
218
+ ```bash
219
+ python test_prompt_template.py --model
220
+ ```
221
+
222
+ #### 2. Test xLAM Model with a Manually Served Endpoint
223
+
224
+ a. Serve the model with vLLM:
225
+
226
+ ```bash
227
+ python -m vllm.entrypoints.openai.api_server --model Salesforce/xLAM-1b-fc-r --served-model-name xlam-1b-fc-r --dtype bfloat16 --port 8001
228
+ ```
229
+
230
+ b. Run the test script:
231
+
232
+ ```bash
233
+ python test_xlam_model_with_endpoint.py --model_name xlam-1b-fc-r --port 8001 [OPTIONS]
234
+ ```
235
+
236
+ Options:
237
+ - `--temperature`: Default 0.3
238
+ - `--top_p`: Default 1.0
239
+ - `--max_tokens`: Default 512
240
+
241
+ This test script provides a handler implementation that can be easily applied to your customized function-calling applications.
242
+
243
+ #### 3. Test xLAM Model by Directly Using vLLM Library
244
+
245
+ To test the xLAM model directly with the vLLM library, run:
246
+
247
+ ```bash
248
+ python test_xlam_model_with_vllm.py --model Salesforce/xLAM-1b-fc-r [OPTIONS]
249
+ ```
250
+
251
+ Options are the same as for the endpoint test. This test script also provides a handler implementation that can be easily applied to your customized function-calling applications.
252
+
253
+ #### Customization
254
+
255
+ These examples are designed to be flexible and easily integrated into your own projects. Feel free to modify the scripts to suit your specific needs and applications. You can adjust test queries or API definitions in each script to test different scenarios or model capabilities.
256
+
257
+ Additional customization tips:
258
+ - Modify the `--dtype` parameter when serving the model based on your GPU capacity.
259
+ - Refer to the vLLM documentation for more detailed configuration options.
260
+ - Explore the `demo.ipynb` file for a comprehensive description of the entire workflow, including how to execute APIs.
261
+
262
+ These resources provide a robust foundation for integrating xLAM models into your applications, allowing for tailored and efficient deployment.
263
+
264
+ ## License
265
+
266
+ `xLAM-1b-fc-r` is distributed under the CC-BY-NC-4.0 license, with additional terms specified in the [Deepseek license](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/LICENSE-MODEL).
267
+
268
+ ## Citation
269
+
270
+ If you find this repo helpful, please cite our paper:
271
+ ```bibtex
272
+ @article{liu2024apigen,
273
+ title={APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
274
+ author={Liu, Zuxin and Hoang, Thai and Zhang, Jianguo and Zhu, Ming and Lan, Tian and Kokane, Shirley and Tan, Juntao and Yao, Weiran and Liu, Zhiwei and Feng, Yihao and others},
275
+ journal={arXiv preprint arXiv:2406.18518},
276
+ year={2024}
277
+ }
278
+ ```
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 32013,
8
+ "eos_token_id": 32021,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 2048,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 5504,
13
+ "max_position_embeddings": 16384,
14
+ "model_type": "llama",
15
+ "num_attention_heads": 16,
16
+ "num_hidden_layers": 24,
17
+ "num_key_value_heads": 16,
18
+ "pretraining_tp": 1,
19
+ "rms_norm_eps": 1e-06,
20
+ "rope_scaling": {
21
+ "factor": 4.0,
22
+ "type": "linear"
23
+ },
24
+ "rope_theta": 100000,
25
+ "tie_word_embeddings": false,
26
+ "torch_dtype": "bfloat16",
27
+ "transformers_version": "4.40.0",
28
+ "use_cache": false,
29
+ "vocab_size": 32256
30
+ }
examples/demo.ipynb ADDED
@@ -0,0 +1,565 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "metadata": {},
6
+ "source": [
7
+ "# xLAM Model Function-Calling Capabilities Demo\n",
8
+ "\n",
9
+ "This notebook demonstrates the function-calling capabilities of the xLAM model. The xLAM model is designed to handle various tasks by generating appropriate function calls based on the given query and available tools.\n",
10
+ "\n",
11
+ "We will cover the following steps:\n",
12
+ "1. Setup and Initialization\n",
13
+ "2. Example Usage with Provided Demo APIs\n",
14
+ "3. Executing Real-Time Weather API Calls\n",
15
+ "\n",
16
+ "Let's get started!"
17
+ ]
18
+ },
19
+ {
20
+ "cell_type": "markdown",
21
+ "metadata": {},
22
+ "source": [
23
+ "## 1. Setup and Initialization\n",
24
+ "\n",
25
+ "First, we need to set up the environment and initialize the xLAMHandler class. Ensure you have all the necessary dependencies installed:\n",
26
+ "- `vllm`\n",
27
+ "- `jinja2`\n",
28
+ "- `requests`"
29
+ ]
30
+ },
31
+ {
32
+ "cell_type": "markdown",
33
+ "metadata": {},
34
+ "source": [
35
+ "Next, we'll import the necessary modules and define the xLAMHandler class and utility functions. You can find the script provided earlier in the cell below."
36
+ ]
37
+ },
38
+ {
39
+ "cell_type": "code",
40
+ "execution_count": 1,
41
+ "metadata": {},
42
+ "outputs": [
43
+ {
44
+ "name": "stderr",
45
+ "output_type": "stream",
46
+ "text": [
47
+ "/export/home/conda/envs/rl/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
48
+ " from .autonotebook import tqdm as notebook_tqdm\n",
49
+ "2024-07-18 07:25:11,294\tINFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output.\n"
50
+ ]
51
+ },
52
+ {
53
+ "name": "stdout",
54
+ "output_type": "stream",
55
+ "text": [
56
+ "INFO 07-18 07:25:13 llm_engine.py:161] Initializing an LLM engine (v0.5.0) with config: model='Salesforce/xLAM-1b-fc-r', speculative_config=None, tokenizer='Salesforce/xLAM-1b-fc-r', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=Salesforce/xLAM-1b-fc-r)\n"
57
+ ]
58
+ },
59
+ {
60
+ "name": "stderr",
61
+ "output_type": "stream",
62
+ "text": [
63
+ "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
64
+ ]
65
+ },
66
+ {
67
+ "name": "stdout",
68
+ "output_type": "stream",
69
+ "text": [
70
+ "INFO 07-18 07:25:24 weight_utils.py:218] Using model weights format ['*.safetensors']\n",
71
+ "INFO 07-18 07:25:24 weight_utils.py:261] No model.safetensors.index.json found in remote.\n",
72
+ "INFO 07-18 07:25:25 model_runner.py:159] Loading model weights took 2.5583 GB\n",
73
+ "INFO 07-18 07:25:31 gpu_executor.py:83] # GPU blocks: 10075, # CPU blocks: 1365\n",
74
+ "INFO 07-18 07:25:40 model_runner.py:878] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.\n",
75
+ "INFO 07-18 07:25:40 model_runner.py:882] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.\n",
76
+ "INFO 07-18 07:26:02 model_runner.py:954] Graph capturing finished in 22 secs.\n"
77
+ ]
78
+ }
79
+ ],
80
+ "source": [
81
+ "import json\n",
82
+ "import time\n",
83
+ "from typing import List, Dict\n",
84
+ "\n",
85
+ "from vllm import LLM, SamplingParams\n",
86
+ "from jinja2 import Template\n",
87
+ "\n",
88
+ "\n",
89
+ "TASK_INSTRUCTION = \"\"\"\n",
90
+ "You are an expert in composing functions. You are given a question and a set of possible functions. \n",
91
+ "Based on the question, you will need to make one or more function/tool calls to achieve the purpose. \n",
92
+ "If none of the functions can be used, point it out and refuse to answer. \n",
93
+ "If the given question lacks the parameters required by the function, also point it out.\n",
94
+ "\"\"\".strip()\n",
95
+ "\n",
96
+ "FORMAT_INSTRUCTION = \"\"\"\n",
97
+ "The output MUST strictly adhere to the following JSON format, and NO other text MUST be included.\n",
98
+ "The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'\n",
99
+ "```\n",
100
+ "{\n",
101
+ " \"tool_calls\": [\n",
102
+ " {\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},\n",
103
+ " ... (more tool calls as required)\n",
104
+ " ]\n",
105
+ "}\n",
106
+ "```\n",
107
+ "\"\"\".strip()\n",
108
+ "\n",
109
+ "class XLAMHandler:\n",
110
+ " def __init__(self, \n",
111
+ " model: str, \n",
112
+ " temperature: float = 0.3, \n",
113
+ " top_p: float = 1, \n",
114
+ " max_tokens: int = 512,\n",
115
+ " tensor_parallel_size: int = 1,\n",
116
+ " dtype: str = \"bfloat16\"):\n",
117
+ " \n",
118
+ " # Initialize LLM with GPU specifications\n",
119
+ " self.llm = LLM(model=model,\n",
120
+ " tensor_parallel_size=tensor_parallel_size,\n",
121
+ " dtype=dtype)\n",
122
+ " \n",
123
+ " self.sampling_params = SamplingParams(\n",
124
+ " temperature=temperature,\n",
125
+ " top_p=top_p,\n",
126
+ " max_tokens=max_tokens\n",
127
+ " )\n",
128
+ " self.chat_template = self.llm.get_tokenizer().chat_template\n",
129
+ " \n",
130
+ " @staticmethod\n",
131
+ " def apply_chat_template(template, messages):\n",
132
+ " jinja_template = Template(template)\n",
133
+ " return jinja_template.render(messages=messages)\n",
134
+ "\n",
135
+ " def process_query(self, query: str, tools: list, task_instruction: str, format_instruction: str):\n",
136
+ " # Convert tools to XLAM format\n",
137
+ " xlam_tools = self.convert_to_xlam_tool(tools)\n",
138
+ "\n",
139
+ " # Build the input prompt\n",
140
+ " prompt = self.build_prompt(query, xlam_tools, task_instruction, format_instruction)\n",
141
+ "\n",
142
+ " messages = [\n",
143
+ " {\"role\": \"user\", \"content\": prompt}\n",
144
+ " ]\n",
145
+ " formatted_prompt = self.apply_chat_template(self.chat_template, messages)\n",
146
+ "\n",
147
+ " # Make inference\n",
148
+ " start_time = time.time()\n",
149
+ " outputs = self.llm.generate([formatted_prompt], self.sampling_params)\n",
150
+ " latency = time.time() - start_time\n",
151
+ "\n",
152
+ " # Calculate tokens per second\n",
153
+ " tokens_generated = sum(len(output.text.split()) for output in outputs[0].outputs)\n",
154
+ " tokens_per_second = tokens_generated / latency\n",
155
+ "\n",
156
+ " # Parse response\n",
157
+ " result = outputs[0].outputs[0].text\n",
158
+ " parsed_result, success, _ = self.parse_response(result)\n",
159
+ "\n",
160
+ " # Prepare metadata\n",
161
+ " metadata = {\n",
162
+ " \"latency\": latency,\n",
163
+ " \"tokens_per_second\": tokens_per_second,\n",
164
+ " \"success\": success,\n",
165
+ " }\n",
166
+ "\n",
167
+ " return parsed_result, metadata\n",
168
+ "\n",
169
+ " def convert_to_xlam_tool(self, tools):\n",
170
+ " if isinstance(tools, dict):\n",
171
+ " return {\n",
172
+ " \"name\": tools[\"name\"],\n",
173
+ " \"description\": tools[\"description\"],\n",
174
+ " \"parameters\": {k: v for k, v in tools[\"parameters\"].get(\"properties\", {}).items()}\n",
175
+ " }\n",
176
+ " elif isinstance(tools, list):\n",
177
+ " return [self.convert_to_xlam_tool(tool) for tool in tools]\n",
178
+ " else:\n",
179
+ " return tools\n",
180
+ "\n",
181
+ " def build_prompt(self, query, tools, task_instruction=TASK_INSTRUCTION, format_instruction=FORMAT_INSTRUCTION):\n",
182
+ " prompt = f\"[BEGIN OF TASK INSTRUCTION]\\n{task_instruction}\\n[END OF TASK INSTRUCTION]\\n\\n\"\n",
183
+ " prompt += f\"[BEGIN OF AVAILABLE TOOLS]\\n{json.dumps(tools)}\\n[END OF AVAILABLE TOOLS]\\n\\n\"\n",
184
+ " prompt += f\"[BEGIN OF FORMAT INSTRUCTION]\\n{format_instruction}\\n[END OF FORMAT INSTRUCTION]\\n\\n\"\n",
185
+ " prompt += f\"[BEGIN OF QUERY]\\n{query}\\n[END OF QUERY]\\n\\n\"\n",
186
+ " return prompt\n",
187
+ "\n",
188
+ " def parse_response(self, response):\n",
189
+ " try:\n",
190
+ " data = json.loads(response)\n",
191
+ " tool_calls = data.get('tool_calls', []) if isinstance(data, dict) else data\n",
192
+ " result = [\n",
193
+ " {tool_call['name']: tool_call['arguments']}\n",
194
+ " for tool_call in tool_calls if isinstance(tool_call, dict)\n",
195
+ " ]\n",
196
+ " return result, True, []\n",
197
+ " except json.JSONDecodeError:\n",
198
+ " return [], False, [\"Failed to parse JSON response\"]\n",
199
+ "\n",
200
+ "handler = XLAMHandler(model=\"Salesforce/xLAM-1b-fc-r\")"
201
+ ]
202
+ },
203
+ {
204
+ "cell_type": "markdown",
205
+ "metadata": {},
206
+ "source": [
207
+ "## 2. Example Usage with Demo APIs\n",
208
+ "\n",
209
+ "In this section, we'll demonstrate how to use the xLAMHandler class with some example APIs. We'll start by defining several API tools and some test queries."
210
+ ]
211
+ },
212
+ {
213
+ "cell_type": "code",
214
+ "execution_count": 3,
215
+ "metadata": {},
216
+ "outputs": [
217
+ {
218
+ "name": "stdout",
219
+ "output_type": "stream",
220
+ "text": [
221
+ "Query: What's the weather like in New York in Fahrenheit?\n"
222
+ ]
223
+ },
224
+ {
225
+ "name": "stderr",
226
+ "output_type": "stream",
227
+ "text": [
228
+ "Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 4.51it/s, Generation Speed: 176.89 toks/s]\n"
229
+ ]
230
+ },
231
+ {
232
+ "name": "stdout",
233
+ "output_type": "stream",
234
+ "text": [
235
+ "Result: [\n",
236
+ " {\n",
237
+ " \"get_weather\": {\n",
238
+ " \"location\": \"New York\",\n",
239
+ " \"unit\": \"fahrenheit\"\n",
240
+ " }\n",
241
+ " }\n",
242
+ "]\n",
243
+ "Latency: 0.22673869132995605\n",
244
+ "Speed: 39.69326958363258\n",
245
+ "--------------------------------------------------\n",
246
+ "Query: What is the stock price of CRM?\n"
247
+ ]
248
+ },
249
+ {
250
+ "name": "stderr",
251
+ "output_type": "stream",
252
+ "text": [
253
+ "Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 5.86it/s, Generation Speed: 182.37 toks/s]\n"
254
+ ]
255
+ },
256
+ {
257
+ "name": "stdout",
258
+ "output_type": "stream",
259
+ "text": [
260
+ "Result: [\n",
261
+ " {\n",
262
+ " \"get_stock_price\": {\n",
263
+ " \"symbol\": \"CRM\"\n",
264
+ " }\n",
265
+ " }\n",
266
+ "]\n",
267
+ "Latency: 0.17523670196533203\n",
268
+ "Speed: 34.23940266341585\n",
269
+ "--------------------------------------------------\n",
270
+ "Query: Tell me the temperature in London in Celsius\n"
271
+ ]
272
+ },
273
+ {
274
+ "name": "stderr",
275
+ "output_type": "stream",
276
+ "text": [
277
+ "Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 5.08it/s, Generation Speed: 183.60 toks/s]"
278
+ ]
279
+ },
280
+ {
281
+ "name": "stdout",
282
+ "output_type": "stream",
283
+ "text": [
284
+ "Result: [\n",
285
+ " {\n",
286
+ " \"get_weather\": {\n",
287
+ " \"location\": \"London\",\n",
288
+ " \"unit\": \"celsius\"\n",
289
+ " }\n",
290
+ " }\n",
291
+ "]\n",
292
+ "Latency: 0.20116281509399414\n",
293
+ "Speed: 39.768781304148916\n",
294
+ "--------------------------------------------------\n"
295
+ ]
296
+ },
297
+ {
298
+ "name": "stderr",
299
+ "output_type": "stream",
300
+ "text": [
301
+ "\n"
302
+ ]
303
+ }
304
+ ],
305
+ "source": [
306
+ "get_weather_api = {\n",
307
+ " \"name\": \"get_weather\",\n",
308
+ " \"description\": \"Get the current weather for a location\",\n",
309
+ " \"parameters\": {\n",
310
+ " \"type\": \"object\",\n",
311
+ " \"properties\": {\n",
312
+ " \"location\": {\n",
313
+ " \"type\": \"string\",\n",
314
+ " \"description\": \"The city and state, e.g. San Francisco, New York\"\n",
315
+ " },\n",
316
+ " \"unit\": {\n",
317
+ " \"type\": \"string\",\n",
318
+ " \"enum\": [\"celsius\", \"fahrenheit\"],\n",
319
+ " \"description\": \"The unit of temperature to return\"\n",
320
+ " }\n",
321
+ " },\n",
322
+ " \"required\": [\"location\"]\n",
323
+ " }\n",
324
+ "}\n",
325
+ "\n",
326
+ "search_api = {\n",
327
+ " \"name\": \"search\",\n",
328
+ " \"description\": \"Search for information on the internet\",\n",
329
+ " \"parameters\": {\n",
330
+ " \"type\": \"object\",\n",
331
+ " \"properties\": {\n",
332
+ " \"query\": {\n",
333
+ " \"type\": \"string\",\n",
334
+ " \"description\": \"The search query, e.g. 'latest news on AI'\"\n",
335
+ " }\n",
336
+ " },\n",
337
+ " \"required\": [\"query\"]\n",
338
+ " }\n",
339
+ "}\n",
340
+ "\n",
341
+ "get_stock_price_api = {\n",
342
+ " \"name\": \"get_stock_price\",\n",
343
+ " \"description\": \"Get the current stock price for a company\",\n",
344
+ " \"parameters\": {\n",
345
+ " \"type\": \"object\",\n",
346
+ " \"properties\": {\n",
347
+ " \"symbol\": {\n",
348
+ " \"type\": \"string\",\n",
349
+ " \"description\": \"The stock symbol, e.g. 'AAPL' for Apple Inc.\"\n",
350
+ " }\n",
351
+ " },\n",
352
+ " \"required\": [\"symbol\"]\n",
353
+ " }\n",
354
+ "}\n",
355
+ "\n",
356
+ "get_news_api = {\n",
357
+ " \"name\": \"get_news\",\n",
358
+ " \"description\": \"Get the latest news headlines\",\n",
359
+ " \"parameters\": {\n",
360
+ " \"type\": \"object\",\n",
361
+ " \"properties\": {\n",
362
+ " \"topic\": {\n",
363
+ " \"type\": \"string\",\n",
364
+ " \"description\": \"The news topic, e.g. 'technology', 'sports'\"\n",
365
+ " }\n",
366
+ " },\n",
367
+ " \"required\": [\"topic\"]\n",
368
+ " }\n",
369
+ "}\n",
370
+ "\n",
371
+ "all_apis = [get_weather_api, search_api, get_stock_price_api, get_news_api]\n",
372
+ "\n",
373
+ "test_queries = [\n",
374
+ " \"What's the weather like in New York in Fahrenheit?\",\n",
375
+ " \"What is the stock price of CRM?\",\n",
376
+ " \"Tell me the temperature in London in Celsius\",\n",
377
+ "]\n",
378
+ "\n",
379
+ "for query in test_queries:\n",
380
+ " print(f\"Query: {query}\")\n",
381
+ " result, metadata = handler.process_query(query, all_apis, TASK_INSTRUCTION, FORMAT_INSTRUCTION)\n",
382
+ " print(f\"Result: {json.dumps(result, indent=2)}\")\n",
383
+ " print(\"Latency: \", metadata[\"latency\"])\n",
384
+ " print(\"Speed: \", metadata[\"tokens_per_second\"])\n",
385
+ " print(\"-\" * 50)"
386
+ ]
387
+ },
388
+ {
389
+ "cell_type": "markdown",
390
+ "metadata": {},
391
+ "source": [
392
+ "## 3. Executing Real-Time Weather API Calls\n",
393
+ "\n",
394
+ "To make real-time weather API calls, we'll use the `requests` library to fetch data from a weather service. After obtaining the weather data, we will ask our xLAM model to summarize the results."
395
+ ]
396
+ },
397
+ {
398
+ "cell_type": "code",
399
+ "execution_count": 4,
400
+ "metadata": {},
401
+ "outputs": [
402
+ {
403
+ "name": "stdout",
404
+ "output_type": "stream",
405
+ "text": [
406
+ "The current weather in San Francisco is 16.0 celsius\n"
407
+ ]
408
+ }
409
+ ],
410
+ "source": [
411
+ "import ast\n",
412
+ "import requests\n",
413
+ "\n",
414
+ "def get_weather(location, unit):\n",
415
+ " \"\"\"\n",
416
+ " Get the current weather for a specified location.\n",
417
+ "\n",
418
+ " Args:\n",
419
+ " location (str): The city and state, e.g. San Francisco, New York.\n",
420
+ " unit (str): The unit of temperature to return, either 'celsius' or 'fahrenheit'.\n",
421
+ "\n",
422
+ " Returns:\n",
423
+ " float: The temperature in the corresponding unit.\n",
424
+ " \"\"\"\n",
425
+ " base_url = \"https://wttr.in\"\n",
426
+ " unit_param = \"m\" if unit == \"celsius\" else \"u\"\n",
427
+ " params = {\n",
428
+ " \"format\": \"j1\",\n",
429
+ " \"unit\": unit_param\n",
430
+ " }\n",
431
+ " response = requests.get(f\"{base_url}/{location}\", params=params)\n",
432
+ " if response.status_code == 200:\n",
433
+ " weather_data = response.json()[\"current_condition\"][0]\n",
434
+ " return float(weather_data[\"temp_C\"]) if unit == \"celsius\" else float(weather_data[\"temp_F\"])\n",
435
+ " else:\n",
436
+ " return {\"error\": \"Failed to retrieve weather data\"}\n",
437
+ " \n",
438
+ "def execute_function_calls(function_calls):\n",
439
+ " \"\"\"\n",
440
+ " Convert the dictionary function_calls to executable Python code and execute the corresponding functions.\n",
441
+ "\n",
442
+ " Args:\n",
443
+ " function_calls (list): A list of dictionaries containing function calls and their arguments.\n",
444
+ "\n",
445
+ " Returns:\n",
446
+ " list: A list of results from executing the functions.\n",
447
+ " \"\"\"\n",
448
+ " results = []\n",
449
+ " for function_call in function_calls:\n",
450
+ " for func_name, args in function_call.items():\n",
451
+ " if func_name in globals() and callable(globals()[func_name]):\n",
452
+ " try:\n",
453
+ " # Safely evaluate the arguments\n",
454
+ " safe_args = ast.literal_eval(str(args))\n",
455
+ " print(safe_args)\n",
456
+ " # Call the function with unpacked arguments\n",
457
+ " func_result = globals()[func_name](**safe_args)\n",
458
+ " results.append(func_result)\n",
459
+ " except Exception as e:\n",
460
+ " results.append(f\"Error {str(e)}\")\n",
461
+ " else:\n",
462
+ " results.append(\"Error: Function not found or not callable\")\n",
463
+ " \n",
464
+ " return results\n",
465
+ "\n",
466
+ "# Example usage\n",
467
+ "location = \"San Francisco\"\n",
468
+ "unit = \"celsius\"\n",
469
+ "weather_data = get_weather(location, unit)\n",
470
+ "print(f\"The current weather in {location} is {weather_data} {unit}\")"
471
+ ]
472
+ },
473
+ {
474
+ "cell_type": "code",
475
+ "execution_count": 5,
476
+ "metadata": {},
477
+ "outputs": [
478
+ {
479
+ "name": "stderr",
480
+ "output_type": "stream",
481
+ "text": [
482
+ "Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 4.86it/s, Generation Speed: 180.67 toks/s]\n"
483
+ ]
484
+ },
485
+ {
486
+ "name": "stdout",
487
+ "output_type": "stream",
488
+ "text": [
489
+ "The function call result: [\n",
490
+ " {\n",
491
+ " \"get_weather\": {\n",
492
+ " \"location\": \"San Francisco\",\n",
493
+ " \"unit\": \"celsius\"\n",
494
+ " }\n",
495
+ " }\n",
496
+ "]\n",
497
+ "{'location': 'San Francisco', 'unit': 'celsius'}\n",
498
+ "Execution results: [16.0]\n",
499
+ "--------------------------------------------------\n"
500
+ ]
501
+ },
502
+ {
503
+ "name": "stderr",
504
+ "output_type": "stream",
505
+ "text": [
506
+ "Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 4.67it/s, Generation Speed: 183.21 toks/s]\n"
507
+ ]
508
+ },
509
+ {
510
+ "name": "stdout",
511
+ "output_type": "stream",
512
+ "text": [
513
+ "The function call result: [\n",
514
+ " {\n",
515
+ " \"get_weather\": {\n",
516
+ " \"location\": \"New York\",\n",
517
+ " \"unit\": \"fahrenheit\"\n",
518
+ " }\n",
519
+ " }\n",
520
+ "]\n",
521
+ "{'location': 'New York', 'unit': 'fahrenheit'}\n",
522
+ "Execution results: [74.0]\n"
523
+ ]
524
+ }
525
+ ],
526
+ "source": [
527
+ "# Example 1\n",
528
+ "query = \"I want to know the weather in San Francisco in Celsius\"\n",
529
+ "function_calls, metadata = handler.process_query(query, all_apis, TASK_INSTRUCTION, FORMAT_INSTRUCTION)\n",
530
+ "print(f\"The function call result: {json.dumps(function_calls, indent=2)}\")\n",
531
+ "execution_results = execute_function_calls(function_calls)\n",
532
+ "print(\"Execution results: \", execution_results)\n",
533
+ "print(\"-\" * 50)\n",
534
+ "\n",
535
+ "# Example 2\n",
536
+ "query = \"Tell me the temperature in New York in Fahrenheit\"\n",
537
+ "function_calls, metadata = handler.process_query(query, all_apis, TASK_INSTRUCTION, FORMAT_INSTRUCTION)\n",
538
+ "print(f\"The function call result: {json.dumps(function_calls, indent=2)}\")\n",
539
+ "execution_results = execute_function_calls(function_calls)\n",
540
+ "print(\"Execution results: \", execution_results)"
541
+ ]
542
+ }
543
+ ],
544
+ "metadata": {
545
+ "kernelspec": {
546
+ "display_name": "Python 3",
547
+ "language": "python",
548
+ "name": "python3"
549
+ },
550
+ "language_info": {
551
+ "codemirror_mode": {
552
+ "name": "ipython",
553
+ "version": 3
554
+ },
555
+ "file_extension": ".py",
556
+ "mimetype": "text/x-python",
557
+ "name": "python",
558
+ "nbconvert_exporter": "python",
559
+ "pygments_lexer": "ipython3",
560
+ "version": "3.10.14"
561
+ }
562
+ },
563
+ "nbformat": 4,
564
+ "nbformat_minor": 4
565
+ }
examples/test_prompt_template.py ADDED
@@ -0,0 +1,161 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import json
3
+ from typing import Dict
4
+
5
+ from jinja2 import Template
6
+ from transformers import AutoTokenizer
7
+
8
+ # Default prompts
9
+ TASK_INSTRUCTION = """
10
+ You are an expert in composing functions. You are given a question and a set of possible functions.
11
+ Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
12
+ If none of the functions can be used, point it out and refuse to answer.
13
+ If the given question lacks the parameters required by the function, also point it out.
14
+ """.strip()
15
+
16
+ FORMAT_INSTRUCTION = """
17
+ The output MUST strictly adhere to the following JSON format, and NO other text MUST be included.
18
+ The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'
19
+ ```
20
+ {
21
+ "tool_calls": [
22
+ {"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},
23
+ ... (more tool calls as required)
24
+ ]
25
+ }
26
+ ```
27
+ """.strip()
28
+
29
+ class PromptAssembler:
30
+ def __init__(self, model: str):
31
+ tokenizer = AutoTokenizer.from_pretrained(model)
32
+ self.chat_template = tokenizer.chat_template
33
+
34
+ @staticmethod
35
+ def apply_chat_template(template, messages):
36
+ jinja_template = Template(template)
37
+ return jinja_template.render(messages=messages)
38
+
39
+ def assemble_prompt(self, query: str, tools: Dict, task_instruction: str, format_instruction: str):
40
+ # Convert tools to XLAM format
41
+ xlam_tools = self.convert_to_xlam_tool(tools)
42
+
43
+ # Build the input prompt
44
+ prompt = self.build_prompt(query, xlam_tools, task_instruction, format_instruction)
45
+
46
+ messages = [
47
+ {"role": "user", "content": prompt}
48
+ ]
49
+ formatted_prompt = self.apply_chat_template(self.chat_template, messages)
50
+
51
+ return formatted_prompt
52
+
53
+ def convert_to_xlam_tool(self, tools):
54
+ if isinstance(tools, dict):
55
+ return {
56
+ "name": tools["name"],
57
+ "description": tools["description"],
58
+ "parameters": {k: v for k, v in tools["parameters"].get("properties", {}).items()}
59
+ }
60
+ elif isinstance(tools, list):
61
+ return [self.convert_to_xlam_tool(tool) for tool in tools]
62
+ else:
63
+ return tools
64
+
65
+ def build_prompt(self, query, tools, task_instruction=TASK_INSTRUCTION, format_instruction=FORMAT_INSTRUCTION):
66
+ prompt = f"[BEGIN OF TASK INSTRUCTION]\n{task_instruction}\n[END OF TASK INSTRUCTION]\n\n"
67
+ prompt += f"[BEGIN OF AVAILABLE TOOLS]\n{json.dumps(tools)}\n[END OF AVAILABLE TOOLS]\n\n"
68
+ prompt += f"[BEGIN OF FORMAT INSTRUCTION]\n{format_instruction}\n[END OF FORMAT INSTRUCTION]\n\n"
69
+ prompt += f"[BEGIN OF QUERY]\n{query}\n[END OF QUERY]\n\n"
70
+ return prompt
71
+
72
+ def print_prompt_template(self):
73
+ template = self.chat_template.replace("{{", "{").replace("}}", "}")
74
+ print("Prompt Template with Placeholders:")
75
+ print(template)
76
+
77
+ if __name__ == "__main__":
78
+ parser = argparse.ArgumentParser(description="Assemble prompts using chat template")
79
+ parser.add_argument("--model", required=True, help="Name of the model (for chat template)")
80
+
81
+ args = parser.parse_args()
82
+
83
+ # Initialize the PromptAssembler
84
+ assembler = PromptAssembler(args.model)
85
+
86
+ # Print the prompt template with placeholders
87
+ assembler.print_prompt_template()
88
+
89
+ # Test case 1: Weather API, follows the OpenAI format: https://platform.openai.com/docs/guides/function-calling
90
+ weather_api = {
91
+ "name": "get_weather",
92
+ "description": "Get the current weather for a location",
93
+ "parameters": {
94
+ "type": "object",
95
+ "properties": {
96
+ "location": {
97
+ "type": "string",
98
+ "description": "The city and state, e.g. San Francisco, CA"
99
+ },
100
+ "unit": {
101
+ "type": "string",
102
+ "enum": ["celsius", "fahrenheit"],
103
+ "description": "The unit of temperature to return"
104
+ }
105
+ },
106
+ "required": ["location"]
107
+ }
108
+ }
109
+
110
+ # Test queries
111
+ test_queries = [
112
+ "What's the weather like in New York?",
113
+ "Tell me the temperature in London in Celsius",
114
+ "What's the weather forecast for Tokyo?",
115
+ "What is the stock price of CRM?", # the model should return an empty list
116
+ "What's the current temperature in Paris in Fahrenheit?"
117
+ ]
118
+
119
+ # Run test cases
120
+ for query in test_queries:
121
+ print(f"\nQuery: {query}")
122
+ formatted_prompt = assembler.assemble_prompt(query, weather_api, TASK_INSTRUCTION, FORMAT_INSTRUCTION)
123
+ print("Formatted Prompt:")
124
+ print(formatted_prompt)
125
+ print("-" * 50)
126
+
127
+ # Test case 2: Multiple APIs, follows the OpenAI format: https://platform.openai.com/docs/guides/function-calling
128
+ calculator_api = {
129
+ "name": "calculate",
130
+ "description": "Perform a mathematical calculation",
131
+ "parameters": {
132
+ "type": "object",
133
+ "properties": {
134
+ "operation": {
135
+ "type": "string",
136
+ "enum": ["add", "subtract", "multiply", "divide"],
137
+ "description": "The mathematical operation to perform"
138
+ },
139
+ "x": {
140
+ "type": "number",
141
+ "description": "The first number"
142
+ },
143
+ "y": {
144
+ "type": "number",
145
+ "description": "The second number"
146
+ }
147
+ },
148
+ "required": ["operation", "x", "y"]
149
+ }
150
+ }
151
+
152
+ multi_api_query = "What's the weather in Miami and what's 15 multiplied by 7?"
153
+ print(f"\nMulti-API Query: {multi_api_query}")
154
+ multi_api_formatted_prompt = assembler.assemble_prompt(
155
+ multi_api_query,
156
+ [weather_api, calculator_api],
157
+ TASK_INSTRUCTION,
158
+ FORMAT_INSTRUCTION
159
+ )
160
+ print("Formatted Prompt:")
161
+ print(multi_api_formatted_prompt)
examples/test_xlam_model_with_endpoint.py ADDED
@@ -0,0 +1,188 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import json
3
+ import time
4
+ from openai import OpenAI
5
+
6
+ # Default prompts
7
+ TASK_INSTRUCTION = """
8
+ You are an expert in composing functions. You are given a question and a set of possible functions.
9
+ Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
10
+ If none of the functions can be used, point it out and refuse to answer.
11
+ If the given question lacks the parameters required by the function, also point it out.
12
+ """.strip()
13
+
14
+ FORMAT_INSTRUCTION = """
15
+ The output MUST strictly adhere to the following JSON format, and NO other text MUST be included.
16
+ The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'
17
+ ```
18
+ {
19
+ "tool_calls": [
20
+ {"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},
21
+ ... (more tool calls as required)
22
+ ]
23
+ }
24
+ ```
25
+ """.strip()
26
+
27
+
28
+ class XLAMHandler:
29
+ def __init__(self, model_name, temperature=0.3, top_p=1, max_tokens=512, port=8000):
30
+ self.model_name = model_name
31
+ self.temperature = temperature
32
+ self.top_p = top_p
33
+ self.max_tokens = max_tokens
34
+ base_url = f"http://localhost:{port}/v1"
35
+ self.client = OpenAI(api_key="Empty", base_url=base_url)
36
+
37
+ def process_query(self, query, tools, task_instruction, format_instruction):
38
+ # Convert tools to XLAM format
39
+ xlam_tools = self.convert_to_xlam_tool(tools)
40
+
41
+ # Build the input prompt
42
+ prompt = self.build_prompt(query, xlam_tools, task_instruction, format_instruction)
43
+
44
+ # Create message for API call
45
+ message = [{"role": "user", "content": prompt}]
46
+
47
+ # Make API call
48
+ start_time = time.time()
49
+ response = self.client.chat.completions.create(
50
+ messages=message,
51
+ model=self.model_name,
52
+ temperature=self.temperature,
53
+ max_tokens=self.max_tokens,
54
+ top_p=self.top_p,
55
+ )
56
+ latency = time.time() - start_time
57
+
58
+ # Parse response
59
+ result = response.choices[0].message.content
60
+ parsed_result, success, _ = self.parse_response(result)
61
+
62
+ # Prepare metadata
63
+ metadata = {
64
+ "input_tokens": response.usage.prompt_tokens,
65
+ "output_tokens": response.usage.completion_tokens,
66
+ "latency": latency
67
+ }
68
+
69
+ return parsed_result, metadata
70
+
71
+ def convert_to_xlam_tool(self, tools):
72
+ if isinstance(tools, dict):
73
+ return {
74
+ "name": tools["name"],
75
+ "description": tools["description"],
76
+ "parameters": {k: v for k, v in tools["parameters"].get("properties", {}).items()}
77
+ }
78
+ elif isinstance(tools, list):
79
+ return [self.convert_to_xlam_tool(tool) for tool in tools]
80
+ else:
81
+ return tools
82
+
83
+ def build_prompt(self, query, tools, task_instruction=TASK_INSTRUCTION, format_instruction=FORMAT_INSTRUCTION):
84
+ prompt = f"[BEGIN OF TASK INSTRUCTION]\n{task_instruction}\n[END OF TASK INSTRUCTION]\n\n"
85
+ prompt += f"[BEGIN OF AVAILABLE TOOLS]\n{json.dumps(tools)}\n[END OF AVAILABLE TOOLS]\n\n"
86
+ prompt += f"[BEGIN OF FORMAT INSTRUCTION]\n{format_instruction}\n[END OF FORMAT INSTRUCTION]\n\n"
87
+ prompt += f"[BEGIN OF QUERY]\n{query}\n[END OF QUERY]\n\n"
88
+ return prompt
89
+
90
+ def parse_response(self, response):
91
+ try:
92
+ data = json.loads(response)
93
+ tool_calls = data.get('tool_calls', []) if isinstance(data, dict) else data
94
+ result = [
95
+ {tool_call['name']: tool_call['arguments']}
96
+ for tool_call in tool_calls if isinstance(tool_call, dict)
97
+ ]
98
+ return result, True, []
99
+ except json.JSONDecodeError:
100
+ return [], False, ["Failed to parse JSON response"]
101
+
102
+ if __name__ == "__main__":
103
+ parser = argparse.ArgumentParser(description="Test XLAM model with endpoint")
104
+ parser.add_argument("--model_name", default="xlam-1b-fc-r", help="Name of the model")
105
+ parser.add_argument("--port", type=int, default=8001, help="Port number for the endpoint")
106
+ parser.add_argument("--temperature", type=float, default=0.3, help="Temperature for sampling")
107
+ parser.add_argument("--top_p", type=float, default=1.0, help="Top p for sampling")
108
+ parser.add_argument("--max_tokens", type=int, default=512, help="Maximum number of tokens to generate")
109
+
110
+ args = parser.parse_args()
111
+
112
+ # Initialize the XLAMHandler with command-line arguments
113
+ handler = XLAMHandler(args.model_name, temperature=args.temperature, top_p=args.top_p, max_tokens=args.max_tokens, port=args.port)
114
+
115
+ # Test case 1: Weather API, follows the OpenAI format: https://platform.openai.com/docs/guides/function-calling
116
+ weather_api = {
117
+ "name": "get_weather",
118
+ "description": "Get the current weather for a location",
119
+ "parameters": {
120
+ "type": "object",
121
+ "properties": {
122
+ "location": {
123
+ "type": "string",
124
+ "description": "The city and state, e.g. San Francisco, CA"
125
+ },
126
+ "unit": {
127
+ "type": "string",
128
+ "enum": ["celsius", "fahrenheit"],
129
+ "description": "The unit of temperature to return"
130
+ }
131
+ },
132
+ "required": ["location"]
133
+ }
134
+ }
135
+
136
+ # Test queries
137
+ test_queries = [
138
+ "What's the weather like in New York?",
139
+ "Tell me the temperature in London in Celsius",
140
+ "What's the weather forecast for Tokyo?",
141
+ "What is the stock price of CRM?", # the model should return an empty list, meaning that it refuse to answer this irrelevant query and tools.
142
+ "What's the current temperature in Paris in Fahrenheit?"
143
+ ]
144
+
145
+ # Run test cases
146
+ for query in test_queries:
147
+ print(f"Query: {query}")
148
+ result, metadata = handler.process_query(query, weather_api, TASK_INSTRUCTION, FORMAT_INSTRUCTION)
149
+ print(f"Result: {json.dumps(result, indent=2)}")
150
+ print(f"Metadata: {json.dumps(metadata, indent=2)}")
151
+ print("-" * 50)
152
+
153
+ # Test case 2: Multiple APIs, follows the OpenAI format: https://platform.openai.com/docs/guides/function-calling
154
+ calculator_api = {
155
+ "name": "calculate",
156
+ "description": "Perform a mathematical calculation",
157
+ "parameters": {
158
+ "type": "object",
159
+ "properties": {
160
+ "operation": {
161
+ "type": "string",
162
+ "enum": ["add", "subtract", "multiply", "divide"],
163
+ "description": "The mathematical operation to perform"
164
+ },
165
+ "x": {
166
+ "type": "number",
167
+ "description": "The first number"
168
+ },
169
+ "y": {
170
+ "type": "number",
171
+ "description": "The second number"
172
+ }
173
+ },
174
+ "required": ["operation", "x", "y"]
175
+ }
176
+ }
177
+
178
+ multi_api_query = "What's the weather in Miami and what's 15 multiplied by 7?"
179
+ multi_api_result, multi_api_metadata = handler.process_query(
180
+ multi_api_query,
181
+ [weather_api, calculator_api],
182
+ TASK_INSTRUCTION,
183
+ FORMAT_INSTRUCTION
184
+ )
185
+
186
+ print("Multi-API Query Result:")
187
+ print(json.dumps(multi_api_result, indent=2))
188
+ print(f"Metadata: {json.dumps(multi_api_metadata, indent=2)}")
examples/test_xlam_model_with_vllm.py ADDED
@@ -0,0 +1,193 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import json
2
+ import time
3
+ import argparse
4
+ from typing import List, Dict
5
+
6
+ from vllm import LLM, SamplingParams
7
+ from jinja2 import Template
8
+
9
+
10
+ # Default prompts
11
+ TASK_INSTRUCTION = """
12
+ You are an expert in composing functions. You are given a question and a set of possible functions.
13
+ Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
14
+ If none of the functions can be used, point it out and refuse to answer.
15
+ If the given question lacks the parameters required by the function, also point it out.
16
+ """.strip()
17
+
18
+ FORMAT_INSTRUCTION = """
19
+ The output MUST strictly adhere to the following JSON format, and NO other text MUST be included.
20
+ The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'
21
+ ```
22
+ {
23
+ "tool_calls": [
24
+ {"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},
25
+ ... (more tool calls as required)
26
+ ]
27
+ }
28
+ ```
29
+ """.strip()
30
+
31
+
32
+ class XLAMHandler:
33
+ def __init__(self, model: str, temperature: float = 0.3, top_p: float = 1, max_tokens: int = 512):
34
+ self.llm = LLM(model=model)
35
+ self.sampling_params = SamplingParams(
36
+ temperature=temperature,
37
+ top_p=top_p,
38
+ max_tokens=max_tokens
39
+ )
40
+ self.chat_template = self.llm.get_tokenizer().chat_template
41
+
42
+ @staticmethod
43
+ def apply_chat_template(template, messages):
44
+ jinja_template = Template(template)
45
+ return jinja_template.render(messages=messages)
46
+
47
+ def process_query(self, query: str, tools: Dict, task_instruction: str, format_instruction: str):
48
+ # Convert tools to XLAM format
49
+ xlam_tools = self.convert_to_xlam_tool(tools)
50
+
51
+ # Build the input prompt
52
+ prompt = self.build_prompt(query, xlam_tools, task_instruction, format_instruction)
53
+
54
+ messages = [
55
+ {"role": "user", "content": prompt}
56
+ ]
57
+ formatted_prompt = self.apply_chat_template(self.chat_template, messages)
58
+
59
+ # Make inference
60
+ start_time = time.time()
61
+ outputs = self.llm.generate([formatted_prompt], self.sampling_params)
62
+ latency = time.time() - start_time
63
+
64
+ # Parse response
65
+ result = outputs[0].outputs[0].text
66
+ parsed_result, success, _ = self.parse_response(result)
67
+
68
+ # Prepare metadata
69
+ metadata = {
70
+ "latency": latency,
71
+ "success": success,
72
+ }
73
+
74
+ return parsed_result, metadata
75
+
76
+ def convert_to_xlam_tool(self, tools):
77
+ if isinstance(tools, dict):
78
+ return {
79
+ "name": tools["name"],
80
+ "description": tools["description"],
81
+ "parameters": {k: v for k, v in tools["parameters"].get("properties", {}).items()}
82
+ }
83
+ elif isinstance(tools, list):
84
+ return [self.convert_to_xlam_tool(tool) for tool in tools]
85
+ else:
86
+ return tools
87
+
88
+ def build_prompt(self, query, tools, task_instruction=TASK_INSTRUCTION, format_instruction=FORMAT_INSTRUCTION):
89
+ prompt = f"[BEGIN OF TASK INSTRUCTION]\n{task_instruction}\n[END OF TASK INSTRUCTION]\n\n"
90
+ prompt += f"[BEGIN OF AVAILABLE TOOLS]\n{json.dumps(tools)}\n[END OF AVAILABLE TOOLS]\n\n"
91
+ prompt += f"[BEGIN OF FORMAT INSTRUCTION]\n{format_instruction}\n[END OF FORMAT INSTRUCTION]\n\n"
92
+ prompt += f"[BEGIN OF QUERY]\n{query}\n[END OF QUERY]\n\n"
93
+ return prompt
94
+
95
+ def parse_response(self, response):
96
+ try:
97
+ data = json.loads(response)
98
+ tool_calls = data.get('tool_calls', []) if isinstance(data, dict) else data
99
+ result = [
100
+ {tool_call['name']: tool_call['arguments']}
101
+ for tool_call in tool_calls if isinstance(tool_call, dict)
102
+ ]
103
+ return result, True, []
104
+ except json.JSONDecodeError:
105
+ return [], False, ["Failed to parse JSON response"]
106
+
107
+
108
+ if __name__ == "__main__":
109
+ parser = argparse.ArgumentParser(description="Test XLAM model with vLLM")
110
+ parser.add_argument("--model", required=True, help="Path to the model")
111
+ parser.add_argument("--temperature", type=float, default=0.3, help="Temperature for sampling")
112
+ parser.add_argument("--top_p", type=float, default=1.0, help="Top p for sampling")
113
+ parser.add_argument("--max_tokens", type=int, default=512, help="Maximum number of tokens to generate")
114
+
115
+ args = parser.parse_args()
116
+
117
+ # Initialize the XLAMHandler with command-line arguments
118
+ handler = XLAMHandler(args.model, temperature=args.temperature, top_p=args.top_p, max_tokens=args.max_tokens)
119
+
120
+ # Test case 1: Weather API, follows the OpenAI format: https://platform.openai.com/docs/guides/function-calling
121
+ weather_api = {
122
+ "name": "get_weather",
123
+ "description": "Get the current weather for a location",
124
+ "parameters": {
125
+ "type": "object",
126
+ "properties": {
127
+ "location": {
128
+ "type": "string",
129
+ "description": "The city and state, e.g. San Francisco, CA"
130
+ },
131
+ "unit": {
132
+ "type": "string",
133
+ "enum": ["celsius", "fahrenheit"],
134
+ "description": "The unit of temperature to return"
135
+ }
136
+ },
137
+ "required": ["location"]
138
+ }
139
+ }
140
+
141
+ # Test queries
142
+ test_queries = [
143
+ "What's the weather like in New York?",
144
+ "Tell me the temperature in London in Celsius",
145
+ "What's the weather forecast for Tokyo?",
146
+ "What is the stock price of CRM?", # the model should return an empty list
147
+ "What's the current temperature in Paris in Fahrenheit?"
148
+ ]
149
+
150
+ # Run test cases
151
+ for query in test_queries:
152
+ print(f"Query: {query}")
153
+ result, metadata = handler.process_query(query, weather_api, TASK_INSTRUCTION, FORMAT_INSTRUCTION)
154
+ print(f"Result: {json.dumps(result, indent=2)}")
155
+ print(f"Metadata: {json.dumps(metadata, indent=2)}")
156
+ print("-" * 50)
157
+
158
+ # Test case 2: Multiple APIs, follows the OpenAI format: https://platform.openai.com/docs/guides/function-calling
159
+ calculator_api = {
160
+ "name": "calculate",
161
+ "description": "Perform a mathematical calculation",
162
+ "parameters": {
163
+ "type": "object",
164
+ "properties": {
165
+ "operation": {
166
+ "type": "string",
167
+ "enum": ["add", "subtract", "multiply", "divide"],
168
+ "description": "The mathematical operation to perform"
169
+ },
170
+ "x": {
171
+ "type": "number",
172
+ "description": "The first number"
173
+ },
174
+ "y": {
175
+ "type": "number",
176
+ "description": "The second number"
177
+ }
178
+ },
179
+ "required": ["operation", "x", "y"]
180
+ }
181
+ }
182
+
183
+ multi_api_query = "What's the weather in Miami and what's 15 multiplied by 7?"
184
+ multi_api_result, multi_api_metadata = handler.process_query(
185
+ multi_api_query,
186
+ [weather_api, calculator_api],
187
+ TASK_INSTRUCTION,
188
+ FORMAT_INSTRUCTION
189
+ )
190
+
191
+ print("Multi-API Query Result:")
192
+ print(json.dumps(multi_api_result, indent=2))
193
+ print(f"Metadata: {json.dumps(multi_api_metadata, indent=2)}")
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 32013,
4
+ "eos_token_id": 32021,
5
+ "transformers_version": "4.40.0"
6
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:540b4f58ab7f35c7d4b81e425b52844511f37e74b6388f5c228ad48ea526cb84
3
+ size 2692969128
special_tokens_map.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ {
4
+ "content": "<|EOT|>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false
9
+ }
10
+ ],
11
+ "bos_token": {
12
+ "content": "<|begin▁of▁sentence|>",
13
+ "lstrip": false,
14
+ "normalized": true,
15
+ "rstrip": false,
16
+ "single_word": false
17
+ },
18
+ "eos_token": {
19
+ "content": "<|EOT|>",
20
+ "lstrip": false,
21
+ "normalized": true,
22
+ "rstrip": false,
23
+ "single_word": false
24
+ },
25
+ "pad_token": {
26
+ "content": "<|end▁of▁sentence|>",
27
+ "lstrip": false,
28
+ "normalized": true,
29
+ "rstrip": false,
30
+ "single_word": false
31
+ }
32
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,198 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "added_tokens_decoder": {
5
+ "32000": {
6
+ "content": "õ",
7
+ "lstrip": false,
8
+ "normalized": true,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": false
12
+ },
13
+ "32001": {
14
+ "content": "÷",
15
+ "lstrip": false,
16
+ "normalized": true,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": false
20
+ },
21
+ "32002": {
22
+ "content": "Á",
23
+ "lstrip": false,
24
+ "normalized": true,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": false
28
+ },
29
+ "32003": {
30
+ "content": "ý",
31
+ "lstrip": false,
32
+ "normalized": true,
33
+ "rstrip": false,
34
+ "single_word": false,
35
+ "special": false
36
+ },
37
+ "32004": {
38
+ "content": "À",
39
+ "lstrip": false,
40
+ "normalized": true,
41
+ "rstrip": false,
42
+ "single_word": false,
43
+ "special": false
44
+ },
45
+ "32005": {
46
+ "content": "ÿ",
47
+ "lstrip": false,
48
+ "normalized": true,
49
+ "rstrip": false,
50
+ "single_word": false,
51
+ "special": false
52
+ },
53
+ "32006": {
54
+ "content": "ø",
55
+ "lstrip": false,
56
+ "normalized": true,
57
+ "rstrip": false,
58
+ "single_word": false,
59
+ "special": false
60
+ },
61
+ "32007": {
62
+ "content": "ú",
63
+ "lstrip": false,
64
+ "normalized": true,
65
+ "rstrip": false,
66
+ "single_word": false,
67
+ "special": false
68
+ },
69
+ "32008": {
70
+ "content": "þ",
71
+ "lstrip": false,
72
+ "normalized": true,
73
+ "rstrip": false,
74
+ "single_word": false,
75
+ "special": false
76
+ },
77
+ "32009": {
78
+ "content": "ü",
79
+ "lstrip": false,
80
+ "normalized": true,
81
+ "rstrip": false,
82
+ "single_word": false,
83
+ "special": false
84
+ },
85
+ "32010": {
86
+ "content": "ù",
87
+ "lstrip": false,
88
+ "normalized": true,
89
+ "rstrip": false,
90
+ "single_word": false,
91
+ "special": false
92
+ },
93
+ "32011": {
94
+ "content": "ö",
95
+ "lstrip": false,
96
+ "normalized": true,
97
+ "rstrip": false,
98
+ "single_word": false,
99
+ "special": false
100
+ },
101
+ "32012": {
102
+ "content": "û",
103
+ "lstrip": false,
104
+ "normalized": true,
105
+ "rstrip": false,
106
+ "single_word": false,
107
+ "special": false
108
+ },
109
+ "32013": {
110
+ "content": "<|begin▁of▁sentence|>",
111
+ "lstrip": false,
112
+ "normalized": true,
113
+ "rstrip": false,
114
+ "single_word": false,
115
+ "special": true
116
+ },
117
+ "32014": {
118
+ "content": "<|end▁of▁sentence|>",
119
+ "lstrip": false,
120
+ "normalized": true,
121
+ "rstrip": false,
122
+ "single_word": false,
123
+ "special": true
124
+ },
125
+ "32015": {
126
+ "content": "<|fim▁hole|>",
127
+ "lstrip": false,
128
+ "normalized": true,
129
+ "rstrip": false,
130
+ "single_word": false,
131
+ "special": false
132
+ },
133
+ "32016": {
134
+ "content": "<|fim▁begin|>",
135
+ "lstrip": false,
136
+ "normalized": true,
137
+ "rstrip": false,
138
+ "single_word": false,
139
+ "special": false
140
+ },
141
+ "32017": {
142
+ "content": "<|fim▁end|>",
143
+ "lstrip": false,
144
+ "normalized": true,
145
+ "rstrip": false,
146
+ "single_word": false,
147
+ "special": false
148
+ },
149
+ "32018": {
150
+ "content": "<pad>",
151
+ "lstrip": false,
152
+ "normalized": true,
153
+ "rstrip": false,
154
+ "single_word": false,
155
+ "special": false
156
+ },
157
+ "32019": {
158
+ "content": "<|User|>",
159
+ "lstrip": false,
160
+ "normalized": true,
161
+ "rstrip": false,
162
+ "single_word": false,
163
+ "special": false
164
+ },
165
+ "32020": {
166
+ "content": "<|Assistant|>",
167
+ "lstrip": false,
168
+ "normalized": true,
169
+ "rstrip": false,
170
+ "single_word": false,
171
+ "special": false
172
+ },
173
+ "32021": {
174
+ "content": "<|EOT|>",
175
+ "lstrip": false,
176
+ "normalized": false,
177
+ "rstrip": false,
178
+ "single_word": false,
179
+ "special": true
180
+ }
181
+ },
182
+ "additional_special_tokens": [
183
+ "<|EOT|>"
184
+ ],
185
+ "bos_token": "<|begin▁of▁sentence|>",
186
+ "chat_template": "{% set system_message = 'You are an AI assistant for function calling.For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer\\n' %}{% if messages[0]['role'] == 'system' %}{% set system_message = messages[0]['content'] %}{% endif %}{% if system_message is defined %}{{ system_message }}{% endif %}{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '### Instruction:\\n' + content + '\\n### Response:' }}{% elif message['role'] == 'assistant' %}{{ '\\n' + content + '\\n<|EOT|>\\n' }}{% endif %}{% endfor %}",
187
+ "clean_up_tokenization_spaces": false,
188
+ "eos_token": "<|EOT|>",
189
+ "legacy": true,
190
+ "model_max_length": 16384,
191
+ "pad_token": "<|end▁of▁sentence|>",
192
+ "padding_side": "right",
193
+ "sp_model_kwargs": {},
194
+ "split_special_tokens": false,
195
+ "tokenizer_class": "LlamaTokenizer",
196
+ "unk_token": null,
197
+ "use_default_system_prompt": false
198
+ }