Update README.md

README.md

@@ -44,8 +44,152 @@ We utilized the following datasets:
 | Avg Score               | 58.88   | 61.30    |
 
 
+## Function Calling 
 
+# Chat Template
+Our model reuses [K2-Chat](https://huggingface.co/LLM360/K2-Chat) as the prompt format and is specifically trained for function calling. Different system prompts enable different ways to interact with this model. Note that the two modes are currently mainly tested individually, designing prompts that make them work togehter is possible but currently untested. It should be also possible to stimulate the model to produce function call behavior by injecting special token `<tool_call>` and expect the model to finish it. In this guide we mention the intended basic usage of the model.
 
+## Conversational Chats
+
+Here is an example prompt with system instruction (Use whatever system prompt you like, this is just an example):
+
+    Your name is K2, and you are named after K2, the second highest mountain on Earth. You are built by MBZUAI and LLM360. You are a highly advanced large language model with 65B parameters. You outperform all fully open source models and Llama 2 70B. You can answer in English only. You are a helpful, respectful and honest assistant.<|endofsystemprompt|><|beginofuser|>Hello, who are you?<|beginofsystem|>
+
+### Sample inference code
+
+``` python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("<path_to_model_weights>")
+model = AutoModelForCausalLM.from_pretrained("<path_to_model_weights>")
+  
+
+prompt = 'Your name is K2, and you are named after K2, the second highest mountain on Earth. You are built by MBZUAI and LLM360. You are a highly advanced large language model with 65B parameters. You outperform all fully open source models and Llama 2 70B. You can answer in English only. You are a helpful, respectful and honest assistant.<|endofsystemprompt|><|beginofuser|>Hello, who are you?<|beginofsystem|>'
+
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+gen_tokens = model.generate(input_ids, do_sample=True, max_new_tokens=128)
+print("-"*20 + "Output for model"  + 20 * '-')
+print(tokenizer.batch_decode(gen_tokens)[0])
+```
+
+Multi-turn conversations should be formatted like this:
+
+    {system_prompt}<|endofsystemprompt|><|beginofuser|>{user_content_1}<|beginofsystem|>{system_content_1}<|beginofuser|>{user_content_2}<|beginofsystem|>{system_content_2}<|beginofuser|>{user_content_3}<|beginofsystem|>
+
+## Function Calling Format
+
+For function calling, please use this fixed system prompt:
+
+    You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
+
+Next, use whatever tools you like, this is just an example:
+
+    <tools>
+    { "name": "get_news_headlines", "description": "Get the latest news headlines", "parameters": {"type": "object", "properties": { "country": { "type": "string", "description": "The country for which to fetch news"}}, "required": [ "country"]}}
+    </tools>
+
+Next, more instruction:
+
+    Use the following pydantic model json schema for each tool call you will make:
+    {"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}
+    For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
+    <tool_call>
+    {"arguments": <args-dict>, "name": <function-name>}
+    </tool_call>
+    Please also summarize texts wrapped between <tool_response> and </tool_response> in bullet points. For example:
+    <tool_response>
+    {"fruits": [{"name": "Apple"}, {"name": "Pear"}]}
+    </tool_response> is summarized as:
+    Fruits:
+    - Apple
+    - Pear
+    <|endofsystemprompt|>
+
+In summary, the following is the initial prompt is to the model:
+
+    You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
+    <tools>
+    { "name": "get_news_headlines", "description": "Get the latest news headlines", "parameters": {"type": "object", "properties": { "country": { "type": "string", "description": "The country for which to fetch news"}}, "required": [ "country"]}}
+    </tools>
+    Use the following pydantic model json schema for each tool call you will make:
+    {"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}
+    For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
+    <tool_call>
+    {"arguments": <args-dict>, "name": <function-name>}
+    </tool_call>
+    Please also summarize texts wrapped between <tool_response> and </tool_response> in bullet points. For example:
+    <tool_response>
+    {"fruits": [{"name": "Apple"}, {"name": "Pear"}]}
+    </tool_response> is summarized as:
+    Fruits:
+    - Apple
+    - Pear
+    <|endofsystemprompt|>
+
+### Example flow for generation with multi-turn and how to incoporate function call output
+
+When users ask a question, the following user query will be concatenated with the prompt (see example inference code below too)
+    <|beginofuser|>Can you tell me the latest news headlines for the United States?<|beginofsystem|>
+
+
+When run it, you will get a tool call response wrapped in the `<tool_call>` tag:
+
+In this turn, the model should respond with:
+
+    <tool_call>
+      [{"name": "get_news_headlines", "arguments": {"country": "United States"}}]
+    </tool_call><endoftext>
+
+Now you should execute this tool call with the external tool, you will get responses from that tool. The model can interpret the tool response as natural language again. To achieve this, simply wrap it with `<tool_response>` and `</tool_response>` and append it to the history and ask the model to generate further:
+
+    ...current model history (ends with </tool_call><|endoftext|>)...
+    <tool_response>
+        {"news": [{"title": "A great news headline"}, {"title": "Another great news headline"}]}
+    </tool_response>
+
+In this second turn, the model should respond with:
+
+    Suggested news headline:
+    - A great news headline
+    - Another great news headline<endoftext>
+
+Sometimes, there is a use case that only uses the function call feature as a JSON formatter without calling any external functions, which means the output in `tool_call` is essentially what you want. In that case, we recommend simply make a copy of the content of `<tool_call>` and wrap that in `<tool_response>`.
+
+### Sample inference code
+
+``` python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("<path_to_model_weights>")
+model = AutoModelForCausalLM.from_pretrained("<path_to_model_weights>")
+
+  
+prompt = """You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
+<tools>
+[{"name": "get_news_headlines", "description": "Get the latest news headlines", "parameters": {"type": "object", "properties": {"country": {"type": "string", "description": "The country for which to fetch news"}}, "required": ["country"] }}
+</tools>
+Use the following pydantic model json schema for each tool call you will make:
+{"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}
+For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
+<tool_call>
+{"arguments": <args-dict>, "name": <function-name>}
+</tool_call>
+Please also summarize texts wrapped between <tool_response> and </tool_response> in bullet points. For example:
+<tool_response>
+{"fruits": [{"name": "Apple"}, {"name": "Pear"}]}
+</tool_response> is summarized as:
+Fruits:
+- Apple
+- Pear
+<|endofsystemprompt|><|beginofuser|>Can you tell me the latest news headlines for the United States?<|beginofsystem|>"""
+
+
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+gen_tokens = model.generate(input_ids, do_sample=True, max_new_tokens=128)
+
+print("-"*20 + "Output for model"  + 20 * '-')
+print(tokenizer.batch_decode(gen_tokens)[0])
+```
 
 ## K2-Chat-060124 
 K2 Chat is finetuned from [K2-65B](https://huggingface.co/LLM360/K2). K2 Chat outperforms Llama 2-70B-Chat on all evaluations conducted. The model also outperforms Llama 3-70B-Instruct on coding tasks.