cognitivecomputations

/

Dolphin3.0-Llama3.1-8B

like 172

Follow

Cognitive Computations 2.08k

The model supports tools.
Don't know about ollama 's integration. You will have to ask them in their support channel.

Super, thanks ! and how does it compare to Qwen 2.5 7b ? thanks !

@apiro93 you may have to add a GGUF manually by creating a modelfile. The reason it is returning an error is because of the template, even though the model supports tools. I haven't checked what chat template it uses, but if it is the same as normal llama3.1/3 you can just copy and paste the template.

this is the template it uses https://ollama.com/library/dolphin3/blobs/4b01fbe300da:

{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 -}}
<|im_start|>{{ .Role }}
{{ .Content }}{{ if not $last }}<|im_end|>
{{ end }}
{{- if and (ne .Role "assistant") $last }}<|im_end|>
<|im_start|>assistant
{{ end }}
{{- end }}

maybe I can use https://ollama.com/library/llama3.3/blobs/948af2743fc7 :
{{- if or .System .Tools }}<|start_header_id|>system<|end_header_id|>
{{- if .System }}

{{ .System }}
{{- end }}
{{- if .Tools }}

Cutting Knowledge Date: December 2023

When you receive a tool call response, use the output to format an answer to the orginal user question.

You are a helpful assistant with tool calling capabilities.
{{- end }}<|eot_id|>
{{- end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}
{{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|>
{{- if and $.Tools $last }}

Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.

Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.

{{ range $.Tools }}
{{- . }}
{{ end }}
Question: {{ .Content }}<|eot_id|>
{{- else }}

{{ .Content }}<|eot_id|>
{{- end }}{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|>
{{- if .ToolCalls }}
{{ range .ToolCalls }}
{"name": "{{ .Function.Name }}", "parameters": {{ .Function.Arguments }}}{{ end }}
{{- else }}

{{ .Content }}
{{- end }}{{ if not $last }}<|eot_id|>{{ end }}
{{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|>

{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- end }}
{{- end }}

?

you can use Qwen's template if you like, that should make ollama stop complaining.

https://ollama.com/library/qwen2.5-coder:32b/blobs/e94a8ecb9327

Hello, I am running the model via VLLM and cannot make tool calling work. I am using this chat template (becuase the default one in the doc - chatlm - doesn't seem to support tool calling): https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_llama3.1_json.jinja

Hovewer, this gives me very strange results, e.g.:

1. Even the first message such as "Hi there!" it interprets as an attempt to call a tool
2. In some cases it ends up in infinite circles - calling tool tavily_search over and over again until reaching GRAPH_RECURSION_LIMIT error.

Is there a documentation somewhere that covers how to use VLLM + This model + Tool Calling?

Here is also a sample of my LangGraph implementation:

search_tool = TavilySearch(max_results=2)
tools = [search_tool]
llm_with_tools = llm.bind_tools(tools, tool_choice='auto')

async def chatbot(state: State):
    response = await llm_with_tools.ainvoke(state["messages"])
    return {"messages": [response]}

graph_builder.add_node("chatbot", chatbot)
tool_node = ToolNode(tools=tools)
graph_builder.add_node("tools", tool_node)
graph_builder.add_conditional_edges(
    "chatbot",
    tools_condition,
)

#Build the graph
graph_builder.add_edge("tools", "chatbot")
graph_builder.add_edge(START, "chatbot")
graph = graph_builder.compile()
return graph