Hi, how should I deploy this model? It seems that there is an error when I deploy it using vllm.
#1
by
wingchi
- opened
Deploy method:
python -m vllm.entrypoints.openai.api_server --served-model-name medfound --model medicalai/MedFound-7B --host 0.0.0.0 --port 8111 --gpu-memory-utilization 0.9 --trust-remote-code
Request Body:
{
"model": "medfound",
"temperature": 0.5,
"top_p": 0.8,
"repetition_penalty": 1.1,
"messages": [
{
"role": "user",
"content": "你好，你是谁"
}
]
}
Reponse:
{
"object": "error",
"message": "As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one.",
"type": "BadRequestError",
"param": null,
"code": 400
}