File size: 2,324 Bytes
6e4e9d1 b25f8b7 6e4e9d1 b25f8b7 6e4e9d1 b25f8b7 6e4e9d1 fcaea19 75a98eb 6e4e9d1 75a98eb 6e4e9d1 fcaea19 3eb2551 d3d4e31 fcaea19 d3d4e31 eeca0c9 fcaea19 d3d4e31 0fa6d9b fcaea19 6e4e9d1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
---
license: cc-by-4.0
datasets:
- lksy/ru_instruct_gpt4
- IlyaGusev/ru_turbo_saiga
- IlyaGusev/ru_sharegpt_cleaned
- IlyaGusev/oasst1_ru_main_branch
language:
- ru
pipeline_tag: text-generation
---
This is a generative model converted to fp16 format based on [IlyaGusev/saiga_mistral_7b_lora](https://huggingface.co/IlyaGusev/saiga_mistral_7b_lora)
Install vLLM:
```bash
pip install vllm
```
Start server:
```bash
python -u -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --model Gaivoronsky/Mistral-7B-Saiga
```
Client:
```python
import openai
openai.api_base = "http://localhost:8000/v1"
openai.api_key = "none"
DEFAULT_MESSAGE_TEMPLATE = "<s>{role}\n{content}</s>"
DEFAULT_RESPONSE_TEMPLATE = "<s>bot\n"
DEFAULT_SYSTEM_PROMPT = "Ты — Сайга, русскоязычный автоматический ассистент. Ты разговариваешь с людьми и помогаешь им."
class Conversation:
def __init__(
self,
message_template=DEFAULT_MESSAGE_TEMPLATE,
system_prompt=DEFAULT_SYSTEM_PROMPT,
response_template=DEFAULT_RESPONSE_TEMPLATE
):
self.message_template = message_template
self.response_template = response_template
self.messages = [{
"role": "system",
"content": system_prompt
}]
def add_user_message(self, message):
self.messages.append({
"role": "user",
"content": message
})
def add_bot_message(self, message):
self.messages.append({
"role": "bot",
"content": message
})
def get_prompt(self):
final_text = ""
for message in self.messages:
message_text = self.message_template.format(**message)
final_text += message_text
final_text += DEFAULT_RESPONSE_TEMPLATE
return final_text.strip()
query = "Сколько весит жираф?"
conversation = Conversation()
conversation.add_user_message(query)
prompt = conversation.get_prompt()
response = openai.ChatCompletion.create(
model="Gaivoronsky/Mistral-7B-Saiga",
messages=[{"role": "user", "content": prompt}],
system=DEFAULT_SYSTEM_PROMPT,
max_tokens=512,
stop=['</s>']
)
response['choices'][0]['message']['content']
``` |