Upload README.md
Browse files
README.md
ADDED
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
CuteGPT is an open-source conversational language model that supports both Chinese and English, developed by Fudan University KnowledgeWorks Laboratory. It is based on the original Llama model structure, and has a scale of 13B (13 billion) parameters. It can perform int8 precision inference on a single 3090 graphics card. CuteGPT expands the Chinese vocabulary and performs pre-training on the Llama model, improving its ability to understand Chinese. Subsequently, it is fine-tuned with conversational instructions to enhance the model's ability to understand instructions. Based on the KW-CuteGPT-7b version, KW-CuteGPT-13b has improved accuracy in knowledge, understanding of complex instructions, ability to comprehend long texts, reasoning ability, faithful question answering, and other capabilities. Currently, the KW-CuteGPT-13b version model outperforms the majority of models of similar scale in certain evaluation tasks.
|
2 |
+
|
3 |
+
**Note: Ask The FAIR team of Meta AI for the license for LLAMA usage first.**
|
4 |
+
|
5 |
+
```python
|
6 |
+
from transformers import LlamaForCausalLM, LlamaTokenizer
|
7 |
+
import torch
|
8 |
+
|
9 |
+
def generate_prompt(query, history, input=None):
|
10 |
+
prompt = ""
|
11 |
+
for i, (old_query, response) in enumerate(history):
|
12 |
+
prompt += "{}{}\n<end>".format(old_query, response)
|
13 |
+
prompt += "{}".format(query)
|
14 |
+
return prompt
|
15 |
+
|
16 |
+
|
17 |
+
|
18 |
+
# Load model
|
19 |
+
device = torch.device("cuda:0")
|
20 |
+
model_name = "/data/dell/xuyipei/my_llama/my_llama_13b/llama_13b_112_sft_v1"
|
21 |
+
tokenizer = LlamaTokenizer.from_pretrained(model_name)
|
22 |
+
model = LlamaForCausalLM.from_pretrained(
|
23 |
+
model_name,
|
24 |
+
torch_dtype=torch.float16
|
25 |
+
)
|
26 |
+
model.eval()
|
27 |
+
model = model.to(device)
|
28 |
+
|
29 |
+
|
30 |
+
# Inference
|
31 |
+
history = []
|
32 |
+
queries = ['请推荐五本名著,依次列出作品名、作者\n', '请再来三本\n']
|
33 |
+
memory_limit = 3 # the number of (query, response) to remember
|
34 |
+
for query in queries:
|
35 |
+
prompt = generate_prompt(prompt, history)
|
36 |
+
input_ids = tokenizer(query, return_tensors="pt", padding=False, truncation=False, add_special_tokens=False)
|
37 |
+
input_ids = input_ids["input_ids"].to(device)
|
38 |
+
|
39 |
+
with torch.no_grad():
|
40 |
+
outputs=model.generate(
|
41 |
+
input_ids=input_ids,
|
42 |
+
top_p=0.8,
|
43 |
+
top_k=50,
|
44 |
+
repetition_penalty=1.1,
|
45 |
+
max_new_tokens = 256,
|
46 |
+
early_stopping = True,
|
47 |
+
eos_token_id = tokenizer.convert_tokens_to_ids('<end>'),
|
48 |
+
pad_token_id = tokenizer.eos_token_id,
|
49 |
+
min_length = input_ids.shape[1] + 1
|
50 |
+
)
|
51 |
+
s = outputs[0]
|
52 |
+
response=tokenizer.decode(s)
|
53 |
+
response = response.replace('<s>', '').replace('<end>', '').replace('</s>', '')
|
54 |
+
print(response)
|
55 |
+
history.append((query, response))
|
56 |
+
history = history[-memory_limit:]
|
57 |
+
```
|