| CuteGPT is an open-source conversational language model that supports both Chinese and English, developed by Fudan University KnowledgeWorks Laboratory. It is based on the original Llama model structure, and has a scale of 13B (13 billion) parameters. It can perform int8 precision inference on a single 3090 graphics card. CuteGPT expands the Chinese vocabulary and performs pre-training on the Llama model, improving its ability to understand Chinese. Subsequently, it is fine-tuned with conversational instructions to enhance the model's ability to understand instructions. Based on the KW-CuteGPT-7b version, KW-CuteGPT-13b has improved accuracy in knowledge, understanding of complex instructions, ability to comprehend long texts, reasoning ability, faithful question answering, and other capabilities. Currently, the KW-CuteGPT-13b version model outperforms the majority of models of similar scale in certain evaluation tasks. | |
| **Note: Ask The FAIR team of Meta AI for the license for LLAMA usage first.** | |
| ```python | |
| from transformers import LlamaForCausalLM, LlamaTokenizer | |
| import torch | |
| def generate_prompt(query, history, input=None): | |
| prompt = "" | |
| for i, (old_query, response) in enumerate(history): | |
| prompt += "{}{}\n<end>".format(old_query, response) | |
| prompt += "{}".format(query) | |
| return prompt | |
| # Load model | |
| device = torch.device("cuda:0") | |
| model_name = "/data/dell/xuyipei/my_llama/my_llama_13b/llama_13b_112_sft_v1" | |
| tokenizer = LlamaTokenizer.from_pretrained(model_name) | |
| model = LlamaForCausalLM.from_pretrained( | |
| model_name, | |
| torch_dtype=torch.float16 | |
| ) | |
| model.eval() | |
| model = model.to(device) | |
| # Inference | |
| history = [] | |
| queries = ['请推荐五本名著，依次列出作品名、作者\n', '请再来三本\n'] | |
| memory_limit = 3 # the number of (query, response) to remember | |
| for query in queries: | |
| prompt = generate_prompt(prompt, history) | |
| input_ids = tokenizer(query, return_tensors="pt", padding=False, truncation=False, add_special_tokens=False) | |
| input_ids = input_ids["input_ids"].to(device) | |
| with torch.no_grad(): | |
| outputs=model.generate( | |
| input_ids=input_ids, | |
| top_p=0.8, | |
| top_k=50, | |
| repetition_penalty=1.1, | |
| max_new_tokens = 256, | |
| early_stopping = True, | |
| eos_token_id = tokenizer.convert_tokens_to_ids('<end>'), | |
| pad_token_id = tokenizer.eos_token_id, | |
| min_length = input_ids.shape[1] + 1 | |
| ) | |
| s = outputs[0] | |
| response=tokenizer.decode(s) | |
| response = response.replace('<s>', '').replace('<end>', '').replace('</s>', '') | |
| print(response) | |
| history.append((query, response)) | |
| history = history[-memory_limit:] | |
| ``` |