Upload README.md

README.md

@@ -29,6 +29,17 @@ It was a proof of concept for merging LLMs trained in other languages, and paid
 
 As far as I know, Swallow is the full set Llama 2 model(7B, 13B, 70B) that can output the most beautiful Japanese. Therefore, I used it as the base model for merging this time. Thank you for their wonderful work.
 
+## Test environment
+
+This model was tested using [text-generation-webui](https://github.com/oobabooga/text-generation-webui/tree/main). I use preset `simple-1` for Generation.
+
+Users reported that setting **repetition_penalty** is important to prevent repeated output. If you run into any issues, be sure to check your settings.
+
+- temperature: 0.7
+- top_p: 0.9
+- **repetition_penalty: 1.15**
+- top_k: 20
+
 ## Prompt template: Swallow (Alpaca format)
 
 ```
@@ -40,6 +51,78 @@ As far as I know, Swallow is the full set Llama 2 model(7B, 13B, 70B) that can o
 ### 応答:
 ```
 
+## Use the instruct model
+
+```
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_name = "nitky/Superswallow-13b-v0.1"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="auto")
+
+
+PROMPT_DICT = {
+ "prompt_input": (
+ "以下に、あるタスクを説明する指示があり、それに付随する入力が更なる文脈を提供しています。"
+ "リクエストを適切に完了するための回答を記述してください。\n\n"
+ "### 指示:\n{instruction}\n\n### 入力:\n{input}\n\n### 応答:"
+
+ ),
+ "prompt_no_input": (
+ "以下に、あるタスクを説明する指示があります。"
+ "リクエストを適切に完了するための回答を記述してください。\n\n"
+ "### 指示:\n{instruction}\n\n### 応答:"
+ ),
+}
+
+def create_prompt(instruction, input=None):
+ """
+ Generates a prompt based on the given instruction and an optional input.
+ If input is provided, it uses the 'prompt_input' template from PROMPT_DICT.
+ If no input is provided, it uses the 'prompt_no_input' template.
+
+ Args:
+ instruction (str): The instruction describing the task.
+ input (str, optional): Additional input providing context for the task. Default is None.
+
+ Returns:
+ str: The generated prompt.
+ """
+ if input:
+ # Use the 'prompt_input' template when additional input is provided
+ return PROMPT_DICT["prompt_input"].format(instruction=instruction, input=input)
+ else:
+ # Use the 'prompt_no_input' template when no additional input is provided
+ return PROMPT_DICT["prompt_no_input"].format(instruction=instruction)
+
+# Example usage
+instruction_example = "以下のトピックに関する簡潔な情報を提供してください。"
+input_example = "東京工業大学の主なキャンパスの一覧を、リスト形式で教えてください"
+prompt = create_prompt(instruction_example, input_example)
+
+input_ids = tokenizer.encode(
+ prompt,
+ add_special_tokens=False,
+ return_tensors="pt"
+)
+
+tokens = model.generate(
+ input_ids.to(device=model.device),
+ max_new_tokens=200,
+ temperature=0.7,
+ top_p=0.9,
+ repetition_penalty=1.15,
+ top_k=20,
+ do_sample=True,
+)
+
+out = tokenizer.decode(tokens[0], skip_special_tokens=True)
+print(out)
+
+```
+
 ## Merge Details
 ### Merge Method