Best Configuration to Reduced Model Repetition
#6
by
ronmasas
- opened
FYI: It was fun to play with, but I don't think this model can be useful at this stage.
There is no usage example provided in the model card for generation. However, I found that utilizing the default settings can result in repetition.
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "yam-peleg/Hebrew-Gemma-11B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="cuda")
chat = [
{ "role": "user", "content": "讻转讜讘 诇讬 砖诇讜砖讛 讻讜转专讜转 诇讘诇讜讙 注诇 住讬讬讘专 讜驻讬爪讛." }
]
# stop token
eos_token_id = tokenizer.convert_tokens_to_ids('<end_of_turn>')
prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer.encode(prompt, return_tensors="pt")
model_device = next(model.parameters()).device
input_ids = input_ids.to(model_device)
output_ids = model.generate(input_ids, max_length=150, eos_token_id=eos_token_id)
output_text = tokenizer.decode(output_ids[0], skip_special_tokens=False)
print(output_text)
<bos><start_of_turn>user
讻转讜讘 诇讬 砖诇讜砖讛 讻讜转专讜转 诇讘诇讜讙 注诇 住讬讬讘专 讜驻讬爪讛.<end_of_turn>
<start_of_turn>model
1. "住讬讬讘专 讜驻讬爪讛: 砖讬诇讜讘 驻讬爪讛 注诐 讟讻谞讜诇讜讙讬讛"
2. "住讬讬讘专 讜驻讬爪讛: 讻讬爪讚 讟讻谞讜诇讜讙讬讛 讬讻讜诇讛 诇讛驻讜讱 讗转 讛驻讬爪讛 诇诪讜爪诇讞转 讬讜转专"
3. "住讬讬讘专 讜驻讬爪讛: 讻讬爪讚 讟讻谞讜诇讜讙讬讛 讬讻讜诇讛 诇讛驻讜讱 讗转 讛驻讬爪讛 诇诪讜爪诇讞转 讬讜转专"<end_of_turn>
Setting do_sample=True and top_p=0.80 worked best for me.
chat = [
{ "role": "user", "content": "讻转讜讘 诇讬 砖诇讜砖讛 讻讜转专讜转 诇讘诇讜讙 注诇 住讬讬讘专 讜驻讬爪讛." }
]
prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer.encode(prompt, return_tensors="pt")
model_device = next(model.parameters()).device
input_ids = input_ids.to(model_device)
output_ids = model.generate(input_ids, max_length=150, eos_token_id=eos_token_id, do_sample=True, top_p=0.80)
output_text = tokenizer.decode(output_ids[0], skip_special_tokens=False)
print(output_text)
<bos><start_of_turn>user
讻转讜讘 诇讬 砖诇讜砖讛 讻讜转专讜转 诇讘诇讜讙 注诇 住讬讬讘专 讜驻讬爪讛.<end_of_turn>
<start_of_turn>model
1. 驻讬爪讛 注诇 讛讗拽专讬诐: 住讬讬讘专, 驻讬爪讛 讜讞讬诇讜抓-爪注专
2. 住讬讬讘专, 驻讬爪讛 讜拽住诐: 拽讞 讗转 讛讚专讬讻讜谞讬诐 砖诇讱 诇注讜诇诐 讛讗诪讬转讬
3. 驻讬爪讛 讜-S.E.E.F: 住讬讬讘专, 驻讬爪讛, 注讬专讘讜谉 讜讛转讞砖讘谞讜转 讗转讬拽讛.<end_of_turn>