tirthadagr8 commited on
Commit
46a0b74
Β·
verified Β·
1 Parent(s): 372dfcc

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -0
README.md CHANGED
@@ -12,6 +12,20 @@ Tokenizer used is from Gemma 2-2B-JPN-IT which is trained on japanese dataset fr
12
 
13
  Model usage:-
14
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
15
 
16
  ```bibtex
17
  @ARTICLE{pryzant_jesc_2018,
 
12
 
13
  Model usage:-
14
 
15
+ ```python
16
+ from transformers import AutoTokenizer,AutoModelForCausalLM
17
+ tokenizer=AutoTokenizer.from_pretrained('tirthadagr8/Japanese_to_english_gpt2CasualLM_GemmaTokenizer')
18
+ model=AutoModelForCausalLM.from_pretrained('tirthadagr8/Japanese_to_english_gpt2CasualLM_GemmaTokenizer')
19
+ model.cuda()
20
+ src_text='あγͺたとは遊びたくγͺい'
21
+ print(tokenizer.batch_decode(model.generate(tokenizer.encode(f"Translate the following Japanese sentence to English:\n\nJapanese:{src_text}\nEnglish:",return_tensors='pt')[:,:-1].cuda(),max_length=128))[0])
22
+
23
+ OUTPUT:
24
+ ```
25
+ <bos>Translate the following Japanese sentence to English:
26
+
27
+ Japanese:あγͺたとは遊びたくγͺい
28
+ English:i don't want to play with you.<eos>
29
 
30
  ```bibtex
31
  @ARTICLE{pryzant_jesc_2018,