tirthadagr8
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -12,6 +12,20 @@ Tokenizer used is from Gemma 2-2B-JPN-IT which is trained on japanese dataset fr
|
|
12 |
|
13 |
Model usage:-
|
14 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
15 |
|
16 |
```bibtex
|
17 |
@ARTICLE{pryzant_jesc_2018,
|
|
|
12 |
|
13 |
Model usage:-
|
14 |
|
15 |
+
```python
|
16 |
+
from transformers import AutoTokenizer,AutoModelForCausalLM
|
17 |
+
tokenizer=AutoTokenizer.from_pretrained('tirthadagr8/Japanese_to_english_gpt2CasualLM_GemmaTokenizer')
|
18 |
+
model=AutoModelForCausalLM.from_pretrained('tirthadagr8/Japanese_to_english_gpt2CasualLM_GemmaTokenizer')
|
19 |
+
model.cuda()
|
20 |
+
src_text='γγͺγγ¨γ―ιγ³γγγͺγ'
|
21 |
+
print(tokenizer.batch_decode(model.generate(tokenizer.encode(f"Translate the following Japanese sentence to English:\n\nJapanese:{src_text}\nEnglish:",return_tensors='pt')[:,:-1].cuda(),max_length=128))[0])
|
22 |
+
|
23 |
+
OUTPUT:
|
24 |
+
```
|
25 |
+
<bos>Translate the following Japanese sentence to English:
|
26 |
+
|
27 |
+
Japanese:γγͺγγ¨γ―ιγ³γγγͺγ
|
28 |
+
English:i don't want to play with you.<eos>
|
29 |
|
30 |
```bibtex
|
31 |
@ARTICLE{pryzant_jesc_2018,
|