Cenker Sisman
commited on
Commit
•
85c636c
1
Parent(s):
ac847b2
Update README.md
Browse files
README.md
CHANGED
@@ -24,8 +24,7 @@ widget:
|
|
24 |
|
25 |
<!-- Provide a quick summary of what the model is/does. -->
|
26 |
|
27 |
-
|
28 |
-
|
29 |
## Model Details
|
30 |
|
31 |
### Model Description
|
@@ -34,12 +33,11 @@ This modelcard aims to be a base template for new models. It has been generated
|
|
34 |
|
35 |
|
36 |
|
37 |
-
- **Developed by:**
|
38 |
-
- **Shared by [optional]:** [More Information Needed]
|
39 |
- **Model type:** [More Information Needed]
|
40 |
- **Language(s) (NLP):** [More Information Needed]
|
41 |
- **License:** [More Information Needed]
|
42 |
-
- **Finetuned from model
|
43 |
|
44 |
### Model Sources [optional]
|
45 |
|
@@ -85,9 +83,41 @@ Users (both direct and downstream) should be made aware of the risks, biases and
|
|
85 |
|
86 |
## How to Get Started with the Model
|
87 |
|
88 |
-
|
89 |
-
|
90 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
91 |
|
92 |
## Training Details
|
93 |
|
|
|
24 |
|
25 |
<!-- Provide a quick summary of what the model is/does. -->
|
26 |
|
27 |
+
Bu model test amaçlı hazırlanmıştır ve fikir vermesi açısından geliştirilmiştir. Model için Vikipedi üzerinden üretilen 40 bin soru cevap GPT ile eğitilmiştir. Daha büyük veri setlerinde daha iyi sonuçlar alınabilir.
|
|
|
28 |
## Model Details
|
29 |
|
30 |
### Model Description
|
|
|
33 |
|
34 |
|
35 |
|
36 |
+
- **Developed by:** Cenker Sisman
|
|
|
37 |
- **Model type:** [More Information Needed]
|
38 |
- **Language(s) (NLP):** [More Information Needed]
|
39 |
- **License:** [More Information Needed]
|
40 |
+
- **Finetuned from model :** redrussianarmy/gpt2-turkish-cased
|
41 |
|
42 |
### Model Sources [optional]
|
43 |
|
|
|
83 |
|
84 |
## How to Get Started with the Model
|
85 |
|
86 |
+
"""Inference"""
|
87 |
+
|
88 |
+
from transformers import PreTrainedTokenizerFast, GPT2LMHeadModel, GPT2TokenizerFast, GPT2Tokenizer
|
89 |
+
|
90 |
+
def load_model(model_path):
|
91 |
+
model = GPT2LMHeadModel.from_pretrained(model_path)
|
92 |
+
return model
|
93 |
+
|
94 |
+
|
95 |
+
def load_tokenizer(tokenizer_path):
|
96 |
+
tokenizer = GPT2Tokenizer.from_pretrained(tokenizer_path)
|
97 |
+
return tokenizer
|
98 |
+
|
99 |
+
def generate_text(model_path, sequence, max_length):
|
100 |
+
model = load_model(model_path)
|
101 |
+
tokenizer = load_tokenizer(model_path)
|
102 |
+
ids = tokenizer.encode(sequence, return_tensors='pt')
|
103 |
+
outputs = model.generate(
|
104 |
+
ids,
|
105 |
+
do_sample=True,
|
106 |
+
max_length=max_length,
|
107 |
+
pad_token_id=model.config.eos_token_id,
|
108 |
+
top_k=1,
|
109 |
+
top_p=0.99,
|
110 |
+
)
|
111 |
+
converted = tokenizer.convert_ids_to_tokens(outputs[0])
|
112 |
+
valid_tokens = [token if token is not None else '.' for token in converted]
|
113 |
+
generated_text = tokenizer.convert_tokens_to_string(valid_tokens)
|
114 |
+
print(generated_text)
|
115 |
+
|
116 |
+
model2_path = "Cenker/turkishgpt"
|
117 |
+
sequence2 = "[Q] cengiz han kimdir"
|
118 |
+
max_len = 120
|
119 |
+
|
120 |
+
generate_text(model2_path, sequence2, max_len)
|
121 |
|
122 |
## Training Details
|
123 |
|