File size: 2,779 Bytes
4528cc7 3b6f2da e4b109a a6ad7c9 e4b109a a6ad7c9 e4b109a 3b6f2da e4b109a 3b6f2da e4b109a a6ad7c9 4abf7ef a6ad7c9 c6cfc9d e4b109a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
library_name: transformers
language:
- en
license: apache-2.0
base_model: pszemraj/tFINE-850m-24x24-v0.4-flan_aug
metrics:
- rouge
datasets:
- pszemraj/infinity-instruct-7m-T2T_en
pipeline_tag: text2text-generation
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# tFINE-850m-24x24-v0.5-instruct-L1
This model is a fine-tuned version of [pszemraj/tFINE-850m-24x24-v0.4-flan_aug](https://huggingface.co/pszemraj/tFINE-850m-24x24-v0.4-flan_aug) on the pszemraj/infinity-instruct-7m-T2T_en dataset.
It achieves the following results on the evaluation set:
- Loss: 1.1478
- Rouge1: 38.4805
- Rouge2: 22.5971
- Rougel: 31.1093
- Rougelsum: 36.596
- Gen Len: 441.475
- Num Input Tokens Seen: 435513684
## usage
```py
from transformers import pipeline
pipe = pipeline(
"text2text-generation",
model="pszemraj/tFINE-850m-24x24-v0.5-instruct-L1",
device_map="auto",
)
prompt = "write a python script to download a file from a url and save as a local file using requests. explain how it works"
res = pipe(
prompt,
max_new_tokens=192,
top_k=4,
penalty_alpha=0.6,
renormalize_logits=True,
no_repeat_ngram_size=5,
)
print(res[0]["generated_text"])
```
## Quick eval
Quick eval for: `pszemraj/tFINE-850m-24x24-v0.5-instruct-L1`
hf (pretrained=pszemraj/tFINE-850m-24x24-v0.5-instruct-L1,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|-------------|------:|----------------|-----:|-----------|---|-----:|---|------|
|boolq | 2|none | 0|acc |↑ |0.5661|± |0.0087|
|openbookqa | 1|none | 0|acc |↑ |0.1540|± |0.0162|
| | |none | 0|acc_norm |↑ |0.2960|± |0.0204|
|piqa | 1|none | 0|acc |↑ |0.6094|± |0.0114|
| | |none | 0|acc_norm |↑ |0.5952|± |0.0115|
|social_iqa | 0|none | 0|acc |↑ |0.3900|± |0.0110|
|tinyArc | 0|none | 25|acc_norm |↑ |0.2903|± | N/A|
|tinyGSM8k | 0|flexible-extract| 5|exact_match|↑ |0.0471|± | N/A|
| | |strict-match | 5|exact_match|↑ |0.0339|± | N/A|
|tinyHellaswag| 0|none | 10|acc_norm |↑ |0.2490|± | N/A|
|tinyMMLU | 0|none | 0|acc_norm |↑ |0.3021|± | N/A|
|winogrande | 1|none | 0|acc |↑ |0.4925|± |0.0141|
|