An mT5-small model on the pnSummary dataset to summarize articles.
Eval results
The following table summarizes the ROUGE scores obtained by the model for the validation set.
+-----------+------+-----------+--------+-----------+
| Score | Type | Precision | Recall | F-Measure |
+-----------+------+-----------+--------+-----------+
| rouge1 | low | 45.13 | 42.92 | 42.52 |
| rouge1 | mid | 45.67 | 43.47 | 43.04 |
| rouge1 | high | 46.26 | 44.04 | 43.57 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rouge2 | low | 26.08 | 24.73 | 24.56 |
| rouge2 | mid | 26.65 | 25.29 | 25.10 |
| rouge2 | high | 27.25 | 25.84 | 25.63 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rougeL | low | 38.72 | 36.86 | 36.55 |
| rougeL | mid | 39.28 | 37.38 | 37.05 |
| rougeL | high | 39.86 | 37.96 | 37.60 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rougeLsum | low | 38.67 | 36.80 | 36.50 |
| rougeLsum | mid | 39.25 | 37.36 | 37.02 |
| rougeLsum | high | 39.88 | 37.94 | 37.59 |
+-----------+------+-----------+--------+-----------+
Test results
The following table summarizes the ROUGE scores obtained by the model for the test set.
+-----------+------+-----------+--------+-----------+
| Score | Type | Precision | Recall | F-Measure |
+-----------+------+-----------+--------+-----------+
| rouge1 | low | 45.09 | 42.67 | 42.43 |
| rouge1 | mid | 45.63 | 43.19 | 42.87 |
| rouge1 | high | 46.20 | 43.78 | 43.40 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rouge2 | low | 26.10 | 24.57 | 24.49 |
| rouge2 | mid | 26.68 | 25.13 | 25.04 |
| rouge2 | high | 27.30 | 25.73 | 25.63 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rougeL | low | 38.60 | 36.49 | 36.29 |
| rougeL | mid | 39.15 | 37.06 | 36.82 |
| rougeL | high | 39.75 | 37.63 | 37.37 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rougeLsum | low | 38.57 | 36.49 | 36.27 |
| rougeLsum | mid | 39.15 | 37.07 | 36.82 |
| rougeLsum | high | 39.76 | 37.56 | 37.36 |
+-----------+------+-----------+--------+-----------+
- Downloads last month
- 3
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.