autoevaluator
HF staff
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator
e455aeb
| language: en | |
| tags: | |
| - summarization | |
| model-index: | |
| - name: google/pegasus-xsum | |
| results: | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: samsum | |
| type: samsum | |
| config: samsum | |
| split: train | |
| metrics: | |
| - type: rouge | |
| value: 21.8096 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTM5ZTVlMTk2M2JiM2Y3ZTZkOWQyODQwZTQ1Nzc3ZDJiZDc5NzhhNTViNGRlYTI2MjkyMDBjMDMzNGU3MjFhYiIsInZlcnNpb24iOjF9.cETaihI9hH44D3QZh_yXTjEhOgumUlKr5wZqIZ_IlnE_jg0_q_KKWM986acH1K8nfHfJbDH_uZbQqqJUEuGXDQ | |
| - type: rouge | |
| value: 4.2525 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDU2YTMwYmRmMGJlMzhlZDczNzczZWM4ZjBjMzE5ZmM0ZmIwYzA2MWQ2MTBhMTUwODM4YjdhYjFmZmY1OGNjOSIsInZlcnNpb24iOjF9.PAtZSns7ubNapkT-P3PzfvWUGET1dHN8AnsjYbYoaSZJ8AUl-5d3rCutt0bEAFffHFCpq23YT3Go2KOaE16PDg | |
| - type: rouge | |
| value: 17.4469 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2VmMTRlNmIzYzlhNzY0Y2ZlYzAyOGJhNDQwNjk1YTdhZTZjNDU2ZmJhZDc5MTlhYjgyYjg4NWE0ZTA1NWExMiIsInZlcnNpb24iOjF9.CznDprjeWu6XWUwgFjGmsV2Z1W7xXuy9jDRpH5NMFlZCHAkqzoZSlMbUTwPnJF51LVJrS_WXJaXfAxKEX_LCCQ | |
| - type: rouge | |
| value: 18.8907 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2I2ZDZjMzUzNzFjNzdmMGVmODJmNzY0ZGIxY2Q3MWEwNThmODNjN2FhZTE4N2M4NjlhZjU1ZTBiODI0NThiYiIsInZlcnNpb24iOjF9.JgWVNm7FZ0r6Kn-cGAfzYszUEuYooLxbxSxSDQUvqiZVuI7tjoDp-dP8-DkgTH0yH2sqNSGBNsBnXNo63YP-AQ | |
| - type: loss | |
| value: 3.0317161083221436 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzg3MjRmYzRjYmIzYjk3NTZhZGM2Mzc3NjNkNTIzY2IwYmEwYWZlMGE4NTUwNjQ5YWFkNTdhMjRjNjIzZmFhYSIsInZlcnNpb24iOjF9.nns3EN07Q66USOa_TIx465wYQajhg1DcTQjU47EeKayi1uxbroU6fi90I-9k52xCGIr8RmeT2W7eNkkfFW2sCw | |
| - type: gen_len | |
| value: 20.3122 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTgxZWNhZjM4ZjE1YTY2MDBhN2ZmYWNiODU1YjNjMjQxMjVhMWYwYzNkMWU4MDM0NTYzN2IxMjIyYjNhZGIxZiIsInZlcnNpb24iOjF9.yMAFwoI4EebWho-7Jy9vAFFUHNytiAAPxr_2a99jL_d72sKjZdVQdU0Ag3fg_dRnVhqDeTVEIqwIkAJfIr8PDA | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: xsum | |
| type: xsum | |
| config: default | |
| split: test | |
| metrics: | |
| - type: rouge | |
| value: 46.8623 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2NjMjg2Y2Q4NjNlNmZmNmZkODlkOWU1OTg1YTcwODVmNDBiZmY4MmIwYmIyNjFiMzRkMDhkNzM2ZjFiMDYwYyIsInZlcnNpb24iOjF9.aunhAPHEDSc_V855wQiRALNEsOyD_ZTbFvi_Mu-CeZYMXO05G3ERYD6mUvdtaIP7X_Yo0vPq0y2uTQ9nVVmEDA | |
| - type: rouge | |
| value: 24.4533 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjNiZTI0ODUyNGJlOTEwMGU5YjcyYzA2MzA3NThlY2ZjZTUzYTc1NThlNWZmYTNkYmQxYmI4NGUzODNlMTIzNCIsInZlcnNpb24iOjF9.8NPuJo_8Gx8EW66O835sxyYz0pteEYJ2jwwF3R24gadj0cdi7IM1LGUIPyS0IRGmYYWtqu6ybKx8xClL0TO-Ag | |
| - type: rouge | |
| value: 39.0548 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2EyZDk4MWUyNDQ1MDA3N2QxZTMwOWM4YTJiMzM2NzZhYTE5YmE0NTYyMjg0M2NhMmQzZGYxNjQ4YjIxNDYyNyIsInZlcnNpb24iOjF9.4zORYWUO7fMkwR_HkDnd-g17Ar_MOJF_8T2rLHujBIW1HnCrNIzXHSIp1vgvn4zh6fO0tnjZDQvCD9yEfzrtDw | |
| - type: rouge | |
| value: 39.0994 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZDAxY2MxZTYzNWRhNWM3YTc2NjE5NzRmMzEwYTgyNDI0NmRlYjIyOTJiNzJjZWYzYThjZjczNTQzOTQzODZmNSIsInZlcnNpb24iOjF9.q3avkS3k-ljplHJbGl4nKhhH0coxe5ybHlwAJy_I1mtLoAW12k_iafy8kpXwQCmiKjqE6bgmFNPs0kkOo6JDAQ | |
| - type: loss | |
| value: 1.5717021226882935 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmJlMzZiNjExNGU4ZjliOTgwZGZiZDFjNGI2N2NjYWZhNmEyMTI3YmZiYWRmNGE1MmZiNjcwZDZmOWFmNmMwZiIsInZlcnNpb24iOjF9.T5ecj6AYQ8eK-IDNqqtvkOWFvW2Q-f5kswto85P1fX_UMy9rgr2IwBRud-FAcQMnKktHK1ld94mjnDO9LcUcCQ | |
| - type: gen_len | |
| value: 22.8821 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzNiN2ExMDJhYjNmMDU2NmIxMTgwZjM5NGNmMzIxZGVmNGVkNzJmNGU4OWEwNjI1NmRmOTYyYjM5NTA1YWE0YSIsInZlcnNpb24iOjF9.pI2ulnlSyqN-EUcl9v5xbKiK-z6FDKeMpIjwqhw0WvjMf623WnqzSJZo-foFbf6DM7yFhMScgVbOOC3JEsKNCQ | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: cnn_dailymail | |
| type: cnn_dailymail | |
| config: 3.0.0 | |
| split: test | |
| metrics: | |
| - type: rouge | |
| value: 22.2062 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjFiNGZkNDExNjUwMjhkNzY3M2NmMDgxNGVhMzZmN2E4YjIzN2Y0ZjMxNDUyOGU2NGYwNDNlOTkzMWI1OTYwNSIsInZlcnNpb24iOjF9.F_6E8QID6kjOGEFCFaUKiBmgUTtSDTQKf4F-L2NDLSDX7drF6HvTMtm9oXUeP_P9cSUNX5OvDUSJPSlfl-C0Cw | |
| - type: rouge | |
| value: 7.6701 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzRhMzVmYjcyNTRlZmVhMzM4NjY3ODczZjc2M2MwM2Y5MTRhNjYxZjY4YTQ3ZmU2YzA2MGMxMzE2YjZiNGMxZSIsInZlcnNpb24iOjF9.U7IY76kNrcYgGh-Uy3H9DtaSRugShwK6Lp-I9Ov6g_r3X16Pi2cI8ZS68BaQiRvowXOKIi9R5iJLLUK3C4J2Cg | |
| - type: rouge | |
| value: 15.4046 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzZjNDUxYWJjNWU0ZGY4M2I2NTBmYTBmODhiMDJiNjIzNzViNGY4YjVlMTM5Y2FlNjUyMWFkYTljYjRhM2NlZCIsInZlcnNpb24iOjF9.tyvSIMTYmFFsD6zzo_CD2tYIZiovH5Ll26qGO78kUZEvYRWfxPMhUwtrylaOsaNdTdwDz-M-TF6IQxpSXl8eBw | |
| - type: rouge | |
| value: 19.2182 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2Q3N2ZiYjVlZTY1OTYzZWRiY2RmZDkwMjM0YzIxYTUyYmJlMWQ2NDBhNTVjYjM0NGNmNWFjOWJmYzc4NDkwYiIsInZlcnNpb24iOjF9.LV3UbhyIL4EP1n5KBqAc29GMh7IIxo5tFbzPBV9Mc8Ew08d6zMs1U5aZwc0an7ClBP20B-haYHwk8JNDBZHcCg | |
| - type: loss | |
| value: 2.681241273880005 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2FjNzRjNGY0ZDkzZDM1NDIzNTQwYmQyNzE3NDQzNTBlMjIxZTFhYmZlZTRmYWI2YjRiYTA5NDYyZTM4NjM3MSIsInZlcnNpb24iOjF9.h1WwEclg4cWWYlCH8O6ifsRvOVFZ2uZj6nk7_m89rHllmOhUtVxCeSTPdNEynyv9B_SKUWPLaS_5Uj55FpF2AA | |
| - type: gen_len | |
| value: 25.0234 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjAxM2Y1ZDgzMTc3ZmVlOWI5YzYyMmExMzc3Y2JkYzJjZDRkODY5MWE4ZGQzZjFmNTFjNDZlMTcyZmQ3MWM5MyIsInZlcnNpb24iOjF9.6XKMtrC2IqCcphlUI8aJp673IRfsnjJhOJrvftxyA_k3YxSDI_3Iyo_mURpXAVo4llb52lB638bRHZv1tvdvDg | |
| ### Pegasus Models | |
| See Docs: [here](https://huggingface.co/transformers/master/model_doc/pegasus.html) | |
| Original TF 1 code [here](https://github.com/google-research/pegasus) | |
| Authors: Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019 | |
| Maintained by: [@sshleifer](https://twitter.com/sam_shleifer) | |
| Task: Summarization | |
| The following is copied from the authors' README. | |
| # Mixed & Stochastic Checkpoints | |
| We train a pegasus model with sampled gap sentence ratios on both C4 and HugeNews, and stochastically sample important sentences. The updated the results are reported in this table. | |
| | dataset | C4 | HugeNews | Mixed & Stochastic| | |
| | ---- | ---- | ---- | ----| | |
| | xsum | 45.20/22.06/36.99 | 47.21/24.56/39.25 | 47.60/24.83/39.64| | |
| | cnn_dailymail | 43.90/21.20/40.76 | 44.17/21.47/41.11 | 44.16/21.56/41.30| | |
| | newsroom | 45.07/33.39/41.28 | 45.15/33.51/41.33 | 45.98/34.20/42.18| | |
| | multi_news | 46.74/17.95/24.26 | 47.52/18.72/24.91 | 47.65/18.75/24.95| | |
| | gigaword | 38.75/19.96/36.14 | 39.12/19.86/36.24 | 39.65/20.47/36.76| | |
| | wikihow | 43.07/19.70/34.79 | 41.35/18.51/33.42 | 46.39/22.12/38.41 *| | |
| | reddit_tifu | 26.54/8.94/21.64 | 26.63/9.01/21.60 | 27.99/9.81/22.94| | |
| | big_patent | 53.63/33.16/42.25 | 53.41/32.89/42.07 | 52.29/33.08/41.66 *| | |
| | arxiv | 44.70/17.27/25.80 | 44.67/17.18/25.73 | 44.21/16.95/25.67| | |
| | pubmed | 45.49/19.90/27.69 | 45.09/19.56/27.42 | 45.97/20.15/28.25| | |
| | aeslc | 37.69/21.85/36.84 | 37.40/21.22/36.45 | 37.68/21.25/36.51| | |
| | billsum | 57.20/39.56/45.80 | 57.31/40.19/45.82 | 59.67/41.58/47.59| | |
| The "Mixed & Stochastic" model has the following changes: | |
| - trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples). | |
| - trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity). | |
| - the model uniformly sample a gap sentence ratio between 15% and 45%. | |
| - importance sentences are sampled using a 20% uniform noise to importance scores. | |
| - the sentencepiece tokenizer is updated to be able to encode newline character. | |
| (*) the numbers of wikihow and big_patent datasets are not comparable because of change in tokenization and data: | |
| - wikihow dataset contains newline characters which is useful for paragraph segmentation, the C4 and HugeNews model's sentencepiece tokenizer doesn't encode newline and loose this information. | |
| - we update the BigPatent dataset to preserve casing, some format cleanings are also changed, please refer to change in TFDS. | |
| The "Mixed & Stochastic" model has the following changes (from pegasus-large in the paper): | |
| trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples). | |
| trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity). | |
| the model uniformly sample a gap sentence ratio between 15% and 45%. | |
| importance sentences are sampled using a 20% uniform noise to importance scores. | |
| the sentencepiece tokenizer is updated to be able to encode newline character. | |
| Citation | |
| ``` | |
| @misc{zhang2019pegasus, | |
| title={PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization}, | |
| author={Jingqing Zhang and Yao Zhao and Mohammad Saleh and Peter J. Liu}, | |
| year={2019}, | |
| eprint={1912.08777}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CL} | |
| } | |
| ``` |