--- language: en tags: - summarization model-index: - name: google/pegasus-xsum results: - task: type: summarization name: Summarization dataset: name: samsum type: samsum config: samsum split: train metrics: - type: rouge value: 21.8096 name: ROUGE-1 verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTM5ZTVlMTk2M2JiM2Y3ZTZkOWQyODQwZTQ1Nzc3ZDJiZDc5NzhhNTViNGRlYTI2MjkyMDBjMDMzNGU3MjFhYiIsInZlcnNpb24iOjF9.cETaihI9hH44D3QZh_yXTjEhOgumUlKr5wZqIZ_IlnE_jg0_q_KKWM986acH1K8nfHfJbDH_uZbQqqJUEuGXDQ - type: rouge value: 4.2525 name: ROUGE-2 verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDU2YTMwYmRmMGJlMzhlZDczNzczZWM4ZjBjMzE5ZmM0ZmIwYzA2MWQ2MTBhMTUwODM4YjdhYjFmZmY1OGNjOSIsInZlcnNpb24iOjF9.PAtZSns7ubNapkT-P3PzfvWUGET1dHN8AnsjYbYoaSZJ8AUl-5d3rCutt0bEAFffHFCpq23YT3Go2KOaE16PDg - type: rouge value: 17.4469 name: ROUGE-L verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2VmMTRlNmIzYzlhNzY0Y2ZlYzAyOGJhNDQwNjk1YTdhZTZjNDU2ZmJhZDc5MTlhYjgyYjg4NWE0ZTA1NWExMiIsInZlcnNpb24iOjF9.CznDprjeWu6XWUwgFjGmsV2Z1W7xXuy9jDRpH5NMFlZCHAkqzoZSlMbUTwPnJF51LVJrS_WXJaXfAxKEX_LCCQ - type: rouge value: 18.8907 name: ROUGE-LSUM verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2I2ZDZjMzUzNzFjNzdmMGVmODJmNzY0ZGIxY2Q3MWEwNThmODNjN2FhZTE4N2M4NjlhZjU1ZTBiODI0NThiYiIsInZlcnNpb24iOjF9.JgWVNm7FZ0r6Kn-cGAfzYszUEuYooLxbxSxSDQUvqiZVuI7tjoDp-dP8-DkgTH0yH2sqNSGBNsBnXNo63YP-AQ - type: loss value: 3.0317161083221436 name: loss verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzg3MjRmYzRjYmIzYjk3NTZhZGM2Mzc3NjNkNTIzY2IwYmEwYWZlMGE4NTUwNjQ5YWFkNTdhMjRjNjIzZmFhYSIsInZlcnNpb24iOjF9.nns3EN07Q66USOa_TIx465wYQajhg1DcTQjU47EeKayi1uxbroU6fi90I-9k52xCGIr8RmeT2W7eNkkfFW2sCw - type: gen_len value: 20.3122 name: gen_len verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTgxZWNhZjM4ZjE1YTY2MDBhN2ZmYWNiODU1YjNjMjQxMjVhMWYwYzNkMWU4MDM0NTYzN2IxMjIyYjNhZGIxZiIsInZlcnNpb24iOjF9.yMAFwoI4EebWho-7Jy9vAFFUHNytiAAPxr_2a99jL_d72sKjZdVQdU0Ag3fg_dRnVhqDeTVEIqwIkAJfIr8PDA - task: type: summarization name: Summarization dataset: name: xsum type: xsum config: default split: test metrics: - type: rouge value: 46.8623 name: ROUGE-1 verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2NjMjg2Y2Q4NjNlNmZmNmZkODlkOWU1OTg1YTcwODVmNDBiZmY4MmIwYmIyNjFiMzRkMDhkNzM2ZjFiMDYwYyIsInZlcnNpb24iOjF9.aunhAPHEDSc_V855wQiRALNEsOyD_ZTbFvi_Mu-CeZYMXO05G3ERYD6mUvdtaIP7X_Yo0vPq0y2uTQ9nVVmEDA - type: rouge value: 24.4533 name: ROUGE-2 verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjNiZTI0ODUyNGJlOTEwMGU5YjcyYzA2MzA3NThlY2ZjZTUzYTc1NThlNWZmYTNkYmQxYmI4NGUzODNlMTIzNCIsInZlcnNpb24iOjF9.8NPuJo_8Gx8EW66O835sxyYz0pteEYJ2jwwF3R24gadj0cdi7IM1LGUIPyS0IRGmYYWtqu6ybKx8xClL0TO-Ag - type: rouge value: 39.0548 name: ROUGE-L verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2EyZDk4MWUyNDQ1MDA3N2QxZTMwOWM4YTJiMzM2NzZhYTE5YmE0NTYyMjg0M2NhMmQzZGYxNjQ4YjIxNDYyNyIsInZlcnNpb24iOjF9.4zORYWUO7fMkwR_HkDnd-g17Ar_MOJF_8T2rLHujBIW1HnCrNIzXHSIp1vgvn4zh6fO0tnjZDQvCD9yEfzrtDw - type: rouge value: 39.0994 name: ROUGE-LSUM verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZDAxY2MxZTYzNWRhNWM3YTc2NjE5NzRmMzEwYTgyNDI0NmRlYjIyOTJiNzJjZWYzYThjZjczNTQzOTQzODZmNSIsInZlcnNpb24iOjF9.q3avkS3k-ljplHJbGl4nKhhH0coxe5ybHlwAJy_I1mtLoAW12k_iafy8kpXwQCmiKjqE6bgmFNPs0kkOo6JDAQ - type: loss value: 1.5717021226882935 name: loss verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmJlMzZiNjExNGU4ZjliOTgwZGZiZDFjNGI2N2NjYWZhNmEyMTI3YmZiYWRmNGE1MmZiNjcwZDZmOWFmNmMwZiIsInZlcnNpb24iOjF9.T5ecj6AYQ8eK-IDNqqtvkOWFvW2Q-f5kswto85P1fX_UMy9rgr2IwBRud-FAcQMnKktHK1ld94mjnDO9LcUcCQ - type: gen_len value: 22.8821 name: gen_len verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzNiN2ExMDJhYjNmMDU2NmIxMTgwZjM5NGNmMzIxZGVmNGVkNzJmNGU4OWEwNjI1NmRmOTYyYjM5NTA1YWE0YSIsInZlcnNpb24iOjF9.pI2ulnlSyqN-EUcl9v5xbKiK-z6FDKeMpIjwqhw0WvjMf623WnqzSJZo-foFbf6DM7yFhMScgVbOOC3JEsKNCQ - task: type: summarization name: Summarization dataset: name: cnn_dailymail type: cnn_dailymail config: 3.0.0 split: test metrics: - type: rouge value: 22.2062 name: ROUGE-1 verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjFiNGZkNDExNjUwMjhkNzY3M2NmMDgxNGVhMzZmN2E4YjIzN2Y0ZjMxNDUyOGU2NGYwNDNlOTkzMWI1OTYwNSIsInZlcnNpb24iOjF9.F_6E8QID6kjOGEFCFaUKiBmgUTtSDTQKf4F-L2NDLSDX7drF6HvTMtm9oXUeP_P9cSUNX5OvDUSJPSlfl-C0Cw - type: rouge value: 7.6701 name: ROUGE-2 verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzRhMzVmYjcyNTRlZmVhMzM4NjY3ODczZjc2M2MwM2Y5MTRhNjYxZjY4YTQ3ZmU2YzA2MGMxMzE2YjZiNGMxZSIsInZlcnNpb24iOjF9.U7IY76kNrcYgGh-Uy3H9DtaSRugShwK6Lp-I9Ov6g_r3X16Pi2cI8ZS68BaQiRvowXOKIi9R5iJLLUK3C4J2Cg - type: rouge value: 15.4046 name: ROUGE-L verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzZjNDUxYWJjNWU0ZGY4M2I2NTBmYTBmODhiMDJiNjIzNzViNGY4YjVlMTM5Y2FlNjUyMWFkYTljYjRhM2NlZCIsInZlcnNpb24iOjF9.tyvSIMTYmFFsD6zzo_CD2tYIZiovH5Ll26qGO78kUZEvYRWfxPMhUwtrylaOsaNdTdwDz-M-TF6IQxpSXl8eBw - type: rouge value: 19.2182 name: ROUGE-LSUM verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2Q3N2ZiYjVlZTY1OTYzZWRiY2RmZDkwMjM0YzIxYTUyYmJlMWQ2NDBhNTVjYjM0NGNmNWFjOWJmYzc4NDkwYiIsInZlcnNpb24iOjF9.LV3UbhyIL4EP1n5KBqAc29GMh7IIxo5tFbzPBV9Mc8Ew08d6zMs1U5aZwc0an7ClBP20B-haYHwk8JNDBZHcCg - type: loss value: 2.681241273880005 name: loss verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2FjNzRjNGY0ZDkzZDM1NDIzNTQwYmQyNzE3NDQzNTBlMjIxZTFhYmZlZTRmYWI2YjRiYTA5NDYyZTM4NjM3MSIsInZlcnNpb24iOjF9.h1WwEclg4cWWYlCH8O6ifsRvOVFZ2uZj6nk7_m89rHllmOhUtVxCeSTPdNEynyv9B_SKUWPLaS_5Uj55FpF2AA - type: gen_len value: 25.0234 name: gen_len verified: true verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjAxM2Y1ZDgzMTc3ZmVlOWI5YzYyMmExMzc3Y2JkYzJjZDRkODY5MWE4ZGQzZjFmNTFjNDZlMTcyZmQ3MWM5MyIsInZlcnNpb24iOjF9.6XKMtrC2IqCcphlUI8aJp673IRfsnjJhOJrvftxyA_k3YxSDI_3Iyo_mURpXAVo4llb52lB638bRHZv1tvdvDg --- ### Pegasus Models See Docs: [here](https://huggingface.co/transformers/master/model_doc/pegasus.html) Original TF 1 code [here](https://github.com/google-research/pegasus) Authors: Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019 Maintained by: [@sshleifer](https://twitter.com/sam_shleifer) Task: Summarization The following is copied from the authors' README. # Mixed & Stochastic Checkpoints We train a pegasus model with sampled gap sentence ratios on both C4 and HugeNews, and stochastically sample important sentences. The updated the results are reported in this table. | dataset | C4 | HugeNews | Mixed & Stochastic| | ---- | ---- | ---- | ----| | xsum | 45.20/22.06/36.99 | 47.21/24.56/39.25 | 47.60/24.83/39.64| | cnn_dailymail | 43.90/21.20/40.76 | 44.17/21.47/41.11 | 44.16/21.56/41.30| | newsroom | 45.07/33.39/41.28 | 45.15/33.51/41.33 | 45.98/34.20/42.18| | multi_news | 46.74/17.95/24.26 | 47.52/18.72/24.91 | 47.65/18.75/24.95| | gigaword | 38.75/19.96/36.14 | 39.12/19.86/36.24 | 39.65/20.47/36.76| | wikihow | 43.07/19.70/34.79 | 41.35/18.51/33.42 | 46.39/22.12/38.41 *| | reddit_tifu | 26.54/8.94/21.64 | 26.63/9.01/21.60 | 27.99/9.81/22.94| | big_patent | 53.63/33.16/42.25 | 53.41/32.89/42.07 | 52.29/33.08/41.66 *| | arxiv | 44.70/17.27/25.80 | 44.67/17.18/25.73 | 44.21/16.95/25.67| | pubmed | 45.49/19.90/27.69 | 45.09/19.56/27.42 | 45.97/20.15/28.25| | aeslc | 37.69/21.85/36.84 | 37.40/21.22/36.45 | 37.68/21.25/36.51| | billsum | 57.20/39.56/45.80 | 57.31/40.19/45.82 | 59.67/41.58/47.59| The "Mixed & Stochastic" model has the following changes: - trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples). - trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity). - the model uniformly sample a gap sentence ratio between 15% and 45%. - importance sentences are sampled using a 20% uniform noise to importance scores. - the sentencepiece tokenizer is updated to be able to encode newline character. (*) the numbers of wikihow and big_patent datasets are not comparable because of change in tokenization and data: - wikihow dataset contains newline characters which is useful for paragraph segmentation, the C4 and HugeNews model's sentencepiece tokenizer doesn't encode newline and loose this information. - we update the BigPatent dataset to preserve casing, some format cleanings are also changed, please refer to change in TFDS. The "Mixed & Stochastic" model has the following changes (from pegasus-large in the paper): trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples). trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity). the model uniformly sample a gap sentence ratio between 15% and 45%. importance sentences are sampled using a 20% uniform noise to importance scores. the sentencepiece tokenizer is updated to be able to encode newline character. Citation ``` @misc{zhang2019pegasus, title={PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization}, author={Jingqing Zhang and Yao Zhao and Mohammad Saleh and Peter J. Liu}, year={2019}, eprint={1912.08777}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```