Hugging Face's logo

google
/
pegasus-xsum

Summarization
Transformers
PyTorch
TensorFlow
JAX
English
pegasus
text2text-generation
Eval Results
Inference Endpoints
Model card Files Community
16
pegasus-xsum / README.md
autoevaluator's picture
autoevaluator HF staff
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator
e455aeb
preview code
|
raw
history blame
10 kB
metadata
language: en
tags:
  - summarization
model-index:
  - name: google/pegasus-xsum
    results:
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: samsum
          type: samsum
          config: samsum
          split: train
        metrics:
          - type: rouge
            value: 21.8096
            name: ROUGE-1
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTM5ZTVlMTk2M2JiM2Y3ZTZkOWQyODQwZTQ1Nzc3ZDJiZDc5NzhhNTViNGRlYTI2MjkyMDBjMDMzNGU3MjFhYiIsInZlcnNpb24iOjF9.cETaihI9hH44D3QZh_yXTjEhOgumUlKr5wZqIZ_IlnE_jg0_q_KKWM986acH1K8nfHfJbDH_uZbQqqJUEuGXDQ
          - type: rouge
            value: 4.2525
            name: ROUGE-2
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDU2YTMwYmRmMGJlMzhlZDczNzczZWM4ZjBjMzE5ZmM0ZmIwYzA2MWQ2MTBhMTUwODM4YjdhYjFmZmY1OGNjOSIsInZlcnNpb24iOjF9.PAtZSns7ubNapkT-P3PzfvWUGET1dHN8AnsjYbYoaSZJ8AUl-5d3rCutt0bEAFffHFCpq23YT3Go2KOaE16PDg
          - type: rouge
            value: 17.4469
            name: ROUGE-L
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2VmMTRlNmIzYzlhNzY0Y2ZlYzAyOGJhNDQwNjk1YTdhZTZjNDU2ZmJhZDc5MTlhYjgyYjg4NWE0ZTA1NWExMiIsInZlcnNpb24iOjF9.CznDprjeWu6XWUwgFjGmsV2Z1W7xXuy9jDRpH5NMFlZCHAkqzoZSlMbUTwPnJF51LVJrS_WXJaXfAxKEX_LCCQ
          - type: rouge
            value: 18.8907
            name: ROUGE-LSUM
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2I2ZDZjMzUzNzFjNzdmMGVmODJmNzY0ZGIxY2Q3MWEwNThmODNjN2FhZTE4N2M4NjlhZjU1ZTBiODI0NThiYiIsInZlcnNpb24iOjF9.JgWVNm7FZ0r6Kn-cGAfzYszUEuYooLxbxSxSDQUvqiZVuI7tjoDp-dP8-DkgTH0yH2sqNSGBNsBnXNo63YP-AQ
          - type: loss
            value: 3.0317161083221436
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzg3MjRmYzRjYmIzYjk3NTZhZGM2Mzc3NjNkNTIzY2IwYmEwYWZlMGE4NTUwNjQ5YWFkNTdhMjRjNjIzZmFhYSIsInZlcnNpb24iOjF9.nns3EN07Q66USOa_TIx465wYQajhg1DcTQjU47EeKayi1uxbroU6fi90I-9k52xCGIr8RmeT2W7eNkkfFW2sCw
          - type: gen_len
            value: 20.3122
            name: gen_len
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTgxZWNhZjM4ZjE1YTY2MDBhN2ZmYWNiODU1YjNjMjQxMjVhMWYwYzNkMWU4MDM0NTYzN2IxMjIyYjNhZGIxZiIsInZlcnNpb24iOjF9.yMAFwoI4EebWho-7Jy9vAFFUHNytiAAPxr_2a99jL_d72sKjZdVQdU0Ag3fg_dRnVhqDeTVEIqwIkAJfIr8PDA
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: xsum
          type: xsum
          config: default
          split: test
        metrics:
          - type: rouge
            value: 46.8623
            name: ROUGE-1
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2NjMjg2Y2Q4NjNlNmZmNmZkODlkOWU1OTg1YTcwODVmNDBiZmY4MmIwYmIyNjFiMzRkMDhkNzM2ZjFiMDYwYyIsInZlcnNpb24iOjF9.aunhAPHEDSc_V855wQiRALNEsOyD_ZTbFvi_Mu-CeZYMXO05G3ERYD6mUvdtaIP7X_Yo0vPq0y2uTQ9nVVmEDA
          - type: rouge
            value: 24.4533
            name: ROUGE-2
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjNiZTI0ODUyNGJlOTEwMGU5YjcyYzA2MzA3NThlY2ZjZTUzYTc1NThlNWZmYTNkYmQxYmI4NGUzODNlMTIzNCIsInZlcnNpb24iOjF9.8NPuJo_8Gx8EW66O835sxyYz0pteEYJ2jwwF3R24gadj0cdi7IM1LGUIPyS0IRGmYYWtqu6ybKx8xClL0TO-Ag
          - type: rouge
            value: 39.0548
            name: ROUGE-L
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2EyZDk4MWUyNDQ1MDA3N2QxZTMwOWM4YTJiMzM2NzZhYTE5YmE0NTYyMjg0M2NhMmQzZGYxNjQ4YjIxNDYyNyIsInZlcnNpb24iOjF9.4zORYWUO7fMkwR_HkDnd-g17Ar_MOJF_8T2rLHujBIW1HnCrNIzXHSIp1vgvn4zh6fO0tnjZDQvCD9yEfzrtDw
          - type: rouge
            value: 39.0994
            name: ROUGE-LSUM
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZDAxY2MxZTYzNWRhNWM3YTc2NjE5NzRmMzEwYTgyNDI0NmRlYjIyOTJiNzJjZWYzYThjZjczNTQzOTQzODZmNSIsInZlcnNpb24iOjF9.q3avkS3k-ljplHJbGl4nKhhH0coxe5ybHlwAJy_I1mtLoAW12k_iafy8kpXwQCmiKjqE6bgmFNPs0kkOo6JDAQ
          - type: loss
            value: 1.5717021226882935
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmJlMzZiNjExNGU4ZjliOTgwZGZiZDFjNGI2N2NjYWZhNmEyMTI3YmZiYWRmNGE1MmZiNjcwZDZmOWFmNmMwZiIsInZlcnNpb24iOjF9.T5ecj6AYQ8eK-IDNqqtvkOWFvW2Q-f5kswto85P1fX_UMy9rgr2IwBRud-FAcQMnKktHK1ld94mjnDO9LcUcCQ
          - type: gen_len
            value: 22.8821
            name: gen_len
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzNiN2ExMDJhYjNmMDU2NmIxMTgwZjM5NGNmMzIxZGVmNGVkNzJmNGU4OWEwNjI1NmRmOTYyYjM5NTA1YWE0YSIsInZlcnNpb24iOjF9.pI2ulnlSyqN-EUcl9v5xbKiK-z6FDKeMpIjwqhw0WvjMf623WnqzSJZo-foFbf6DM7yFhMScgVbOOC3JEsKNCQ
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: cnn_dailymail
          type: cnn_dailymail
          config: 3.0.0
          split: test
        metrics:
          - type: rouge
            value: 22.2062
            name: ROUGE-1
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjFiNGZkNDExNjUwMjhkNzY3M2NmMDgxNGVhMzZmN2E4YjIzN2Y0ZjMxNDUyOGU2NGYwNDNlOTkzMWI1OTYwNSIsInZlcnNpb24iOjF9.F_6E8QID6kjOGEFCFaUKiBmgUTtSDTQKf4F-L2NDLSDX7drF6HvTMtm9oXUeP_P9cSUNX5OvDUSJPSlfl-C0Cw
          - type: rouge
            value: 7.6701
            name: ROUGE-2
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzRhMzVmYjcyNTRlZmVhMzM4NjY3ODczZjc2M2MwM2Y5MTRhNjYxZjY4YTQ3ZmU2YzA2MGMxMzE2YjZiNGMxZSIsInZlcnNpb24iOjF9.U7IY76kNrcYgGh-Uy3H9DtaSRugShwK6Lp-I9Ov6g_r3X16Pi2cI8ZS68BaQiRvowXOKIi9R5iJLLUK3C4J2Cg
          - type: rouge
            value: 15.4046
            name: ROUGE-L
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzZjNDUxYWJjNWU0ZGY4M2I2NTBmYTBmODhiMDJiNjIzNzViNGY4YjVlMTM5Y2FlNjUyMWFkYTljYjRhM2NlZCIsInZlcnNpb24iOjF9.tyvSIMTYmFFsD6zzo_CD2tYIZiovH5Ll26qGO78kUZEvYRWfxPMhUwtrylaOsaNdTdwDz-M-TF6IQxpSXl8eBw
          - type: rouge
            value: 19.2182
            name: ROUGE-LSUM
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2Q3N2ZiYjVlZTY1OTYzZWRiY2RmZDkwMjM0YzIxYTUyYmJlMWQ2NDBhNTVjYjM0NGNmNWFjOWJmYzc4NDkwYiIsInZlcnNpb24iOjF9.LV3UbhyIL4EP1n5KBqAc29GMh7IIxo5tFbzPBV9Mc8Ew08d6zMs1U5aZwc0an7ClBP20B-haYHwk8JNDBZHcCg
          - type: loss
            value: 2.681241273880005
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2FjNzRjNGY0ZDkzZDM1NDIzNTQwYmQyNzE3NDQzNTBlMjIxZTFhYmZlZTRmYWI2YjRiYTA5NDYyZTM4NjM3MSIsInZlcnNpb24iOjF9.h1WwEclg4cWWYlCH8O6ifsRvOVFZ2uZj6nk7_m89rHllmOhUtVxCeSTPdNEynyv9B_SKUWPLaS_5Uj55FpF2AA
          - type: gen_len
            value: 25.0234
            name: gen_len
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjAxM2Y1ZDgzMTc3ZmVlOWI5YzYyMmExMzc3Y2JkYzJjZDRkODY5MWE4ZGQzZjFmNTFjNDZlMTcyZmQ3MWM5MyIsInZlcnNpb24iOjF9.6XKMtrC2IqCcphlUI8aJp673IRfsnjJhOJrvftxyA_k3YxSDI_3Iyo_mURpXAVo4llb52lB638bRHZv1tvdvDg

Pegasus Models

See Docs: here

Original TF 1 code here

Authors: Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019

Maintained by: @sshleifer

Task: Summarization

The following is copied from the authors' README.

Mixed & Stochastic Checkpoints

We train a pegasus model with sampled gap sentence ratios on both C4 and HugeNews, and stochastically sample important sentences. The updated the results are reported in this table.

dataset C4 HugeNews Mixed & Stochastic
xsum 45.20/22.06/36.99 47.21/24.56/39.25 47.60/24.83/39.64
cnn_dailymail 43.90/21.20/40.76 44.17/21.47/41.11 44.16/21.56/41.30
newsroom 45.07/33.39/41.28 45.15/33.51/41.33 45.98/34.20/42.18
multi_news 46.74/17.95/24.26 47.52/18.72/24.91 47.65/18.75/24.95
gigaword 38.75/19.96/36.14 39.12/19.86/36.24 39.65/20.47/36.76
wikihow 43.07/19.70/34.79 41.35/18.51/33.42 46.39/22.12/38.41 *
reddit_tifu 26.54/8.94/21.64 26.63/9.01/21.60 27.99/9.81/22.94
big_patent 53.63/33.16/42.25 53.41/32.89/42.07 52.29/33.08/41.66 *
arxiv 44.70/17.27/25.80 44.67/17.18/25.73 44.21/16.95/25.67
pubmed 45.49/19.90/27.69 45.09/19.56/27.42 45.97/20.15/28.25
aeslc 37.69/21.85/36.84 37.40/21.22/36.45 37.68/21.25/36.51
billsum 57.20/39.56/45.80 57.31/40.19/45.82 59.67/41.58/47.59

The "Mixed & Stochastic" model has the following changes:

  • trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples).
  • trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity).
  • the model uniformly sample a gap sentence ratio between 15% and 45%.
  • importance sentences are sampled using a 20% uniform noise to importance scores.
  • the sentencepiece tokenizer is updated to be able to encode newline character.

(*) the numbers of wikihow and big_patent datasets are not comparable because of change in tokenization and data:

  • wikihow dataset contains newline characters which is useful for paragraph segmentation, the C4 and HugeNews model's sentencepiece tokenizer doesn't encode newline and loose this information.
  • we update the BigPatent dataset to preserve casing, some format cleanings are also changed, please refer to change in TFDS.

The "Mixed & Stochastic" model has the following changes (from pegasus-large in the paper):

trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples). trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity). the model uniformly sample a gap sentence ratio between 15% and 45%. importance sentences are sampled using a 20% uniform noise to importance scores. the sentencepiece tokenizer is updated to be able to encode newline character.

Citation



@misc{zhang2019pegasus,
    title={PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization},
    author={Jingqing Zhang and Yao Zhao and Mohammad Saleh and Peter J. Liu},
    year={2019},
    eprint={1912.08777},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}