MikaSie commited on
Commit
06d59a7
1 Parent(s): e6420b1

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +105 -95
README.md CHANGED
@@ -1,100 +1,110 @@
1
  ---
2
- base_model: google/pegasus-large
3
  tags:
4
- - generated_from_trainer
5
- datasets:
6
- - arrow
 
 
 
 
7
  model-index:
8
- - name: RoBERTa_Pegasus_dependent_V1
9
- results: []
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  ---
11
 
12
- <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
- should probably proofread and complete it, then remove this comment. -->
14
-
15
- # RoBERTa_Pegasus_dependent_V1
16
-
17
- This model is a fine-tuned version of [google/pegasus-large](https://huggingface.co/google/pegasus-large) on the arrow dataset.
18
- It achieves the following results on the evaluation set:
19
- - Loss: 2.1922
20
-
21
- ## Model description
22
-
23
- More information needed
24
-
25
- ## Intended uses & limitations
26
-
27
- More information needed
28
-
29
- ## Training and evaluation data
30
-
31
- More information needed
32
-
33
- ## Training procedure
34
-
35
- ### Training hyperparameters
36
-
37
- The following hyperparameters were used during training:
38
- - learning_rate: 5e-05
39
- - train_batch_size: 2
40
- - eval_batch_size: 1
41
- - seed: 42
42
- - distributed_type: multi-GPU
43
- - num_devices: 4
44
- - gradient_accumulation_steps: 2
45
- - total_train_batch_size: 16
46
- - total_eval_batch_size: 4
47
- - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
48
- - lr_scheduler_type: linear
49
- - lr_scheduler_warmup_ratio: 0.1
50
- - num_epochs: 40
51
-
52
- ### Training results
53
-
54
- | Training Loss | Epoch | Step | Validation Loss |
55
- |:-------------:|:-------:|:----:|:---------------:|
56
- | 5.0795 | 0.9927 | 68 | 3.9061 |
57
- | 3.6946 | 2.0 | 137 | 3.2272 |
58
- | 3.1058 | 2.9927 | 205 | 2.6658 |
59
- | 2.6695 | 4.0 | 274 | 2.4699 |
60
- | 2.511 | 4.9927 | 342 | 2.3796 |
61
- | 2.3613 | 6.0 | 411 | 2.3305 |
62
- | 2.3187 | 6.9927 | 479 | 2.3024 |
63
- | 2.2335 | 8.0 | 548 | 2.2800 |
64
- | 2.2252 | 8.9927 | 616 | 2.2639 |
65
- | 2.1548 | 10.0 | 685 | 2.2515 |
66
- | 2.1536 | 10.9927 | 753 | 2.2412 |
67
- | 2.0982 | 12.0 | 822 | 2.2338 |
68
- | 2.0988 | 12.9927 | 890 | 2.2248 |
69
- | 2.0441 | 14.0 | 959 | 2.2204 |
70
- | 2.0552 | 14.9927 | 1027 | 2.2164 |
71
- | 2.0084 | 16.0 | 1096 | 2.2111 |
72
- | 2.0178 | 16.9927 | 1164 | 2.2076 |
73
- | 1.9719 | 18.0 | 1233 | 2.2051 |
74
- | 1.9901 | 18.9927 | 1301 | 2.2043 |
75
- | 1.9464 | 20.0 | 1370 | 2.2009 |
76
- | 1.9631 | 20.9927 | 1438 | 2.2000 |
77
- | 1.9233 | 22.0 | 1507 | 2.1981 |
78
- | 1.9394 | 22.9927 | 1575 | 2.1961 |
79
- | 1.9012 | 24.0 | 1644 | 2.1947 |
80
- | 1.9213 | 24.9927 | 1712 | 2.1943 |
81
- | 1.8843 | 26.0 | 1781 | 2.1942 |
82
- | 1.9025 | 26.9927 | 1849 | 2.1942 |
83
- | 1.8715 | 28.0 | 1918 | 2.1950 |
84
- | 1.8906 | 28.9927 | 1986 | 2.1936 |
85
- | 1.8559 | 30.0 | 2055 | 2.1939 |
86
- | 1.8774 | 30.9927 | 2123 | 2.1934 |
87
- | 1.8468 | 32.0 | 2192 | 2.1913 |
88
- | 1.8685 | 32.9927 | 2260 | 2.1916 |
89
- | 1.834 | 34.0 | 2329 | 2.1924 |
90
- | 1.8582 | 34.9927 | 2397 | 2.1929 |
91
- | 1.8333 | 36.0 | 2466 | 2.1925 |
92
- | 1.859 | 36.9927 | 2534 | 2.1922 |
93
-
94
-
95
- ### Framework versions
96
-
97
- - Transformers 4.40.1
98
- - Pytorch 2.2.1+cu121
99
- - Datasets 2.17.1
100
- - Tokenizers 0.19.1
 
1
  ---
2
+ language: en
3
  tags:
4
+ - summarization
5
+ - abstractive
6
+ - hybrid
7
+ - multistep
8
+ datasets: dennlinger/eur-lex-sum
9
+ pipeline_tag: summarization
10
+ base_model: Pegasus
11
  model-index:
12
+ - name: BART
13
+ results:
14
+ - task:
15
+ type: summarization
16
+ name: Long, Legal Document Summarization
17
+ dataset:
18
+ name: eur-lex-sum
19
+ type: dennlinger/eur-lex-sum
20
+ metrics:
21
+ - type: ROUGE-1
22
+ value: 0.30669095170825544
23
+ - type: ROUGE-2
24
+ value: 0.09110966989374848
25
+ - type: ROUGE-L
26
+ value: 0.20213752168630658
27
+ - type: BERTScore
28
+ value: 0.8435331912680045
29
+ - type: BARTScore
30
+ value: -1.275050522228859
31
+ - type: BLANC
32
+ value: 0.1249752365647217
33
  ---
34
 
35
+ # Model Card for RoBERTa_Pegasus_dependent_V1
36
+
37
+ ## Model Details
38
+ ---
39
+ ### Model Description
40
+
41
+ This model is a fine-tuned version of Pegasus. The research involves a multi-step summarization approach to long, legal documents. Many decisions in the renewables energy space are heavily dependent on regulations. But these regulations are often long and complicated. The proposed architecture first uses one or more extractive summarization steps to compress the source text, before the final summary is created by the abstractive summarization model. This fine-tuned abstractive model has been trained on a dataset, pre-processed through extractive summarization by RoBERTa with dependent ratio. The research has used multiple extractive-abstractive model combinations, which can be found on https://huggingface.co/MikaSie. To obtain optimal results, feed the model an extractive summary as input as it was designed this way!
42
+
43
+ The dataset used by this model is the [EUR-lex-sum](https://huggingface.co/datasets/dennlinger/eur-lex-sum) dataset. The evaluation metrics can be found in the metadata of this model card.
44
+ This paper was introduced by the master thesis of Mika Sie at the University Utrecht in collaboration with Power2x. More information can be found in PAPER_LINK.
45
+
46
+ - **Developed by:** Mika Sie
47
+ - **Funded by:** University Utrecht & Power2X
48
+ - **Language (NLP):** English
49
+ - **Finetuned from model:** Pegasus
50
+
51
+
52
+ ### Model Sources
53
+
54
+ - **Repository**: https://github.com/MikaSie/Thesis
55
+ - **Paper**: PAPER_LINK
56
+ - **Streamlit demo**: STREAMLIT_LINK
57
+
58
+ ## Uses
59
+ ---
60
+ ### Direct Use
61
+
62
+ This model can be directly used for summarizing long, legal documents. However, it is recommended to first use an extractive summarization tool, such as RoBERTa, to compress the source text before feeding it to this model. This model has been specifically designed to work with extractive summaries.
63
+ An example using the Huggingface pipeline could be:
64
+
65
+ ```python
66
+ pip install bert-extractive-summarizer
67
+
68
+ from summarizer import Summarizer
69
+ from transformers import pipeline
70
+
71
+ extractive_model = Summarizer()
72
+
73
+ text = 'Original document text to be summarized'
74
+
75
+ extractive_summary = Summarizer(text)
76
+
77
+ abstractive_model = pipeline('summarization', model = 'MikaSie/RoBERTa_Pegasus_dependent_V1', tokenizer = 'MikaSie/RoBERTa_Pegasus_dependent_V1')
78
+
79
+ result = pipeline(extractive_summary)
80
+ ```
81
+
82
+ But more information of implementation can be found in the Thesis report.
83
+ ### Out-of-Scope Use
84
+
85
+ Using this model without an extractive summarization step may not yield optimal results. It is recommended to follow the proposed multi-step summarization approach outlined in the model description for best performance.
86
+
87
+ ## Bias, Risks, and Limitations
88
+ ---
89
+
90
+ ### Bias
91
+
92
+ As with any language model, this model may inherit biases present in the training data. It is important to be aware of potential biases in the source text and to critically evaluate the generated summaries.
93
+
94
+ ### Risks
95
+
96
+ - The model may not always generate accurate or comprehensive summaries, especially for complex legal documents.
97
+ - The model may not generate truthful information.
98
+
99
+ ### Limitations
100
+
101
+ - The model may produce summaries that are overly abstractive or fail to capture important details.
102
+ - The model's performance may vary depending on the quality and relevance of the extractive summaries used as input.
103
+
104
+ ### Recommendations
105
+
106
+ - Carefully review and validate the generated summaries before relying on them for critical tasks.
107
+ - Consider using the model in conjunction with human review or other validation mechanisms to ensure the accuracy and completeness of the summaries.
108
+ - Experiment with different extractive summarization models or techniques to find the most suitable input for the abstractive model.
109
+ - Provide feedback and contribute to the ongoing research and development of the model to help improve its performance and address its limitations.
110
+ - Any actions taken based on this content are at your own risk.