README.md

---
license:
- apache-2.0
- bsd-3-clause
tags:
- summarization
- summary
- booksum
- long-document
- long-form
- tglobal-xl
- XL
datasets:
- kmfoda/booksum
metrics:
- rouge
inference: false
model-index:
- name: pszemraj/long-t5-tglobal-xl-16384-book-summary
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: multi_news
      type: multi_news
      config: default
      split: test
    metrics:
    - type: rouge
      value: 36.2043
      name: ROUGE-1
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzRmMmUyOTVjMmJmZTRiZDcyYzY3MTQ1MmUyNDA5NjVhYzEzYzBiNzcxYTRhMDQ3OTlhMGZjYmJlNDM1M2NjYyIsInZlcnNpb24iOjF9._uArOQ1_0znXDPXMq7unA1OHB-XbgqzzKRbFRcVUzTUJdWk26LiSa2pEEVNNmJPg6Uo7CAvONmhpEswLvl9TAg
    - type: rouge
      value: 8.424
      name: ROUGE-2
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzg0MzljYjVjYWQ3MmRkZDBlOGI5M2RiMGU0M2UwZGUzMDg2NTU0NjcwMTNiN2ZmODEzNTQ0MmEwNDA3NDA5MSIsInZlcnNpb24iOjF9.Dzj85ld6TjosQ8KyUdoadzicMLedEFrICC6Q-08O3qx28d9B9Uke1zw-VWabiuesPEDTRGbWuBgPA5vxYWUZAw
    - type: rouge
      value: 17.3721
      name: ROUGE-L
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDA3ZjZmODAwMTNlM2RlZmJlMDI5MGVkMGRkMTBjMTYzNDk5ZjFiNTY5MWE1MDUwNWI2MDE4ZDA2YWMwMmI2NCIsInZlcnNpb24iOjF9.MOV_nId0XAK1eMQssG5GN9DsitZaTrxl4jdCJnOg9EZ0-vAw227ln599YV5YfZ1OPJnWwek6rneqqyONiHn9AQ
    - type: rouge
      value: 32.3994
      name: ROUGE-LSUM
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmY3MDMwOTZjNWI0YTk1MDgwMzJkYTFiN2U5YWU0Mzc0MWRiMzc1NzZlMDhjMWUwMmY2ODI2MjI5ODBkYWUxOSIsInZlcnNpb24iOjF9._BwGIZbcA4pUBkEAL0cW-JPPta0KSoGug4Z7vogHacUz-AEhIOI5ICUldZh0pt9OK67MpUSzpShJOu3rSt5YDQ
    - type: loss
      value: 2.0843334197998047
      name: loss
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOWFhMmE5ZjA3ODM4YmVjMDMyMjk5YjNlMjA1MGMzOWY0NTRlYzk1YjZiMzQxMDMxOTMwMjFkNTdmNjM1NDcyMyIsInZlcnNpb24iOjF9.3wbXV4CIIgnfXAnnRztdOR12PwsWsEfiglQQ09K-C1EgW4gai4x9l-wTE2OZ7CTWkuk_tr4tL_uqOCXLZRMtCQ
    - type: gen_len
      value: 248.3572
      name: gen_len
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMWZhOGMwMDJjNGU2MzA2YzI1OWU1ZDY5N2NjZmM1YTA5NDg1MzUwNmU1YTBhNjQyNWYwYzA3OGNmODFjMmE2NSIsInZlcnNpb24iOjF9.Rc9u89zCdbFnjsnmq65l_JvCtUwOX_ZWapKJpTZ-rC8HxcUVfi2Ash2QfvvvxHH_YWhwklxxdnNa0HCm46qLAA
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: billsum
      type: billsum
      config: default
      split: test
    metrics:
    - name: ROUGE-1
      type: rouge
      value: 41.3645
      verified: true
    - name: ROUGE-2
      type: rouge
      value: 16.144
      verified: true
    - name: ROUGE-L
      type: rouge
      value: 24.2981
      verified: true
    - name: ROUGE-LSUM
      type: rouge
      value: 35.3234
      verified: true
    - name: loss
      type: loss
      value: 1.282260775566101
      verified: true
    - name: gen_len
      type: gen_len
      value: 291.8158
      verified: true
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: ccdv/arxiv-summarization
      type: ccdv/arxiv-summarization
      config: document
      split: test
    metrics:
    - name: ROUGE-1
      type: rouge
      value: 36.3225
      verified: true
    - name: ROUGE-2
      type: rouge
      value: 9.3743
      verified: true
    - name: ROUGE-L
      type: rouge
      value: 19.8396
      verified: true
    - name: ROUGE-LSUM
      type: rouge
      value: 32.2532
      verified: true
    - name: loss
      type: loss
      value: 2.146871566772461
      verified: true
    - name: gen_len
      type: gen_len
      value: 186.2966
      verified: true
---

# long-t5-tglobal-xl + BookSum

<a href="https://colab.research.google.com/gist/pszemraj/c19e32baf876deb866c31cd46c86e893/long-t5-xl-accelerate-test.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

Summarize long text and get a SparkNotes-like summary of any topic!

- Generalizes reasonably well to academic & narrative text.
- This is the XL checkpoint, which **produces even better summaries [from a human evaluation perspective](https://long-t5-xl-book-summary-examples.netlify.app/)**.

A simple example/use case with [the base model](https://huggingface.co/pszemraj/long-t5-tglobal-base-16384-book-summary) on ASR is [here](https://longt5-booksum-example.netlify.app/).

## Cheeky Proof-of-Concept

A summary of the [infamous navy seals copypasta](https://knowyourmeme.com/memes/navy-seal-copypasta):

> In this chapter, the monster explains how he intends to exact revenge on "the little b\*\*\*\*" who insulted him. He tells the kiddo that he is a highly trained and experienced killer who will use his arsenal of weapons--including his access to the internet--to exact justice on the little brat.

While this is a crude example, try running this copypasta through other summarization models to see the difference in comprehension (_even though it's not even a "long" text!_).

* * *

**Contents**

<!-- TOC -->

- [Description](#description)
- [How-To in Python](#how-to-in-python)
    - [Beyond the basics](#beyond-the-basics)
        - [Adjusting parameters](#adjusting-parameters)
        - [LLM.int8 Quantization](#llmint8-quantization)
- [About](#about)
    - [Intended uses & limitations](#intended-uses--limitations)
    - [Training and evaluation data](#training-and-evaluation-data)
    - [Eval results](#eval-results)
- [FAQ](#faq)
    - [How can I run inference with this on CPU?](#how-can-i-run-inference-with-this-on-cpu)
    - [How to run inference over a very long (30k+ tokens) document in batches?](#how-to-run-inference-over-a-very-long-30k-tokens-document-in-batches)
    - [How to fine-tune further?](#how-to-fine-tune-further)
    - [Are there simpler ways to run this?](#are-there-simpler-ways-to-run-this)
- [Training procedure](#training-procedure)
    - [Updates](#updates)
    - [Training hyperparameters](#training-hyperparameters)
    - [Framework versions](#framework-versions)

<!-- /TOC -->

* * *

## Description

A fine-tuned version of [google/long-t5-tglobal-xl](https://huggingface.co/google/long-t5-tglobal-xl) on the `kmfoda/booksum` dataset.

Read the paper by Guo et al. here: [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://arxiv.org/pdf/2112.07916.pdf)

## How-To in Python

install/update transformers `pip install -U transformers`

summarize text with pipeline:

```python
import torch
from transformers import pipeline

summarizer = pipeline(
    "summarization",
    "pszemraj/long-t5-tglobal-xl-16384-book-summary",
    device=0 if torch.cuda.is_available() else -1,
)
long_text = "Here is a lot of text I don't want to read. Replace me"

result = summarizer(long_text)
print(result[0]["summary_text"])
```

### Beyond the basics

There are two additional points to consider beyond simple inference: adjusting decoding parameters for improved performance, and quantization for reduced memory consumption.

#### Adjusting parameters

Pass [other parameters related to beam search textgen](https://huggingface.co/blog/how-to-generate) when calling `summarizer` to get even higher quality results.

#### LLM.int8 Quantization

> alternative section title: how to get this monster to run inference on free colab runtimes

Via [this PR](https://github.com/huggingface/transformers/pull/20341) LLM.int8 is now supported for `long-t5` models. 

- per **initial tests** the summarization quality seems to hold while using _significantly_ less memory! \*
- a version of this model quantized to int8 is [already on the hub here](https://huggingface.co/pszemraj/long-t5-tglobal-xl-16384-book-summary-8bit) so if you're using the 8-bit version anyway, you can start there for a 3.5 gb download only!

First, make sure you have the latest versions of the relevant packages:
```bash
pip install -U transformers bitsandbytes accelerate
```

load in 8-bit (_magic completed by `bitsandbytes` behind the scenes_)

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained(
    "pszemraj/long-t5-tglobal-xl-16384-book-summary"
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    "pszemraj/long-t5-tglobal-xl-16384-book-summary",
    load_in_8bit=True,
    device_map="auto",
)
```

The above is already present in the Colab demo linked at the top of the model card.

\* More rigorous metrics-based research comparing beam-search summarization with and without LLM.int8 will take place over time.

* * *

## About

### Intended uses & limitations

While this model seems to improve factual consistency, **don't take summaries as foolproof and check things that seem odd**.

Specifically: negation statements (i.e., the model says: _this thing does not have [ATTRIBUTE]_, when instead it should have said _this thing has lots of [ATTRIBUTE]_).

- I'm sure someone will write a paper on this eventually (if there isn't one already), but you can usually check this by comparing a particular statement with what the surrounding sentences imply.

### Training and evaluation data

`kmfoda/booksum` dataset on HuggingFace - read [the original paper here](https://arxiv.org/abs/2105.08209).

- For **initial fine-tuning**, only input text with 12288 input tokens or less and 1024 output tokens or less was used (_i.e. lines longer than that were dropped before training_) for memory reasons. After a quick analysis, summaries in the 12288-16384 range are in the **small** minority in this dataset.
    - In addition, this initial training combined the training and validation sets and trained on them in aggregate to increase the functional dataset size. **Therefore, take the validation set results with a grain of salt; primary metrics should (always) be the test set.**.
- The **final stages of fine-tuning** used the standard 16384 input/1024 output conventions, preserving the standard in/out lengths (_and truncating longer sequences_). This did not seem to change the loss/performance much.

### Eval results

Official results with the [model evaluator](https://huggingface.co/spaces/autoevaluate/model-evaluator) will be computed and posted here.

**Please read the note above, as due to the training methods, the performance on the validation set looks better than the results on the test set will be**. The model achieves the following results on the evaluation set:

-   eval_loss: 1.2756
-   eval_rouge1: 41.8013
-   eval_rouge2: 12.0895
-   eval_rougeL: 21.6007
-   eval_rougeLsum: 39.5382
-   eval_gen_len: 387.2945
-   eval_runtime: 13908.4995
-   eval_samples_per_second: 0.107
-   eval_steps_per_second: 0.027


    ***** predict/test metrics (initial) *****
      predict_gen_len            =   506.4368
      predict_loss               =      2.028
      predict_rouge1             =    36.8815
      predict_rouge2             =     8.0625
      predict_rougeL             =    17.6161
      predict_rougeLsum          =    34.9068
      predict_runtime            = 2:04:14.37
      predict_samples            =       1431
      predict_samples_per_second =      0.192
      predict_steps_per_second   =      0.048

\* evaluating big model not as easy as it seems. Doing a bit more investigating

* * *

## FAQ

### How can I run inference with this on CPU?

lol

### How to run inference over a very long (30k+ tokens) document in batches?

See `summarize.py` in [the code for my hf space Document Summarization](https://huggingface.co/spaces/pszemraj/document-summarization/blob/main/summarize.py) :)

You can also use the same code to split a document into batches of 4096, etc., and iterate over them with the model. This is useful in situations where CUDA memory is limited.

**Update:** see the section on the `textsum` package below.

### How to fine-tune further?

See [train with a script](https://huggingface.co/docs/transformers/run_scripts) and [the summarization scripts](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization)

### Are there simpler ways to run this?

For this reason, I created a Python package utility. It's called [textsum](https://github.com/pszemraj/textsum), and you can use it to load models and summarize things in a few lines of code.

```sh
pip install textsum
```

Use `textsum` in python with this model:

```python
from textsum.summarize import Summarizer

summarizer = Summarizer(
    model_name_or_path="pszemraj/long-t5-tglobal-xl-16384-book-summary"
)

long_string = "This is a long string of text that will be summarized."
out_str = summarizer.summarize_string(long_string)
print(f"summary: {out_str}")
```

This package provides easy-to-use interfaces for applying summarization models to text documents of arbitrary length. Currently implemented interfaces include a Python API, a CLI, and a shareable demo application.

For details, explanations, and documentation, see the README (_linked above_) or the [wiki](https://github.com/pszemraj/textsum/wiki).

* * *

## Training procedure

### Updates

Updates to this model/model card will be posted here when relevant. The model seems to be fairly converged; if updates/improvements are possible using the `BookSum` dataset, this repo will be updated.

### Training hyperparameters

The following hyperparameters were used during training:

-   learning_rate: 0.0006
-   train_batch_size: 1
-   eval_batch_size: 1
-   seed: 10350
-   distributed_type: multi-GPU
-   num_devices: 4
-   gradient_accumulation_steps: 32
-   total_train_batch_size: 128
-   total_eval_batch_size: 4
-   optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-   lr_scheduler_type: constant
-   num_epochs: 1.0

\*_Prior training sessions used roughly similar parameters (learning rates were higher); multiple sessions were required as this takes eons to train._

### Framework versions

-   Transformers 4.25.0.dev0
-   Pytorch 1.13.0+cu117
-   Datasets 2.6.1
-   Tokenizers 0.13.1

* * *