Hugging Face's logo

Salesforce
/
dialogstudio-t5-base-v1.0

Text2Text Generation
Transformers
PyTorch
Safetensors
t5
dialog
text-generation-inference
Inference Endpoints
Model card Files Community
1
dialogstudio-t5-base-v1.0 / README.md
jianguozhang's picture
jianguozhang
Create README.md
b2dd99b
preview code
|
raw
history blame
12.5 kB
metadata
language:
  - en
  - fr
  - ro
  - de
  - multilingual
widget:
  - text: >-
      Continue the dialogue as a task-oriented dialogue system called SYSTEM.
      The answer of SYSTEM should follow the ACTION provided next while
      answering the USER's last utterance: 

      <USER> Hello, I am looking for a restaurant in Cambridge. I believe it is
      called Golden Wok. 

      <EXTERNAL KNOWLEDGE> ACTION: {'Restaurant-Inform': [['address', '191
      Histon Road Chesterton']]}
    example_title: Dialog Act to Response Generation
  - text: 'Translate to German:  My name is Arthur'
    example_title: Translation
  - text: >-
      Please answer to the following question. Who is going to be the next
      Ballon d'or?
    example_title: Question Answering
  - text: >-
      Please answer the following question. What is the boiling point of
      Nitrogen?
    example_title: Scientific knowledge
  - text: >-
      Answer the following yes/no question. Can you write 200 words in a single
      tweet?
    example_title: Yes/no question
  - text: >-
      Answer the following yes/no question by reasoning step-by-step. Can you
      write 200 words in a single tweet?
    example_title: Reasoning task
  - text: >-
      Q: Is the statement ( `Jianguo is a research scientist at Salesforce AI`
      and `Jianguo is a student at UIC` ) True or Flase? A: Let's think step by
      step
    example_title: Boolean Expressions
  - text: >-
      The square root of x is the cube root of y. What is y to the power of 2,
      if x = 4?
    example_title: Math reasoning
  - text: >-
      Premise:  At my age you will probably have learnt one lesson. Hypothesis: 
      It's not certain how many lessons you'll learn by your thirties. Does the
      premise entail the hypothesis?
    example_title: Premise and hypothesis
inference:
  parameters:
    max_length: 256
tags:
  - text2text-generation
  - dialog
datasets:
  - Salesforce/dialogstudio
  - flan
license: apache-2.0

Model Card for DialogStudio-T5 base

drawing

Table of Contents

  1. TL;DR
  2. Model Details
  3. Usage
  4. Uses
  5. Bias, Risks, and Limitations
  6. Training Details
  7. Evaluation
  8. Environmental Impact
  9. Citation
  10. Model Card Authors

TL;DR

If you already know T5 and Flan-T5, DialogStudio-T5 is better at many things. With the same number of parameters, the models are fine-tuned from a selected amount of dialogues from DialogStudio and also 1000 additional tasks.

Disclaimer: Content from this model card are modified from contents written by the Hugging Face team, and parts of it were copy pasted from the T5 model card and Flan-T5 model card.

Follow the DialogStudio GitHub repository for latest information.

Model Details

Data

We sample a small amount of dialogues from each commercial supported dataset under three categories of DialogStudio, i.e., KG-Dial, TOD and Open-Domain dialogues. Additionally, we sample at most 150 examples for each non-translation task from FLAN.

Note that this version does not incorporate datasets utilized for training large-scale models (>=7B) like Alpaca, ShareGPT, GPT4ALL, UltraChat from OpenAI's 'GPT-3.5/4', or other datasets such as OASST1 and WizardCoder.

drawing

Model Description

  • Model type: Language model
  • Language(s) (NLP): English, Spanish, Japanese, Persian, Hindi, French, Chinese, Bengali, Gujarati, German, Telugu, Italian, Arabic, Polish, Tamil, Marathi, Malayalam, Oriya, Panjabi, Portuguese, Urdu, Galician, Hebrew, Korean, Catalan, Thai, Dutch, Indonesian, Vietnamese, Bulgarian, Filipino, Central Khmer, Lao, Turkish, Russian, Croatian, Swedish, Yoruba, Kurdish, Burmese, Malay, Czech, Finnish, Somali, Tagalog, Swahili, Sinhala, Kannada, Zhuang, Igbo, Xhosa, Romanian, Haitian, Estonian, Slovak, Lithuanian, Greek, Nepali, Assamese, Norwegian
  • License: Apache 2.0
  • Related Models: All DialogStudio-T5 Checkpoints
  • Resources for more information:
  • Maximum model length::
    • Maximum input length: 1200
    • Maximum output length: 256
  • Training formats:
    • We process dialogue data into below input format :
      • With instruction and external knowledge: Instruction: your instruction <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N <EXTERNAL KNOWLEDGE> your external knowledge
      • Without instruction: <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N <EXTERNAL KNOWLEDGE> your external knowledge
      • Without external knowledge: Instruction: your instruction <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N
      • Without both: <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N
      • Note: output is final the system response; <USER>, <SYSTEM> and <EXTERNAL KNOWLEDGE> are special tokens
    • For sampled FLAN data:
      • We follow their original data format, i.e., we did not set special tokens to separate in-context learning examples.
    • In summary:
      • We recommend you use our format and add our special tokens (such as <USER> and <SYSTEM> ) to get better performance. However, you may not necessary need to exactly follow our format if you do observe random behavios.
      • We found that T5 model series such as Flan-t5 and DialogStudio-T5 may generate repetitive tokens during inference. If you find such repetition issues, you can set the repetition_penalty in model.generate(), such as 1.5, to mitigate them. Note that repetition_penalty=1.0 by default.

Usage

Find below some example scripts on how to use the model in transformers:

Using the Pytorch model

Running the model on a CPU

Click to expand 

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-base-v1.0")
model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-base-v1.0")

input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

outputs = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Running the model on a GPU

Click to expand 
# pip install accelerate
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-base-v1.0")
model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-base-v1.0", device_map="auto")

input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Running the model on a GPU using different precisions

FP16

Click to expand 
# pip install accelerate
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-base-v1.0")
model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-base-v1.0", device_map="auto", torch_dtype=torch.float16)

input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

INT8

Click to expand 
# pip install bitsandbytes accelerate
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-base-v1.0")
model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-base-v1.0", device_map="auto", load_in_8bit=True)

input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Uses

Direct Use and Downstream Use

The primary use is research on language models, including: research on zero-shot NLP tasks and in-context few-shot learning NLP tasks, such as dialogue response generation, reasoning, and question answering; advancing fairness and safety research, and understanding limitations of current large language models

Out-of-Scope Use

More information needed.

Bias, Risks, and Limitations

The information below in this section are copied and modified from Flan-T5's models card:

Language models, including DialogStudio-T5, can potentially be used for language generation in a harmful way, according to Rae et al. (2021). DialogStudio-T5 should not be used directly in any application, without a prior assessment of safety and fairness concerns specific to the application.

Ethical considerations and risks

DialogStudio-T5 is fine-tuned on a large corpus of text data that was not filtered for explicit content or assessed for existing biases. As a result the model itself is potentially vulnerable to generating equivalently inappropriate content or replicating inherent biases in the underlying data.

Known Limitations

DialogStudio-T5 has not been tested in real world applications.

Sensitive Use:

DialogStudio-T5 should not be applied for any unacceptable use cases, e.g., generation of abusive speech.

Training Details

Training Data

We sample a small amount of dialogues from each commercial supported dataset under three categories of DialogStudio, i.e., KG-Dial, TOD and Open-Domain dialogues. Additionally, we sample at most 150 examples for each non-translation task from FLAN.

Note that this version does not incorporate datasets utilized for training large-scale models (>=7B) like Alpaca, ShareGPT, GPT4ALL, UltraChat from OpenAI's 'GPT-3.5/4', or other datasets such as OASST1 and WizardCoder.

See above Training formats: for details of the training formats.

Training Procedure

These models are based on Flan-T5 and are fine-tuned with instructions for better zero-shot and few-shot performance. There is one fine-tuned DialogStudio model per T5 model size.

The model has been trained on 16 A100 GPUs, each with 40G memory, using public transformer codebase.

Evaluation

Testing Data, Factors & Metrics

The authors evaluated the model on several dialogue tasks and general tasks such as 0-shot/5-shot MMLU and 3-shot BBH.

Results

For full results for DialogStudio, see the research paper.

Environmental Impact

More information needed.

Citation

BibTeX:

@misc{zhang2023dialogstudio,
      title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI}, 
      author={Jianguo Zhang and Kun Qian and Zhiwei Liu and Shelby Heinecke and Rui Meng and Ye Liu and Zhou Yu and and Huan Wang and Silvio Savarese and Caiming Xiong},
      year={2023},
      eprint={2307.10172},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}