gauneg/deberta-v3-base-absa-ate-sentiment

Training

This model is designed for token classification tasks, enabling it to extract aspect terms and predict the sentiment polarity associated with the extracted aspect terms. The extracted aspect terms will be the span(s) from the input text on which a sentiment is being expressed.

Datasets

This model has been trained on the following datasets:

1. Aspect Based Sentiment Analysis SemEval Shared Tasks (2014, 2015, 2016)
2. Multi-Aspect Multi-Sentiment MAMS

Use

• Making end-to-end inference with a pipeline

from transformers import pipeline

ate_sent_pipeline = pipeline(task='ner', 
                  aggregation_strategy='simple',
                  model="gauneg/deberta-v3-base-absa-ate-sentiment")

text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
ate_sent_pipeline(text_input)

Expected output

[{'entity_group': 'pos', #sentiment polarity
  'score': 0.87505656,
  'word': 'food', # aspect term
  'start': 25,
  'end': 30},
 {'entity_group': 'neg',# sentiment polarity
  'score': 0.4558051,
  'word': 'service', #aspect term
  'start': 55,
  'end': 63}]

OR

• Making token level inferences with Auto classes

from transformers import AutoTokenizer, AutoModelForTokenClassification
model_id = "gauneg/deberta-v3-base-absa-ate-sentiment"
tokenizer = AutoTokenizer.from_pretrained(model_id)



# the sequence of labels used during training
labels = {"B-neu": 1, "I-neu": 2, "O": 0, "B-neg": 3, "B-con": 4, "I-pos": 5, "B-pos": 6, "I-con": 7, "I-neg": 8, "X": -100}
id2lab = {idx: lab for lab, idx in labels.items()}
lab2id = {lab: idx for lab, idx in labels.items()}

model = AutoModelForTokenClassification.from_pretrained("../models/deberta-v3-base-bio-w-pol/", 
                                                        num_labels=len(labels), id2label=id2lab, label2id=lab2id)

# making one prediction at a time (should be padded/batched and truncated for efficiency)
text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
tok_inputs = tokenizer(text_input, return_tensors="pt")


y_pred = model(**tok_inputs) # predicting the logits


# selecting the most favoured labels for each token from the logits
y_pred_fin = y_pred.logits.argmax(dim=-1)[0]


# since first and the last tokens are excluded ([CLS] and [SEP]) they have to be removed before decoding
decoded_pred = [id2lab[logx.item()] for logx in y_pred_fin[1:-1]]

## displaying the input tokens with predictions and skipping [CLS] and [SEP] tokens at the beginning and the end respectively
decoded_toks = tok_inputs['input_ids'][0][1:-1]
tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(decoded_toks), decoded_pred))

Expected output

[('▁Been', 'O'),
 ('▁here', 'O'),
 ('▁a', 'O'),
 ('▁few', 'O'),
 ('▁times', 'O'),
 ('▁and', 'O'),
 ('▁food', 'B-pos'),
 ('▁has', 'O'),
 ('▁always', 'O'),
 ('▁been', 'O'),
 ('▁good', 'O'),
 ('▁but', 'O'),
 ('▁service', 'B-neg'),
 ('▁really', 'O'),
 ('▁suffers', 'O'),
 ('▁when', 'O'),
 ('▁it', 'O'),
 ('▁gets', 'O'),
 ('▁crowded', 'O'),
 ('.', 'O')]

Evaluation on Benchmark Test Datasets

The first evaluation is for token-extraction task without considering the polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens on which the sentiments have been expressed. (scores are expressed as micro-averages of B-I-O labels)

ATE (Aspect Term Extraction Only)

Test Dataset	Base Model	Fine-tuned Model	Precision	Recall	F1 Score
hotel reviews (SemEval 2015)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	71.16	73.92	71.6
hotel reviews (SemEval 2015)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	70.92	72.28	71.07
hotel reviews (SemEval 2015)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	64.05	79.69	70.0
hotel reviews (SemEval 2015)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	66.29	72.78	68.92
------------	----------	----------------	---------	------	--------
laptop reviews (SemEval 2014)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	70.58	61.52	64.21
laptop reviews (SemEval 2014)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	66.38	50.62	54.31
laptop reviews (SemEval 2014)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	70.82	48.97	52.08
laptop reviews (SemEval 2014)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	73.61	46.38	49.87
------------	----------	----------------	---------	------	--------
MAMS-ATE (2019)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	81.07	79.66	80.35
MAMS-ATE (2019)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	79.91	78.95	79.39
MAMS-ATE (2019)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	74.46	84.5	78.75
MAMS-ATE (2019)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	77.8	79.81	78.75
------------	----------	----------------	---------	------	--------
restaurant reviews (SemEval 2014)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	88.59	87.0	87.45
restaurant reviews (SemEval 2014)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	92.26	82.95	86.57
restaurant reviews (SemEval 2014)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	93.07	81.95	86.32
restaurant reviews (SemEval 2014)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	92.94	81.71	86.01
------------	----------	----------------	---------	------	--------
restaurant reviews (SemEval 2015)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	72.91	75.4	72.74
restaurant reviews (SemEval 2015)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	70.54	77.48	72.63
restaurant reviews (SemEval 2015)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	68.32	79.84	72.28
restaurant reviews (SemEval 2015)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	71.94	74.75	71.84
------------	----------	----------------	---------	------	--------
restaurant reviews (SemEval 2016)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	70.22	75.83	71.84
restaurant reviews (SemEval 2016)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	71.54	73.38	71.2
restaurant reviews (SemEval 2016)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	71.35	72.78	70.85
restaurant reviews (SemEval 2016)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	66.68	77.97	70.79

Aspect Sentiment Evaluation

This evaluation considers token-extraction task with polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens on which the sentiments have been expressed along with the polarity of the sentiments. (scores are expressed as macro-averages)

Test Dataset	Base Model	Fine-tuned Model	Precision	Recall	F1 Score
hotel reviews (SemEval 2015)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	51.92	65.55	54.94
hotel reviews (SemEval 2015)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	54.62	53.65	54.08
hotel reviews (SemEval 2015)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	55.43	56.53	54.03
hotel reviews (SemEval 2015)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	52.88	55.19	53.85
------------	----------	----------------	---------	------	--------
laptop reviews (SemEval 2014)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	44.25	41.55	42.81
laptop reviews (SemEval 2014)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	46.15	33.23	37.09
laptop reviews (SemEval 2014)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	41.7	34.38	36.93
laptop reviews (SemEval 2014)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	44.98	31.87	35.67
------------	----------	----------------	---------	------	--------
MAMS-ATE (2019)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	72.06	72.98	72.49
MAMS-ATE (2019)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	72.97	71.63	72.26
MAMS-ATE (2019)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	69.34	73.3	71.07
MAMS-ATE (2019)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	65.74	75.11	69.77
------------	----------	----------------	---------	------	--------
restaurant reviews (SemEval 2014)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	61.15	58.46	59.74
restaurant reviews (SemEval 2014)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	60.13	56.81	58.13
restaurant reviews (SemEval 2014)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	56.79	59.3	57.93
restaurant reviews (SemEval 2014)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	58.99	54.76	56.45
------------	----------	----------------	---------	------	--------
restaurant reviews (SemEval 2015)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	53.89	55.7	54.11
restaurant reviews (SemEval 2015)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	54.36	55.38	53.6
restaurant reviews (SemEval 2015)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	51.67	56.58	53.29
restaurant reviews (SemEval 2015)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	54.55	53.68	53.12
------------	----------	----------------	---------	------	--------
restaurant reviews (SemEval 2016)	FacebookAI/roberta-large	gauneg/roberta-large-absa-ate-sentiment-lora-adapter	53.7	60.49	55.05
restaurant reviews (SemEval 2016)	FacebookAI/roberta-base	gauneg/roberta-base-absa-ate-sentiment	52.31	54.58	52.33
restaurant reviews (SemEval 2016)	(this) microsoft/deberta-v3-base	gauneg/deberta-v3-base-absa-ate-sentiment	52.07	54.58	52.15
restaurant reviews (SemEval 2016)	microsoft/deberta-v3-large	gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter	49.07	56.5	51.25