ICEF-NLP/bcms-bertic-senticomments-sr-fourway

BERTić-SentiComments-SR-Four-way

BERTić-SentiComments-SR-Four-way is a variant of the BERTić model, fine-tuned on the task of four-way sentiment classification of Serbian short texts. It differentiates between objective (NS), ambiguous/mixed (M), positive (+1), and negative texts (-1). The model was fine-tuned for 5 epochs on the SentiComments.SR dataset.

Benchmarking

This model was evaluated on the task of four-way sentiment classification of short texts in Serbian from the SentiComments.SR dataset and compared to multilingual BERT. Different lengths of fine-tuning were considered, ranging from 1 to 5 epochs. Linear classifiers relying on bag-of-words (BOW) and/or bag-of-embeddings (BOE) features were used as baselines.

Since the dataset is imbalanced, weighted F1 measure was utilized as the performance metric. Model fine-tuning and evaluation were performed using 10-fold stratified cross-validation. The code and data to run these experiments are available on the SentiComments.SR GitHub repository.

Results

Model	Weighted F1
Baseline - Linear classifier with BOW features	0.640
Baseline - Linear classifier with BOE features	0.628
Baseline - Linear classifier with BOW+BOE features	0.655
Multilingual BERT, 1 epoch	0.584
BERTić-SentiComments-SR-Four-way, 1 epoch	0.753
Multilingual BERT, 3 epochs	0.662
BERTić-SentiComments-SR-Four-way, 3 epochs	0.782
Multilingual BERT, 5 epochs	0.670
BERTić-SentiComments-SR-Four-way, 5 epochs	0.797

References

If you wish to use this model in your paper or project, please cite the following papers:

• A versatile framework for resource-limited sentiment articulation, annotation, and analysis of short texts, Vuk Batanović, Miloš Cvetanović, Boško Nikolić, PLoS ONE, 15(11): e0242050 (2020).
• Semantic Similarity and Sentiment Analysis of Short Texts in Serbian, Vuk Batanović, in Proceedings of the 29th Telecommunications Forum (TELFOR 2021), Belgrade, Serbia (2021).