|
--- |
|
base_model: DeepPavlov/rubert-base-cased |
|
tags: |
|
- generated_from_trainer |
|
- sentiment |
|
metrics: |
|
- f1 |
|
model-index: |
|
- name: vashkontrol-sentiment-rubert |
|
results: [] |
|
license: mit |
|
datasets: |
|
- kartashoffv/vash_kontrol_reviews |
|
language: |
|
- ru |
|
pipeline_tag: text-classification |
|
widget: |
|
- text: "Отзывчивые и понимающие работники, обслуживание очень понравилось, специалист проявила большое терпение чтобы восстановить пароль от Госуслуг. Спасибо!" |
|
--- |
|
|
|
|
|
# Sentimental assessment of portal reviews "VashKontrol" |
|
|
|
The model is designed to evaluate the tone of reviews from the [VashKontrol portal](https://vashkontrol.ru/). |
|
|
|
This model is a fine-tuned version of [DeepPavlov/rubert-base-cased](https://huggingface.co/DeepPavlov/rubert-base-cased) on a following dataset: [kartashoffv/vash_kontrol_reviews](https://huggingface.co/datasets/kartashoffv/vash_kontrol_reviews). |
|
|
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.1085 |
|
- F1: 0.9461 |
|
|
|
## Model description |
|
|
|
The model predicts a sentiment label (positive, neutral, negative) for a submitted text review. |
|
|
|
|
|
## Training and evaluation data |
|
|
|
The model was trained on the corpus of reviews of the [VashControl portal](https://vashkontrol.ru/), left by users in the period from 2020 to 2022 inclusive. |
|
The total number of reviews was 17,385. The sentimental assessment of the dataset was carried out by the author manually by dividing the general dataset into positive/neutral/negative reviews. |
|
|
|
The resulting classes: |
|
0 (positive): 13045 |
|
1 (neutral): 1196 |
|
2 (negative): 3144 |
|
|
|
Class weighting was used to solve the class imbalance. |
|
|
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 10 |
|
- eval_batch_size: 10 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 5 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | F1 | |
|
|:-------------:|:-----:|:----:|:---------------:|:------:| |
|
| 0.0992 | 1.0 | 1391 | 0.0737 | 0.9337 | |
|
| 0.0585 | 2.0 | 2782 | 0.0616 | 0.9384 | |
|
| 0.0358 | 3.0 | 4173 | 0.0787 | 0.9441 | |
|
| 0.0221 | 4.0 | 5564 | 0.0918 | 0.9488 | |
|
| 0.0106 | 5.0 | 6955 | 0.1085 | 0.9461 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.31.0 |
|
- Pytorch 2.0.1+cu118 |
|
- Datasets 2.14.1 |
|
- Tokenizers 0.13.3 |
|
|
|
|
|
### Usage |
|
|
|
``` |
|
import torch |
|
from transformers import AutoModelForSequenceClassification |
|
from transformers import BertTokenizerFast |
|
|
|
tokenizer = BertTokenizerFast.from_pretrained('kartashoffv/vashkontrol-sentiment-rubert') |
|
model = AutoModelForSequenceClassification.from_pretrained('kartashoffv/vashkontrol-sentiment-rubert', return_dict=True) |
|
|
|
@torch.no_grad() |
|
def predict(review): |
|
inputs = tokenizer(review, max_length=512, padding=True, truncation=True, return_tensors='pt') |
|
outputs = model(**inputs) |
|
predicted = torch.nn.functional.softmax(outputs.logits, dim=1) |
|
pred_label = torch.argmax(predicted, dim=1).numpy() |
|
return pred_label |
|
``` |
|
### Labels |
|
|
|
``` |
|
0: POSITIVE |
|
1: NEUTRAL |
|
2: NEGATIVE |
|
``` |
|
|
|
|
