README.md

metadata

license: cc-by-nc-4.0
language:
  - hu
  - en
metrics:
  - accuracy
  - f1
model-index:
  - name: Hun_Eng_RoBERTa_base_Plain
    results:
      - task:
          type: text-classification
        metrics:
          - type: accuracy
            value: 0.75 (hu) / 0.65 (en)
          - type: f1
            value: 0.74 (hu) / 0.64 (en)
widget:
  - text: >-
      A tanúsítvány meghatározott adatainak a 2008/118/EK irányelv IV. fejezete
      szerinti szállításához szükséges adminisztratív okmányban...
    example_title: Incomprehensible
  - text: >-
      Az AEO-engedély birtokosainak listáján – keresésre – megjelenő
      információk: az engedélyes neve, az engedélyt kibocsátó ország...
    example_title: Comprehensible

Model description

Cased fine-tuned XLM-RoBERTa-base model for Hungarian and English, trained on datasets provided by the National Tax and Customs Administration - Hungary (NAV) and translated versions of the same dataset using Google Translate API.

Intended uses & limitations

The model is designed to classify sentences as either "comprehensible" or "not comprehensible" (according to Plain Language guidelines):

• Label_0 - "comprehensible" - The sentence is in Plain Language.
• Label_1 - "not comprehensible" - The sentence is not in Plain Language.

Training

Fine-tuned version of the original xlm-roberta-base model, trained on a dataset of Hungarian legal and administrative texts. The model was also trained on the translated version of this dataset (via Google Translate API) for English classification.

Eval results

Hungarian Results:

Class	Precision	Recall	F1-Score
Comprehensible / Label_0	0.82	0.62	0.70
Not comprehensible / Label_1	0.71	0.88	0.78
accuracy			0.75
macro avg	0.77	0.75	0.74
weighted avg	0.76	0.75	0.74

English Results:

Class	Precision	Recall	F1-Score
Comprehensible / Label_0	0.70	0.50	0.58
Not comprehensible / Label_1	0.63	0.80	0.70
accuracy			0.65
macro avg	0.66	0.65	0.64
weighted avg	0.66	0.65	0.64

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("uvegesistvan/Hun_Eng_RoBERTa_base_Plain")
model = AutoModelForSequenceClassification.from_pretrained("uvegesistvan/Hun_Eng_RoBERTa_base_Plain")

BibTeX entry and citation info

If you use the model, please cite the following dissertation (to be submitted for workshop discussion):

Bibtex:

@PhDThesis{ Uveges:2024,
  author = {{"U}veges, Istv{\'a}n},
  title  = {K{\"o}z{\'e}rthet{\"o} és automatiz{\'a}ci{\'o} - k{\'i}s{\'e}rletek a jog, term{\'e}szetesnyelv-feldolgoz{\'a}s {\'e}s informatika hat{\'a}r{\'a}n.},
  year   = {2024},
  school = {Szegedi Tudom{\'a}nyegyetem}
}