README.md

---
language: 
- pt
tags:
- generated_from_trainer
datasets:
- lener_br
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: checkpoints
  results:
  - task:
      name: Token Classification
      type: token-classification
    dataset:
      name: lener_br
      type: lener_br
    metrics:
    - name: F1
      type: f1
      value: 0.8716487228203504
    - name: Precision
      type: precision
      value: 0.8559286898839138
    - name: Recall
      type: recall
      value: 0.8879569892473118
    - name: Accuracy
      type: accuracy
      value: 0.9755893153732458
    - name: Loss
      type: loss
      value: 0.1133928969502449
widget:
- text: "Acrescento que não há de se falar em violação do artigo 114, § 3º, da Constituição Federal, posto que referido dispositivo revela-se impertinente, tratando da possibilidade de ajuizamento de dissídio coletivo pelo Ministério Público do Trabalho nos casos de greve em atividade essencial."
---

## (BERT base) NER model in the legal domain in Portuguese (LeNER-Br)

**ner-bert-base-portuguese-cased-lenerbr** is a NER model (token classification) in the legal domain in Portuguese that was finetuned on 16/12/2021 in Google Colab from the model [BERTimbau base](https://huggingface.co/neuralmind/bert-base-portuguese-cased) on the dataset [LeNER_br](https://huggingface.co/datasets/lener_br) by using a NER objective.

Note: due to the small size of BERTimbau base and finetuning dataset, the model overfitted before to reach the end of training. Here are the overall final metrics on the validation dataset (*note: see the paragraph "Validation metrics by Named Entity" to get detailed metrics*):
  - **f1**: 0.8716487228203504
  - **precision**: 0.8559286898839138
  - **recall**: 0.8879569892473118
  - **accuracy**: 0.9755893153732458
  - **loss**: 0.1133928969502449
  
## Widget & APP

You can test this model into the widget of this page.

## Using the model for inference in production
````
# install pytorch: check https://pytorch.org/
# !pip install transformers 
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

# parameters
model_name = "ner-bert-base-portuguese-cased-lenebr"
model = AutoModelForTokenClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

input_text = "EMENTA: APELAÇÃO CÍVEL - AÇÃO DE INDENIZAÇÃO POR DANOS MORAIS - PRELIMINAR - ARGUIDA PELO MINISTÉRIO PÚBLICO EM GRAU RECURSAL - NULIDADE - AUSÊNCIA DE IN- TERVENÇÃO DO PARQUET NA INSTÂNCIA A QUO - PRESENÇA DE INCAPAZ - PREJUÍZO EXISTENTE - PRELIMINAR ACOLHIDA - NULIDADE RECONHECIDA."

# tokenization
inputs = tokenizer(input_text, max_length=512, truncation=True, return_tensors="pt")
tokens = inputs.tokens()

# get predictions
outputs = model(**inputs).logits
predictions = torch.argmax(outputs, dim=2)

# print predictions
for token, prediction in zip(tokens, predictions[0].numpy()):
    print((token, model.config.id2label[prediction]))
````
You can use pipeline, too. However, it seems to have an issue regarding to the max_length of the input sequence.
````
!pip install transformers
import transformers
from transformers import pipeline

model_name = "ner-bert-base-portuguese-cased-lenebr"

ner = pipeline(
    "ner",
    model=model_name
) 

ner(input_text)
````
## Training procedure

### Training results

````
Num examples = 7828
Num Epochs = 3
Instantaneous batch size per device = 8
Total train batch size (w. parallel, distributed & accumulation) = 8
Gradient Accumulation steps = 1
Total optimization steps = 2937

Step	Training Loss	Validation Loss	Precision	Recall    	F1      	Accuracy
290 	0.315100     	0.141881       	0.764542 	0.709462  	0.735973	0.960550
580	 0.089100	     0.137700       	0.729155 	0.810538  	0.767695	0.959940
870	 0.071700	     0.122069       	0.780277 	0.872903  	0.823995	0.967955
1160	0.047500     	0.125950       	0.800312 	0.881720  	0.839046	0.968367
1450	0.034900     	0.129228       	0.763666 	0.910323  	0.830570	0.969068
1740	0.036100     	0.113393       	0.855929 	0.887957  	0.871649	0.975589
2030	0.037800     	0.121275       	0.817230 	0.889462  	0.851818	0.970393
2320	0.018700     	0.115745       	0.836066 	0.877419  	0.856243	0.973136
2610	0.017100     	0.118826       	0.822488 	0.888817  	0.854367	0.973471
````

### Validation metrics by Named Entity
````
Num examples = 1177

{'JURISPRUDENCIA': {'f1': 0.6641509433962263,
  'number': 657,
  'precision': 0.6586826347305389,
  'recall': 0.669710806697108},
 'LEGISLACAO': {'f1': 0.8489082969432314,
  'number': 571,
  'precision': 0.8466898954703833,
  'recall': 0.851138353765324},
 'LOCAL': {'f1': 0.8066037735849058,
  'number': 194,
  'precision': 0.7434782608695653,
  'recall': 0.8814432989690721},
 'ORGANIZACAO': {'f1': 0.8540462427745664,
  'number': 1340,
  'precision': 0.8277310924369747,
  'recall': 0.8820895522388059},
 'PESSOA': {'f1': 0.9845722300140253,
  'number': 1072,
  'precision': 0.9868791002811621,
  'recall': 0.9822761194029851},
 'TEMPO': {'f1': 0.9527794381350867,
  'number': 816,
  'precision': 0.9299883313885647,
  'recall': 0.9767156862745098},
 'overall_accuracy': 0.9755893153732458,
 'overall_f1': 0.8716487228203504,
 'overall_precision': 0.8559286898839138,
 'overall_recall': 0.8879569892473118}
 ````