---
pipeline_tag: fill-mask
widget:
 - text: "đậu xanh rau <mask>"
---
# <a name="introduction"></a> ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing (EMNLP 2023 - Main)
**Disclaimer**: The paper contains actual comments on social networks that might be construed as abusive, offensive, or obscene.

ViSoBERT is the state-of-the-art language model for Vietnamese social media tasks:

 - ViSoBERT is the first monolingual MLM (XLM-R architecture) from scratch specifically for Vietnamese social media text.
 - ViSoBERT outperforms previous monolingual, multilingual, and multilingual social media approaches, obtaining new state-of-the-art performances on four downstream Vietnamese social media tasks.

The general architecture and experimental results of ViSoBERT can be found in our [paper](https://openreview.net/forum?id=gqkg54QNDY):

    @inproceedings{
    anonymous2023plmvismt,
    title={{PLM}4Vi{SMT}: A Pre-Trained Language Model for Vietnamese Social Media Text Processing},
    author={Anonymous},
    booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
    year={2023},
    url={https://openreview.net/forum?id=gqkg54QNDY}
    }
    

**Please CITE** our paper when ViSoBERT is used to help produce published results or is incorporated into other software.

**Installation** Install `transformers` with pip: `pip install transformers` and `SentencePiece` with `pip install SentencePiece`

**Example usage**
from transformers import AutoModel,AutoTokenizer
import torch

model= AutoModel.from_pretrained('uitnlp/visobert')
tokenizer = AutoTokenizer.from_pretrained('uitnlp/visobert')

encoding = tokenizer('dau xanh rau ma',return_tensors='pt')

with torch.no_grad():
  output = model(**encoding)