README.md

---
language: 
  - en
  - ja
  - pt
  - es
  - ko
  - ar
  - tr
  - th
  - fr
  - id
  - ru
  - de
  - fa
  - it
  - zh
  - pl
  - hi
  - ur
  - nl
  - el
  - ms
  - ca
  - sr
  - sv
  - uk
  - he
  - fi
  - cs
  - ta
  - ne
  - vi
  - hu
  - eo
  - bn
  - mr
  - ml
  - hr
  - no
  - sw
  - sl
  - te
  - az
  - da
  - ro
  - gl
  - gu
  - ps
  - mk
  - kn
  - bg
  - lv
  - eu
  - pa
  - et
  - mn
  - sq
  - si
  - sd
  - la
  - is
  - jv
  - lt
  - ku
  - am
  - bs
  - hy
  - or
  - sk
  - uz
  - cy
  - my
  - su
  - br
  - as
  - af
  - be
  - fy
  - kk
  - ga
  - lo
  - ka
  - km
  - sa
  - mg
  - so
  - ug
  - ky
  - gd
  - yi
tags:
  - Twitter
  - Multilingual
license: "apache-2.0"
mask_token: "<mask>"
---

# TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-green.svg?style=flat-square)](http://makeapullrequest.com)
[![arXiv](https://img.shields.io/badge/arXiv-2203.15827-b31b1b.svg)](https://arxiv.org/abs/2209.07562)


This repo contains models, code and pointers to datasets from our paper: [TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations](https://arxiv.org/abs/2209.07562).
[[PDF]](https://arxiv.org/pdf/2209.07562.pdf)
[[HuggingFace Models]](https://huggingface.co/Twitter)

### Overview
TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).

TwHIN-BERT can be used as a drop-in replacement for BERT in a variety of NLP and recommendation tasks. It not only outperforms similar models semantic understanding tasks such text classification), but also **social recommendation** tasks such as predicting user to Tweet engagement.

## 1. Pretrained Models

We initially release two pretrained TwHIN-BERT models (base and large) that are compatible wit the [HuggingFace BERT models](https://github.com/huggingface/transformers).


| Model | Size | Download Link (🤗 HuggingFace) |
| ------------- | ------------- | --------- |
| TwHIN-BERT-base   | 280M parameters | [Twitter/TwHIN-BERT-base](https://huggingface.co/Twitter/twhin-bert-base) |
| TwHIN-BERT-large  | 550M parameters | [Twitter/TwHIN-BERT-large](https://huggingface.co/Twitter/twhin-bert-large) |


To use these models in 🤗 Transformers:
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('Twitter/twhin-bert-large')
model = AutoModel.from_pretrained('Twitter/twhin-bert-large')
inputs = tokenizer("I'm using TwHIN-BERT! #TwHIN-BERT #NLP", return_tensors="pt")
outputs = model(**inputs)
```



<!-- ## 2. Set up environment and data
### Environment
TBD


## 3. Fine-tune TwHIN-BERT

TBD -->


## Citation
If you use TwHIN-BERT or out datasets in your work, please cite the following:
```bib
@article{zhang2022twhin,
  title={TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations},
  author={Zhang, Xinyang and Malkov, Yury and Florez, Omar and Park, Serim and McWilliams, Brian and Han, Jiawei and El-Kishky, Ahmed},
  journal={arXiv preprint arXiv:2209.07562},
  year={2022}
}
```