| | --- |
| | language: |
| | - en |
| | base_model: |
| | - FacebookAI/roberta-large |
| | pipeline_tag: text-classification |
| | --- |
| | # Graded Word Sense Disambiguation (WSD) Model |
| |
|
| | ## Model Summary |
| | This model is a **fine-tuned version of RoBERTa-Large** for **Graded Word Sense Disambiguation (WSD)**. It is designed to predict the **degree of applicability** (1-4) of a word sense in context by leveraging **large-scale sense-annotated corpora**. The model is based on the work outlined in: |
| |
|
| | **Reference Paper:** |
| | Pierluigi Cassotti, Nina Tahmasebi (2025). Sense-specific Historical Word Usage Generation. |
| |
|
| |
|
| | This model has been trained to handle **graded WSD tasks**, providing **continuous-valued predictions** instead of hard classification, making it useful for nuanced applications in lexicography, computational linguistics, and historical text analysis. |
| |
|
| | --- |
| |
|
| | ## Model Details |
| | - **Base Model:** `roberta-large` |
| | - **Task:** Graded Word Sense Disambiguation (WSD) |
| | - **Fine-tuning Dataset:** Oxford English Dictionary (OED) sense-annotated corpus |
| | - **Training Steps:** |
| | - Tokenizer augmented with special tokens (`<t>`, `</t>`) for marking target words in context. |
| | - Dataset preprocessed with **sense annotations** and **word offsets**. |
| | - Sentences containing sense-annotated words were split into **train (90%)** and **validation (10%)** sets. |
| | - **Objective:** Predicting a continuous label representing the applicability of a sense. |
| | - **Evaluation Metric:** Root Mean Squared Error (RMSE). |
| | - **Batch Size:** 32 |
| | - **Learning Rate:** 2e-5 |
| | - **Epochs:** 1 |
| | - **Optimizer:** AdamW with weight decay of 0.01 |
| | - **Evaluation Strategy:** Steps-based (every 10% of the dataset). |
| |
|
| | --- |
| |
|
| | ## Training & Fine-Tuning |
| | Fine-tuning was performed using the **Hugging Face `Trainer` API** with a **custom dataset loader**. The dataset was processed as follows: |
| |
|
| | 1. **Preprocessing** |
| | - Example sentences were extracted from the OED and augmented with **definitions**. |
| | - The target word was **highlighted** with special tokens (`<t>`, `</t>`). |
| | - Each instance was labeled with a **graded similarity score**. |
| |
|
| | 2. **Tokenization & Encoding** |
| | - Tokenized with `AutoTokenizer.from_pretrained("roberta-large")`. |
| | - Definitions were concatenated using the `</s></s>` separator for **cross-sentence representation**. |
| |
|
| | 3. **Training Pipeline** |
| | - Model fine-tuned on the **regression task** with a single **linear output head**. |
| | - Used **Mean Squared Error (MSE) loss**. |
| | - Evaluation on validation set using **Root Mean Squared Error (RMSE)**. |
| |
|
| | --- |
| |
|
| | ## Usage |
| | ### Example Code |
| | ```python |
| | from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| | import torch |
| | |
| | tokenizer = AutoTokenizer.from_pretrained("ChangeIsKey/graded-wsd") |
| | model = AutoModelForSequenceClassification.from_pretrained("ChangeIsKey/graded-wsd") |
| | |
| | sentence = "The <t>bank</t> of the river was eroding due to the storm." |
| | target_word = "bank" |
| | definition = "The land alongside a river or a stream." |
| | |
| | tokenized_input = tokenizer(f"{sentence} </s></s> {definition}", truncation=True, padding=True, return_tensors="pt") |
| | with torch.no_grad(): |
| | output = model(**tokenized_input) |
| | score = output.logits.item() |
| | |
| | print(f"Graded Sense Score: {score}") |
| | ``` |
| |
|
| | ### Input Format |
| | - Sentence: Contextual usage of the word. |
| | - Target Word: The word to be disambiguated. |
| | - Definition: The dictionary definition of the intended sense. |
| |
|
| | ### Output |
| | - **A continuous score** (between 1 and 4) indicating the **similarity** of the given definition with respect to the word in its current context. |
| |
|
| | --- |
| |
|
| | ## Citation |
| | If you use this model, please cite the following paper: |
| |
|
| | ``` |
| | @article{10.1162/tacl_a_00761, |
| | author = {Cassotti, Pierluigi and Tahmasebi, Nina}, |
| | title = {Sense-specific Historical Word Usage Generation}, |
| | journal = {Transactions of the Association for Computational Linguistics}, |
| | volume = {13}, |
| | pages = {690-708}, |
| | year = {2025}, |
| | month = {07}, |
| | abstract = {Large-scale sense-annotated corpora are important for a range of tasks but are hard to come by. Dictionaries that record and describe the vocabulary of a language often offer a small set of real-world example sentences for each sense of a word. However, on their own, these sentences are too few to be used as diachronic sense-annotated corpora. We propose a targeted strategy for training and evaluating generative models producing historically and semantically accurate word usages given any word, sense definition, and year triple. Our results demonstrate that fine-tuned models can generate usages with the same properties as real-world example sentences from a reference dictionary. Thus the generated usages will be suitable for training and testing computational models where large-scale sense-annotated corpora are needed but currently unavailable.}, |
| | issn = {2307-387X}, |
| | doi = {10.1162/tacl_a_00761}, |
| | url = {https://doi.org/10.1162/tacl\_a\_00761}, |
| | eprint = {https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl\_a\_00761/2535111/tacl\_a\_00761.pdf}, |
| | } |
| | ``` |
