|
--- |
|
license: llama3.1 |
|
language: |
|
- de |
|
- en |
|
- es |
|
- fr |
|
- it |
|
tags: |
|
- text-generation-inference |
|
base_model: |
|
- meta-llama/Llama-3.1-8B-Instruct |
|
--- |
|
# Model Card for LLM-wsd-TT-10000 |
|
|
|
## Model description |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
**LLM-wsd-TT-10000** is a *Large Language Model (LLM)* instruction-tuned over **meta-llama/Meta-Llama-3.1-8B-Instruct**. |
|
This model has been trained for the **WSD** task over a balanced training dataset (10000 instances per language), with machine-translation. It is capable of providing the definition of a word in a given sentence. Specifically, it can answer both: |
|
1) **Open-ended questions**, where the model will generate the definition of the target word; |
|
2) **Closed-ended questions**, where the model will generate the identifier of the correct option out of a list of alternatives. |
|
|
|
More details regarding the training procedure (e.g. hyperparameters, dataset construction, and so on) can be found in Section 4.2 of the [paper](https://arxiv.org/abs/2503.08662). |
|
|
|
- **Developed by:** Pierpaolo Basile, Lucia Siciliani, Elio Musacchio |
|
- **Model type:** LLaMA 3.1 Instruct |
|
- **Language(s) (NLP):** English, French, German, Italian and Spanish |
|
- **License:** [LLAMA 3.1 COMMUNITY LICENSE AGREEMENT](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct/blob/main/LICENSE) |
|
- **Finetuned from model:** [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) |
|
|
|
## Prompt Format |
|
|
|
The model has been trained using several instructions depending on language, task (open-ended or closed-ended) and number of occurences of target word in the sentence. In [Instructions](#instructions), we provide the instructions used for all cases. The following placeholder variables have to be replaced: |
|
- {target_word}: the target word in the input to disambiguate; |
|
- {options}: options to provide to the model for the closed-ended task only. The options should be newline separated and each option should be identified by a number. Refer to the [closed-ended example](#closed-ended) for an example of options formatting; |
|
- {occurrence}: the ordinal number of the {target_word} occurrence (e.g. "second"). This is required only when the input sentence contains multiple occurrences of {target_word}. |
|
|
|
Please note that the complete prompt also has the following string after the instruction: |
|
|
|
```python |
|
" Input: \"{sentence}\"" |
|
``` |
|
|
|
where {sentence} is the input sentence containing the word to disambiguate. |
|
|
|
## How to Get Started with the Model |
|
|
|
Below you can find two examples of model usage, for open-ended and closed-ended generation respectively. |
|
|
|
### Open-ended |
|
|
|
```python |
|
import torch |
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
from transformers.trainer_utils import set_seed |
|
|
|
target_word = "long" |
|
instruction = f"Give a brief definition of the word \"{target_word}\" in the sentence given as input. Generate only the definition." |
|
input_sentence = "How long has it been since you reviewed the objectives of your benefit and service program?" |
|
|
|
model_id = "swap-uniba/LLM-wsd-TT-10000" |
|
|
|
set_seed(42) |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False) |
|
|
|
tokenizer.padding_side = "left" |
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_id, |
|
device_map='cuda', |
|
torch_dtype=torch.bfloat16, |
|
).eval() |
|
|
|
terminators = [ |
|
tokenizer.eos_token_id, |
|
tokenizer.convert_tokens_to_ids("<|eot_id|>") |
|
] |
|
|
|
messages = [ |
|
{"role": "user", "content": instruction + " Input: \"" + input_sentence + "\""}, |
|
] |
|
|
|
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt") |
|
|
|
outputs = model.generate( |
|
input_ids.to('cuda'), |
|
max_new_tokens=512, |
|
eos_token_id=terminators, |
|
num_beams=1, |
|
do_sample=False |
|
) |
|
|
|
print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)) |
|
``` |
|
|
|
### Closed-ended |
|
|
|
```python |
|
import torch |
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
from transformers.trainer_utils import set_seed |
|
|
|
target_word = "hurry" |
|
instruction = f"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n1) Move very fast\n2) Urge to an unnatural speed\n\nGenerate only the number of the selected option." |
|
input_sentence = "If you hurry you might beat the headquarters boys." |
|
|
|
model_id = "swap-uniba/LLM-wsd-TT-10000" |
|
|
|
set_seed(42) |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False) |
|
|
|
tokenizer.padding_side = "left" |
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_id, |
|
device_map='cuda', |
|
torch_dtype=torch.bfloat16, |
|
).eval() |
|
|
|
terminators = [ |
|
tokenizer.eos_token_id, |
|
tokenizer.convert_tokens_to_ids("<|eot_id|>") |
|
] |
|
|
|
messages = [ |
|
{"role": "user", "content": instruction + " Input: \"" + input_sentence + "\""}, |
|
] |
|
|
|
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt") |
|
|
|
outputs = model.generate( |
|
input_ids.to('cuda'), |
|
max_new_tokens=512, |
|
eos_token_id=terminators, |
|
num_beams=1, |
|
do_sample=False |
|
) |
|
|
|
print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)) |
|
``` |
|
|
|
## Citation |
|
|
|
If you use this model in your research, please cite the following: |
|
|
|
```bibtex |
|
@misc{basile2025exploringwordsensedisambiguation, |
|
title={Exploring the Word Sense Disambiguation Capabilities of Large Language Models}, |
|
author={Pierpaolo Basile and Lucia Siciliani and Elio Musacchio and Giovanni Semeraro}, |
|
year={2025}, |
|
eprint={2503.08662}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2503.08662}, |
|
} |
|
``` |
|
|
|
## Instructions |
|
|
|
### Single occurrence of target word (open-ended) |
|
|
|
#### English |
|
```python |
|
"Give a brief definition of the word \"{target_word}\" in the sentence given as input. Generate only the definition." |
|
``` |
|
|
|
#### French |
|
```python |
|
"Donnez une brève définition du mot \"{target_word}\" dans la phrase d’entrée donnée. Ne donnez que la définition." |
|
``` |
|
|
|
#### German |
|
```python |
|
"Geben Sie eine kurze Definition des Wortes \"{target_word}\" in dem gegebenen Satz an. Erzeugen Sie nur die Definition." |
|
``` |
|
|
|
#### Italian |
|
```python |
|
"Fornisci una breve definizione della parola \"{target_word}\" nella frase data in input. Genera solo la definizione." |
|
``` |
|
|
|
#### Spanish |
|
```python |
|
"Proporciona una definición breve de la palabra \"{target_word}\" en la frase dada en entrada. Genera solo la definición." |
|
``` |
|
|
|
### Multiple occurences of target word (open-ended) |
|
|
|
#### English |
|
```python |
|
"Give a brief definition of the {occurrence} occurrence of the word \"{target_word}\" in the sentence given as input. Generate only the definition." |
|
``` |
|
|
|
#### French |
|
```python |
|
"Donnez une brève définition de l'occurrence {occurrence} du mot \"{target_word}\" dans la phrase d’entrée donnée. Ne donnez que la définition." |
|
``` |
|
|
|
#### German |
|
```python |
|
"Geben Sie eine kurze Definition des {occurrence} Vorkommens des Wortes \"{target_word}\" in dem gegebenen Eingabesatz an. Erzeugen Sie nur die Definition." |
|
``` |
|
|
|
#### Italian |
|
```python |
|
"Fornisci una breve definizione della {occurrence} occorrenza della parola \"{target_word}\" nella frase data in input. Genera solo la definizione." |
|
``` |
|
|
|
#### Spanish |
|
```python |
|
"Proporciona una definición breve de la {occurrence} ocurrencia de la palabra \"{target_word}\" en la frase dada en entrada. Genera solo la definición." |
|
``` |
|
|
|
### Single occurrence of target word (closed-ended) |
|
|
|
#### English |
|
```python |
|
"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n{options}\n\nGenerate only the number of the selected option." |
|
``` |
|
|
|
#### French |
|
```python |
|
"Étant donné le mot \"{target_word}\" dans la phrase saisie, choisissez la signification correcte parmi les suivantes:\n{options}\n\nNe donnez que le numéro de l’option sélectionnée." |
|
``` |
|
|
|
#### German |
|
```python |
|
"Wählen Sie für das Wort \"{target_word}\" im Eingabesatz die richtige Bedeutung aus den folgenden Angaben:\n{options}\n\nErzeugt nur die Nummer der ausgewählten Option" |
|
``` |
|
|
|
#### Italian |
|
```python |
|
"Data la parola \"{target_word}\" nella frase in input, scegli il significato corretto tra i seguenti:\n{options}\n\nGenera solo il numero dell'opzione selezionata." |
|
``` |
|
|
|
#### Spanish |
|
```python |
|
"Dada la palabra \"{target_word}\" en la frase de entrada, elija el significado correcto entre los siguientes:\n{options}\n\nGenera solo el número de la opción seleccionada." |
|
``` |
|
|
|
### Multiple occurrences of target word (closed-ended) |
|
|
|
#### English |
|
```python |
|
"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n{options}\n\nGenerate only the number of the selected option." |
|
``` |
|
|
|
#### French |
|
```python |
|
"Étant donné l'occurrence {occurrence} du mot \"{target_word}\" dans la phrase d'entrée, choisissez la signification correcte parmi les suivantes:\n{options}\n\nNe donnez que le numéro de l’option sélectionnée." |
|
``` |
|
|
|
#### German |
|
```python |
|
"Wählen Sie angesichts des {occurrence} Vorkommens des Wortes \"{target_word}\" im Eingabesatz die richtige Bedeutung aus der folgenden Liste aus:\n{options}\n\nErzeugt nur die Nummer der ausgewählten Option." |
|
``` |
|
|
|
#### Italian |
|
```python |
|
"Data la {occurrence} occorrenza della parola \"{target_word}\" nella frase in input, scegli il significato corretto tra i seguenti:\n{options}\n\nGenera solo il numero dell'opzione selezionata." |
|
``` |
|
|
|
#### Spanish |
|
```python |
|
"Dada la {occurrence} ocurrencia de la palabra \"{target_word}\" en la frase de entrada, elije el significado correcto entre los siguientes:\n{options}\n\nGenera solo el número de la opción seleccionada." |
|
``` |