TOSBert
TOSBert is a fine-tuned BERT model for sequence classification tasks. It is trained on a custom dataset for multi-label classification.
Model Details
- Model Name: TOSBert
- Model Architecture: BERT
- Framework: Hugging Face Transformers
- Model Type: Sequence Classification (Multi-label Classification)
Dataset
The model is trained on the online_terms_of_service dataset hosted on Hugging Face. This dataset consists of text sequences extracted from various online terms of service documents. Each sequence is labeled with multiple categories related to legal and privacy terms.
Training
The model was fine-tuned using the following parameters:
- Number of Epochs: 3
- Batch Size: 16 (both for training and evaluation)
- Warmup Steps: 500
- Weight Decay: 0.01
- Learning Rate: Automatically adjusted
Usage
Installation
To use this model, you need to install the transformers
library from Hugging Face:
pip install transformers
Loading the Model
You can load the model using the following code:
from transformers import BertForSequenceClassification, BertTokenizer
model_name = "CodeHima/TOSBert"
model = BertForSequenceClassification.from_pretrained(model_name)
tokenizer = BertTokenizer.from_pretrained(model_name)
Inference
Here is an example of how to use the model for inference:
from transformers import pipeline
classifier = pipeline("text-classification", model=model, tokenizer=tokenizer, return_all_scores=True)
text = "Your input text here"
predictions = classifier(text)
print(predictions)
Training Script
Below is an example script used for training the model:
from transformers import Trainer, TrainingArguments, BertForSequenceClassification, BertTokenizer
import torch
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
# Define the model
model_name = "bert-base-uncased"
model = BertForSequenceClassification.from_pretrained(model_name, num_labels=3)
# Define the tokenizer
tokenizer = BertTokenizer.from_pretrained(model_name)
# Load your dataset
# train_dataset and eval_dataset should be instances of torch.utils.data.Dataset
# Example: train_dataset = YourDataset(train_data)
# Define training arguments
training_args = TrainingArguments(
output_dir='./results',
num_train_epochs=3,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
warmup_steps=500,
weight_decay=0.01,
logging_dir='./logs',
logging_steps=10,
eval_strategy="epoch"
)
# Custom data collator to convert labels to floats
def data_collator(features):
batch = {}
first = features[0]
if 'label' in first and first['label'] is not None:
dtype = torch.float32
batch['labels'] = torch.tensor([f['label'] for f in features], dtype=dtype)
for k, v in first.items():
if k != 'label' and v is not None and not isinstance(v, str):
batch[k] = torch.stack([f[k] for f in features])
return batch
# Define the compute metrics function
def compute_metrics(pred):
labels = pred.label_ids
preds = (pred.predictions > 0.5).astype(int)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='micro')
acc = accuracy_score(labels, preds)
return {
'accuracy': acc,
'f1': f1,
'precision': precision,
'recall': recall
}
# Initialize the Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
compute_metrics=compute_metrics,
data_collator=data_collator
)
# Train the model
trainer.train()
Evaluation
To evaluate the model on the validation set, you can use the following code:
results = trainer.evaluate()
print(results)
License
This project is licensed under the MIT License. See the LICENSE file for more details.
Citation
If you use this model in your research, please cite it as follows:
@misc{TOSBert,
author = {Himanshu Mohanty},
title = {TOSBert: Fine-tuned BERT model for multi-label classification},
year = {2024},
publisher = {Hugging Face},
journal = {Hugging Face Model Hub},
howpublished = {\url{https://huggingface.co/CodeHima/TOSBert}}
}
Acknowledgements
This project uses the Hugging Face Transformers library. Special thanks to the developers and contributors of this library.
- Downloads last month
- 15