llm_firewall_distilbert-base-uncased

This model was trained from scratch on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1218
• Accuracy: 0.9451

Latest finetune 5 Dec 2023

{'eval_loss': 0.12179878354072571, 'eval_accuracy': 0.9450980392156862, 'eval_runtime': 5.8053, 'eval_samples_per_second': 43.925, 'eval_steps_per_second': 2.756, 'epoch': 20.0}

Model description

Finetuned distilbert-uncased on prompts that are either malicious or benign.

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.3191	1.0	64	0.5996	0.7255
0.5065	2.0	128	0.4536	0.8
0.4134	3.0	192	0.3856	0.8275
0.3294	4.0	256	0.2654	0.8824
0.2536	5.0	320	0.1977	0.9216
0.2001	6.0	384	0.1671	0.9412
0.2144	7.0	448	0.1670	0.9373
0.2017	8.0	512	0.1575	0.9333
0.1819	9.0	576	0.1866	0.9294
0.143	10.0	640	0.1834	0.9373
0.153	11.0	704	0.1589	0.9412
0.1469	12.0	768	0.1347	0.9451
0.1568	13.0	832	0.1425	0.9451
0.139	14.0	896	0.1438	0.9451
0.1889	15.0	960	0.1330	0.9451
0.1185	16.0	1024	0.1323	0.9451
0.1166	17.0	1088	0.1280	0.9451
0.1475	18.0	1152	0.1233	0.9451
0.1145	19.0	1216	0.1225	0.9451
0.1121	20.0	1280	0.1218	0.9451

Framework versions

• Transformers 4.35.2
• Pytorch 2.1.1
• Datasets 2.15.0
• Tokenizers 0.15.0