Royal-lobster/Slither-LLM-Auditor-LoRA-Adapter

Slither Auditor

This model is a fine-tuned version of Phind/Phind-CodeLlama-34B-v2 on the Royal-lobster/Slither-Audited-Solidity-QA dataset.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 6
• gradient_accumulation_steps: 3
• total_train_batch_size: 18
• total_eval_batch_size: 6
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
0.2069	0.1	20	nan
0.0986	0.21	40	nan
0.1101	0.31	60	nan
0.072	0.41	80	nan
0.1258	0.52	100	nan
0.0675	0.62	120	nan
0.0728	0.72	140	nan
0.115	0.83	160	nan
0.0769	0.93	180	nan
0.0609	1.03	200	nan
0.0881	1.14	220	nan
0.0674	1.24	240	nan
0.0476	1.34	260	nan
0.0259	1.45	280	nan
0.0534	1.55	300	nan
0.0449	1.65	320	nan
0.0325	1.76	340	nan
0.03	1.86	360	nan
0.0416	1.96	380	nan

Framework versions

• Transformers 4.35.0.dev0
• Pytorch 2.0.1+cu118
• Datasets 2.14.5
• Tokenizers 0.14.1