skoda/Vanity-SBU-Mistral-7B-Instruct-v0.2

Model Details

This model was fine-tuned on a curated dataset of 10,000 license plate examples, including vanity and standard plates, to classify plates as vanity or non-vanity and assess their acceptance or rejection. It leverages Chain of Thought (CoT) reasoning for better interpretability of complex and nuanced semantic patterns in vanity plates. The training utilized the PEFT library with Low-Rank Adaptation (LoRA) for efficient memory and computation.

Model Description

• Developed by: Anonymous Author(s)
• Model type: Text classification and interpretation
• Language(s) (NLP): English
• Finetuned from model [optional]: mistralai/Mistral-7B-Instruct-v0.2

Model Sources [optional]

• Repository: https://drive.google.com/drive/folders/1_12PC4b5PdDW4Dv0rF1mBDx7U2Q-QYIW
• Paper [optional]: https://drive.google.com/drive/folders/1Pm3dU1B_Ya6qRCXQNTE4ZcF0SGOYXZ-o?usp=sharing

Uses

The model is intended for research and development, focusing on the semantic and contextual interpretation of vanity license plates. It can classify plates, determine their vanity status, and generate explanations for approval or rejection decisions.

Training Details

Training Data

The training dataset includes 10,000 license plates, categorized into accepted and rejected vanity plates, along with standard plates. Semantic features like Named Entity Recognition (NER) and lexical features were integrated for contextual insights. Dataset link - https://drive.google.com/drive/folders/1_12PC4b5PdDW4Dv0rF1mBDx7U2Q-QYIW

Training Procedure

The model was fine-tuned using Low-Rank Adaptation (LoRA) with NF4 quantization for memory efficiency and gradient checkpointing to reduce resource usage. The training employed the Paged AdamW optimizer with a learning rate of 2e-4, batch size of 4, and ran for 5 epochs on NVIDIA GPUs in Google Colab. The process focused on minimizing the causal language modeling loss, enabling effective handling of nuanced semantic tasks.

Training Hyperparameters

• Training regime: Mixed precision (fp16)
• Quantization: NF4 with gradient checkpointing
• Optimizer: Paged AdamW
• Learning rate: 2e-4
• Batch size: 4
• Epochs: 5

Evaluation

Testing Data, Factors & Metrics

Testing Data

20% of the annotated dataset was used as the test set, alongside evaluations on external datasets like the NY License Plate dataset.

Metrics

Metrics included accuracy, precision, recall, and F1 score for classification. BLEU and ROUGE scores were used to evaluate interpretative explanations.

Results

• Vanity classification accuracy: 78.32%
• Approval classification accuracy: 75.18%
• BLEU for explanations: 60.25

Framework versions

• PEFT 0.13.2