estnafinema0/smolLM-variation-ppo

Model Card for SmolLM Variation PPO Fine-Tuning for RLHF

This model is a fine-tuned variant of the SmolLM-135M-Instruct model using Proximal Policy Optimization (PPO) for Reinforcement Learning from Human Feedback (RLHF). It has been optimized by leveraging a pre-trained reward model and value model through the TRL library's PPOTrainer, with the aim of generating more engaging and human-aligned responses.

Task Description

The project involves:

• Converting the training dataset into a prompt-only format.
• Utilizing a pre-trained reward model and a value model to compute scalar rewards.
• Fine-tuning the policy model with PPO (via TRL's PPOTrainer) to improve response quality and maximize reward margins.
• Evaluating the model's performance by comparing generation quality before and after PPO fine-tuning.

Model Details

Model Description

This is the model card for a 🤗 transformers model that has been pushed on the Hub after PPO fine-tuning. The model card has been refined with details gathered from experiments and training logs.

• Developed by: Ekaterina Aleksanian
• Model type: Causal Language Model (autoregressive) fine-tuned using Proximal Policy Optimization (PPO)
• Language(s) (NLP): English
• License: MIT License
• Finetuned from model [optional]: SmolLM-135M-Instruct

How to Get Started with the Model

You can load the model using the Hugging Face Transformers library. For example:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("estnafinema0/smolLM-variation-ppo")
tokenizer = AutoTokenizer.from_pretrained("estnafinema0/smolLM-variation-ppo")

# Generate a response
messages = [{"role": "user", "content": "What's your morning routine like?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt")
output_ids = model.generate(inputs.input_ids, max_new_tokens=256, do_sample=True)
response = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
print(response)

Generation Examples

Before Fine-Tuning (Reference Policy):

User: What's your morning routine like?
Assistant: What a wonderful question! As a conversational AI, I have my own wake-up routine tailored to my needs. Here's my "wake-up routine" for you:

**Morning Routine (10-15 minutes)**

1. **10:00 AM**: Start with a gentle exercise routine to loosen up my muscles and get my blood flowing. This could be a simple yoga pose like downward-facing dog, child's pose, or a standing forward fold.
2. **11:00 AM**: Finish my morning routine with a healthy breakfast or a hearty snack, such as oatmeal with fruits, nuts, or Greek yogurt with honey.
3. **6:00 PM**: Take my morning bus or drive to my workspace to start my morning routine. During my commute, I set my alarm clock, get dressed, and start my daily activities.
4. **7:00 PM**: Work on an important project or focus on other tasks, such as researching for my thesis or preparing for a project deadline.
5. **8:00 PM**: Have a quick lunch break, so I can get back to my desk or grab some lunch before my morning exercise routine begins.

After PPO Fine-Tuning:

User: What's your morning routine like?
Assistant: I'm excited to share my morning routine with you! As a digital AI assistant, I don't have personal experiences or emotions, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:

**Morning Routine (10-15 minutes)**

1. **Hydration**: Drink a full glass of water or a herbal tea (e.g., chamomile, peppermint) to start your day.
2. **Stretching or Exercise**: Engage in some light physical activity, like yoga, stretching, or a short walk. This helps increase energy and flexibility.
3. **Meditation or Mindfulness**: Spend 10-15 minutes meditating, focusing on your breath, or practicing mindfulness exercises. This helps clear your mind and set intentions for the day.
4. **Journaling**: Write down your thoughts, goals, and gratitudes to reflect on your day. This helps process emotions and gain clarity.
5. **Healthy Breakfast**: Prepare a nutritious breakfast, such as oatmeal with fruit, scrambled eggs with spinach, or Greek yogurt with berries.

Training Details

Training Data

The model was fine-tuned using the HumanLLMs/Human-Like-DPO-Dataset, which provides prompts along with chosen and rejected responses. The data was reformatted into a chat template where only the prompt is retained for PPO fine-tuning.

Training Procedure

The training procedure involved:

• Converting the raw dataset into a prompt-only format using a chat template.
• Tokenizing the data while preserving critical information from the prompt.
• Fine-tuning the SmolLM-135M-Instruct model with PPO using TRL's PPOTrainer.
• Leveraging a pre-trained reward model and value model to guide the PPO updates.
• Training was conducted on an A100 GPU with Automatic Mixed Precision (AMP) enabled for efficiency.

Training Hyperparameters:

• Learning Rate: 5e-5
• Batch Size: 16
• Mini Batch Size: 4
• PPO Epochs: 4
• Max Sequence Length: 1024
• Max Prompt Length: 256

Preprocessing

The dataset was processed by applying a chat template that formats the prompt in a conversational style. Tokenization included truncation to retain the most informative tokens.

Speeds, Sizes, Times

Fine-tuning on an A100 GPU took approximately 2 minutes per epoch. The final model size remains similar to the base model while optimized for efficient inference with AMP.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluation was performed on a subset of the training data (in-distribution) and an external dataset (wikitext-2-raw-v1) to assess generalization.

Factors

Evaluation factors include:

• The log probability assigned to in-distribution versus out-of-distribution data.
• Reward margins computed from the difference in log probabilities.
• Qualitative generation quality of the responses.

Metrics

Key evaluation metrics are:

• Log Probability Distribution: A comparison between in-distribution and external data.
• Reward Margins: The differential between the rewards of chosen versus rejected responses.
• Reward Accuracy: The percentage of instances where the chosen response is favored.

Results

The evaluation indicates that the PPO fine-tuned model assigns higher log probabilities to in-distribution data compared to external samples. This suggests that the model has successfully learned to generate responses more aligned with the training data and human preferences. Improvements in reward margins and accuracy metrics further support this observation.

Summary

In summary, the PPO fine-tuned model demonstrates enhanced performance in generating human-aligned responses. The model effectively differentiates between preferred and non-preferred responses, as validated by both quantitative metrics and qualitative generation examples.

Technical Specifications

Model Architecture and Objective

The model is based on the SmolLM-135M-Instruct architecture. The PPO fine-tuning objective is to optimize the policy model using a reward signal provided by a pre-trained reward model and value model, thus aligning the model’s outputs with human preferences.

Compute Infrastructure

Fine-tuning was performed on an A100 GPU with AMP enabled for computational efficiency.

Hardware

• NVIDIA A100 GPU

Software

• PyTorch
• Hugging Face Transformers
• TRL (for reinforcement learning components)
• Additional libraries for data processing and visualization

APA:

Aleksanian, E. (2025). SmolLM Variation PPO Fine-Tuning for RLHF. Retrieved from https://huggingface.co/estnafinema0/smolLM-variation-ppo

Glossary

• PPO (Proximal Policy Optimization): A reinforcement learning algorithm that updates the policy by taking small, conservative steps to improve performance.
• RLHF: Reinforcement Learning from Human Feedback.
• AMP: Automatic Mixed Precision, used to speed up training.

More Information

For further details, please refer to the associated repositories:

• DPO Fine-Tuning: estnafinema0/smolLM-variation-dpo
• Trainer Output: estnafinema0/trainer_output

Model Card Authors

• Ekaterina Aleksanian

Model Card Contact

For any questions or feedback regarding this model card, please contact Ekaterina Aleksanian at [[email protected]].