Create README.md
Browse files# parseny/TinyLlama1.1B-Nvidia-QA
This repository contains the parseny/TinyLlama1.1B-Nvidia-QA model, a fine-tuned version of the TinyLlama language model designed for generating coherent and contextually relevant responses in a chat-based environment for NVIDIA documentation questions. The model was fine-tuned on a [dataset of question-answer pairs](https://www.kaggle.com/datasets/gondimalladeepesh/nvidia-documentation-question-and-answer-pairs) and evaluated using several metrics to ensure high performance.
## Model Details
- **Model ID**: parseny/TinyLlama1.1B-Nvidia-QA
- **Model Type**: Causal Language Model
- **Base Model**: TinyLlama-1.1B
- **Quantization**: 4-bit quantization using BitsAndBytes
- **Fine-Tuning Framework**: Hugging Face Transformers and PEFT
## Training Configuration
The model was fine-tuned with the following training arguments:
```python
training_arguments = TrainingArguments(
output_dir="./logs",
per_device_train_batch_size=16,
gradient_accumulation_steps=4,
optim="paged_adamw_32bit",
fp16=True,
evaluation_strategy="epoch",
save_strategy="epoch",
num_train_epochs=5,
load_best_model_at_end=True,
learning_rate=5e-4
)
```
## Evaluation Metrics
The performance of the fine-tuned model was evaluated using the following metrics:
- **ROUGE Scores**:
- **ROUGE-1**: 0.3122
- **ROUGE-2**: 0.1228
- **ROUGE-L**: 0.2599
- **ROUGE-Lsum**: 0.2600
- **METEOR Score**: 0.27
These scores indicate that the model performs reasonably well in generating responses that are lexically and semantically similar to the reference answers.
## Model Usage
You can use this model to generate responses for chat-based applications. Below is an example of how to load and use the model for generating responses:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel, PeftConfig
import torch
# Load the model and tokenizer
model_name = "parseny/TinyLlama1.1B-Nvidia-QA"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
model.to('cuda')
# Generate a response
generation_config = GenerationConfig(
penalty_alpha=0.6, do_sample=True,
top_k=5, temperature=0.5, repetition_penalty=1.2,
max_new_tokens=47, pad_token_id=tokenizer.eos_token_id
)
def generate_response(prompt):
try:
inputs = tokenizer(prompt, return_tensors="pt").to('cuda')
outputs = model.generate(**inputs, generation_config=generation_config)
generated_response = tokenizer.decode(outputs[0], skip_special_tokens=True)
start_idx = generated_response.find('<|im_start|>assistant\n') + len('<|im_start|>assistant\n')
generated_response = generated_response[start_idx:]
end_idx = generated_response.find('<|im_end|>')
generated_response = generated_response[:end_idx]
return generated_response
except:
return ""
# Example usage
prompt = "What was the purpose of setting up the DGX RAID memory in version 2 of the pipeline?"
response = generate_response(prompt)
print(response)
```
## Training Procedure
The model was fine-tuned using a dataset of question-answer pairs. The fine-tuning process involved:
1. Loading the pre-trained TinyLlama-1.1B model.
2. Quantizing the model to 4-bit precision to reduce memory usage and increase inference speed.
3. Fine-tuning the model using the `SFTTrainer` with the specified training arguments.
4. Evaluating the model at the end of each epoch and saving the best-performing model.
## How to Cite
If you use this model in your research or applications, please cite it as follows:
```
@misc
{TinyLlama1.1B-Nvidia-QA,
author = {parseny},
title = {TinyLlama1.1B-Nvidia-QA: NVIDIA documentation helper},
year = {2024},
publisher = {Hugging Face},
url = {https://huggingface.co/parseny/TinyLlama1.1B-Nvidia-QA},
}
```
## Contact
For any questions or issues, please open an issue on the Hugging Face model repository.