---
library_name: transformers
tags:
- code
- gemma-2b
- finetune
- qlora
license: apache-2.0
datasets:
- SaikatM/Code-Platypus
language:
- en
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
This model is a fine-tuned version of google/gemma-2b on an SaikatM/Code-Platypus dataset.



## Model Details

### Model Description

- **Finetuned from model:** google/gemma-2b

### Model Sources

Training code can be found here: https://github.com/Saikat-M/LLM-Finetuning

### Direct Use

* Code generation tasks

### Training Data
* Dataset: https://huggingface.co/datasets/SaikatM/Code-Platypus

* Source Dataset: https://huggingface.co/datasets/garage-bAInd/Open-Platypus

### Training Procedure

Used QLoRA from PEFT and used SFTTrainer.

#### Preprocessing 

From the Open-Platypus dataset filtering-out rows which has leetcode_ne in it's data_source column. 

#### Training Hyperparameters

LoraConfig(

    r=4,
    lora_alpha=2,
    target_modules=modules,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

TrainingArguments(

    output_dir="gemma-2b-code-platypus",      
    num_train_epochs=1,                    
    per_device_train_batch_size=4,            
    gradient_accumulation_steps=4,            
    gradient_checkpointing=True,              
    optim="paged_adamw_8bit",                 
    logging_steps=1,                          
    save_strategy="epoch",                    
    bf16=False,
    tf32=False,
    learning_rate=2e-4,                       
    max_steps= 100,
    max_grad_norm=0.3,                        
    warmup_ratio=0.03,                        
    lr_scheduler_type="constant",             
    push_to_hub=False,                        
    report_to="tensorboard",                  
)

SFTTrainer(

    model=model,
    train_dataset=train_data,
    eval_dataset=test_data,
    dataset_text_field="text",
    peft_config=lora_config,
    max_seq_length=512,
    tokenizer=tokenizer,
    args=training_arguments,
)

#### Speeds, Sizes, Times 

Took around 1 hour to train.

### Results
* Test Result 1:
```
Write a fucntion to sort a list in python

Answer:

def sort_list(list):
    return sorted(list)<eos>
Response:  None
```
* Test Result 2:
```
Write a function to count Consonants in a Given Word in Python

Response:  None
```
* Test Result 3:
```
Write a function to count the number of vowels in a given string in Python.

Example 1:

Input:  s =  "leetcodeisgreat"
Output:  5
Explanation:  The vowels are  'e', 'i', 'a', 'o', and 'u'.

Example 2:

Input:  s =  "leetcodeisgreat"
Output:  0
Explanation:  The vowels are  'e', 'i', 'a', 'o', and 'u'.

Constraints:

*   1 <= s.length <= 100
*   s consists of lowercase English letters.


def countVowels(s):
    count = 0
    for c in s:
        if c in 'aeiou':
            count += 1
    return count
<eos>
Response:  None
```

### Compute Infrastructure

Trained in Google Colab

#### Hardware

T4 GPU Hardware accelerator.