Model Card for Model ID

This model is a fine-tuned version of google/gemma-2b on an SaikatM/Code-Platypus dataset.

Model Details

Model Description

  • Finetuned from model: google/gemma-2b

Model Sources

Training code can be found here: https://github.com/Saikat-M/LLM-Finetuning

Direct Use

  • Code generation tasks

Training Data

Training Procedure

Used QLoRA from PEFT and used SFTTrainer.

Preprocessing

From the Open-Platypus dataset filtering-out rows which has leetcode_ne in it's data_source column.

Training Hyperparameters

LoraConfig(

r=4,
lora_alpha=2,
target_modules=modules,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"

)

TrainingArguments(

output_dir="gemma-2b-code-platypus",      
num_train_epochs=1,                    
per_device_train_batch_size=4,            
gradient_accumulation_steps=4,            
gradient_checkpointing=True,              
optim="paged_adamw_8bit",                 
logging_steps=1,                          
save_strategy="epoch",                    
bf16=False,
tf32=False,
learning_rate=2e-4,                       
max_steps= 100,
max_grad_norm=0.3,                        
warmup_ratio=0.03,                        
lr_scheduler_type="constant",             
push_to_hub=False,                        
report_to="tensorboard",                  

)

SFTTrainer(

model=model,
train_dataset=train_data,
eval_dataset=test_data,
dataset_text_field="text",
peft_config=lora_config,
max_seq_length=512,
tokenizer=tokenizer,
args=training_arguments,

)

Speeds, Sizes, Times

Took around 1 hour to train.

Results

  • Test Result 1:
Write a fucntion to sort a list in python

Answer:

def sort_list(list):
    return sorted(list)<eos>
Response:  None
  • Test Result 2:
Write a function to count Consonants in a Given Word in Python

Response:  None
  • Test Result 3:
Write a function to count the number of vowels in a given string in Python.

Example 1:

Input:  s =  "leetcodeisgreat"
Output:  5
Explanation:  The vowels are  'e', 'i', 'a', 'o', and 'u'.

Example 2:

Input:  s =  "leetcodeisgreat"
Output:  0
Explanation:  The vowels are  'e', 'i', 'a', 'o', and 'u'.

Constraints:

*   1 <= s.length <= 100
*   s consists of lowercase English letters.


def countVowels(s):
    count = 0
    for c in s:
        if c in 'aeiou':
            count += 1
    return count
<eos>
Response:  None

Compute Infrastructure

Trained in Google Colab

Hardware

T4 GPU Hardware accelerator.

Downloads last month
15
Safetensors
Model size
2.51B params
Tensor type
FP16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train SaikatM/Code-Gemma-v1