|
--- |
|
library_name: transformers |
|
license: bigcode-openrail-m |
|
tags: |
|
- code |
|
--- |
|
|
|
# Model Card for StarCoderBase1B-Racket-SelfInstruct |
|
|
|
Each commit to this repository has a checkpoint (one per epoch) for a fine-tuned |
|
StarCoderBase-1B. The dataset for fine-tuning is a Racket self-instruction |
|
dataset. As shown in Evaluation below, self-instruction was not effective, and |
|
this model is barely any better than StarCoderBase-1B. |
|
|
|
## Finetuning Dataset and Hyperparameters |
|
|
|
- Dataset: https://huggingface.co/datasets/nuprl/MultiPL-T-racket-selfinstruct |
|
- Base Model: https://huggingface.co/bigcode/starcoderbase-1b |
|
- Batch size: 8 |
|
- Epochs: 7 |
|
- Learning rate: 3e-05 |
|
- Warmup steps: 100 |
|
- Sequence length: 2048 |
|
|
|
## Evaluation |
|
|
|
The results on MultiPL-HumanEval-Racket are as follows: |
|
|
|
``` |
|
Dataset,Pass@k,Estimate,NumProblems,MinCompletions,MaxCompletions |
|
humaneval-rkt-checkpoint_1494-0.2-reworded,1,7.70,161,50,50 |
|
humaneval-rkt-checkpoint_1992-0.2-reworded,1,6.86,161,50,50 |
|
humaneval-rkt-checkpoint_2490-0.2-reworded,1,6.82,161,50,50 |
|
humaneval-rkt-checkpoint_2988-0.2-reworded,1,6.91,161,50,50 |
|
humaneval-rkt-checkpoint_498-0.2-reworded,1,6.19,161,50,50 |
|
humaneval-rkt-checkpoint_6973-0.2-reworded,1,6.53,161,50,50 |
|
humaneval-rkt-checkpoint_996-0.2-reworded,1,7.08,161,50,50 |
|
``` |
