--- license: apache-2.0 datasets: - JetBrains/KStack-clean base_model: meta-llama/CodeLlama-7b-hf results: - task: type: text-generation dataset: name: MultiPL-HumanEval (Kotlin) type: openai_humaneval metrics: - name: pass@1 type: pass@1 value: 37.89 tags: - code --- **Exllamav2** quant (**exl2** / **5.0 bpw**) made with ExLlamaV2 v0.0.21 Other EXL2 quants: | **Quant** | **Model Size** | **lm_head** | | ----- | ---------- | ------- | |
**[2.2](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-2_2bpw_exl2)**
|
2055 MB
|
6
| |
**[2.5](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-2_5bpw_exl2)**
|
2279 MB
|
6
| |
**[3.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-3_0bpw_exl2)**
|
2663 MB
|
6
| |
**[3.5](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-3_5bpw_exl2)**
|
3047 MB
|
6
| |
**[3.75](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-3_75bpw_exl2)**
|
3244 MB
|
6
| |
**[4.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-4_0bpw_exl2)**
|
3437 MB
|
6
| |
**[4.25](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-4_25bpw_exl2)**
|
3629 MB
|
6
| |
**[5.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-5_0bpw_exl2)**
|
4209 MB
|
6
| |
**[6.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-6_0bpw_exl2)**
|
5006 MB
|
8
| |
**[6.5](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-6_5bpw_exl2)**
|
5383 MB
|
8
| |
**[8.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-8_0bpw_exl2)**
|
6176 MB
|
8
| # Model description This is a repository for the **CodeLlama-7b** model fine-tuned on the [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset with rule-based filtering, in the *Hugging Face Transformers* format. KStack-clean is a small subset of [KStack](https://huggingface.co/datasets/JetBrains/KStack), the largest collection of permissively licensed Kotlin code, automatically filtered to include files that have the highest "educational value for learning algorithms in Kotlin". # How to use ```python from transformers import AutoModelForCausalLM, AutoTokenizer # Load pre-trained model and tokenizer model_name = 'JetBrains/CodeLlama-7B-KStack-clean' tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda') # Create and encode input input_text = """\ This function takes an integer n and returns factorial of a number: fun factorial(n: Int): Int {\ """ input_ids = tokenizer.encode( input_text, return_tensors='pt' ).to('cuda') # Generate output = model.generate( input_ids, max_length=60, num_return_sequences=1, pad_token_id=tokenizer.eos_token_id ) # Decode output generated_text = tokenizer.decode(output[0], skip_special_tokens=True) print(generated_text) ``` As with the base model, we can use FIM. To do this, the following format must be used: ``` '
 ' + prefix + '  ' + suffix + ' '
```

# Training setup

The model was trained on one A100 GPU with following hyperparameters:

|         **Hyperparameter**           |             **Value**              |
|:---------------------------:|:----------------------------------------:|
|        `warmup`            |           100 steps            |
|        `max_lr`        |          5e-5          |
|        `scheduler`        |          linear          |
|        `total_batch_size`        |        32 (~30K tokens per step)          |
|        `num_epochs`        |          2          |

More details about fine-tuning can be found in the technical report (coming soon!).

# Fine-tuning data

For tuning the model, we used 25K exmaples from the [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset, selected from the larger [KStack](https://huggingface.co/datasets/JetBrains/KStack) dataset according to educational value for learning algorithms. In total, the dataset contains about 23M tokens. 

# Evaluation 

For evaluation, we used the [Kotlin HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval) dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the [datasets's page](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval).

Here are the results of our evaluation:

|         **Model name**           |             **Kotlin HumanEval Pass Rate**              |
|:---------------------------:|:----------------------------------------:|
|           `CodeLlama-7B`            |           26.89            |
|        `CodeLlama-7B-KStack-clean`        |          **37.89**        |

# Ethical Considerations and Limitations

CodeLlama-7B-KStack-clean is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack-clean's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack-clean, developers should perform safety testing and tuning tailored to their specific applications of the model.