|
--- |
|
task_categories: |
|
- text-generation |
|
--- |
|
# Description |
|
This language model is the version 0.0 of a Gradio Coding Assistant. It is an instruction fine-tuned version of [StarCoder](https://huggingface.co/bigcode/starcoder) that is |
|
design to provide assistance to developers who are using [gradio](https://www.gradio.app). |
|
|
|
# Dataset |
|
The dataset is multi-source. Its content comes from the following sources |
|
- The stack |
|
|
|
More precisely, we looked into [the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) which contain codes permissive licenses. We shortlisted the files whose |
|
content incorporated the keyword `gradio`. |
|
- GitHub Issues |
|
|
|
We scrapped all the issues of the official repository [the-gradio-app/gradio](https://github.com/gradio-app/gradio) and added them to our training dataset. |
|
- Spaces on Hugging Face Hub |
|
|
|
We used the [HuggingFace_Hub API](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) to scrape the data from the spaces which are designed with gradio. We kept track of those |
|
with permissive licenses, namely MIT and Apache 2.0. This set of code was further deduplicated. |
|
|
|
# Training setting and hyperparameters |
|
For our fine-tuning, we decided to follow a 2-step strategy. |
|
- Pretraining (Fine-tuning) with next token prediction on the previously built gradio dataset (this step should familiarize the model with the gradio syntax.). |
|
- Instruction fine-tuning on an instruction dataset (this step should make the model conversational.). |
|
For both steps, we made use of parameter-efficient fine-tuning via the library [PEFT](https://github.com/huggingface/peft), more precisely [LoRA](https://arxiv.org/abs/2106.09685). Our |
|
training script is the famous [starcoder fine-tuning script](https://github.com/bigcode-project/starcoder). |
|
|
|
## Resources |
|
Our training was done of 8 A100 GPUs of 80GB. |
|
|
|
## Pretraining |
|
These are the parameters that we used : |
|
- learning rate : 5e-4 |
|
- warmup_steps : |
|
- gradient_accumulation_steps : 4 |
|
- batch_size : 1 |
|
- sequence length : 2048 |
|
- max_steps : 1000 |
|
- warmup_steps : 5 |
|
- weight_decay : 0.05 |
|
|
|
LORA PARAMETERS : |
|
- r = 16 |
|
- alpha = 32 |
|
- dropout = 0.05 |
|
|
|
We stopped the training before the end and kept the *checkpoint-100* for the second step. |
|
|
|
## Fine-tuning |
|
This step consisted into the instruction fine-tuning of the previous checkpoint. For that purpose, we used a modified version of [openassistant-guanaco](https://huggingface.co/datasets/timdettmers/openassistant-guanaco). |
|
The template for the instruction fine-tuning was `Question: {question}\n\nAnswer: {answer}`. We used exactly the same parameters we used during the pretraining and we kept the *checkpoint-50*. |
|
|
|
# Usage |
|
The usage is straightforward and very similar to any other instruction fine-tuned model. |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
checkpoint_name="ArmelR/starcoder-gradio-v0" |
|
model = AutoModelForCausalLM.from_pretrained(checkpoint_name) |
|
tokenizer = AutoTokenizer.from_pretrained(checkpoint_name) |
|
|
|
prompt = "Create a gradio application that help to convert temperature in celcius into temperature in Fahrenheit" |
|
inputs = tokenizer(f"Question: {prompt}\n\nAnswer: ", return_tensors="pt") |
|
|
|
outputs = model.generate( |
|
inputs["input_ids"], |
|
temperature=0.2, |
|
top_p=0.95, |
|
max_new_tokens=200 |
|
) |
|
|
|
input_len=len(inputs["input_ids"]) |
|
print(tokenizer.decode(outputs[0][input_len:])) |
|
``` |
|
|
|
# More information |
|
For further information, refer to [StarCoder](https://huggingface.co/bigcode/starcoder). |
