Update README.md

README.md

@@ -19,9 +19,46 @@ with permissive licenses, namely MIT and Apache 2.0. This set of code was furthe
 
 # Training setting and hyperparameters
 For our fine-tuning, we decided to follow a 2-step strategy.
-- Pretraining (Fine-tuning) with next token prediction on the previously built gradio dataset (this step should familiarize the model with the gradio syntax.)
-- Instruction fine-tuning on an instruction fine-tuning (this step should make the model conversational)
- ## Pretraining
-  Gradio - ready 50 steps
-- Fine-tuning
-  Oasst Guanaco 100 steps
+- Pretraining (Fine-tuning) with next token prediction on the previously built gradio dataset (this step should familiarize the model with the gradio syntax.).
+- Instruction fine-tuning on an instruction dataset (this step should make the model conversational.).
+For both steps, we made use of parameter-efficient fine-tuning via the library [PEFT](https://github.com/huggingface/peft), more precisely [LoRa](https://arxiv.org/abs/2106.09685). Our
+training script is the famous [starcoder fine-tuning script](https://github.com/bigcode-project/starcoder).
+
+## Resources
+Our training was done of 8 A100 GPUs of 80GB.
+
+## Pretraining
+These are the parameters that we used :
+- learning rate : 5e-4
+- warmup_steps :
+- gradient_accumulation_steps : 4
+- batch_size : 1
+- sequence length : 2048
+- max_steps : 1000
+- warmup_steps : 5
+- weight_decay : 0.05
+
+LORA PARAMETERS :
+- r = 16
+- alpha = 32
+- dropout = 0.05
+
+We stopped the training before the end and kept the *checkpoint-100* for the second step.
+
+## Fine-tuning
+This step consisted into the instruction fine-tuning of the previous checkpoint. For that purpose, we used a modified version of [openassistant-guanaco](https://huggingface.co/datasets/timdettmers/openassistant-guanaco).
+The template for the instruction fine-tuning was `Question: {question}\n\nAnswer: {answer}`. We used exactly the same parameters we used during the pretraining and we kept the *checkpoint-50*.
+
+## Usage
+The usage is straightforward an very similar to any other instruction fine-tuned model
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+checkpoint_name="ArmelR/starcoder-gradio-v0"
+model = AutoModelForCausalLM.from_pretrained(checkpoint_name)
+tokenizer = AutoTokenizer.from_pretrained(checkpoint_name)
+prompt = "Create a gradio application that help to convert temperature in celcius into temperature in Fahrenheit"
+inputs = tokenizer(f"Question: {prompt}\n\nAnswer: ", return_tensors="pt")
+outputs = model.generate(inputs["input_ids"], temperature=0.2, top_p=0.95)
+print(tokenizer.decode(outputs))
+```