Update README.md

README.md

@@ -15,8 +15,7 @@ base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
-
+Quantum Research Bot is a model fined tuned over the latest research data in quantum science. It contains data from the second half of 2024 making it more performant than base models.
 
 
 ## Model Details
@@ -87,13 +86,17 @@ Use the code below to get started with the model.
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+Initially trained on a bit less than 3k entries, it was later expanded t 5k high quality questions and answers to make the best of supervised fine tuning.
+
+The dataset was generated by crawling the https://quantum-journal.org/ site, and passing data into the OpenAI gpt-4-turbo model with various prompts to ensure high quality data generation.
 
 [More Information Needed]
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+Many training procedures were tried alongside with multiple models.
+
+After exensive grid search, supervised fine tuning of Llama 3.1-8B with LORA+ resulted in the best training and evaluation cross entropy.
 
 #### Preprocessing [optional]
 
@@ -102,7 +105,19 @@ Use the code below to get started with the model.
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Training regime:** [More Information Needed]
+- bfloat16 precision
+- LORA rank: 8
+- LORA alpha: 16
+- LORA droput: 0.1
+- Unfreezed nodes are attention, MLP, and embeddings
+- Optimizer: AdamW
+- LR: 1e-4
+- LR scheduler: cosine
+- NEFT enabled: true
+- Batch size: 8
+- Number of epochs: 3
+
 
 #### Speeds, Sizes, Times [optional]