Update README.md

README.md

@@ -10,8 +10,8 @@ widget:
 ---
 ![](ft_sections.png)
 
-This is a small language model designed for scientific research application. It is fine tuned to analyzing randomized clinical trial abstracts and to classify sentences into four key sections: Background, Methods, Results, and Conclusion. 
-This makes it easier and faster for researchers to understand and organize important information from clinical studies.
+A small language model designed for scientific research applications. Phi2 was fine tuned to analyzing randomized clinical trial abstracts and to classify sentences into four key sections: Background, Methods, Results, and Conclusion. 
+This model facilitates researchers in understanding and organizing key information from clinical studies.
 
 ## Model Details
 
@@ -19,7 +19,7 @@ This makes it easier and faster for researchers to understand and organize impor
 The publication rate of Randomized Controlled Trials (RCTs) is consistently increasing,
 with more than 1 million RCTs already published. 
 Approximately half of these publications are listed in PubMed,
-posing a significant challenge for medical researchers seeking specific information.
+posing a significant data-volume challenge for medical researchers seeking specific information.
 
 When searching for prior studies, such as for writing systematic reviews, 
 researchers often skim through abstracts to quickly determine if the papers meet their criteria of interest. 
@@ -27,8 +27,8 @@ This task is facilitated when abstracts are structured, meaning the text within
 like objective, method, result, and conclusion.
 However, more than half of the RCT abstracts published are unstructured, complicating the rapid identification of relevant information.
 
-This model classifies each sentence of an abstract into a corresponding heading can greatly accelerate the process of locating the desired information. 
-This classification not only aids researchers but also benefits various downstream applications, including automatic text summarization, information extraction, and information retrieval.
+This model classifies each sentence of an abstract into a corresponding 'canonical 'section, greatly accelerating the process of locating the desired information. 
+This classification not only aids researchers but may also benefit other downstream applications, including automatic text summarization, information extraction, and information retrieval.
 
 
 - **Developed by: Salvatore Saporito
@@ -56,26 +56,60 @@ Prompt Format:
 
 Usage:
 
-    from peft import PeftModel, PeftConfig
+    import torch
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+    from transformers import BitsAndBytesConfig
+    from peft import PeftModel
 
-    #Load the model weights from hub
+    #Load base model weight
+    tokenizer_name = "microsoft/phi-2"
+    basemodel_name = "microsoft/phi-2"
     model_id = "SaborDay/Phi2_RCT1M-ft-heading"
-    trained_model = PeftModel.from_pretrained(model, model_id)
-
-    #Run inference
-    outputs = trained_model.generate(**inputs, max_length=1000)
+    
+    #Load base model weight & tokenizer
+    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name,trust_remote_code=True)
+  
+    model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto', trust_remote_code=True)
+    
+    #Load adapter
+    fine_tuned_model = PeftModel.from_pretrained(model, model_id)
+  
+    #Run inference    
+    outputs = fine_tuned_model.generate(**inputs, max_length=1000)
+    
     text = tokenizer.batch_decode(outputs,skip_special_tokens=True)[0]
     print(text)
+    
 
 
 Example:
 Application on unseen data 
 
-        PROMPT: '###Unstruct:\nKawasaki disease (KD) is a systemic vasculitis that causes abnormalities in the coronary arteries. Interleukin (IL)-41 is a novel immunoregulatory cytokine involved in the pathogenesis of some inflammatory and immune-related diseases. However, the role of IL-41 in KD is unclear. The purpose of this study was to detect the expression of IL-41 in the plasma of children with KD and its relationship with the disease.\nA total of 44 children with KD and 37 healthy controls (HC) were recruited for this study. Plasma concentrations of IL-41 were determined by ELISA. Correlations between plasma IL-41 levels and KD-related clinical parameters were analyzed by Pearson correlation and multivariate linear regression analysis. Receiver operating characteristic curve analysis was used to assess the clinical value of IL-41 in the diagnosis of KD.\nOur results showed that plasma IL-41 levels were significantly elevated in children with KD compared with HC. Correlation analysis demonstrated that IL-41 levels were positively correlated with D-dimer and N-terminal pro-B-type natriuretic peptide, and negatively correlated with IgM, mean corpuscular hemoglobin concentration, total protein, albumin and pre-albumin. Multivariable linear regression analysis revealed that IgM and mean corpuscular hemoglobin concentrations were associated with IL-41. Receiver operating characteristic curve analysis showed that the area under the curve of IL-41 was 0.7101, with IL-41 providing 88.64 % sensitivity and 54.05 % specificity.\nOur study indicated that plasma IL-41 levels in children with KD were significantly higher than those in HC, and may provide a potential diagnostic biomarker for KD.\n###Struct:\n
+        PROMPT: '###Unstruct:\nKawasaki disease (KD) is a systemic vasculitis that causes abnormalities in the coronary arteries.
+        Interleukin (IL)-41 is a novel immunoregulatory cytokine involved in the pathogenesis of some inflammatory and immune-related diseases. 
+        However, the role of IL-41 in KD is unclear.
+        The purpose of this study was to detect the expression of IL-41 in the plasma of children with KD and its relationship with the disease.
+        A total of 44 children with KD and 37 healthy controls (HC) were recruited for this study. Plasma concentrations of IL-41 were determined by ELISA. 
+        Correlations between plasma IL-41 levels and KD-related clinical parameters were analyzed by Pearson correlation and multivariate linear regression analysis.
+        Receiver operating characteristic curve analysis was used to assess the clinical value of IL-41 in the diagnosis of KD.
+        Our results showed that plasma IL-41 levels were significantly elevated in children with KD compared with HC. 
+        Correlation analysis demonstrated that IL-41 levels were positively correlated with D-dimer and N-terminal pro-B-type natriuretic peptide, and negatively correlated with IgM, mean corpuscular hemoglobin concentration, total protein, albumin and pre-albumin. Multivariable linear regression analysis revealed that IgM and mean corpuscular hemoglobin concentrations were associated with IL-41. Receiver operating characteristic curve analysis showed that the area under the curve of IL-41 was 0.7101, with IL-41 providing 88.64 % sensitivity and 54.05 % specificity.
+        Our study indicated that plasma IL-41 levels in children with KD were significantly higher than those in HC, and may provide a potential diagnostic biomarker for KD.
+        ###Struct:'
 
         Answer Phi2_RCT1M-ft-heading:
 
-        BACKGROUND: Kawasaki disease (KD) is a systemic vasculitis that causes abnormalities in the coronary arteries. Interleukin (IL)-41 is a novel immunoregulatory cytokine involved in the pathogenesis of some inflammatory and immune-related diseases. However, the role of IL-41 in KD is unclear. The purpose of this study was to detect the expression of IL-41 in the plasma of children with KD and its relationship with the disease. METHODS: A total of 44 children with KD and 37 healthy controls (HC) were recruited for this study. Plasma concentrations of IL-41 were determined by ELISA. Correlations between plasma IL-41 levels and KD-related clinical parameters were analyzed by Pearson correlation and multivariate linear regression analysis. Receiver operating characteristic curve analysis was used to assess the clinical value of IL-41 in the diagnosis of KD. RESULTS: Our results showed that plasma IL-41 levels were significantly elevated in children with KD compared with HC. Correlation analysis demonstrated that IL-41 levels were positively correlated with D-dimer and N-terminal pro-B-type natriuretic peptide, and negatively correlated with IgM, mean corpuscular hemoglobin concentration, total protein, albumin and pre-albumin. Multivariable linear regression analysis revealed that IgM and mean corpuscular hemoglobin concentrations were associated with IL-41. Receiver operating characteristic curve analysis showed that the area under the curve of IL-41 was 0.7101, with IL-41 providing 88.64 % sensitivity and 54.05 % specificity. CONCLUSIONS: Our study indicated that plasma IL-41 levels in children with KD were significantly higher than those in HC, and may provide a potential diagnostic biomarker for KD.']
+        BACKGROUND: Kawasaki disease (KD) is a systemic vasculitis that causes abnormalities in the coronary arteries. 
+        Interleukin (IL)-41 is a novel immunoregulatory cytokine involved in the pathogenesis of some inflammatory and immune-related diseases. 
+        However, the role of IL-41 in KD is unclear. T
+        he purpose of this study was to detect the expression of IL-41 in the plasma of children with KD and its relationship with the disease. 
+        METHODS: A total of 44 children with KD and 37 healthy controls (HC) were recruited for this study. 
+        Plasma concentrations of IL-41 were determined by ELISA.
+        Correlations between plasma IL-41 levels and KD-related clinical parameters were analyzed by Pearson correlation and multivariate linear regression analysis.
+        Receiver operating characteristic curve analysis was used to assess the clinical value of IL-41 in the diagnosis of KD. 
+        RESULTS: Our results showed that plasma IL-41 levels were significantly elevated in children with KD compared with HC.
+        Correlation analysis demonstrated that IL-41 levels were positively correlated with D-dimer and N-terminal pro-B-type natriuretic peptide, and negatively correlated with IgM, mean corpuscular hemoglobin concentration, total protein, albumin and pre-albumin. Multivariable linear regression analysis revealed that IgM and mean corpuscular hemoglobin concentrations were associated with IL-41. Receiver operating characteristic curve analysis showed that the area under the curve of IL-41 was 0.7101, with IL-41 providing 88.64 % sensitivity and 54.05 % specificity. 
+        CONCLUSIONS: Our study indicated that plasma IL-41 levels in children with KD were significantly higher than those in HC, and may provide a potential diagnostic biomarker for KD.
     
 ## Training Details
 
@@ -96,10 +130,16 @@ Generation of dedicated prompt for Causal_LM modelling.
                                 bnb_4bit_quant_type='nf4',
                                 bnb_4bit_compute_dtype=torch.bfloat16,
                                 bnb_4bit_use_double_quant=True)
-                                
+
+#### Training Run metrics
+
+    https://wandb.ai/salvatore-saporito-phd/huggingface/runs/5fcnxthk?nw=nwusersalvatoresaporitophd
+
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+The model was evaluated over a subset of previously considered abstracts (20k RCT).
+Each individual evaluation sample was verified not to be present in training set using corresponding PMID/
+https://github.com/Franck-Dernoncourt/pubmed-rct/tree/master/PubMed_20k_RCT
 
 ### Testing Data, Factors & Metrics
 
@@ -109,7 +149,7 @@ Generation of dedicated prompt for Causal_LM modelling.
 
 #### Metrics
 
-Coming soon
+[WIP]
 
 ## Technical Specifications [optional]
 
@@ -118,12 +158,11 @@ Coming soon
     LoraConfig(
         r=16,
         lora_alpha=32,
-        target_modules=[
-        'q_proj','k_proj','v_proj','dense','fc1','fc2'], 
-    bias="none",
-    lora_dropout=0.05,
-    task_type="CAUSAL_LM",
-    )
+        target_modules=['q_proj','k_proj','v_proj','dense','fc1','fc2'], 
+        bias="none",
+        lora_dropout=0.05,
+        task_type="CAUSAL_LM",
+        )
 
 ### Compute Infrastructure
 
@@ -137,7 +176,10 @@ Coming soon
 
 ## Model Card Contact
 
+Salvatore Saporito - [email protected]
 
 ## References
 
 https://arxiv.org/abs/1710.06071
+https://arxiv.org/abs/2106.09685
+https://arxiv.org/pdf/2309.05463