File size: 2,503 Bytes
bdaf98c 1bae770 bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 4b32dec bdaf98c 1637b4e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 |
---
library_name: transformers
language:
- en
base_model: microsoft/phi-2
pipeline_tag: text-generation
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
- **Developed by:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** English
- **Finetuned from model [optional]:** https://huggingface.co/microsoft/phi-2
### Model Sources [optional]
- **Repository:** Coming soon
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
Automatic identification of sections in (clinical trial) abstracts.
## How to Get Started with the Model
Prompt Format:
'''
###Unstruct:
{abstract}
###Struct:
'''
## Training Details
### Training Data
50k randomly sampled RCT abstract within period [1970-2023]
### Training Procedure
Generation of (unstructured, structured) pairs for structured abstracts.
Generation of dedicated prompt for Causal_LM modelling.
#### Training Hyperparameters
bnb_config = BitsAndBytesConfig(load_in_4bit=True,
bnb_4bit_quant_type='nf4',
#bnb_4bit_compute_dtype='float16',
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True)
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
10k randomly sampled RCT abstract within period [1970-2023]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Technical Specifications [optional]
### Model Architecture and Objective
LoraConfig(
r=16,
lora_alpha=32,
target_modules=[
'q_proj',
'k_proj',
'v_proj',
'dense',
'fc1',
'fc2',
], #print(model) will show the modules to use
bias="none",
lora_dropout=0.05,
task_type="CAUSAL_LM",
)
### Compute Infrastructure
#### Hardware
1 x RTX4090 - 24 GB
#### Software
torch einops transformers bitsandbytes accelerate peft
## Model Card Contact |