---
library_name: transformers
tags: []
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->



## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

- **Developed by:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** English
- **Finetuned from model [optional]:** https://huggingface.co/microsoft/phi-2

### Model Sources [optional]

- **Repository:** Coming soon
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

Automatic identification of sections in (clinical trial) abstracts.

## How to Get Started with the Model

Prompt Format:

'''
###Unstruct:
{abstract}
###Struct:
'''

## Training Details

### Training Data

50k randomly sampled RCT abstract within period [1970-2023]

### Training Procedure 

Generation of (unstructured, structured) pairs for structured abstracts.
Generation of dedicated prompt for Causal_LM modelling.


#### Training Hyperparameters

bnb_config = BitsAndBytesConfig(load_in_4bit=True,
                                bnb_4bit_quant_type='nf4',
                                #bnb_4bit_compute_dtype='float16',
                                bnb_4bit_compute_dtype=torch.bfloat16,
                                bnb_4bit_use_double_quant=True)
                                
## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

10k randomly sampled RCT abstract within period [1970-2023]

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

### Results

[More Information Needed]

#### Summary


## Technical Specifications [optional]

### Model Architecture and Objective

LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
    'q_proj',
    'k_proj',
    'v_proj',
    'dense',
    'fc1',
    'fc2',
    ], #print(model) will show the modules to use
    bias="none",
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

### Compute Infrastructure

#### Hardware

1 x RTX4090 - 24 GB

#### Software

torch einops transformers bitsandbytes accelerate peft 

## Model Card Contact