|
--- |
|
language: |
|
- en |
|
license: apache-2.0 |
|
library_name: transformers |
|
tags: |
|
- medical |
|
- pharmacovigilance |
|
- vaccines |
|
datasets: |
|
- chrisvoncsefalvay/vaers-outcomes |
|
metrics: |
|
- accuracy |
|
- f1 |
|
- precision |
|
- recall |
|
dataset: chrisvoncsefalvay/vaers-outcomes |
|
pipeline_tag: text-classification |
|
widget: |
|
- text: Patient is a 90 y.o. male with a PMH of IPF, HFpEF, AFib (Eliquis), Metastatic |
|
Prostate Cancer who presented to Hospital 10/28/2023 following an unwitnessed |
|
fall at his assisted living. He was found to have an AKI, pericardial effusion, |
|
hypoxia, AMS, and COVID-19. His hospital course was complicated by delirium and |
|
aspiration, leading to acute hypoxic respiratory failure requiring BiPAP and transfer |
|
to the ICU. Palliative Care had been following, and after goals of care conversations |
|
on 11/10/2023 the patient was transitioned to DNR-CC. Patient expired at 0107 |
|
11/12/23. |
|
example_title: VAERS 2727645 (hospitalisation, death) |
|
- text: 'hospitalized for paralytic ileus a week after the vaccination; This serious |
|
case was reported by a physician via call center representative and described |
|
the occurrence of ileus paralytic in a patient who received Rota (Rotarix liquid |
|
formulation) for prophylaxis. On an unknown date, the patient received the 1st |
|
dose of Rotarix liquid formulation. On an unknown date, less than 2 weeks after |
|
receiving Rotarix liquid formulation, the patient experienced ileus paralytic |
|
(Verbatim: hospitalized for paralytic ileus a week after the vaccination) (serious |
|
criteria hospitalization and GSK medically significant). The outcome of the ileus |
|
paralytic was not reported. It was unknown if the reporter considered the ileus |
|
paralytic to be related to Rotarix liquid formulation. It was unknown if the company |
|
considered the ileus paralytic to be related to Rotarix liquid formulation. Additional |
|
Information: GSK Receipt Date: 27-DEC-2023 Age at vaccination and lot number were |
|
not reported. The patient of unknown age and gender was hospitalized for paralytic |
|
ileus a week after the vaccination. The reporting physician was in charge of the |
|
patient.' |
|
example_title: VAERS 2728408 (hospitalisation) |
|
- text: Patient received Pfizer vaccine 7 days beyond BUD. According to Pfizer manufacturer |
|
research data, vaccine is stable and effective up to 2 days after BUD. Waiting |
|
for more stability data from PFIZER to determine if revaccination is necessary. |
|
example_title: VAERS 2728394 (no event) |
|
- text: Fever of 106F rectally beginning 1 hr after immunizations and lasting <24 |
|
hrs. Seen at ER treated w/tylenol & cool baths. |
|
example_title: VAERS 25042 (ER attendance) |
|
- text: I had the MMR shot last week, and I felt a little dizzy afterwards, but it |
|
passed after a few minutes and I'm doing fine now. |
|
example_title: 'Non-sample example: simulated informal patient narrative (no event)' |
|
- text: My niece had the COVID vaccine. A few weeks later, she was T-boned by a drunk |
|
driver. She called me from the ER. She's fully recovered now, though. |
|
example_title: 'Non-sample example: simulated informal patient narrative (ER attendance, |
|
albeit unconnected)' |
|
model-index: |
|
- name: daedra |
|
results: |
|
- task: |
|
type: text-classification |
|
dataset: |
|
name: vaers-outcomes |
|
type: vaers-outcomes |
|
metrics: |
|
- type: accuracy_microaverage |
|
value: 0.885 |
|
name: Accuracy, microaveraged |
|
verified: false |
|
- type: f1_microaverage |
|
value: 0.885 |
|
name: F1 score, microaveraged |
|
verified: false |
|
- type: precision_macroaverage |
|
value: 0.769 |
|
name: Precision, macroaveraged |
|
verified: false |
|
- type: recall_macroaverage |
|
value: 0.688 |
|
name: Recall, macroaveraged |
|
verified: false |
|
--- |
|
|
|
# DAEDRA: Determining Adverse Event Disposition for Regulatory Affairs |
|
|
|
This model is a fine-tuned version of [dmis-lab/biobert-base-cased-v1.2](https://huggingface.co/dmis-lab/biobert-base-cased-v1.2) trained on the [VAERS adversome outcomes data set](https://huggingface.com/datasets/chrisvoncsefalvay/vaers-outcomes). |
|
|
|
# Table of Contents |
|
|
|
- [Model Details](#model-details) |
|
- [Uses](#uses) |
|
- [Bias, Risks, and Limitations](#bias-risks-and-limitations) |
|
- [Training Details](#training-details) |
|
- [Evaluation](#evaluation) |
|
- [Environmental Impact](#environmental-impact) |
|
- [Technical Specifications](#technical-specifications-optional) |
|
- [Citation](#citation) |
|
|
|
|
|
# Model Details |
|
|
|
## Model Description |
|
|
|
<!-- Provide a longer summary of what this model is/does. --> |
|
|
|
DAEDRA is a model for the identification of adverse event dispositions (outcomes) from passive pharmacovigilance data. |
|
The model is trained on a real-world adversomics data set spanning over three decades (1990-2023) and comprising over 1.8m records for a total corpus of 173,093,850 words constructed from a subset of reports submitted to VAERS. |
|
It is intended to identify, based on the narrative, whether any, or any combination, of three serious outcomes -- death, hospitalisation and ER attendance -- have occurred. |
|
|
|
|
|
- **Developed by:** Chris von Csefalvay |
|
- **Model type:** Language model |
|
- **Language(s) (NLP):** en |
|
- **License:** apache-2.0 |
|
- **Parent Model:** [dmis-lab/biobert-base-cased-v1.2](https://huggingface.co/dmis-lab/biobert-base-cased-v1.2) |
|
- **Resources for more information:** |
|
- [GitHub Repo](https://github.com/chrisvoncsefalvay/daedra) |
|
|
|
|
|
# Uses |
|
|
|
This model was designed to facilitate the coding of passive adverse event reports into severity outcome categories. |
|
|
|
## Direct Use |
|
|
|
Load the model via the `transformers` library: |
|
|
|
``` |
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("chrisvoncsefalvay/daedra") |
|
model = AutoModel.from_pretrained("chrisvoncsefalvay/daedra") |
|
``` |
|
|
|
## Out-of-Scope Use |
|
|
|
This model is not intended for the diagnosis or treatment of any disease. |
|
|
|
|
|
# Bias, Risks, and Limitations |
|
|
|
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. |
|
|
|
|
|
|
|
# Training Details |
|
|
|
## Training Data |
|
|
|
The model was trained on the [VAERS adversome outcomes data set](https://huggingface.com/datasets/chrisvoncsefalvay/vaers-outcomes), which comprises 1,814,920 reports from the FDA's Vaccine Adverse Events Reporting System (VAERS). Reports were split into a 70% training set and a 15% test set and 15% validation set after age and gender matching. |
|
|
|
## Training Procedure |
|
|
|
Training was conducted on an Azure `Standard_NC24s_v3` instance in `us-east`, with 4x Tesla V100-PCIE-16GB GPUs and 24x Intel Xeon E5-2690 v4 CPUs at 2.60GHz. |
|
|
|
### Speeds, Sizes, Times |
|
|
|
Training took 15 hours and 10 minutes. |
|
|
|
|
|
## Testing Data, Factors & Metrics |
|
|
|
### Testing Data |
|
|
|
The model was tested on the `test` partition of the [VAERS adversome outcomes data set](https://huggingface.com/datasets/chrisvoncsefalvay/vaers-outcomes). |
|
|
|
## Results |
|
|
|
On the test set, the model achieved the following results: |
|
|
|
* `f1`: 0.885 |
|
* `precision` and `recall`, microaveraged: 0.885 |
|
* `precision`, macroaveraged: 0.769 |
|
* `recall`, macroaveraged: 0.688 |
|
|
|
|
|
# Environmental Impact |
|
|
|
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
|
|
- **Hardware Type:** 4 x Tesla V100-PCIE-16GB |
|
- **Hours used:** 15.166 |
|
- **Cloud Provider:** Azure |
|
- **Compute Region:** us-east |
|
- **Carbon Emitted:** 6.72 kg CO2eq (offset by provider) |
|
|
|
|
|
# Citation |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
**BibTeX:** |
|
|
|
Forthcoming -- watch this space. |
|
|
|
# Model Card Authors |
|
|
|
<!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. --> |
|
|
|
Chris von Csefalvay |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 64 |
|
- eval_batch_size: 64 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 3 |
|
|
|
### Framework versions |
|
|
|
- Transformers 4.37.2 |
|
- Pytorch 2.1.2+cu121 |
|
- Datasets 2.3.2 |
|
- Tokenizers 0.15.1 |
|
|
