metadata

library_name: transformers
tags:
  - clinical
  - healthcare
  - NLP
  - BERT
  - MIMIC-IV
  - MedNLI
  - transformer
language:
  - en
metrics:
  - accuracy
base_model:
  - mosaicml/mosaic-bert-base
model-index:
  - name: ClinicalMosaic
    results:
      - task:
          type: classification
        dataset:
          name: MedNLI
          type: MedNLI
        metrics:
          - name: Accuracy
            type: accuracy
            value: 86.5
license: mit
datasets:
  - bigbio/mednli
pipeline_tag: fill-mask

Model Card for Clinical Mosaic

Clinical Mosaic is a transformer-based language model built on the Mosaic BERT architecture. It has been optimized for clinical language understanding by incorporating recent innovations such as Attention with Linear Biases (ALiBi) for long‐sequence extrapolation and Gated Linear Units (GLU) to capture complex clinical patterns. The model is pre-trained on 331,794 deidentified clinical notes from the MIMIC-IV-NOTES 2.2 database and demonstrates strong performance on clinical reasoning tasks.

Model Details

Developed by: Sifal Klioui, Sana Sellami, and Youssef Trardi (Aix-Marseille Univ, LIS, CNRS, Marseille, France)
Funded by: PICOMALE project (AMIDEX)
Base Model: Mosaic BERT
License: MIMIC Data Use Agreement (requires compliance with original DUA)
Repository: PatientTrajectoryForecasting
Paper: Patient Trajectory Prediction: Integrating Clinical Notes with Transformers (PDF)

Uses

Direct Use

Clinical Mosaic can be used directly as a clinical language model for:

Natural language inference and clinical reasoning tasks.
Serving as a backbone for further fine-tuning on clinical NLP applications (e.g., clinical note summarization, diagnosis classification).

Downstream Use

The model can be integrated into larger systems for tasks such as patient trajectory prediction or decision support, provided that additional fine-tuning and rigorous validation are performed.

Out-of-Scope Use

Clinical Decision-Making: The model is for research use only and should not be deployed for direct patient care without further validation and regulatory approval.
Medical Advice Generation: It is not intended to replace expert clinical judgment.

Bias, Risks, and Limitations

Clinical Mosaic was pre-trained on deidentified clinical notes from MIMIC-IV-NOTES 2.2—a dataset from a single U.S. institution. This may introduce biases related to local clinical practices and patient demographics. Although extensive care was taken to deidentify data and prevent PHI leakage, users must ensure that the model is not used to inadvertently reidentify sensitive information. When applying the model to new populations or clinical settings, performance may vary, so further fine-tuning and bias audits are recommended.

Recommendations

Text Normalization Consistency: Any input text must undergo identical preprocessing as applied during training
Evaluation: Users should evaluate the model’s performance on their target population.
Fine-Tuning: Consider additional fine-tuning or domain adaptation if deploying in a different clinical context.
Bias Audits: Regularly perform bias audits to monitor for potential disparities in performance.

How to Get Started with the Model

Install the Hugging Face Transformers library and load the model as follows:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("path/to/clinical-mosaic")
model = AutoModel.from_pretrained("path/to/clinical-mosaic")

Further instructions and example scripts are provided in the model’s repository.

Training Details

Training Data

Data Source: 331,794 clinical notes from the MIMIC-IV-NOTES 2.2 dataset.
Preprocessing:
- Following (Alsentzer et al., 2019), clinical notes were preprocessed by unifying medical abbreviations (e.g., “hr”, “hrs” to “hours”), removing accents, converting special characters, and normalizing text to lowercase. These steps help mitigate variations caused by subword tokenizers.
- For additional details, please refer to the PatientTrajectoryForecasting GitHub repository.

Training Procedure

Setup: Distributed training using 7 NVIDIA A40 GPUs.
Hyperparameters:
- Effective Batch Size: 224
- Training Steps: 80,000
- Sequence Length: 512 tokens
- Optimizer: ADAMW
- Initial Learning Rate: 5e-4
- Learning Rate Schedule: Linear warmup for 33,000 steps, followed by cosine annealing for 46,000 steps (final LR 1e-5)
- Masking Probability: 30%

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on the MedNLI dataset, which comprises 14,049 clinical premise-hypothesis pairs derived from MIMIC-III notes.

Factors

Evaluation disaggregated performance across clinical language understanding, with a focus on natural language inference.

Results

Clinical Mosaic outperformed comparable models:

BERT: 77.6%
BioBERT: 80.8%
Clinical Discharge BERT: 84.1%
Bio+Clinical BERT: 82.7%
Clinical Mosaic: 86.5%

These results indicate improved clinical reasoning and language understanding.

Summary

The model demonstrates robust performance on clinical natural language inference tasks and serves as a strong foundation for further clinical NLP applications.

Environmental Impact

Hardware Type: 7 NVIDIA A40 GPUs
Hours used: 1008 (GPU hours)
Provider: Private Infrastructure
Carbon Emitted: 108.86 kg CO2 eq.

Acknowledgments

We would like to thank the LIS laboratory for providing the GPU resources necessary for pretraining and conducting extensive experiments. Additionally, we acknowledge CEDRE for supporting early-stage experiments and hosting part of the computational infrastructure.

Citation

BibTeX:

To be added

More Information

For further details, please refer to the model’s repository and supplementary documentation.

Model Card Contact

For questions or further information, please contact [[email protected]].