drawing

BioChat Model

Source Paper: BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
BioChat is a language model fine-tuned using the ChatDoctor dataset from ChatDoctor-5k. Specifically designed for medical conversations, BioChat enables users to engage in interactive discussions with a virtual doctor. Whether you are seeking advice about symptoms you are experiencing, exploring possible health conditions, or looking for general medical insights, BioChat is built to assist in a reliable and informative manner.
NOTE: We are still in the early stages of exploring the generation capabilities and limitations of this model. It is important to emphasize that its text generation features are intended solely for research purposes and are not yet suitable for production use.
Finetuned from model: BioMistral-7B.

Using BioChat

You can use BioMistral with Hugging Face's Transformers library as follow.

Loading the model and tokenizer :

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BioMistral/BioMistral-7B")
model = AutoModelForCausalLM.from_pretrained(
    "BioMistral/BioMistral-7B",
    load_in_8bit=True,
    device_map="auto",
    output_hidden_states=True  # Ensure hidden states are available
)
model = PeftModel.from_pretrained(model, "Indah1/BioChat10")

Fine-Tuning Data

The fine-tuning data used for BioChat is derived from the ChatDoctor-5k dataset. This dataset contains a collection of medical conversations tailored to simulate doctor-patient interactions, making it an ideal source for training a medical conversational model. The dataset was carefully curated to ensure relevance and diversity in medical topics.

Training Hyperparameters

Hyperparameter	Value
Weigh Decay	0.01
Learning Rate	2e-05
Training Batch Size	8
Batch Size	8
Number of GPU	1
Optimizer	AdamW_8Bit
Warm Up Ratio	0.03
Scheduler	Cosine
Number of Epoch	5, 10, 15

Evaluation

To determine the best model for fine-tuning, I used perplexity as a metric to evaluate performance and select the most optimal version. By leveraging the model's capabilities, I aim to evaluate its behavior and responses using tools like the Word Embedding Association Test (WEAT). Below are the WEAT scores and perplexity values for the model at epochs 5, 10, and 15, which helped in determining the best-performing version. It is important to emphasize that its text generation features are intended solely for research purposes and are not yet suitable for production use. By releasing this model, we aim to drive advancements in biomedical NLP applications and contribute to best practices for the responsible development of domain-specific language models. Ensuring reliability, fairness, accuracy, and explainability remains a top priority for us.

Model Name	Perplexity Score	WEAT Score	Effect Size
BioChat5	4.5799	-0.00652	-0.4059
BioChat10	4.5873	0.002351	0.06176
BioChat15	4.8864	0.00859	0.43890

Framework versions

PEFT 0.11.1

Indah1
/

BioChat10