Kabatubare/web-md-llama2-7b-3000

Medical3000 Model Card

This is a model card for web-md-llama2-7b-3000 , a fine-tuned version of Llama-2-7B, specifically aimed at medical dialogues.

Covered areas:

General Medicine: Basic medical advice, symptoms, general treatments.

Cardiology: Questions related to heart diseases, blood circulation.

Neurology: Topics around brain health, neurological disorders.

Gastroenterology: Issues related to the digestive system.

Oncology: Questions about different types of cancers, treatments.

Endocrinology: Topics related to hormones, diabetes, thyroid.

Orthopedics: Bone health, joint issues.

Pediatrics: Child health, vaccinations, growth and development.

Mental Health: Depression, anxiety, stress, and other mental health issues.

Women's Health: Pregnancy, menstrual health, menopause.

Model Details

Base Model

• Name: Llama-2-7B

Fine-tuned Model

• Name: web-md-llama2-7b-3000
• Fine-tuned on: Kabatubare/medical-guanaco-3000
• Description: This model is fine-tuned to specialize in medical dialogues and healthcare applications.

Architecture and Training Parameters

Architecture

• LoRA Attention Dimension: 64
• LoRA Alpha Parameter: 16
• LoRA Dropout: 0.1
• Precision: 4-bit (bitsandbytes)
• Quantization Type: nf4

Training Parameters

• Epochs: 3
• Batch Size: 4
• Gradient Accumulation Steps: 1
• Max Gradient Norm: 0.3
• Learning Rate: 3e-4
• Weight Decay: 0.001
• Optimizer: paged_adamw_32bit
• LR Scheduler: cosine
• Warmup Ratio: 0.03
• Logging Steps: 25

Datasets

Fine-tuning Dataset

• Name: Kabatubare/medical-guanaco-3000
• Description: This is a reduced and balanced dataset curated from a larger medical dialogue dataset using derived from 24,000 WebMD question and answer dialogue sessions . It aims to cover a broad range of medical topics and is suitable for training healthcare chatbots and conducting medical NLP research.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Yo!Medical3000")
model = AutoModelForCausalLM.from_pretrained("Yo!Medical3000")

# Use the model for inference