altaidevorg/gemma-women-health-merged

Model Summary

This model is fine-tuned from gemma-2-2b-it using a custom dataset created by our team, providing a concise overview of its key details and purpose.

Motivation

Women's health is a critical and often underserved topic, with limited accessible resources where women can ask questions and receive reliable answers about their health. Recognizing this gap, we embarked on this project to develop a compact language model (LLM) tailored to address these needs. Our goal is to create a model that not only provides valuable insights but also enables efficient on-device inference, ensuring greater accessibility and usability for women worldwide.

Usage

from unsloth import FastLanguageModel
import torch
from transformers import TextStreamer


max_seq_length = 2048
dtype = None
load_in_4bit = False
lora_path = "altaidevorg/gemma-women-health-merged"
use_streamer = False

model, tokenizer = FastLanguageModel.from_pretrained(
    lora_path,
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)
FastLanguageModel.for_inference(model)

text_streamer = TextStreamer(tokenizer, skip_prompt = True)


messages = [
    {"role": "user", "content": user_prompt},
]


input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize = True,
    add_generation_prompt = True,
    return_tensors = "pt",
).cuda()

terminators = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<end_of_turn>")]

outputs = model.generate(input_ids = input_ids,
    streamer = text_streamer if use_streamer else None,
    max_new_tokens = 1024,
    eos_token_id=terminators,
    use_cache=True, do_sample=True, temperature=0.6, top_p=0.9)
if not use_streamer:
    out = outputs[0][input_ids.shape[-1]:]
    generated_text = tokenizer.decode(out, skip_special_tokens=True)
    print(generated_text)

Dataset

The dataset was prepared through a comprehensive process based on our synthetic dataset generation method. Health-related websites were scraped, open-source e-books and PDFs focusing on women's health were collected, and an instruction dataset was created from these sources. To generate high-quality questions, we utilized a technique that involves role-playing between two LLMs and contextually-augmented by RAG. The dataset will be made publicly available through its dedicated repository altaidevorg/women-health-mini. Additionally, the code used for dataset generation will be released in the future to promote transparency and enable reproducibility.

Evaluation Notes

During testing, we observed that the LoRA checkpoint performed better in evaluations compared to the version where the LoRA checkpoint was merged with the base model. Interestingly, the standalone LoRA checkpoint consistently delivered superior results, though we currently lack a concrete explanation for this phenomenon. We are actively investigating the underlying cause, with our best hypothesis being that the merging process may introduce some form of precision loss. Further research is underway to validate this theory and optimize the performance.

Disclaimer

Any output of this model should never be taken as a medical advice, and it is mainly intended for research purposes.