SmolLM-135M-Instruct-medical_meadow_medical_flashcards-10epochs

This model is a fine-tuned version of HuggingFaceTB/SmolLM-135M-Instruct on the medalpaca/medical_meadow_medical_flashcards dataset.

Model description

The primary purpose of this fine-tuning is to enhance the model's medical knowledge, equipping it with the ability to provide more accurate and reliable responses in medical contexts. By leveraging a specialized dataset based on Anki Medical Curriculum flashcards, this model covers a comprehensive range of medical topics, including anatomy, physiology, pathology, and pharmacology.

Designed for local or offline inference, this model is particularly valuable in medical settings where data privacy is paramount. It enables healthcare institutions to utilize advanced language model capabilities without the need to transmit sensitive data externally. The model's compact size allows it to run efficiently on low-cost hardware while maintaining reasonable inference speed, making it an ideal solution for private, on-premise deployment in resource-constrained environments.

Model Performance Comparison

While the base model's responses are general and follow a markdown format, the fine-tuned model appears to have effectively incorporated the specialized skills it was trained on, particularly in replicating the expected response style. However, as someone without a medical background, I am unable to assess the accuracy or quality of the responses provided by either model. Below is a table comparing the responses of the original model, base model, and fine-tuned model for a set of medical questions.

Question	Original Response	Base Model Response	Fine-tuned Model Response
What antimalarial medication has activity against Plasmodium hypnozoites?	Primaquine is the antimalarial medication that has activity against Plasmodium hypnozoites.	Antimalarial medications are a class of drugs that are effective against Plasmodium parasites, which are responsible for causing malaria. The most common antimalarial medications used to treat malaria include: 1. Ampicillin (Amphotericin B): A broad-spectrum antibiotic that is effective against Plasmodium species. 2. Fosamivir (Fosamivir): A broad-spectrum antiviral that is effective against Plasmodium species. 3. Influvir (Influvir): A broad-spectrum antiviral that is effective against Plasmodium species. 4. **Ampicillin	Erythromycin is the antimalarial medication that has activity against Plasmodium hypnozoites.
What is the term used to describe the amount of substance filtered in Bowman's space per unit time?	The amount of substance filtered in Bowman's space per unit time is called the filtered load.	The term "Bowman's space" is a colloquialism used to describe the amount of substance filtered in a Bowman's space, which is a hypothetical space where a spacecraft is launched into orbit. The concept is based on the idea that a spacecraft can be launched into orbit and then return to Earth, with the spacecraft filtering out any remaining particles or debris that may have been present in the space. The term "Bowman's space" is often used to describe the amount of substance filtered in a Bowman's space, which is a hypothetical space where a spacecraft is launched into orbit and then returns to Earth. The	The term used to describe the amount of substance filtered in Bowman's space per unit time is called the filtration efficiency.
What is the term used to describe the blood volume contained in the veins?	The blood volume contained in the veins is called the unstressed volume.	The term "blood volume" refers to the amount of blood that circulates through the body's circulatory system. It is a complex process that involves the exchange of oxygen, nutrients, and waste products between the blood and the tissues. The blood volume is typically measured in liters (L) or milliliters (mL) and is typically distributed throughout the body through the circulatory system. There are several key components that contribute to the blood volume: 1. Red blood cells: These are the cells that carry oxygen from the lungs to the tissues. They have a diameter of about 1.5 mm and are composed of hemoglobin	The term used to describe the blood volume contained in the veins is called venous blood.
What is the most important factor for determining the prognosis of a transitional cell carcinoma?	The most important factor for determining the prognosis of a transitional cell carcinoma is the degree of invasion.	Determining the prognosis of a transitional cell carcinoma (TCC) is crucial, as it can significantly impact the patient's quality of life, treatment options, and overall survival. The most important factor for determining the prognosis of a TCC is the presence of a clear diagnosis, a well-defined tumor, and a favorable prognosis. Here are some key factors that can influence the prognosis of a TCC: 1. Clear diagnosis: A TCC is diagnosed with a clear tumor, which means it has a benign or benign-like growth. A TCC is also diagnosed with a favorable prognosis, as it is unlikely to spread to	The most important factor for determining the prognosis of a transitional cell carcinoma is the presence of a high grade.

Observations

Accuracy of Responses: The fine-tuned model replicates the expected format and style but demonstrates some inaccuracies. For example, erythromycin is incorrectly identified as an antimalarial medication effective against Plasmodium hypnozoites. The correct answer should be primaquine.

Base Model Issues: The base model's responses are overly generalized and sometimes off-topic, especially in medical contexts, as seen in the incorrect explanations about Bowman's space and blood volume.

Usage

def generate_response(model, tokenizer, instruction, device="cpu"):
    """Generate a response from the model based on an instruction."""
    messages = [{"role": "user", "content": instruction}]
    input_text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
    outputs = model.generate(
        inputs, max_new_tokens=128, temperature=0.2, top_p=0.9, do_sample=True
    )
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

device = "cuda" if torch.cuda.is_available() else "cpu"
model_id = "mohres/SmolLM-135M-Instruct-medical_meadow_medical_flashcards-10epochs"

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)

instruction = "What is the term used to describe the blood volume contained in the veins?"
response = generate_response(model, tokenizer, instruction, device)
print(response)

Training procedure

For detailed information and the code used in this process, please refer to the GitHub repository: GitHub

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 10

Framework versions

• PEFT 0.12.0
• Transformers 4.44.1
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1