File size: 5,088 Bytes
50728ae 2646694 50728ae |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 |
---
language:
- en
pipeline_tag: text-generation
tags:
- Pytorch
- Llama3
- M42
- Health
- HealthCare
---
# SandLogic Technologies - Quantized Llama3-Med42-8B Models
## Model Description
We have quantized the Llama3-Med42-8B model into two variants:
1. Q5_KM
2. Q4_KM
These quantized models offer improved efficiency while maintaining performance.
## Original Model Information
- **Name**: [Llama3-Med42-8B](https://huggingface.co/m42-health/Llama3-Med42-8B)
- **Developer**: M42 Health AI Team
- **Base Model**: [Llama3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
- **Model Type**: Clinical large language model (LLM)
- **Parameters**: 8 billion
- **Context Length**: 8k tokens
- **Input**: Text only
- **Output**: Text only
- **License**: Llama 3 Community License Agreement
## Model Capabilities
Llama3-Med42-8B is designed for medical and healthcare-related tasks, including:
- Medical question answering
- Patient record summarization
- Aiding medical diagnosis
- General health Q&A
## Training Data
The model was instruction-tuned using a dataset of approximately 1 billion tokens compiled from various open-access and high-quality sources, including:
- Medical flashcards
- Exam questions
- Open-domain dialogues
## Important Limitations and Safe Use
**DISCLAIMER: This model is not yet ready for clinical use without further testing and validation. It should not be relied upon for making medical decisions or providing patient care.**
- Potential for generating incorrect or harmful information
- Risk of perpetuating biases in training data
- Requires extensive human evaluation to ensure safety
## Use Cases
While not ready for real clinical use, potential applications include:
1. **Medical Education**: Assist in studying and understanding medical concepts
2. **Research Support**: Aid in literature review and hypothesis generation
3. **Health Information**: Provide general health information (with appropriate disclaimers)
4. **Clinical Decision Support**: (Future potential) Enhance clinical decision-making processes
## Model Variants
We offer two quantized versions of the Llama3-Med42-8B model:
1. **Q5_KM**: 5-bit quantization using the KM method
2. **Q4_KM**: 4-bit quantization using the KM method
These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.
## Usage
```bash
pip install llama-cpp-python
```
Please refer to the llama-cpp-python [documentation](https://llama-cpp-python.readthedocs.io/en/latest/) to install with GPU support.
### Basic Text Completion
Here's an example demonstrating how to use the high-level API for basic text completion:
```bash
from llama_cpp import Llama
llm = Llama(
model_path="./models/7B/Llama3-Med42-8B.gguf",
verbose=False,
# n_gpu_layers=-1, # Uncomment to use GPU acceleration
# n_ctx=2048, # Uncomment to increase the context window
)
output = llm.create_chat_completion(
messages =[
{
"role": "system",
"content": (
"You are a helpful, respectful and honest medical assistant. You are a second version of Med42 developed by the AI team at M42, UAE. "
"Always answer as helpfully as possible, while being safe. "
"Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. "
"Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. "
"If you don’t know the answer to a question, please don’t share false information."
),
},
{"role": "user", "content": "What are the symptoms of diabetes?"},
]
)
print(output["choices"][0]['message']['content'])
```
## Download
You can download `Llama` models in `gguf` format directly from Hugging Face using the `from_pretrained` method. This feature requires the `huggingface-hub` package.
To install it, run: `pip install huggingface-hub`
```bash
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="SandLogicTechnologies/Llama3-Med42-8B-GGUF",
filename="*Llama3-Med42-8B-Q5_K_M.gguf",
verbose=False
)
```
By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.
## Ethical Considerations
Users must be aware of the model's limitations and potential biases. It should not be used for direct medical advice or decision-making without proper validation and human oversight.
## Acknowledgements
We thank the M42 Health AI Team and the creators of Llama3 for their contributions to the field of medical AI.Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.
## Contact
For any inquiries or support, please contact us at [email protected] or visit our [support page](https://www.sandlogic.com/LingoForge/support). |