I am learning how to make LoRAs with Oobabooga, these data are for experimental and research purposes.
This is a Medical Knowledge LoRA made for use with this model: llama-2-70b-Guanaco-QLoRA-fp16 https://huggingface.co/TheBloke/llama-2-70b-Guanaco-QLoRA-fp16) (quantized and merged models coming soon).
Model lineage:
https://huggingface.co/timdettmers/guanaco-65b -> https://huggingface.co/Mikael110/llama-2-70b-guanaco-qlora -> https://huggingface.co/TheBloke/llama-2-70b-Guanaco-QLoRA-fp16
Training Data and Formatting:
Training data are garnered from: https://huggingface.co/datasets/BI55/MedText
These training data were then formatted for use with the "Raw text file" training option in the Oobabooga text-generation-webui: (https://github.com/oobabooga/text-generation-webui)
Training parameters are in the training_parameters.json file and there is a screenshot of the UI with the correct settings.
Examples and Additional Information:
Check out the png files in the repo for an example conversation as well as other pieces of information that beginners might find useful.
Current/Future Work:
- Finish training with "Structed Dataset" I have a .json file with a structured dataset for the Guanaco model, but it takes significantly longer to process in the Oobabooga webui.
- Train the vanilla LlamaV2 70B model, with Raw and Structured data.
- Merge LoRA with LLM so you don't need to load the LoRA seperately.
Use at own risk, I am using this repo to both organize my results and potentially help others with LoRA training.
It is not the intention of this repo to purport medical information.
Refer to the reference material for licensing guidance. I don't care how you use this LoRA, but you should reference the licensing requirments of the reference material if you indend on using this for anything other than personal use.
I want to thank and acknowledge the hard work of the people involved in the creation of the dataset and Guanaco models/LoRA! Your work is greatly appreciated <3