Update README.md

README.md

@@ -1,3 +1,156 @@
 ---
 license: apache-2.0
 ---
+
+# HelixNet-LMoE
+
+HelixNet-LMoE is a simple LoRA based Mixture of Experts version of the [HelixNet](https://huggingface.co/migtissera/HelixNet) 3-model system by [Migel Tissera](https://huggingface.co/migtissera). \
+It is a 6bpw multi-lora exl2 model for use with ExLlamaV2.
+
+For each HelixNet model, a separate LoRA adapter was extracted:
+ * lora-actor
+ * lora-critic
+ * lora-regenerator
+
+These are then loaded together with the base [Mistral 7b](https://huggingface.co/mistralai/Mistral-7B-v0.1) model which has been quantized to 6bpw-exl2 to give the combined LMoE model.
+
+As HelixNet processes its inputs using the actor, critic and regenerator actions, the corresponding LoRA adapter is dynamically enabled as required.
+
+It is similar in approach to [Airoboro's MoE implementation](https://github.com/jondurbin/airoboros/tree/main#lmoe) allowing GPU memory requirements in this (6bpw quantized) instance to be reduced from 20GB for the 3 separate 6bpw models to 8GB for the 6bpw multi lora model.
+The LoRAs were extracted based on the process given in [https://github.com/uukuguy/multi_loras](https://github.com/uukuguy/multi_loras), with a Rank of 64 and an Alpha of 128.
+
+# Performance:
+Testing on an RTX-4090 to compare with the separate 6bpw exl2 models from [https://huggingface.co/LoneStriker?search_models=helixnet](https://huggingface.co/LoneStriker?search_models=helixnet) gives:
+
+**3 separate models:** 120 tokens / second, using 20GB GPU\
+**LMoE combined model:** 91 tokens / second, using 8GB GPU 
+
+
+# Prompt format:
+
+```
+SYSTEM: Elaborate on the topic using a Tree of Thoughts and backtrack when necessary to construct a clear, cohesive Chain of Thought reasoning. Always answer without hesitation.
+USER: What is the relationship between Earth's atmosphere, magnetic field and gravity?
+ASSISTANT:
+```
+# Example Usage
+
+The following is a code example on how to use HelixNet-LMoE. No special system-context messages are needed for the `critic` and the `regenerator`. \
+At the **You:** prompt, enter a question such as _What is the relationship between Earth's atmosphere, magnetic field and gravity?_
+
+```python
+import time
+import sys, os
+import dataclasses
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from exllamav2 import(
+    ExLlamaV2,
+    ExLlamaV2Config,
+    ExLlamaV2Cache,
+    ExLlamaV2Tokenizer,
+    ExLlamaV2Lora,    
+)
+
+from exllamav2.generator import (
+    ExLlamaV2BaseGenerator,
+    ExLlamaV2Sampler
+)
+
+
+class ModelClass:
+    def __init__(self, generator, tokenizer, model):
+        self.generator = generator
+        self.tokenizer = tokenizer
+        self.model = model
+
+DEBUG = os.environ.get("DEBUG") and True or False
+
+# Initialize model and cache
+def load_model(model_directory, max_seq_len=8192):
+    """
+    Loads a model from a directory and return the generator and tokenizer
+    """
+    config = ExLlamaV2Config()
+    config.model_dir = model_directory
+    config.max_seq_len = max_seq_len
+    config.prepare()
+
+    model = ExLlamaV2(config)
+    print("Loading model: " + model_directory)
+
+    cache = ExLlamaV2Cache(model, lazy = True, max_seq_len=max_seq_len)
+    model.load_autosplit(cache)
+
+    tokenizer = ExLlamaV2Tokenizer(config)
+    generator = ExLlamaV2BaseGenerator(model, cache, tokenizer)
+    model = ModelClass(generator=generator, tokenizer=tokenizer, model=model)
+    generator.warmup()
+    return model
+
+def generate_text(prompt, lora, settings, max_new_tokens):
+    time_begin = time.time()
+    response = base_model.generator.generate_simple(prompt, settings, max_new_tokens, loras=lora)
+    response = response[len(prompt):]
+    time_end = time.time()
+    time_total = time_end - time_begin
+    tokens = base_model.tokenizer.encode(response)
+    count = tokens.shape[-1]
+    print(f"Response generated in {time_total:.2f} seconds, {count} tokens, {count / time_total:.2f} tokens/second, character len: {len(response)}")
+    return response
+
+base_model = load_model("models/HelixNet-LMoE-6.0bpw-h6-exl2")
+lora_actor = ExLlamaV2Lora.from_directory(base_model.model, "models/HelixNet-LMoE-6.0bpw-h6-exl2/lora-actor")
+lora_critic = ExLlamaV2Lora.from_directory(base_model.model, "models/HelixNet-LMoE-6.0bpw-h6-exl2/lora-critic")
+lora_regenerator = ExLlamaV2Lora.from_directory(base_model.model, "models/HelixNet-LMoE-6.0bpw-h6-exl2/lora-regenerator")
+
+settings = ExLlamaV2Sampler.Settings()
+settings.temperature = 0.75
+settings.top_k = 50
+settings.top_p = 1.0
+max_new_tokens = 2000
+
+system_prompt = "You are HelixNet. Elaborate on the topic using a Tree of Thoughts and backtrack when necessary to construct a clear, cohesive Chain of Thought reasoning. Always answer without hesitation."
+
+while True:
+    user_input = input("You: ")
+
+    prompt_actor = f"SYSTEM: {system_prompt}\nUSER: {user_input}\nASSISTANT: "
+    if DEBUG: print(f"{prompt_actor}\n\n")
+    print("ACTOR:")
+    response_actor = generate_text(prompt_actor, lora_actor, settings, max_new_tokens)
+    if DEBUG: print(f"{response_actor}\n\n")
+    print("="*132)
+
+    prompt_critic = f"SYSTEM: {system_prompt}\nUSER: {user_input}\nRESPONSE: {response_actor}\nCRITIQUE: "
+    if DEBUG: print(f"{prompt_critic}\n\n")
+    print("CRITIQUE:")
+    response_critic = generate_text(prompt_critic, lora_critic, settings, max_new_tokens)
+    if DEBUG: print(f"{response_critic}\n\n")
+    print("="*132)
+
+    prompt_regenerator = f"SYSTEM: {system_prompt}\nUSER: {user_input}\nRESPONSE: {response_actor}\nCRITIQUE: {response_critic}\nREGENERATOR: "
+    if DEBUG: print(f"{prompt_regenerator}\n\n")
+    print("REGENERATION:")
+    response_regenerator = generate_text(prompt_regenerator, lora_regenerator, settings, max_new_tokens)
+    print("="*132)
+    conversation = f"SYSTEM: {system_prompt}\nUSER: {user_input}\nASSISTANT: {response_regenerator}"
+    print(conversation)
+
+```
+
+# LLM Evaluation
+
+Evaluation on a merged version of each base+lora models has yet to be done on the [HuggingFaceH4 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) to see how it compares to the equivalent full HelixNet model.
+
+# HelixNet Details
+
+HelixNet is a Deep Learning architecture consisting of 3 x Mistral-7B LLMs. It has an `actor`, a `critic`, and a `regenerator`. The `actor` LLM produces an initial response to a given system-context and a question. The `critic` then takes in as input, a tuple of (system-context, question, response) and provides a critique based on the provided answer to the given system-context and the question. Its job is not to criticize, but to provide an intelligent critique so that the answer can be modified/regenerated to address the question better. Finally, the `regenerator` takes in a tuple of (system-context, question, response, critique) and regenerates the answer.
+
+HelixNet is insprired from an actor-critic architecture most prominent in Reinforcement Learning algorithms. The name derives from Helix, referring to the spiral structure of a DNA molecule. It symbolizes the intertwined nature of the three networks, working in tandem, much like the strands of a DNA molecule.
+
+HelixNet regenerates very pleasing and accurate responses, due to the entropy preservation of the regenerator. The regenerator was only trained on a dataset of 1000 samples, similar to Meta's LIMA. The actor network here was trained on about 250K very high-quality samples, and the critic network was trained on further 10K samples.
+
+Full details on how HelixNet was trained and evaluated is located at [https://huggingface.co/migtissera/HelixNet](https://huggingface.co/migtissera/HelixNet) \
+The 6bpw separate models for HelixNet are available at [https://huggingface.co/LoneStriker?search_models=helixnet](https://huggingface.co/LoneStriker?search_models=helixnet)
+