Update README.md

README.md

@@ -15,21 +15,96 @@ Luxeai-anu-1-bit-70M
 
 ## Model Description
 The Luxeai-anu-1-bit-70M Large Language Model (LLM) is my first trial to implement one-bit LLM based on the original paper - "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits". I have taken the pre-trained Mistral-7B-v0.3 and abideen/Cosmopedia-100k-pretrain dataset. 
+I used Microsoft Azure Standard_NC6s_v3 6 cores, 112GB RAM, 736GB storage 1 x NVIDIA Tesla V100 to train this initial model. I will be training on a much bigger dataset once I get a sponshorship for a 8x DGX System. I have tested on a sub-set of the same dataset.
 
 ## Intended Use
-- **Task**: Describe the specific tasks (e.g., sentiment analysis, text generation) the model is designed for.
-- **Industries**: Mention any particular industries or applications where the model could be applied.
-- **Users**: Identify the intended users (e.g., researchers, developers).
+- **Task**: text generation
+
 
 ## How to Use
-Provide code examples for loading and using the model:
+Please follow the below code to run and test it in Python Jupyter Notebook
 
 ```python
-from transformers import AutoModel, AutoTokenizer
+pip install transformers sentencepiece
+
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+from transformers.models.llama.modeling_llama import *
+# Load a pretrained BitNet model
+model = "arunb74/Luxeai-anu-1-bit-70M"
+tokenizer = AutoTokenizer.from_pretrained(model)
+model = AutoModelForCausalLM.from_pretrained(model)
+
+def activation_norm_quant(x):
+    x = RMSNorm(x)
+    scale = 127.0 / x.abs().max(dim=-1, keepdim=True).values.clamp_(min=1e-5)
+    y = (x * scale).round().clamp_(-128, 127)
+    y = y / scale
+    return y, scale
+
+def weight_quant(w):
+    scale = 1.0 / w.abs().mean().clamp_(min=1e-5)
+    u = (w * scale).round().clamp_(-1, 1)
+    u = u / scale
+    return u
+
+class BitNetInference(nn.Linear):
+    def forward(self, x):
+        w = self.weight # a weight tensor with shape [d, k]
+        x = x.to(w.device)
+        RMSNorm = LlamaRMSNorm(x.shape[-1]).to(w.device)
+        x_norm = RMSNorm(x)
+        # A trick for implementing Straight−Through−Estimator (STE) using detach()
+        x_quant = x_norm + (activation_norm_quant(x_norm) - x_norm).detach()
+        w_quant = w + (weight_quant(w) - w).detach()
+        y = F.linear(x_quant, w_quant)
+        return y
+
+
+def convert_to_bitnet(model, copy_weights):
+    for name, module in model.named_modules():
+        # Replace linear layers with BitNet
+        if isinstance(module, LlamaSdpaAttention) or isinstance(module, LlamaMLP):
+            for child_name, child_module in module.named_children():
+                if isinstance(child_module, nn.Linear):
+                    bitlinear = BitNetInference(child_module.in_features, child_module.out_features, child_module.bias is not None).to(device="cuda:0")
+                    if copy_weights:
+                        bitlinear.weight = child_module.weight
+                        if child_module.bias is not None:
+                            bitlinear.bias = child_module.bias
+                    setattr(module, child_name, bitlinear)
+        # Remove redundant input_layernorms
+        elif isinstance(module, LlamaDecoderLayer):
+            for child_name, child_module in module.named_children():
+                if isinstance(child_module, LlamaRMSNorm) and child_name == "input_layernorm":
+                    setattr(module, child_name, nn.Identity().to(device="cuda:0"))
+
+
+convert_to_bitnet(model, copy_weights=True)
+
+# Create a text generation pipeline
+pipe = pipeline(
+    "text-generation", 
+    model=model, 
+    tokenizer=tokenizer, 
+    device_map="auto"
+)
+
+prompt = "The LISA Pathfinder scientific collaboration will meet in Trento"
+
+sequences = pipe(
+    f"<s>[INST] {prompt} [/INST]",
+    do_sample=True,
+    max_new_tokens=100, 
+    temperature=0.7, 
+    top_k=50, 
+    top_p=0.95,
+    num_return_sequences=1,
+)
+
+print(sequences[0]['generated_text'])
+
+The output will be as follows - <s>[INST] The LISA Pathfinder scientific collaboration will meet in Trento [/INST]
 
-tokenizer = AutoTokenizer.from_pretrained("username/model_name")
-model = AutoModel.from_pretrained("username/model_name")
+The LISA Pathfinder Biology, a leading provider of biochemistry and molecular biology, provides a comprehensive understanding of the mechanisms and mechanisms of the LISA pathways. The LISA Pathfinder Biology, a researcher specializing in molecular biology, is a clinical trial of the disease, and its pathophysiology, and a combination of the most commonly used and widely used treatments. It is a relatively simple procedure that involves two steps.
 
-# Example usage
-inputs = tokenizer("Hello, world!", return_tensors="pt")
-outputs = model(**inputs)
+# I need community members to help me further for feedback, suitable dataset for further training, testing, evaluation.