aghorbani
/

h2o-danube2-1-8b-chat-json-v0-1-fa2

@@ -1,199 +1,198 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+language:
+- en
 library_name: transformers
+tags:
+- gpt
+- llm
+- large language model
+- h2o-llmstudio
+inference: false
+thumbnail: https://h2o.ai/etc.clientlibs/h2o/clientlibs/clientlib-site/resources/images/favicon.ico
 ---
+# Model Card
+## Summary
+This model was trained using [H2O LLM Studio](https://github.com/h2oai/h2o-llmstudio).
+- Base model: [h2oai/h2o-danube2-1.8b-chat](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat)
+## Usage
+To use the model with the `transformers` library on a machine with GPUs, first make sure you have the `transformers` library installed.
+```bash
+pip install transformers==4.38.2
+```
+Also make sure you are providing your huggingface token to the pipeline if the model is lying in a private repo.
+    - Either leave `token=True` in the `pipeline` and login to hugginface_hub by running
+        ```python
+        import huggingface_hub
+        huggingface_hub.login(<ACCESS_TOKEN>)
+        ```
+    - Or directly pass your <ACCESS_TOKEN> to `token` in the `pipeline`
+```python
+from transformers import pipeline
+generate_text = pipeline(
+    model="aghorbani/h2o-danube2-1-8b-chat-json-v0-1-fa2",
+    torch_dtype="auto",
+    trust_remote_code=True,
+    use_fast=True,
+    device_map={"": "cuda:0"},
+    token=True,
+)
+res = generate_text(
+    "Why is drinking water so healthy?",
+    min_new_tokens=2,
+    max_new_tokens=256,
+    do_sample=False,
+    num_beams=1,
+    temperature=float(0.0),
+    repetition_penalty=float(1.0),
+    renormalize_logits=True
+)
+print(res[0]["generated_text"])
+```
+You can print a sample prompt after the preprocessing step to see how it is feed to the tokenizer:
+```python
+print(generate_text.preprocess("Why is drinking water so healthy?")["prompt_text"])
+```
+```bash
+<|prompt|>Why is drinking water so healthy?</s><|answer|>
+```
+Alternatively, you can download [h2oai_pipeline.py](h2oai_pipeline.py), store it alongside your notebook, and construct the pipeline yourself from the loaded model and tokenizer. If the model and the tokenizer are fully supported in the `transformers` package, this will allow you to set `trust_remote_code=False`.
+```python
+from h2oai_pipeline import H2OTextGenerationPipeline
+from transformers import AutoModelForCausalLM, AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained(
+    "aghorbani/h2o-danube2-1-8b-chat-json-v0-1-fa2",
+    use_fast=True,
+    padding_side="left",
+    trust_remote_code=True,
+)
+model = AutoModelForCausalLM.from_pretrained(
+    "aghorbani/h2o-danube2-1-8b-chat-json-v0-1-fa2",
+    torch_dtype="auto",
+    device_map={"": "cuda:0"},
+    trust_remote_code=True,
+)
+generate_text = H2OTextGenerationPipeline(model=model, tokenizer=tokenizer)
+res = generate_text(
+    "Why is drinking water so healthy?",
+    min_new_tokens=2,
+    max_new_tokens=256,
+    do_sample=False,
+    num_beams=1,
+    temperature=float(0.0),
+    repetition_penalty=float(1.0),
+    renormalize_logits=True
+)
+print(res[0]["generated_text"])
+```
+You may also construct the pipeline from the loaded model and tokenizer yourself and consider the preprocessing steps:
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model_name = "aghorbani/h2o-danube2-1-8b-chat-json-v0-1-fa2"  # either local folder or huggingface model name
+# Important: The prompt needs to be in the same format the model was trained with.
+# You can find an example prompt in the experiment logs.
+prompt = "<|prompt|>How are you?</s><|answer|>"
+tokenizer = AutoTokenizer.from_pretrained(
+    model_name,
+    use_fast=True,
+    trust_remote_code=True,
+)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map={"": "cuda:0"},
+    trust_remote_code=True,
+)
+model.cuda().eval()
+inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
+# generate configuration can be modified to your needs
+tokens = model.generate(
+    input_ids=inputs["input_ids"],
+    attention_mask=inputs["attention_mask"],
+    min_new_tokens=2,
+    max_new_tokens=256,
+    do_sample=False,
+    num_beams=1,
+    temperature=float(0.0),
+    repetition_penalty=float(1.0),
+    renormalize_logits=True
+)[0]
+tokens = tokens[inputs["input_ids"].shape[1]:]
+answer = tokenizer.decode(tokens, skip_special_tokens=True)
+print(answer)
+```
+## Quantization and sharding
+You can load the models using quantization by specifying ```load_in_8bit=True``` or ```load_in_4bit=True```. Also, sharding on multiple GPUs is possible by setting ```device_map=auto```.
+## Model Architecture
+```
+MistralForCausalLM(
+  (model): MistralModel(
+    (embed_tokens): Embedding(32000, 2560, padding_idx=0)
+    (layers): ModuleList(
+      (0-23): 24 x MistralDecoderLayer(
+        (self_attn): MistralFlashAttention2(
+          (q_proj): Linear(in_features=2560, out_features=2560, bias=False)
+          (k_proj): Linear(in_features=2560, out_features=640, bias=False)
+          (v_proj): Linear(in_features=2560, out_features=640, bias=False)
+          (o_proj): Linear(in_features=2560, out_features=2560, bias=False)
+          (rotary_emb): MistralRotaryEmbedding()
+        )
+        (mlp): MistralMLP(
+          (gate_proj): Linear(in_features=2560, out_features=6912, bias=False)
+          (up_proj): Linear(in_features=2560, out_features=6912, bias=False)
+          (down_proj): Linear(in_features=6912, out_features=2560, bias=False)
+          (act_fn): SiLU()
+        )
+        (input_layernorm): MistralRMSNorm()
+        (post_attention_layernorm): MistralRMSNorm()
+      )
+    )
+    (norm): MistralRMSNorm()
+  )
+  (lm_head): Linear(in_features=2560, out_features=32000, bias=False)
+)
+```
+## Model Configuration
+This model was trained using H2O LLM Studio and with the configuration in [cfg.yaml](cfg.yaml). Visit [H2O LLM Studio](https://github.com/h2oai/h2o-llmstudio) to learn how to train your own large language models.
+## Disclaimer
+Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.
+- Biases and Offensiveness: The large language model is trained on a diverse range of internet text data, which may contain biased, racist, offensive, or otherwise inappropriate content. By using this model, you acknowledge and accept that the generated content may sometimes exhibit biases or produce content that is offensive or inappropriate. The developers of this repository do not endorse, support, or promote any such content or viewpoints.
+- Limitations: The large language model is an AI-based tool and not a human. It may produce incorrect, nonsensical, or irrelevant responses. It is the user's responsibility to critically evaluate the generated content and use it at their discretion.
+- Use at Your Own Risk: Users of this large language model must assume full responsibility for any consequences that may arise from their use of the tool. The developers and contributors of this repository shall not be held liable for any damages, losses, or harm resulting from the use or misuse of the provided model.
+- Ethical Considerations: Users are encouraged to use the large language model responsibly and ethically. By using this model, you agree not to use it for purposes that promote hate speech, discrimination, harassment, or any form of illegal or harmful activities.
+- Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
+- Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.
+By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.