Create README.md

README.md

@@ -0,0 +1,86 @@
+---
+license: apache-2.0
+datasets:
+- WizardLM/WizardLM_evol_instruct_V2_196k
+- leemeng/ShareGPT90K_ja_1392
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- nlp
+- llm
+---
+# AmberChat
+
+
+We present AmberChat, an instruction following model finetuned from [LLM360/Amber](https://huggingface.co/LLM360/Amber).
+
+## Model Description
+
+- **Model type:** Language model with the same architecture as LLaMA-7B
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Original Checkpoints:** [Aws bucket with AmberChat checkpoint with all available optimizer states](https://aws.amazon.com/)
+- **Resources for more information:**
+  - [Research paper](https://arxiv.org/)
+  - [GitHub Repo](https://github.com/LLM360)
+  - [Amber pretraining data](https://huggingface.co/)
+
+
+# Loading Amber 
+
+```python
+from transformers import LlamaTokenizer, LlamaForCausalLM
+
+tokenizer = LlamaTokenizer.from_pretrained("LLM360/AmberChat")
+model = LlamaForCausalLM.from_pretrained("LLM360/AmberChat")
+
+input_text = "translate English to German: How old are you?"
+input_ids = tokenizer(input_text, return_tensors="pt").input_ids
+
+outputs = model.generate(input_ids)
+print(tokenizer.decode(outputs[0]))
+```
+
+# AmberChat Finetuning Details
+
+## DataMix
+| Subset      | Number of rows |
+| ----------- | ----------- |
+| WizardLM/WizardLM_evol_instruct_V2_196k      | 143k       |
+| Sharegpt-90k   | 90k        |
+| Total | 233k |
+
+## Hyperparameters
+| Hyperparameter      | Value |
+| ----------- | ----------- |
+| Total Parameters      | 6.7B       |
+| Hidden Size   | 4096        |
+| Intermediate Size (MLPs)   | 11008        |
+| Number of Attention Heads   | 32        |
+| Number of Hidden Lyaers  | 32        |
+| RMSNorm ɛ  | 1e^-6        |
+| Max Seq Length   | 2048        |
+| Vocab Size | 32000 |
+
+
+# Evaluation
+
+| Model                                                | MT-Bench                                                  | 
+|------------------------------------------------------|------------------------------------------------------------|
+| LLM360/Amber 359 | 2.48750 | 
+| **LLM360/AmberChat** | **5.428125** |
+
+# Citation
+
+**BibTeX:**
+
+```bibtex
+@article{xxx,
+  title={XXX},
+  author={XXX},
+  journal={XXX},
+  year={2023}
+}
+```