rustformers
/

bloomz-ggml

+---
+datasets:
+- bigscience/xP3
+license: bigscience-bloom-rail-1.0
+language:
+- ak
+- ar
+- as
+- bm
+- bn
+- ca
+- code
+- en
+- es
+- eu
+- fon
+- fr
+- gu
+- hi
+- id
+- ig
+- ki
+- kn
+- lg
+- ln
+- ml
+- mr
+- ne
+- nso
+- ny
+- or
+- pa
+- pt
+- rn
+- rw
+- sn
+- st
+- sw
+- ta
+- te
+- tn
+- ts
+- tum
+- tw
+- ur
+- vi
+- wo
+- xh
+- yo
+- zh
+- zu
+programming_language:
+- C
+- C++
+- C#
+- Go
+- Java
+- JavaScript
+- Lua
+- PHP
+- Python
+- Ruby
+- Rust
+- Scala
+- TypeScript
+tags:
+- llm-rs
+- ggml
+pipeline_tag: text-generation
+---
+# GGML covnerted Models of [BigScience](https://huggingface.co/bigscience)'s Bloom models
+## Description
+> We present BLOOMZ & mT0, a family of models capable of following human instructions in dozens of languages zero-shot. We finetune BLOOM & mT5 pretrained multilingual language models on our crosslingual task mixture (xP3) and find the resulting models capable of crosslingual generalization to unseen tasks & languages.
+- **Repository:** [bigscience-workshop/xmtf](https://github.com/bigscience-workshop/xmtf)
+- **Paper:** [Crosslingual Generalization through Multitask Finetuning](https://arxiv.org/abs/2211.01786)
+- **Point of Contact:** [Niklas Muennighoff](mailto:[email protected])
+- **Languages:** Refer to [bloom](https://huggingface.co/bigscience/bloom) for pretraining & [xP3](https://huggingface.co/datasets/bigscience/xP3) for finetuning language proportions. It understands both pretraining & finetuning languages.
+### Intended use
+We recommend using the model to perform tasks expressed in natural language. For example, given the prompt "*Translate to English: Je t’aime.*", the model will most likely answer "*I love you.*". Some prompt ideas from our paper:
+- 一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。你认为这句话的立场是赞扬、中立还是批评?
+- Suggest at least five related search terms to "Mạng neural nhân tạo".
+- Write a fairy tale about a troll saving a princess from a dangerous dragon. The fairy tale is a masterpiece that has achieved praise worldwide and its moral is "Heroes Come in All Shapes and Sizes". Story (in Spanish):
+- Explain in a sentence in Telugu what is backpropagation in neural networks.
+## Converted Models
+| Name                                                                                                            | Based on                                                                | Type   | Container   | GGML Version   |
+|:----------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:-------|:------------|:---------------|
+| [bloomz-1b1-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-f16.bin)               | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | F16    | GGML        | V3             |
+| [bloomz-1b1-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q4_0.bin)             | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q4_0   | GGML        | V3             |
+| [bloomz-1b1-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q4_0-ggjt.bin)   | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q4_0   | GGJT        | V3             |
+| [bloomz-1b1-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q5_1-ggjt.bin)   | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q5_1   | GGJT        | V3             |
+| [bloomz-1b7-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-f16.bin)               | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | F16    | GGML        | V3             |
+| [bloomz-1b7-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q4_0.bin)             | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q4_0   | GGML        | V3             |
+| [bloomz-1b7-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q4_0-ggjt.bin)   | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q4_0   | GGJT        | V3             |
+| [bloomz-1b7-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q5_1-ggjt.bin)   | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q5_1   | GGJT        | V3             |
+| [bloomz-3b-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-f16.bin)                 | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | F16    | GGML        | V3             |
+| [bloomz-3b-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q4_0.bin)               | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q4_0   | GGML        | V3             |
+| [bloomz-3b-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q4_0-ggjt.bin)     | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q4_0   | GGJT        | V3             |
+| [bloomz-3b-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q5_1-ggjt.bin)     | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q5_1   | GGJT        | V3             |
+| [bloomz-560m-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-f16.bin)             | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | F16    | GGML        | V3             |
+| [bloomz-560m-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q4_0.bin)           | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q4_0   | GGML        | V3             |
+| [bloomz-560m-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q4_0-ggjt.bin) | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q4_0   | GGJT        | V3             |
+| [bloomz-560m-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q5_1-ggjt.bin) | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q5_1   | GGJT        | V3             |
+## Usage
+### Python via [llm-rs](https://github.com/LLukas22/llm-rs-python):
+#### Installation
+Via pip: `pip install llm-rs`
+#### Run inference
+```python
+from llm_rs import AutoModel
+#Load the model, define any model you like from the list above as the `model_file`
+model = AutoModel.from_pretrained("rustformers/bloomz-ggml",model_file="bloomz-3b-q4_0-ggjt.bin")
+#Generate
+print(model.generate("The meaning of life is"))
+```
+### Rust via [Rustformers/llm](https://github.com/rustformers/llm):
+#### Installation
+```
+git clone --recurse-submodules https://github.com/rustformers/llm.git
+cd llm
+cargo build --release
+```
+#### Run inference
+```
+cargo run --release -- bloom infer -m path/to/model.bin  -p "Tell me how cool the Rust programming language is:"
+```