maria-ai commited on
Commit
364b67b
1 Parent(s): fb4391c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +92 -0
README.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - fr
5
+ - ru
6
+ - de
7
+ - ja
8
+ - ko
9
+ - zh
10
+ - it
11
+ - uk
12
+ - multilingual
13
+ - code
14
+ license: apache-2.0
15
+ library_name: transformers
16
+ tags:
17
+ - mistral
18
+ - gistral
19
+ - gistral-16b
20
+ - multilingual
21
+ - code
22
+ - 128k
23
+ - metamath
24
+ - grok-1
25
+ - anthropic
26
+ - openhermes
27
+ - instruct
28
+ - merge
29
+ - llama-cpp
30
+ - gguf-my-repo
31
+ base_model: ehristoforu/Gistral-16B
32
+ datasets:
33
+ - HuggingFaceH4/grok-conversation-harmless
34
+ - HuggingFaceH4/ultrachat_200k
35
+ - HuggingFaceH4/ultrafeedback_binarized_fixed
36
+ - HuggingFaceH4/cai-conversation-harmless
37
+ - meta-math/MetaMathQA
38
+ - emozilla/yarn-train-tokenized-16k-mistral
39
+ - snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset
40
+ - microsoft/orca-math-word-problems-200k
41
+ - m-a-p/Code-Feedback
42
+ - teknium/openhermes
43
+ - lksy/ru_instruct_gpt4
44
+ - IlyaGusev/ru_turbo_saiga
45
+ - IlyaGusev/ru_sharegpt_cleaned
46
+ - IlyaGusev/oasst1_ru_main_branch
47
+ pipeline_tag: text-generation
48
+ ---
49
+
50
+ # maria-ai/Gistral-16B-Q4_K_S-GGUF
51
+ This model was converted to GGUF format from [`ehristoforu/Gistral-16B`](https://huggingface.co/ehristoforu/Gistral-16B) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
52
+ Refer to the [original model card](https://huggingface.co/ehristoforu/Gistral-16B) for more details on the model.
53
+
54
+ ## Use with llama.cpp
55
+ Install llama.cpp through brew (works on Mac and Linux)
56
+
57
+ ```bash
58
+ brew install llama.cpp
59
+
60
+ ```
61
+ Invoke the llama.cpp server or the CLI.
62
+
63
+ ### CLI:
64
+ ```bash
65
+ llama --hf-repo maria-ai/Gistral-16B-Q4_K_S-GGUF --hf-file gistral-16b-q4_k_s.gguf -p "The meaning to life and the universe is"
66
+ ```
67
+
68
+ ### Server:
69
+ ```bash
70
+ llama-server --hf-repo maria-ai/Gistral-16B-Q4_K_S-GGUF --hf-file gistral-16b-q4_k_s.gguf -c 2048
71
+ ```
72
+
73
+ Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.
74
+
75
+ Step 1: Clone llama.cpp from GitHub.
76
+ ```
77
+ git clone https://github.com/ggerganov/llama.cpp
78
+ ```
79
+
80
+ Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
81
+ ```
82
+ cd llama.cpp && LLAMA_CURL=1 make
83
+ ```
84
+
85
+ Step 3: Run inference through the main binary.
86
+ ```
87
+ ./main --hf-repo maria-ai/Gistral-16B-Q4_K_S-GGUF --hf-file gistral-16b-q4_k_s.gguf -p "The meaning to life and the universe is"
88
+ ```
89
+ or
90
+ ```
91
+ ./server --hf-repo maria-ai/Gistral-16B-Q4_K_S-GGUF --hf-file gistral-16b-q4_k_s.gguf -c 2048
92
+ ```