Update README.md
Browse files
README.md
CHANGED
@@ -25,9 +25,33 @@ However, llama.cpp requires quantized files in the new GGUF format - that's wher
|
|
25 |
it contains a few quantizations of the original weights from Together's fined-tuned model (as indicated by
|
26 |
the file names)
|
27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
28 |
Concerning the license(s):
|
29 |
|
30 |
-
* the [
|
31 |
license](https://ai.meta.com/llama/license/)
|
32 |
* the fine tuned model from Together Computer uses the
|
33 |
[same license](https://huggingface.co/togethercomputer/LLaMA-2-7B-32K/blob/main/README.md)
|
|
|
25 |
it contains a few quantizations of the original weights from Together's fined-tuned model (as indicated by
|
26 |
the file names)
|
27 |
|
28 |
+
## How the Quantization was done ##
|
29 |
+
|
30 |
+
Since the author does not want arbitrary Python stuff loitering on his computer, the quatization was done
|
31 |
+
using [Docker](https://www.docker.com/).
|
32 |
+
|
33 |
+
Assuming that you have the [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed on
|
34 |
+
your system and also have a basic knowledge of how it is used, you mayx just follow the instructions shown
|
35 |
+
below in order to generate your own quantizations:
|
36 |
+
|
37 |
+
> Nota bene: you will need 30+x GB of free disk space, at least - depending on your quantization
|
38 |
+
|
39 |
+
1. create a new folder called `llama.cpp_in_Docker`<br>this folder will later be mounted into the Docker
|
40 |
+
container and store the quantization results
|
41 |
+
2. download the weights for the fine-tuned LLaMA-2 model from
|
42 |
+
[Hugging Face](https://huggingface.co/togethercomputer/LLaMA-2-7B-32K) into a subfolder of `llama.cpp_in_Docker`
|
43 |
+
(let's call the new folder `LLaMA-2-7B-32K`)
|
44 |
+
3. within the Docker Desktop, download search for and download a `basic-python` image - just use one of the
|
45 |
+
most popular ones
|
46 |
+
4. from a terminal session on your host computer (i.e., not a Docker container!), start a new container for the
|
47 |
+
downloaded image which mounts the folder we crated before:<br> <br>``
|
48 |
+
...
|
49 |
+
|
50 |
+
## License ##
|
51 |
+
|
52 |
Concerning the license(s):
|
53 |
|
54 |
+
* the [original model](https://ai.meta.com/llama/) (from Meta AI) was released under a rather [permittive
|
55 |
license](https://ai.meta.com/llama/license/)
|
56 |
* the fine tuned model from Together Computer uses the
|
57 |
[same license](https://huggingface.co/togethercomputer/LLaMA-2-7B-32K/blob/main/README.md)
|