amanpreetsingh459
/

llama-2-7b-chat_q4_quantized_cpp

text-generation-inference

Model card Files Files and versions Community

amanpreetsingh459 commited on Dec 3, 2023

Commit

ba1963f

•

1 Parent(s): d36c2a3

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -9,7 +9,7 @@ tags:
 - This model contains the 4-bit quantized version of [llama2-7B-chat](https://github.com/facebookresearch/llama) model in cpp.
 - This can be run on a local cpu system as a cpp module *(instructions for the same are given below)*.
 - As for the testing, the model has been tested on `Linux(Ubuntu)` os with `12 GB RAM` and `core i5 processor`.
-- The performance is `roughly` **907.46 ms per token**, **1.10 tokens per second**
 # Usage:
 1. Clone the llama C++ repository from github:<br>
   `git clone https://github.com/ggerganov/llama.cpp.git`

 - This model contains the 4-bit quantized version of [llama2-7B-chat](https://github.com/facebookresearch/llama) model in cpp.
 - This can be run on a local cpu system as a cpp module *(instructions for the same are given below)*.
 - As for the testing, the model has been tested on `Linux(Ubuntu)` os with `12 GB RAM` and `core i5 processor`.
+- The performance is `roughly` **~3 tokens per second**
 # Usage:
 1. Clone the llama C++ repository from github:<br>
   `git clone https://github.com/ggerganov/llama.cpp.git`