---
tags:
- ctranslate2
---
"Ctranslate2" is an amazing library that runs these models.  They are faster, more accurate, and use less VRAM/RAM than GGML and GPTQ models.

How to run with instructions: https://github.com/BBC-Esq
- COMING SOON

Learn more about the amazing "ctranslate2" technology:"
- https://github.com/OpenNMT/CTranslate2
- https://opennmt.net/CTranslate2/index.html

COMPARED to GGML:
- The VRAM numbers includes other programs running and a second monitor so people can get a realistic idea of how much VRAM/RAM is needed.
- THE FASTER AND HIGHER-QUALITY INT8 "ctranslate2" 7b model uses the same amount of VRAM as the far-inferior 3-bit "k_m" GGML version!!
						
<style>
    .ctranslate2-row {
        font-weight: bold;
        text-decoration: underline;
    }
</style>
<table border="1" cellpadding="5" cellspacing="0">
    <tr>
        <th>Comments</th>
        <th>Quant. Method</th>
        <th>Quant. Bit</th>
        <th>Model Specifics</th>
        <th>Params.</th>
        <th>VRAM Usage</th>
        <th>Size on disk</th>
    </tr>
    <tr><td>Original llama2 models</td><td></td><td>32-bit</td><td>llama2-13b-chat-hf</td><td>13B</td><td>high</td><td>24.2 (GB)</td></tr>
    <tr><td></td><td></td><td>32-bit</td><td>llama2-7b-chat-hf</td><td>7B</td><td>high</td><td>12.5 (GB)</td></tr>
    <tr><td>Comparable 13B models in terms of quality</td><td>ggml</td><td>8-bit</td><td>llama2-13b-chat_Q8_0</td><td>13B</td><td>21.2 (GB)</td><td>12.8 (GB)</td></tr>
    <tr class="ctranslate2-row"><td></td><td>ctranslate2</td><td>8-bit</td><td>Llama-2-13b-chat-hf-ct2-int8</td><td>13B</td><td>16 (GB)</td><td>6.28 (GB)</td></tr>
    <tr><td>Comparable 7B models in terms of quality</td><td>ggml</td><td>8-bit</td><td>llama2-7b-chat_Q8_0</td><td>7B</td><td>12 (GB)</td><td>6.66 (GB)</td></tr>
    <tr class="ctranslate2-row"><td></td><td>ctranslate2</td><td>8-bit</td><td>Llama-2-7b-chat-hf-ct2-int8</td><td>7B</td><td>10.2 (GB)</td><td>6.28 (GB)</td></tr>
    <tr><td>ggml quants lower than 8-bit for additional comparison</td><td>ggml</td><td>8-bit</td><td>llama2-7b-chat_Q6_k</td><td>7B</td><td>11.3 (GB)</td><td>5.14 (GB)</td></tr>
    <tr><td></td><td>ggml</td><td>5-bit</td><td>llama2-7b-chat_Q5_k_m</td><td>7B</td><td>11.6 (GB)</td><td>4.45 (GB)</td></tr>
    <tr><td></td><td>ggml</td><td>5-bit</td><td>llama2-7b-chat_5_k_s</td><td>7B</td><td>11.4 (GB)</td><td>4.33 (GB)</td></tr>
    <tr><td></td><td>ggml</td><td>4-bit</td><td>llama2-7b-chat_4_k_m</td><td>7B</td><td>11 (GB)</td><td>3.79 (GB)</td></tr>
    <tr><td></td><td>ggml</td><td>4-bit</td><td>llama2-7b-chat_4_k_s</td><td>7B</td><td>10.8 (GB)</td><td>3.56 (GB)</td></tr>
    <tr><td></td><td>ggml</td><td>3-bit</td><td>llama2-7b-chat_3_k_l</td><td>7B</td><td>10.5 (GB)</td><td>3.34 (GB)</td></tr>
    <tr><td></td><td>ggml</td><td>3-bit</td><td>llama2-7b-chat_3_k_m</td><td>7B</td><td>10.3 (GB)</td><td>3.05 (GB)</td></tr>
    <tr><td></td><td>ggml</td><td>3-bit</td><td>llama2-7b-chat_3_k_s</td><td>7B</td><td>10 (GB)</td><td>2.74 (GB)</td></tr>
</table>

Information:

| Format          | Approximate Size Compared to `float32` | Nvidia GPU Required "Compute" | Accuracy Summary |
|-----------------|----------------------------|-----------------|--------------------------|
| `float32`       | 100%                       | 1.0        | Offers more precision and a wider range.  Most un-quantized models use this. |
| `int16`         | 51.37%                     | 1.0        | Same as `int8` but with a larger range. |
| `float16`       | 50.00%                     | 5.3  (e.g. Nvidia 10 Series and Higher)      | Suitable for scientific computations; balance between precision and memory. |
| `bfloat16`      | 50.00%                     | 8.0  (e.g. Nvidia 30 Series and Higher)      | Often used in neural network training; larger exponent range than `float16`. |
| `int8_float32`  | 27.47%                     | test manually (see below)             | Combines low precision integer with high precision float. Useful for mixed data. |
| `int8_float16`  | 26.10%                     | test manually (see below)             | Combines low precision integer with medium precision float. Saves memory. |
| `int8_bfloat16` | 26.10%                     | test manually (see below)             | Combines low precision integer with reduced precision float. Efficient for neural nets. |
| `int8`          | 25%                        | 1.0        | Lower precision, suitable for whole numbers within a specific range. Often used where memory is crucial. |

| Web Link                                                                      | Description                                                                                                         |
|-------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| [CUDA GPUs Supported](https://en.wikipedia.org/wiki/CUDA#GPUs_supported)       | See what level of "compute" your Nvidia GPU supports.                                                               |
| [CTranslate2 Quantization](https://opennmt.net/CTranslate2/quantization.html#implicit-type-conversion-on-load) | Even if your GPU/CPU doesn't support the data type of the model you download, "ctranslate2" will automatically run the model in a way that's compatible. |
| [Bfloat16 Floating-Point Format](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format#bfloat16_floating-point_format) | Visualize data formats.                                                                                                           |
| [Nvidia Floating-Point](https://docs.nvidia.com/cuda/floating-point/index.html) | Technical discussion.                                                                                    |

You can also check compatibility manually AFTER pip installing "ctranslate2." Then, within the virtual environment where "ctranslate2" is installed open a command prompt and run the following commands:

   ```
   python
   ```

   ```python
   import ctranslate2
   ```

Check GPU/CUDA compatibility:

   ```python
   ctranslate2.get_supported_compute_types("cuda")
   ```

Check CPU compatibility:

   ```python
   ctranslate2.get_supported_compute_types("cpu")
   ```

It will print out your CPU/GPU compatibility.