File size: 3,667 Bytes
92ebaf8 348e709 92ebaf8 d126142 92ebaf8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
# starcoderbase-3b-GPTQ
Quantized starcoderbase-3b model to GPTQ format (4-bit precision) using [Auto-GPTQ](https://github.com/AutoGPTQ/AutoGPTQ).
[Quantization script](https://github.com/cosmo3769/Quantized-LLMs/blob/main/notebooks/quantize-starcodebase-3b-gptq.ipynb)
## Benchmark
[Benchmarking script](https://github.com/cosmo3769/Quantized-LLMs/blob/main/notebooks/llmbenchmark-starcodebase-3b-lm-eval-harness.ipynb)
### Baseline starcoderbase-3b model (non-quantized)
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|-----------------------|-------|------|------|---------------|-----:|---|-----:|
|codexglue_code2text |N/A |none |None |smoothed_bleu_4|1.3519|± |0.3067|
| - code2text_go | 1|none |None |smoothed_bleu_4|1.5781|± |0.3734|
| - code2text_java | 1|none |None |smoothed_bleu_4|1.2778|± |0.1991|
| - code2text_javascript| 1|none |None |smoothed_bleu_4|1.1443|± |0.1181|
| - code2text_php | 1|none |None |smoothed_bleu_4|0.5171|± |0.5171|
| - code2text_python | 1|none |None |smoothed_bleu_4|2.8338|± |1.5323|
| - code2text_ruby | 3|none |None |smoothed_bleu_4|0.7601|± |0.7601|
| Groups |Version|Filter|n-shot| Metric |Value | |Stderr|
|-------------------|-------|------|------|---------------|-----:|---|-----:|
|codexglue_code2text|N/A |none |None |smoothed_bleu_4|1.3519|± |0.3067|
| Tasks |Version|Filter|n-shot| Metric |Value| |Stderr|
|---------------------------------------------|------:|------|------|-----------|----:|---|-----:|
|bigbench_code_line_description_generate_until| 1|none |None |exact_match| 0|± | 0|
| Tasks |Version|Filter|n-shot|Metric|Value| |Stderr|
|----------------------------------------------|------:|------|------|------|----:|---|-----:|
|bigbench_code_line_description_multiple_choice| 0|none |None |acc | 0.25|± |0.0564|
### Quantized starcoderbase-3b model to GPTQ format
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|-----------------------|-------|------|------|---------------|-----:|---|-----:|
|codexglue_code2text |N/A |none |None |smoothed_bleu_4|0.9254|± |0.2109|
| - code2text_go | 1|none |None |smoothed_bleu_4|1.4702|± |0.4813|
| - code2text_java | 1|none |None |smoothed_bleu_4|0.6907|± |0.6907|
| - code2text_javascript| 1|none |None |smoothed_bleu_4|0.9469|± |0.0339|
| - code2text_php | 1|none |None |smoothed_bleu_4|0.5171|± |0.5171|
| - code2text_python | 1|none |None |smoothed_bleu_4|1.1676|± |0.2156|
| - code2text_ruby | 3|none |None |smoothed_bleu_4|0.7601|± |0.7601|
| Groups |Version|Filter|n-shot| Metric |Value | |Stderr|
|-------------------|-------|------|------|---------------|-----:|---|-----:|
|codexglue_code2text|N/A |none |None |smoothed_bleu_4|0.9254|± |0.2109|
| Tasks |Version|Filter|n-shot| Metric |Value| |Stderr|
|---------------------------------------------|------:|------|------|-----------|----:|---|-----:|
|bigbench_code_line_description_generate_until| 1|none |None |exact_match| 0|± | 0|
| Tasks |Version|Filter|n-shot|Metric|Value| |Stderr|
|----------------------------------------------|------:|------|------|------|----:|---|-----:|
|bigbench_code_line_description_multiple_choice| 0|none |None |acc | 0.1|± | 0.1| |