| # starcoderbase-3b-GPTQ | |
| Quantized starcoderbase-3b model to GPTQ format (4-bit precision) using [Auto-GPTQ](https://github.com/AutoGPTQ/AutoGPTQ). | |
| [Quantization script](https://github.com/cosmo3769/Quantized-LLMs/blob/main/notebooks/quantize-starcodebase-3b-gptq.ipynb) | |
| ## Benchmark | |
| [Benchmarking script](https://github.com/cosmo3769/Quantized-LLMs/blob/main/notebooks/llmbenchmark-starcodebase-3b-lm-eval-harness.ipynb) | |
| ### Baseline starcoderbase-3b model (non-quantized) | |
| | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| | |
| |-----------------------|-------|------|------|---------------|-----:|---|-----:| | |
| |codexglue_code2text |N/A |none |None |smoothed_bleu_4|1.3519|± |0.3067| | |
| | - code2text_go | 1|none |None |smoothed_bleu_4|1.5781|± |0.3734| | |
| | - code2text_java | 1|none |None |smoothed_bleu_4|1.2778|± |0.1991| | |
| | - code2text_javascript| 1|none |None |smoothed_bleu_4|1.1443|± |0.1181| | |
| | - code2text_php | 1|none |None |smoothed_bleu_4|0.5171|± |0.5171| | |
| | - code2text_python | 1|none |None |smoothed_bleu_4|2.8338|± |1.5323| | |
| | - code2text_ruby | 3|none |None |smoothed_bleu_4|0.7601|± |0.7601| | |
| | Groups |Version|Filter|n-shot| Metric |Value | |Stderr| | |
| |-------------------|-------|------|------|---------------|-----:|---|-----:| | |
| |codexglue_code2text|N/A |none |None |smoothed_bleu_4|1.3519|± |0.3067| | |
| | Tasks |Version|Filter|n-shot| Metric |Value| |Stderr| | |
| |---------------------------------------------|------:|------|------|-----------|----:|---|-----:| | |
| |bigbench_code_line_description_generate_until| 1|none |None |exact_match| 0|± | 0| | |
| | Tasks |Version|Filter|n-shot|Metric|Value| |Stderr| | |
| |----------------------------------------------|------:|------|------|------|----:|---|-----:| | |
| |bigbench_code_line_description_multiple_choice| 0|none |None |acc | 0.25|± |0.0564| | |
| ### Quantized starcoderbase-3b model to GPTQ format | |
| | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| | |
| |-----------------------|-------|------|------|---------------|-----:|---|-----:| | |
| |codexglue_code2text |N/A |none |None |smoothed_bleu_4|0.9254|± |0.2109| | |
| | - code2text_go | 1|none |None |smoothed_bleu_4|1.4702|± |0.4813| | |
| | - code2text_java | 1|none |None |smoothed_bleu_4|0.6907|± |0.6907| | |
| | - code2text_javascript| 1|none |None |smoothed_bleu_4|0.9469|± |0.0339| | |
| | - code2text_php | 1|none |None |smoothed_bleu_4|0.5171|± |0.5171| | |
| | - code2text_python | 1|none |None |smoothed_bleu_4|1.1676|± |0.2156| | |
| | - code2text_ruby | 3|none |None |smoothed_bleu_4|0.7601|± |0.7601| | |
| | Groups |Version|Filter|n-shot| Metric |Value | |Stderr| | |
| |-------------------|-------|------|------|---------------|-----:|---|-----:| | |
| |codexglue_code2text|N/A |none |None |smoothed_bleu_4|0.9254|± |0.2109| | |
| | Tasks |Version|Filter|n-shot| Metric |Value| |Stderr| | |
| |---------------------------------------------|------:|------|------|-----------|----:|---|-----:| | |
| |bigbench_code_line_description_generate_until| 1|none |None |exact_match| 0|± | 0| | |
| | Tasks |Version|Filter|n-shot|Metric|Value| |Stderr| | |
| |----------------------------------------------|------:|------|------|------|----:|---|-----:| | |
| |bigbench_code_line_description_multiple_choice| 0|none |None |acc | 0.1|± | 0.1| |