metadata

language:
  - code
pipeline_tag: text-generation
tags:
  - llama-2
license: llama2
base_model:
  - meta-llama/CodeLlama-7b-hf

Opencsg-CodeLlama-7b-v0.2 [中文] [English]

OpenCSG

[OpenCSG Community] [github] [wechat] [Twitter]

OpenCSG stands for Converged resources, Software refinement, and Generative LM. The 'C' represents Converged resources, indicating the integration and full utilization of hybrid resources. The 'S' stands for Software refinement, signifying software that is refined by large models. The 'G' represents Generative LM, which denotes widespread, inclusive, and democratized generative large models.

The vision of OpenCSG is to empower every industry, every company, and every individual to own their models. We adhere to the principles of openness and open source, making the large model software stack of OpenCSG available to the community. We welcome everyone to use, send feedback, and contribute collaboratively.

Model Description

CodeLlama is a collection of pretrained and fine-tuned generative text models from Llama2, which ranges in scale from 7 billion to 34 billion parameters. Based on CodeLlama, opencsg-CodeLlama-v0.2 is a series of models fintuned througth full-paramters fine-tuning method.

This is the repository for the base 7B version finetuned based on CodeLlama-7b-hf.

Model Size	Base Model
7B	opencsg/Opencsg-CodeLlama-7b-v0.1
13B	opencsg/Opencsg-CodeLlama-13b-v0.1
34B	opencsg/Opencsg-CodeLlama-34b-v0.1
7B	opencsg/Opencsg-CodeLlama-7b-v0.2
13B	opencsg/Opencsg-CodeLlama-13b-v0.2
34B	opencsg/Opencsg-CodeLlama-34b-v0.2

Model Eval

HumanEval is the most common code generation benchmark for evaluating model performance, especially on the compeltion of code exercise cases. Model evaluation is, to some extent, a metaphysics. Different models have different sensitivities to decoding methods, parameters and instructions. It is impratical for us to manually set specific configurations for each fine-tuned model, because a real LLM should master general capabilities despite the parameters being manipulated by users.

Therefore, OpenCSG racked their brains to provide a relatively fair method to compare the fine-tuned models on the HumanEval benchmark. To simplify the comparison, we chosed the Pass@1 metric for the Python language, but our fine-tuning dataset includes samples in multiple languages.

For fairness, we evaluated the original and fine-tuned CodeLlama models based only on the prompts from the original cases, without including any other instructions.

Besides, we use the greedy decoding method for each model during evaluation.

Model	HumanEval python pass@1
CodeLlama-7b-hf	30.5%
opencsg-CodeLlama-7b-v0.1	43.9%
opencsg-CodeLlama-7b-v0.2	50.0%
CodeLlama-13b-hf	36.0%
opencsg-CodeLlama-13b-v0.1	51.2%
opencsg-CodeLlama-13b-v0.2	55.5%
CodeLlama-34b-hf	48.2%
opencsg-CodeLlama-34b-v0.1	56.1%
opencsg-CodeLlama-34b-v0.2	64.0%
CodeLlama-70b-hf	53.0%
CodeLlama-70b-Instruct-hf	67.8%

TODO

We will provide more benchmark scores on fine-tuned models in the future.
We will provide different practical problems to evaluate the performance of fine-tuned models in the field of software engineering.

Model Usage

from transformers import AutoTokenizer
import transformers
import torch

model = "opencsg/opencsg-CodeLlama-7b-v0.2"

tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)
input_text = "#write a quick sort algorithm."
sequences = pipeline(
    input_text,
    do_sample=False,
    top_k=10,
    temperature=0.1,
    top_p=0.95,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=256,
)
for seq in sequences:
    print(seq['generated_text'][len(input_text):])

Training

Hardware

GPUs: 8 Tesla A800
Training time: 4 hours

Software

Orchestration: Deepspeed
Neural networks: PyTorch
BP16 if applicable: apex

OpenCSG介绍

OpenCSG

[OpenCSG 社区] [github] [微信] [推特]

OpenCSG 致力于资源融合、软件求精和生成式 LM。其中，“C”代表资源融合（Converged resources），表示多种混合资源的整合和充分利用。 “S”代表软件求精（Software refinement），表示通过大模型精炼过的软件。 “G”代表生成型语言模型（Generative LM），它表示广泛使用的、包容性的、经过民主化的生成式大模型。

OpenCSG 的愿景是让每个行业、每个公司、每个人都拥有自己的模型。我们坚持开放开源的原则，将OpenCSG的大模型栈提供给社区。欢迎大家积极使用、反馈想法和贡献内容。

模型介绍

CodeLlama 是一系列由 Llama2 经过预训练和微调得到的生成式代码大模型，其规模从 70 亿到 340 亿个参数不等。 opencsg-CodeLlama-v0.2是一系列基于CodeLlama的通过全参数微调方法进行调优的模型。

这是基于 CodeLlama-7b-hf 进行微调的模型版本。

模型大小	基座模型
7B	opencsg/Opencsg-CodeLlama-7b-v0.1
13B	opencsg/Opencsg-CodeLlama-13b-v0.1
34B	opencsg/Opencsg-CodeLlama-34b-v0.1
7B	opencsg/Opencsg-CodeLlama-7b-v0.2
13B	opencsg/Opencsg-CodeLlama-13b-v0.2
34B	opencsg/Opencsg-CodeLlama-34b-v0.2

模型评估

HumanEval 是评估模型在代码生成方面性能的最常见的基准，尤其是在代码习题的补全方面。模型评估在某种程度上是一种玄学。不同的模型对解码方法、参数和指令的敏感度不同，优秀的大模型是具备通用能力的，而不会因为解码参数的调整使得模型的生成表现有很大的差异。

因此，OpenCSG 提供了一个相对公平的方法来在 HumanEval 基准上比较各微调模型。方便起见，我们选择了Python语言Pass@1指标，但要注意的是，我们的微调数据集是包含多种编程语言。

为了公平起见，我们仅根据原始问题的提示来评估原始和微调过的 CodeLlama 模型，不包含任何其他说明。

除此之外，我们在评估过程中对每个模型都使用贪婪解码方法。

模型	HumanEval python pass@1
CodeLlama-7b-hf	30.5%
opencsg-CodeLlama-7b-v0.1	43.9%
opencsg-CodeLlama-7b-v0.2	50.0%
CodeLlama-13b-hf	36.0%
opencsg-CodeLlama-13b-v0.1	51.2%
opencsg-CodeLlama-13b-v0.2	55.5%
CodeLlama-34b-hf	48.2%
opencsg-CodeLlama-34b-v0.1	56.1%
opencsg-CodeLlama-34b-v0.2	64.0%
CodeLlama-70b-hf	53.0%
CodeLlama-70b-Instruct-hf	67.8%