d-matrix/CodeLlama-7b

This is a d-Matrix functional reference of the CODELLAMA-7B model. The reference provides the following functional configurations:

Configuration	Explanation
BASELINE	a reference functionally equivalent to the original model
BASIC	all linear algebraic operands quantized to BFP16-64, and all other operations transformed to approximated kernel simulations

Usage

Install d-Matrix Dmx_Compressor first.

pip install dmx_compressor

The following is an example model and its evaluation.

git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install .

from dmx.compressor.modeling import DmxModel
from bigcode_eval.evaluator import Evaluator

pipe = pipeline(
  task="text-generation",
  model="d-matrix/CodeLlama-7b",
  trust_remote_code=True,
)

# Transform the model with DMX
model = DmxModel.from_torch(pipe.model).to_basic_model()  # Using BASIC configuration
model(torch.randint(1, 100, (1, max_length))) # Assign desired max length of generation

evaluator = Evaluator(accelerator, model, tokenizer, eval_args)

eval_results = evaluator.evaluate(task)  # Assign desired task, i.e. "humaneval"