README.md

<h1 align="center">:star: Multilingual Code Evaluation LeaderBoard Guide</h1>


<h4 align="center">
    <p>
        <a href="#running-the-evaluation">Running Evaluation</a> |
        <a href="#submission-of-results-to-the-leaderboard">Results Submission</a> 
    <p>
</h4>

This is a guide to submit and reproduce the numbers in the [Multilingual Code Evaluation LeaderBoard](https://huggingface.co/spaces/bigcode/multilingual-code-evals).
The LeaderBoard is a demo for evaluating and comparing the performance of language models on code generation tasks.

The LeaderBoard is open for submissions of results produced by the community. If you have a model that you want to submit results for, please follow the instructions below.

## Running the evaluation
We report the passs@1 for [HumanEval](https://huggingface.co/datasets/openai_humaneval) Python benchamrk and some languages from the [MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E) benchmark. We use the same template and parameters for all models.

### 1-Setup
Follow the setup instructions in the evaluation harness [README](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main#setup).

Create two folders `generations_$model` and `metrics_$model` where you will save the generated code and the metrics respectively for your model `$model`.
```bash
cd bigcode-evaluation-harness
mkdir generations_$model
mkdir metrics_$model
```

To run the evaluation, we first generate the code solutions for the target tasks on GPUs, then execute the code on a docker container (only cpus are needed).

### 2- Generation
Below are the instruction for generating the code solutions sequentially or in parallel with slurm. You might need to reduce the batch size for some models or change the precision based on your device.
```bash
# after activating env and setting up accelerate...
langs=(py js java cpp swift php d jl lua r rkt rs)

model=YOUR_MODEL
org=HF_ORGANISATION

for lang in "${langs[@]}"; do
    # use humaneval for py and multipl-e for the rest
    if [ "$lang" == "py" ]; then
        task=humaneval
    else
        task=multiple-$lang
    fi

    echo "Running task $task"
    generations_path=generations_$model/generations_$task\_$model.json
    accelerate launch main.py \
            --model $org/$model \
            --task $task \
            --n_samples 50 \
            --batch_size 50 \
            --max_length_generation 512 \
            --temperature 0.2 \
            --precision bf16 \
            --trust_remote_code \
            --use_auth_token \
            --generation_only \
            --save_generations \
            --save_generations_path $generations_path
    echo "Task $task done"
done
```
This will generate and save the code solutions for all tasks in the `generations_$model` folder.

If you want to submit jobs in parallel with `slurm`, run multiple-eval.slurm with:
```bash
langs=(py js java cpp swift php d jl lua r rkt rs)

model=YOUR_MODEL
org=HF_ORGANISATION
out_path=generations_$model

for lang in "${langs[@]}"; do
    if [ "$lang" == "py" ]; then
        task=humaneval
    else
        task=multiple-$lang
    fi
    echo "Submitting task $task"
    sbatch -J "eval-$model-$task" multiple_evals.slurm "$model" "$task" "$org" "$out_path"
done
```
This will submit one job for each task.

### 3- Execution

We execute and evaluate the solutions inside a docker container, you can either build the image or pull the one we provide:
```bash
# to build it:
# sudo make DOCKERFILE=Dockerfile-multiple all
sudo docker pull ghcr.io/bigcode-project/evaluation-harness-multiple
sudo docker tag ghcr.io/bigcode-project/evaluation-harness-multiple evaluation-harness-multiple
````

Then, you can run the evaluation on the generated code:
```bash
langs=(py js java cpp swift php d jl lua r rkt rs)

model=YOUR_MODEL
org=HF_ORGANISATION
# if you provide absolute paths remove the $(pwd) from the command below
generations_path=generations_$model
metrics_path=metrics_$model

for lang in "${langs[@]}"; do
    if [ "$lang" == "py" ]; then
        task=humaneval
    else
        task=multiple-$lang
    fi

    gen_suffix=generations_$task\_$model.json
    metric_suffix=metrics_$task\_$model.json
    echo "Evaluation of $model on $task benchmark, data in $generations_path/$gen_suffix"

    sudo docker run -v $(pwd)/$generations_path/$gen_suffix:/app/$gen_suffix:ro  -v $(pwd)/$metrics_path:/app/$metrics_path -it evaluation-harness-multiple python3 main.py \
        --model $org/$model \
        --tasks $task \
        --load_generations_path /app/$gen_suffix \
        --metric_output_path /app/$metrics_path/$metric_suffix \
        --allow_code_execution  \
        --use_auth_token \
        --temperature 0.2 \
        --n_samples 50 | tee -a logs_$model.txt
    echo "Task $task done, metric saved at $metrics_path/$metric_suffix"
done
```

## Submission of results to the LeaderBoard
If you followed the steps above you now have a folder `metrics_$model` with `json` files, each containing the result of one task. To submit the results to the LeaderBoard, you need to create a json summarizing these metrics using `group_jsons.py` and submit it [here](https://huggingface.co/spaces/bigcode/multilingual-code-evals). Follow the instruction on `Submit here` section.
```bash
python group_jsons.py --metrics_path metrics_$model --model $model --org $org --username $your_hf_username
```
For credibility, we invite you to add the generations and json metrics to your submission.

Now you're ready to submit your results by opening a PR on the leaderboard, go to `Submit results :rocket:`section for more details.

## Notes
Some models might require some extra arguments, like [CodeGeeX2-6b](https://huggingface.co/THUDM/codegeex2-6b) which requires providing the language tag as a prefix and doing generation under torch 2.0. And [replit-v1-3b](https://huggingface.co/replit/replit-code-v1-3b) that requires adding extra. You can just add the prefix as a new argument
```bash
# define prefixes base on codegeex-2 repo
declare -A langs
langs=( [py]="# Python" [js]="// JavaScript" [java]="// Java" [cpp]="// C++" [swift]="// Swift" [php]="// PHP" [jl]="# Julia" [lua]="// Lua" [r]="# R" [rkt]="; Racket" [rs]="// Rust" [d]="" )

model="codegeex2-6b"
org="THUDM"

for lang in "${!langs[@]}"; do
    prefix="language: ${langs[$lang]}"
    echo "For language $lang, the prefix is: $prefix"
    generations_path=generations_$model/generations_$task\_$model.json
    accelerate launch main.py \
            --model $org/$model \
            --task multiple-l$ang \
            --n_samples 5 \
            --batch_size 5 \
            --limit 8 \
            --max_length_generation 512 \
            --temperature 0.2 \
            --precision bf16 \
            --trust_remote_code \
            --use_auth_token \
            --generation_only \
            --save_generations_path $generations_path \
            --prefix \"$prefix\" \
    echo "Task $task done"
done
```
Replit model command (pull code from this [PR](https://github.com/bigcode-project/bigcode-evaluation-harness/pull/115)):
```bash
accelerate launch main.py \
    --model replit/replit-code-v1-3b \
    --tasks multiple-$lang \
    --max_length_generation 512 \
    --batch_size 50 \
    --n_samples 10 \
    --temperature 0.2 \
    --precision fp16 \
    --allow_code_execution \
    --trust_remote_code \
    --save_generations \
    --use_auth_token \
    --generation_only \
    --save_generations_path /fsx/loubna/code/bigcode-evaluation-harness/multiple_gens_replit/replit-$lang.json \
    --automodel_kwargs '{\
        \"attn_config\": {\
            \"alibi\": true,\
            \"alibi_bias_max\": 8,\
            \"attn_impl\": \"triton\",\
            \"attn_pdrop\": 0,\
            \"attn_type\": \"multihead_attention\",\
            \"attn_uses_sequence_id\": false,\
            \"clip_qkv\": null,\
            \"prefix_lm\": false,\
            \"qk_ln\": false,\
            \"softmax_scale\": null\
        }\
    }'
```

## Bonus
For the throughput and peak memory measurments, we point you to [optimum-benchamrk](https://github.com/huggingface/optimum-benchmark) (checkout commit `49f0924e2bb041cf17d78dd0848d8e2cad31632d` [here](https://github.com/huggingface/optimum-benchmark/commit/49f0924e2bb041cf17d78dd0848d8e2cad31632d)).
You can follow the instructions in the repo, copy our config yaml and run the command below:
```bash
cp throughput_config.yaml optimum-benchmark/examples
device=cuda:0
batch=1
optimum-benchmark --config-dir examples --config-name throughput_config model=$org/$model device=$device benchmark.input_shapes.batch_size=$batch
```