File size: 7,784 Bytes
9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca 9f606aa 74f06ca f0d4174 261234d f0d4174 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 |
---
license: apache-2.0
language:
- ru
- en
base_model:
- jinaai/jina-embeddings-v3
---
## **JinaJudge: Proxy Judgement for Russian LLM Arena**
### **Description**
This model is trained to replicate the judgement patterns of GPT-4-1106-Preview in the [Russian LLM Arena](https://huggingface.co/spaces/Vikhrmodels/arenahardlb), designed for faster and more cost-effective evaluation of language models. While the model's focus is on Russian LLM evaluation, it can also be used for English-centric models.
---
### **Model Details**
This is an iterative update of [kaleinaNyan/jina-v3-rullmarena-judge-300924](https://huggingface.co/kaleinaNyan/jina-v3-rullmarena-judge-300924) model:
- Increased amount of training data (not by much, approaximately 1.5x times).
- Updated data composition to fix erroneous judgements where GPT-4 picked English responses over Russian ones.
- Validation set was updated as well to exclude such errors.
- Test set did not change (no bad judgements in that regard).
---
### **Evaluation**
The validation process was based on **existing judgements** from the Russian LLM Arena, which were already available. These judgements were filtered and simplified to match the three-class structure used in training.
NOTE: values in parenthesis show relative improvement compared to previous model.
**Models evaluated**:
- **gemma-2-9b-it-sppo-iter3**
- **glm-4-9b-chat**
- **gpt-3.5-turbo-1106**
- **mistral-7b-instruct-v0.3**
- **storm-7b**
**Validation Performance (old validation set)**:
- **Accuracy**: 79.97% (-0.78)
- **Precision**: 78.25% (-0.31)
- **Recall**: 78.25% (-1.23)
- **F1-score**: 78.25% (-0.75)
NOTE: will report later what actually caused the drop (the subset of fixed judgements or smth else)
**Validation Performance (new validation set)**:
- **Accuracy**: 83.59% (+2.48)
- **Precision**: 80.97% (+2.14)
- **Recall**: 80.97% (+1.22)
- **F1-score**: 80.97% (+1.77)
For the **test** phase, new judgements were generated using GPT-4 for the `kolibri-mistral-0427-upd` model.
**Test Performance**:
- **Accuracy**: 85.09% (+2.37)
- **Precision**: 83.20% (+3.09)
- **Recall**: 83.20% (+0.78)
- **F1-score**: 83.20% (+2.02)
---
### **Usage Example**
```python
from transformers import AutoModel
jina = AutoModel.from_pretrained("kaleinaNyan/jina-v3-rullmarena-judge-041024", trust_remote_code=True)
prompt_template = """
<user prompt>
{user_prompt}
<end>
<assistant A answer>
{assistant_a}
<end>
<assistant B answer>
{assistant_b}
<end>
""".strip()
prompt = "your prompt"
assistant_a = "assistant a response"
assistant_b = "assistant b response"
example = prompt_template.format(
user_prompt=user_prompt,
assistant_a=assistant_a,
assistant_b=assistant_b,
)
judgement = jina([example])[0].argmax()
judgement_map = {
0: "A is better than B",
1: "A == B",
2: "B is better than A"
}
print(judgement_map[judgement])
```
---
### **Generated ranking**
The ranking was obtained using a modified [Russian LLM Arena code](https://github.com/oKatanaaa/ru_llm_arena).
All judgements were regenerated using the jina-judge model. It takes about 16 minutes to regenerate the whole board (or 23 seconds per model) on an RTX3090.
| Model | Score | 95% CI | Average #Tokens |
|--------------------------------------------------|-------|----------------------|-----------------|
| gpt-4-1106-preview | 82.8 | (-2.2, 2.3) | 541 |
| gpt-4o-mini | 75.3 | (-2.5, 2.9) | 448 |
| qwen-2.5-72b-it | 73.1 | (-3.4, 3.1) | 557 |
| gemma-2-9b-it-sppo-iter3 | 70.6 | (-3.9, 2.8) | 509 |
| gemma-2-27b-it | 68.7 | (-2.8, 3.8) | 472 |
| t-lite-instruct-0.1 | 67.5 | (-3.8, 3.8) | 810 |
| gemma-2-9b-it | 67.0 | (-3.7, 3.3) | 459 |
| suzume-llama-3-8B-multilingual-orpo-borda-half | 62.4 | (-3.5, 3.7) | 682 |
| glm-4-9b-chat | 61.5 | (-3.7, 3.0) | 568 |
| phi-3-medium-4k-instruct | 60.4 | (-3.5, 3.7) | 566 |
| sfr-iterative-dpo-llama-3-8b-r | 57.2 | (-3.9, 2.2) | 516 |
| c4ai-command-r-v01 | 55.0 | (-3.9, 3.1) | 529 |
| suzume-llama-3-8b-multilingual | 51.9 | (-2.8, 3.7) | 641 |
| mistral-nemo-instruct-2407 | 51.9 | (-3.8, 3.7) | 403 |
| yandex_gpt_pro | 50.3 | (-3.4, 3.1) | 345 |
| gpt-3.5-turbo-0125 | 50.0 | (0.0, 0.0) | 220 |
| hermes-2-theta-llama-3-8b | 49.3 | (-3.4, 3.9) | 485 |
| starling-lm-7b-beta | 48.3 | (-3.8, 4.0) | 629 |
| llama-3-8b-saiga-suzume-ties | 47.9 | (-3.9, 5.0) | 763 |
| llama-3-smaug-8b | 47.6 | (-3.6, 3.1) | 524 |
| vikhr-it-5.4-fp16-orpo-v2 | 46.8 | (-2.5, 2.7) | 379 |
| aya-23-8b | 46.1 | (-3.9, 3.9) | 554 |
| saiga_llama3_8b_v6 | 44.8 | (-3.4, 3.3) | 471 |
| qwen2-7b-instruct | 43.6 | (-3.0, 2.7) | 340 |
| vikhr-it-5.2-fp16-cp | 43.6 | (-4.1, 3.3) | 543 |
| openchat-3.5-0106 | 42.8 | (-3.9, 3.3) | 492 |
| kolibri-mistral-0427-upd | 42.3 | (-4.2, 3.2) | 551 |
| paralex-llama-3-8b-sft | 41.8 | (-3.2, 3.7) | 688 |
| llama-3-instruct-8b-sppo-iter3 | 41.7 | (-3.4, 3.3) | 502 |
| gpt-3.5-turbo-1106 | 41.5 | (-2.9, 2.1) | 191 |
| mistral-7b-instruct-v0.3 | 41.1 | (-4.3, 3.5) | 469 |
| gigachat_pro | 40.9 | (-3.4, 3.6) | 294 |
| openchat-3.6-8b-20240522 | 39.1 | (-3.2, 4.1) | 428 |
| vikhr-it-5.3-fp16-32k | 38.8 | (-3.5, 3.3) | 519 |
| hermes-2-pro-llama-3-8b | 38.4 | (-3.2, 3.1) | 463 |
| kolibri-vikhr-mistral-0427 | 34.5 | (-2.9, 3.5) | 489 |
| vikhr-it-5.3-fp16 | 33.5 | (-3.5, 3.8) | 523 |
| llama-3-instruct-8b-simpo | 32.7 | (-3.9, 3.6) | 417 |
| meta-llama-3-8b-instruct | 32.1 | (-3.4, 3.3) | 450 |
| neural-chat-7b-v3-3 | 25.9 | (-2.7, 3.6) | 927 |
| gigachat_lite | 25.4 | (-2.8, 2.5) | 276 |
| snorkel-mistral-pairrm-dpo | 10.3 | (-2.0, 2.3) | 773 |
| storm-7b | 3.7 | (-1.3, 1.6) | 419 |
|