|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- transformers |
|
- mteb |
|
model-index: |
|
- name: mmlw-roberta-base |
|
results: |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: PL-MTEB/8tags-clustering |
|
name: MTEB 8TagsClustering |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 33.08463724780795 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/allegro-reviews |
|
name: MTEB AllegroReviews |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 40.25844930417495 |
|
- type: f1 |
|
value: 35.59685265418916 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana-pl |
|
name: MTEB ArguAna-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 33.073 |
|
- type: map_at_10 |
|
value: 50.223 |
|
- type: map_at_100 |
|
value: 50.942 |
|
- type: map_at_1000 |
|
value: 50.94499999999999 |
|
- type: map_at_3 |
|
value: 45.721000000000004 |
|
- type: map_at_5 |
|
value: 48.413000000000004 |
|
- type: mrr_at_1 |
|
value: 34.424 |
|
- type: mrr_at_10 |
|
value: 50.68899999999999 |
|
- type: mrr_at_100 |
|
value: 51.437999999999995 |
|
- type: mrr_at_1000 |
|
value: 51.441 |
|
- type: mrr_at_3 |
|
value: 46.219 |
|
- type: mrr_at_5 |
|
value: 48.921 |
|
- type: ndcg_at_1 |
|
value: 33.073 |
|
- type: ndcg_at_10 |
|
value: 59.021 |
|
- type: ndcg_at_100 |
|
value: 61.902 |
|
- type: ndcg_at_1000 |
|
value: 61.983999999999995 |
|
- type: ndcg_at_3 |
|
value: 49.818 |
|
- type: ndcg_at_5 |
|
value: 54.644999999999996 |
|
- type: precision_at_1 |
|
value: 33.073 |
|
- type: precision_at_10 |
|
value: 8.684 |
|
- type: precision_at_100 |
|
value: 0.9900000000000001 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 20.555 |
|
- type: precision_at_5 |
|
value: 14.666 |
|
- type: recall_at_1 |
|
value: 33.073 |
|
- type: recall_at_10 |
|
value: 86.842 |
|
- type: recall_at_100 |
|
value: 99.004 |
|
- type: recall_at_1000 |
|
value: 99.644 |
|
- type: recall_at_3 |
|
value: 61.663999999999994 |
|
- type: recall_at_5 |
|
value: 73.329 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/cbd |
|
name: MTEB CBD |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 68.11 |
|
- type: ap |
|
value: 20.916633959031266 |
|
- type: f1 |
|
value: 56.85804802205465 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/cdsce-pairclassification |
|
name: MTEB CDSC-E |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 89.2 |
|
- type: cos_sim_ap |
|
value: 79.1041156765933 |
|
- type: cos_sim_f1 |
|
value: 70.0 |
|
- type: cos_sim_precision |
|
value: 74.11764705882354 |
|
- type: cos_sim_recall |
|
value: 66.3157894736842 |
|
- type: dot_accuracy |
|
value: 88.2 |
|
- type: dot_ap |
|
value: 72.57183688228149 |
|
- type: dot_f1 |
|
value: 67.16417910447761 |
|
- type: dot_precision |
|
value: 63.67924528301887 |
|
- type: dot_recall |
|
value: 71.05263157894737 |
|
- type: euclidean_accuracy |
|
value: 89.3 |
|
- type: euclidean_ap |
|
value: 79.01345533432428 |
|
- type: euclidean_f1 |
|
value: 70.19498607242339 |
|
- type: euclidean_precision |
|
value: 74.55621301775149 |
|
- type: euclidean_recall |
|
value: 66.3157894736842 |
|
- type: manhattan_accuracy |
|
value: 89.3 |
|
- type: manhattan_ap |
|
value: 79.01671381791259 |
|
- type: manhattan_f1 |
|
value: 70.0280112044818 |
|
- type: manhattan_precision |
|
value: 74.8502994011976 |
|
- type: manhattan_recall |
|
value: 65.78947368421053 |
|
- type: max_accuracy |
|
value: 89.3 |
|
- type: max_ap |
|
value: 79.1041156765933 |
|
- type: max_f1 |
|
value: 70.19498607242339 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/cdscr-sts |
|
name: MTEB CDSC-R |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 91.79559442663039 |
|
- type: cos_sim_spearman |
|
value: 92.5438168962641 |
|
- type: euclidean_pearson |
|
value: 92.02981265332856 |
|
- type: euclidean_spearman |
|
value: 92.5548245733484 |
|
- type: manhattan_pearson |
|
value: 91.95296287979178 |
|
- type: manhattan_spearman |
|
value: 92.50279516120241 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-pl |
|
name: MTEB DBPedia-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 7.829999999999999 |
|
- type: map_at_10 |
|
value: 16.616 |
|
- type: map_at_100 |
|
value: 23.629 |
|
- type: map_at_1000 |
|
value: 25.235999999999997 |
|
- type: map_at_3 |
|
value: 12.485 |
|
- type: map_at_5 |
|
value: 14.077 |
|
- type: mrr_at_1 |
|
value: 61.75000000000001 |
|
- type: mrr_at_10 |
|
value: 69.852 |
|
- type: mrr_at_100 |
|
value: 70.279 |
|
- type: mrr_at_1000 |
|
value: 70.294 |
|
- type: mrr_at_3 |
|
value: 68.375 |
|
- type: mrr_at_5 |
|
value: 69.187 |
|
- type: ndcg_at_1 |
|
value: 49.75 |
|
- type: ndcg_at_10 |
|
value: 36.217 |
|
- type: ndcg_at_100 |
|
value: 41.235 |
|
- type: ndcg_at_1000 |
|
value: 48.952 |
|
- type: ndcg_at_3 |
|
value: 41.669 |
|
- type: ndcg_at_5 |
|
value: 38.285000000000004 |
|
- type: precision_at_1 |
|
value: 61.5 |
|
- type: precision_at_10 |
|
value: 28.499999999999996 |
|
- type: precision_at_100 |
|
value: 9.572 |
|
- type: precision_at_1000 |
|
value: 2.025 |
|
- type: precision_at_3 |
|
value: 44.083 |
|
- type: precision_at_5 |
|
value: 36.3 |
|
- type: recall_at_1 |
|
value: 7.829999999999999 |
|
- type: recall_at_10 |
|
value: 21.462999999999997 |
|
- type: recall_at_100 |
|
value: 47.095 |
|
- type: recall_at_1000 |
|
value: 71.883 |
|
- type: recall_at_3 |
|
value: 13.891 |
|
- type: recall_at_5 |
|
value: 16.326999999999998 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa-pl |
|
name: MTEB FiQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 16.950000000000003 |
|
- type: map_at_10 |
|
value: 27.422 |
|
- type: map_at_100 |
|
value: 29.146 |
|
- type: map_at_1000 |
|
value: 29.328 |
|
- type: map_at_3 |
|
value: 23.735999999999997 |
|
- type: map_at_5 |
|
value: 25.671 |
|
- type: mrr_at_1 |
|
value: 33.796 |
|
- type: mrr_at_10 |
|
value: 42.689 |
|
- type: mrr_at_100 |
|
value: 43.522 |
|
- type: mrr_at_1000 |
|
value: 43.563 |
|
- type: mrr_at_3 |
|
value: 40.226 |
|
- type: mrr_at_5 |
|
value: 41.685 |
|
- type: ndcg_at_1 |
|
value: 33.642 |
|
- type: ndcg_at_10 |
|
value: 35.008 |
|
- type: ndcg_at_100 |
|
value: 41.839 |
|
- type: ndcg_at_1000 |
|
value: 45.035 |
|
- type: ndcg_at_3 |
|
value: 31.358999999999998 |
|
- type: ndcg_at_5 |
|
value: 32.377 |
|
- type: precision_at_1 |
|
value: 33.642 |
|
- type: precision_at_10 |
|
value: 9.937999999999999 |
|
- type: precision_at_100 |
|
value: 1.685 |
|
- type: precision_at_1000 |
|
value: 0.22699999999999998 |
|
- type: precision_at_3 |
|
value: 21.142 |
|
- type: precision_at_5 |
|
value: 15.586 |
|
- type: recall_at_1 |
|
value: 16.950000000000003 |
|
- type: recall_at_10 |
|
value: 42.286 |
|
- type: recall_at_100 |
|
value: 68.51899999999999 |
|
- type: recall_at_1000 |
|
value: 87.471 |
|
- type: recall_at_3 |
|
value: 28.834 |
|
- type: recall_at_5 |
|
value: 34.274 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa-pl |
|
name: MTEB HotpotQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 37.711 |
|
- type: map_at_10 |
|
value: 57.867999999999995 |
|
- type: map_at_100 |
|
value: 58.77 |
|
- type: map_at_1000 |
|
value: 58.836999999999996 |
|
- type: map_at_3 |
|
value: 54.400999999999996 |
|
- type: map_at_5 |
|
value: 56.564 |
|
- type: mrr_at_1 |
|
value: 75.449 |
|
- type: mrr_at_10 |
|
value: 81.575 |
|
- type: mrr_at_100 |
|
value: 81.783 |
|
- type: mrr_at_1000 |
|
value: 81.792 |
|
- type: mrr_at_3 |
|
value: 80.50399999999999 |
|
- type: mrr_at_5 |
|
value: 81.172 |
|
- type: ndcg_at_1 |
|
value: 75.422 |
|
- type: ndcg_at_10 |
|
value: 66.635 |
|
- type: ndcg_at_100 |
|
value: 69.85 |
|
- type: ndcg_at_1000 |
|
value: 71.179 |
|
- type: ndcg_at_3 |
|
value: 61.648 |
|
- type: ndcg_at_5 |
|
value: 64.412 |
|
- type: precision_at_1 |
|
value: 75.422 |
|
- type: precision_at_10 |
|
value: 13.962 |
|
- type: precision_at_100 |
|
value: 1.649 |
|
- type: precision_at_1000 |
|
value: 0.183 |
|
- type: precision_at_3 |
|
value: 39.172000000000004 |
|
- type: precision_at_5 |
|
value: 25.691000000000003 |
|
- type: recall_at_1 |
|
value: 37.711 |
|
- type: recall_at_10 |
|
value: 69.811 |
|
- type: recall_at_100 |
|
value: 82.471 |
|
- type: recall_at_1000 |
|
value: 91.29 |
|
- type: recall_at_3 |
|
value: 58.757999999999996 |
|
- type: recall_at_5 |
|
value: 64.227 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco-pl |
|
name: MTEB MSMARCO-PL |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 17.033 |
|
- type: map_at_10 |
|
value: 27.242 |
|
- type: map_at_100 |
|
value: 28.451999999999998 |
|
- type: map_at_1000 |
|
value: 28.515 |
|
- type: map_at_3 |
|
value: 24.046 |
|
- type: map_at_5 |
|
value: 25.840999999999998 |
|
- type: mrr_at_1 |
|
value: 17.493 |
|
- type: mrr_at_10 |
|
value: 27.67 |
|
- type: mrr_at_100 |
|
value: 28.823999999999998 |
|
- type: mrr_at_1000 |
|
value: 28.881 |
|
- type: mrr_at_3 |
|
value: 24.529999999999998 |
|
- type: mrr_at_5 |
|
value: 26.27 |
|
- type: ndcg_at_1 |
|
value: 17.479 |
|
- type: ndcg_at_10 |
|
value: 33.048 |
|
- type: ndcg_at_100 |
|
value: 39.071 |
|
- type: ndcg_at_1000 |
|
value: 40.739999999999995 |
|
- type: ndcg_at_3 |
|
value: 26.493 |
|
- type: ndcg_at_5 |
|
value: 29.701 |
|
- type: precision_at_1 |
|
value: 17.479 |
|
- type: precision_at_10 |
|
value: 5.324 |
|
- type: precision_at_100 |
|
value: 0.8380000000000001 |
|
- type: precision_at_1000 |
|
value: 0.098 |
|
- type: precision_at_3 |
|
value: 11.408999999999999 |
|
- type: precision_at_5 |
|
value: 8.469999999999999 |
|
- type: recall_at_1 |
|
value: 17.033 |
|
- type: recall_at_10 |
|
value: 50.929 |
|
- type: recall_at_100 |
|
value: 79.262 |
|
- type: recall_at_1000 |
|
value: 92.239 |
|
- type: recall_at_3 |
|
value: 33.06 |
|
- type: recall_at_5 |
|
value: 40.747 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 72.31002017484867 |
|
- type: f1 |
|
value: 69.61603671063031 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 75.52790854068594 |
|
- type: f1 |
|
value: 75.4053872472259 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus-pl |
|
name: MTEB NFCorpus-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.877000000000001 |
|
- type: map_at_10 |
|
value: 12.817 |
|
- type: map_at_100 |
|
value: 16.247 |
|
- type: map_at_1000 |
|
value: 17.683 |
|
- type: map_at_3 |
|
value: 9.334000000000001 |
|
- type: map_at_5 |
|
value: 10.886999999999999 |
|
- type: mrr_at_1 |
|
value: 45.201 |
|
- type: mrr_at_10 |
|
value: 52.7 |
|
- type: mrr_at_100 |
|
value: 53.425999999999995 |
|
- type: mrr_at_1000 |
|
value: 53.461000000000006 |
|
- type: mrr_at_3 |
|
value: 50.464 |
|
- type: mrr_at_5 |
|
value: 51.827 |
|
- type: ndcg_at_1 |
|
value: 41.949999999999996 |
|
- type: ndcg_at_10 |
|
value: 34.144999999999996 |
|
- type: ndcg_at_100 |
|
value: 31.556 |
|
- type: ndcg_at_1000 |
|
value: 40.265 |
|
- type: ndcg_at_3 |
|
value: 38.07 |
|
- type: ndcg_at_5 |
|
value: 36.571 |
|
- type: precision_at_1 |
|
value: 44.272 |
|
- type: precision_at_10 |
|
value: 25.697 |
|
- type: precision_at_100 |
|
value: 8.077 |
|
- type: precision_at_1000 |
|
value: 2.084 |
|
- type: precision_at_3 |
|
value: 36.016999999999996 |
|
- type: precision_at_5 |
|
value: 31.703 |
|
- type: recall_at_1 |
|
value: 5.877000000000001 |
|
- type: recall_at_10 |
|
value: 16.986 |
|
- type: recall_at_100 |
|
value: 32.719 |
|
- type: recall_at_1000 |
|
value: 63.763000000000005 |
|
- type: recall_at_3 |
|
value: 10.292 |
|
- type: recall_at_5 |
|
value: 12.886000000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq-pl |
|
name: MTEB NQ-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.476 |
|
- type: map_at_10 |
|
value: 38.67 |
|
- type: map_at_100 |
|
value: 39.784000000000006 |
|
- type: map_at_1000 |
|
value: 39.831 |
|
- type: map_at_3 |
|
value: 34.829 |
|
- type: map_at_5 |
|
value: 37.025000000000006 |
|
- type: mrr_at_1 |
|
value: 28.621000000000002 |
|
- type: mrr_at_10 |
|
value: 41.13 |
|
- type: mrr_at_100 |
|
value: 42.028 |
|
- type: mrr_at_1000 |
|
value: 42.059999999999995 |
|
- type: mrr_at_3 |
|
value: 37.877 |
|
- type: mrr_at_5 |
|
value: 39.763999999999996 |
|
- type: ndcg_at_1 |
|
value: 28.563 |
|
- type: ndcg_at_10 |
|
value: 45.654 |
|
- type: ndcg_at_100 |
|
value: 50.695 |
|
- type: ndcg_at_1000 |
|
value: 51.873999999999995 |
|
- type: ndcg_at_3 |
|
value: 38.359 |
|
- type: ndcg_at_5 |
|
value: 42.045 |
|
- type: precision_at_1 |
|
value: 28.563 |
|
- type: precision_at_10 |
|
value: 7.6450000000000005 |
|
- type: precision_at_100 |
|
value: 1.052 |
|
- type: precision_at_1000 |
|
value: 0.117 |
|
- type: precision_at_3 |
|
value: 17.458000000000002 |
|
- type: precision_at_5 |
|
value: 12.613 |
|
- type: recall_at_1 |
|
value: 25.476 |
|
- type: recall_at_10 |
|
value: 64.484 |
|
- type: recall_at_100 |
|
value: 86.96199999999999 |
|
- type: recall_at_1000 |
|
value: 95.872 |
|
- type: recall_at_3 |
|
value: 45.527 |
|
- type: recall_at_5 |
|
value: 54.029 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: laugustyniak/abusive-clauses-pl |
|
name: MTEB PAC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 65.87315377932232 |
|
- type: ap |
|
value: 76.41966964416534 |
|
- type: f1 |
|
value: 63.64417488639012 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/ppc-pairclassification |
|
name: MTEB PPC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 87.7 |
|
- type: cos_sim_ap |
|
value: 92.81319372631636 |
|
- type: cos_sim_f1 |
|
value: 90.04048582995952 |
|
- type: cos_sim_precision |
|
value: 88.11410459587957 |
|
- type: cos_sim_recall |
|
value: 92.05298013245033 |
|
- type: dot_accuracy |
|
value: 75.0 |
|
- type: dot_ap |
|
value: 83.63089957943261 |
|
- type: dot_f1 |
|
value: 80.76923076923077 |
|
- type: dot_precision |
|
value: 75.43103448275862 |
|
- type: dot_recall |
|
value: 86.9205298013245 |
|
- type: euclidean_accuracy |
|
value: 87.7 |
|
- type: euclidean_ap |
|
value: 92.94772245932825 |
|
- type: euclidean_f1 |
|
value: 90.10458567980692 |
|
- type: euclidean_precision |
|
value: 87.63693270735524 |
|
- type: euclidean_recall |
|
value: 92.71523178807946 |
|
- type: manhattan_accuracy |
|
value: 87.8 |
|
- type: manhattan_ap |
|
value: 92.95330512127123 |
|
- type: manhattan_f1 |
|
value: 90.08130081300813 |
|
- type: manhattan_precision |
|
value: 88.49840255591054 |
|
- type: manhattan_recall |
|
value: 91.72185430463577 |
|
- type: max_accuracy |
|
value: 87.8 |
|
- type: max_ap |
|
value: 92.95330512127123 |
|
- type: max_f1 |
|
value: 90.10458567980692 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/psc-pairclassification |
|
name: MTEB PSC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 96.19666048237477 |
|
- type: cos_sim_ap |
|
value: 98.61237969571302 |
|
- type: cos_sim_f1 |
|
value: 93.77845220030349 |
|
- type: cos_sim_precision |
|
value: 93.35347432024169 |
|
- type: cos_sim_recall |
|
value: 94.20731707317073 |
|
- type: dot_accuracy |
|
value: 94.89795918367348 |
|
- type: dot_ap |
|
value: 97.02853491357943 |
|
- type: dot_f1 |
|
value: 91.85185185185186 |
|
- type: dot_precision |
|
value: 89.33717579250721 |
|
- type: dot_recall |
|
value: 94.51219512195121 |
|
- type: euclidean_accuracy |
|
value: 96.38218923933209 |
|
- type: euclidean_ap |
|
value: 98.58145584134218 |
|
- type: euclidean_f1 |
|
value: 94.04580152671755 |
|
- type: euclidean_precision |
|
value: 94.18960244648318 |
|
- type: euclidean_recall |
|
value: 93.90243902439023 |
|
- type: manhattan_accuracy |
|
value: 96.47495361781077 |
|
- type: manhattan_ap |
|
value: 98.6108221024781 |
|
- type: manhattan_f1 |
|
value: 94.18960244648318 |
|
- type: manhattan_precision |
|
value: 94.47852760736197 |
|
- type: manhattan_recall |
|
value: 93.90243902439023 |
|
- type: max_accuracy |
|
value: 96.47495361781077 |
|
- type: max_ap |
|
value: 98.61237969571302 |
|
- type: max_f1 |
|
value: 94.18960244648318 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_in |
|
name: MTEB PolEmo2.0-IN |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 71.73130193905818 |
|
- type: f1 |
|
value: 71.17731918813324 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_out |
|
name: MTEB PolEmo2.0-OUT |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 46.59919028340081 |
|
- type: f1 |
|
value: 37.216392949948954 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora-pl |
|
name: MTEB Quora-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 66.134 |
|
- type: map_at_10 |
|
value: 80.19 |
|
- type: map_at_100 |
|
value: 80.937 |
|
- type: map_at_1000 |
|
value: 80.95599999999999 |
|
- type: map_at_3 |
|
value: 77.074 |
|
- type: map_at_5 |
|
value: 79.054 |
|
- type: mrr_at_1 |
|
value: 75.88000000000001 |
|
- type: mrr_at_10 |
|
value: 83.226 |
|
- type: mrr_at_100 |
|
value: 83.403 |
|
- type: mrr_at_1000 |
|
value: 83.406 |
|
- type: mrr_at_3 |
|
value: 82.03200000000001 |
|
- type: mrr_at_5 |
|
value: 82.843 |
|
- type: ndcg_at_1 |
|
value: 75.94 |
|
- type: ndcg_at_10 |
|
value: 84.437 |
|
- type: ndcg_at_100 |
|
value: 86.13 |
|
- type: ndcg_at_1000 |
|
value: 86.29299999999999 |
|
- type: ndcg_at_3 |
|
value: 81.07799999999999 |
|
- type: ndcg_at_5 |
|
value: 83.0 |
|
- type: precision_at_1 |
|
value: 75.94 |
|
- type: precision_at_10 |
|
value: 12.953999999999999 |
|
- type: precision_at_100 |
|
value: 1.514 |
|
- type: precision_at_1000 |
|
value: 0.156 |
|
- type: precision_at_3 |
|
value: 35.61 |
|
- type: precision_at_5 |
|
value: 23.652 |
|
- type: recall_at_1 |
|
value: 66.134 |
|
- type: recall_at_10 |
|
value: 92.991 |
|
- type: recall_at_100 |
|
value: 99.003 |
|
- type: recall_at_1000 |
|
value: 99.86 |
|
- type: recall_at_3 |
|
value: 83.643 |
|
- type: recall_at_5 |
|
value: 88.81099999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs-pl |
|
name: MTEB SCIDOCS-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.183 |
|
- type: map_at_10 |
|
value: 10.626 |
|
- type: map_at_100 |
|
value: 12.485 |
|
- type: map_at_1000 |
|
value: 12.793 |
|
- type: map_at_3 |
|
value: 7.531000000000001 |
|
- type: map_at_5 |
|
value: 9.037 |
|
- type: mrr_at_1 |
|
value: 20.5 |
|
- type: mrr_at_10 |
|
value: 30.175 |
|
- type: mrr_at_100 |
|
value: 31.356 |
|
- type: mrr_at_1000 |
|
value: 31.421 |
|
- type: mrr_at_3 |
|
value: 26.900000000000002 |
|
- type: mrr_at_5 |
|
value: 28.689999999999998 |
|
- type: ndcg_at_1 |
|
value: 20.599999999999998 |
|
- type: ndcg_at_10 |
|
value: 17.84 |
|
- type: ndcg_at_100 |
|
value: 25.518 |
|
- type: ndcg_at_1000 |
|
value: 31.137999999999998 |
|
- type: ndcg_at_3 |
|
value: 16.677 |
|
- type: ndcg_at_5 |
|
value: 14.641000000000002 |
|
- type: precision_at_1 |
|
value: 20.599999999999998 |
|
- type: precision_at_10 |
|
value: 9.3 |
|
- type: precision_at_100 |
|
value: 2.048 |
|
- type: precision_at_1000 |
|
value: 0.33999999999999997 |
|
- type: precision_at_3 |
|
value: 15.533 |
|
- type: precision_at_5 |
|
value: 12.839999999999998 |
|
- type: recall_at_1 |
|
value: 4.183 |
|
- type: recall_at_10 |
|
value: 18.862000000000002 |
|
- type: recall_at_100 |
|
value: 41.592 |
|
- type: recall_at_1000 |
|
value: 69.037 |
|
- type: recall_at_3 |
|
value: 9.443 |
|
- type: recall_at_5 |
|
value: 13.028 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/sicke-pl-pairclassification |
|
name: MTEB SICK-E-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 86.32286995515696 |
|
- type: cos_sim_ap |
|
value: 82.04302619416443 |
|
- type: cos_sim_f1 |
|
value: 74.95572086432874 |
|
- type: cos_sim_precision |
|
value: 74.55954897815363 |
|
- type: cos_sim_recall |
|
value: 75.35612535612536 |
|
- type: dot_accuracy |
|
value: 83.9176518548716 |
|
- type: dot_ap |
|
value: 76.8608733580272 |
|
- type: dot_f1 |
|
value: 72.31936654569449 |
|
- type: dot_precision |
|
value: 67.36324523663184 |
|
- type: dot_recall |
|
value: 78.06267806267806 |
|
- type: euclidean_accuracy |
|
value: 86.32286995515696 |
|
- type: euclidean_ap |
|
value: 81.9648986659308 |
|
- type: euclidean_f1 |
|
value: 74.93796526054591 |
|
- type: euclidean_precision |
|
value: 74.59421312632321 |
|
- type: euclidean_recall |
|
value: 75.28490028490027 |
|
- type: manhattan_accuracy |
|
value: 86.30248675091724 |
|
- type: manhattan_ap |
|
value: 81.92853980116878 |
|
- type: manhattan_f1 |
|
value: 74.80968858131489 |
|
- type: manhattan_precision |
|
value: 72.74562584118439 |
|
- type: manhattan_recall |
|
value: 76.99430199430199 |
|
- type: max_accuracy |
|
value: 86.32286995515696 |
|
- type: max_ap |
|
value: 82.04302619416443 |
|
- type: max_f1 |
|
value: 74.95572086432874 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/sickr-pl-sts |
|
name: MTEB SICK-R-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 83.07566183637853 |
|
- type: cos_sim_spearman |
|
value: 79.20198022242548 |
|
- type: euclidean_pearson |
|
value: 81.27875473517936 |
|
- type: euclidean_spearman |
|
value: 79.21560102311153 |
|
- type: manhattan_pearson |
|
value: 81.21559474880459 |
|
- type: manhattan_spearman |
|
value: 79.1537846814979 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (pl) |
|
config: pl |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 36.39657573900194 |
|
- type: cos_sim_spearman |
|
value: 40.36403461037013 |
|
- type: euclidean_pearson |
|
value: 29.143416004776316 |
|
- type: euclidean_spearman |
|
value: 40.43197841306375 |
|
- type: manhattan_pearson |
|
value: 29.18632337290767 |
|
- type: manhattan_spearman |
|
value: 40.50563343395481 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact-pl |
|
name: MTEB SciFact-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 49.428 |
|
- type: map_at_10 |
|
value: 60.423 |
|
- type: map_at_100 |
|
value: 61.037 |
|
- type: map_at_1000 |
|
value: 61.065999999999995 |
|
- type: map_at_3 |
|
value: 56.989000000000004 |
|
- type: map_at_5 |
|
value: 59.041999999999994 |
|
- type: mrr_at_1 |
|
value: 52.666999999999994 |
|
- type: mrr_at_10 |
|
value: 61.746 |
|
- type: mrr_at_100 |
|
value: 62.273 |
|
- type: mrr_at_1000 |
|
value: 62.300999999999995 |
|
- type: mrr_at_3 |
|
value: 59.278 |
|
- type: mrr_at_5 |
|
value: 60.611000000000004 |
|
- type: ndcg_at_1 |
|
value: 52.333 |
|
- type: ndcg_at_10 |
|
value: 65.75 |
|
- type: ndcg_at_100 |
|
value: 68.566 |
|
- type: ndcg_at_1000 |
|
value: 69.314 |
|
- type: ndcg_at_3 |
|
value: 59.768 |
|
- type: ndcg_at_5 |
|
value: 62.808 |
|
- type: precision_at_1 |
|
value: 52.333 |
|
- type: precision_at_10 |
|
value: 9.167 |
|
- type: precision_at_100 |
|
value: 1.0630000000000002 |
|
- type: precision_at_1000 |
|
value: 0.11299999999999999 |
|
- type: precision_at_3 |
|
value: 23.778 |
|
- type: precision_at_5 |
|
value: 16.2 |
|
- type: recall_at_1 |
|
value: 49.428 |
|
- type: recall_at_10 |
|
value: 81.07799999999999 |
|
- type: recall_at_100 |
|
value: 93.93299999999999 |
|
- type: recall_at_1000 |
|
value: 99.667 |
|
- type: recall_at_3 |
|
value: 65.061 |
|
- type: recall_at_5 |
|
value: 72.667 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid-pl |
|
name: MTEB TRECCOVID-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.22100000000000003 |
|
- type: map_at_10 |
|
value: 1.788 |
|
- type: map_at_100 |
|
value: 9.937 |
|
- type: map_at_1000 |
|
value: 24.762999999999998 |
|
- type: map_at_3 |
|
value: 0.579 |
|
- type: map_at_5 |
|
value: 0.947 |
|
- type: mrr_at_1 |
|
value: 78.0 |
|
- type: mrr_at_10 |
|
value: 88.067 |
|
- type: mrr_at_100 |
|
value: 88.067 |
|
- type: mrr_at_1000 |
|
value: 88.067 |
|
- type: mrr_at_3 |
|
value: 87.667 |
|
- type: mrr_at_5 |
|
value: 88.067 |
|
- type: ndcg_at_1 |
|
value: 76.0 |
|
- type: ndcg_at_10 |
|
value: 71.332 |
|
- type: ndcg_at_100 |
|
value: 54.80500000000001 |
|
- type: ndcg_at_1000 |
|
value: 49.504999999999995 |
|
- type: ndcg_at_3 |
|
value: 73.693 |
|
- type: ndcg_at_5 |
|
value: 73.733 |
|
- type: precision_at_1 |
|
value: 82.0 |
|
- type: precision_at_10 |
|
value: 76.8 |
|
- type: precision_at_100 |
|
value: 56.68 |
|
- type: precision_at_1000 |
|
value: 22.236 |
|
- type: precision_at_3 |
|
value: 78.667 |
|
- type: precision_at_5 |
|
value: 79.2 |
|
- type: recall_at_1 |
|
value: 0.22100000000000003 |
|
- type: recall_at_10 |
|
value: 2.033 |
|
- type: recall_at_100 |
|
value: 13.431999999999999 |
|
- type: recall_at_1000 |
|
value: 46.913 |
|
- type: recall_at_3 |
|
value: 0.625 |
|
- type: recall_at_5 |
|
value: 1.052 |
|
language: pl |
|
license: apache-2.0 |
|
widget: |
|
- source_sentence: "zapytanie: Jak dożyć 100 lat?" |
|
sentences: |
|
- "Trzeba zdrowo się odżywiać i uprawiać sport." |
|
- "Trzeba pić alkohol, imprezować i jeździć szybkimi autami." |
|
- "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." |
|
|
|
--- |
|
|
|
<h1 align="center">MMLW-roberta-base</h1> |
|
|
|
MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish. |
|
This is a distilled model that can be used to generate embeddings applicable to many tasks such as semantic similarity, clustering, information retrieval. The model can also serve as a base for further fine-tuning. |
|
It transforms texts to 768 dimensional vectors. |
|
The model was initialized with Polish RoBERTa checkpoint, and then trained with [multilingual knowledge distillation method](https://aclanthology.org/2020.emnlp-main.365/) on a diverse corpus of 60 million Polish-English text pairs. We utilised [English FlagEmbeddings (BGE)](https://huggingface.co/BAAI/bge-base-en) as teacher models for distillation. |
|
|
|
## Usage (Sentence-Transformers) |
|
|
|
⚠️ Our embedding models require the use of specific prefixes and suffixes when encoding texts. For this model, each query should be preceded by the prefix **"zapytanie: "** ⚠️ |
|
|
|
You can use the model like this with [sentence-transformers](https://www.SBERT.net): |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
from sentence_transformers.util import cos_sim |
|
|
|
query_prefix = "zapytanie: " |
|
answer_prefix = "" |
|
queries = [query_prefix + "Jak dożyć 100 lat?"] |
|
answers = [ |
|
answer_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.", |
|
answer_prefix + "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.", |
|
answer_prefix + "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." |
|
] |
|
model = SentenceTransformer("sdadas/mmlw-roberta-base") |
|
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False) |
|
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False) |
|
|
|
best_answer = cos_sim(queries_emb, answers_emb).argmax().item() |
|
print(answers[best_answer]) |
|
# Trzeba zdrowo się odżywiać i uprawiać sport. |
|
``` |
|
|
|
## Evaluation Results |
|
|
|
- The model achieves an **Average Score** of **61.05** on the Polish Massive Text Embedding Benchmark (MTEB). See [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for detailed results. |
|
- The model achieves **NDCG@10** of **53.60** on the Polish Information Retrieval Benchmark. See [PIRB Leaderboard](https://huggingface.co/spaces/sdadas/pirb) for detailed results. |
|
|
|
## Acknowledgements |
|
This model was trained with the A100 GPU cluster support delivered by the Gdansk University of Technology within the TASK center initiative. |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@article{dadas2024pirb, |
|
title={{PIRB}: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods}, |
|
author={Sławomir Dadas and Michał Perełkiewicz and Rafał Poświata}, |
|
year={2024}, |
|
eprint={2402.13350}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
