|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- transformers |
|
- mteb |
|
model-index: |
|
- name: mmlw-e5-base |
|
results: |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: PL-MTEB/8tags-clustering |
|
name: MTEB 8TagsClustering |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 30.249113010261492 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/allegro-reviews |
|
name: MTEB AllegroReviews |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 36.3817097415507 |
|
- type: f1 |
|
value: 32.77742158736663 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana-pl |
|
name: MTEB ArguAna-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 32.646 |
|
- type: map_at_10 |
|
value: 49.488 |
|
- type: map_at_100 |
|
value: 50.190999999999995 |
|
- type: map_at_1000 |
|
value: 50.194 |
|
- type: map_at_3 |
|
value: 44.749 |
|
- type: map_at_5 |
|
value: 47.571999999999996 |
|
- type: mrr_at_1 |
|
value: 34.211000000000006 |
|
- type: mrr_at_10 |
|
value: 50.112 |
|
- type: mrr_at_100 |
|
value: 50.836000000000006 |
|
- type: mrr_at_1000 |
|
value: 50.839 |
|
- type: mrr_at_3 |
|
value: 45.614 |
|
- type: mrr_at_5 |
|
value: 48.242000000000004 |
|
- type: ndcg_at_1 |
|
value: 32.646 |
|
- type: ndcg_at_10 |
|
value: 58.396 |
|
- type: ndcg_at_100 |
|
value: 61.285000000000004 |
|
- type: ndcg_at_1000 |
|
value: 61.358999999999995 |
|
- type: ndcg_at_3 |
|
value: 48.759 |
|
- type: ndcg_at_5 |
|
value: 53.807 |
|
- type: precision_at_1 |
|
value: 32.646 |
|
- type: precision_at_10 |
|
value: 8.663 |
|
- type: precision_at_100 |
|
value: 0.9900000000000001 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 20.128 |
|
- type: precision_at_5 |
|
value: 14.509 |
|
- type: recall_at_1 |
|
value: 32.646 |
|
- type: recall_at_10 |
|
value: 86.629 |
|
- type: recall_at_100 |
|
value: 99.004 |
|
- type: recall_at_1000 |
|
value: 99.57300000000001 |
|
- type: recall_at_3 |
|
value: 60.38400000000001 |
|
- type: recall_at_5 |
|
value: 72.54599999999999 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/cbd |
|
name: MTEB CBD |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 65.53999999999999 |
|
- type: ap |
|
value: 19.75395945379771 |
|
- type: f1 |
|
value: 55.00481388401326 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/cdsce-pairclassification |
|
name: MTEB CDSC-E |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 89.5 |
|
- type: cos_sim_ap |
|
value: 77.26879308078568 |
|
- type: cos_sim_f1 |
|
value: 65.13157894736842 |
|
- type: cos_sim_precision |
|
value: 86.8421052631579 |
|
- type: cos_sim_recall |
|
value: 52.10526315789473 |
|
- type: dot_accuracy |
|
value: 88.0 |
|
- type: dot_ap |
|
value: 69.17235659054914 |
|
- type: dot_f1 |
|
value: 65.71428571428571 |
|
- type: dot_precision |
|
value: 71.875 |
|
- type: dot_recall |
|
value: 60.526315789473685 |
|
- type: euclidean_accuracy |
|
value: 89.5 |
|
- type: euclidean_ap |
|
value: 77.1905400565015 |
|
- type: euclidean_f1 |
|
value: 64.91803278688525 |
|
- type: euclidean_precision |
|
value: 86.08695652173914 |
|
- type: euclidean_recall |
|
value: 52.10526315789473 |
|
- type: manhattan_accuracy |
|
value: 89.5 |
|
- type: manhattan_ap |
|
value: 77.19531778873724 |
|
- type: manhattan_f1 |
|
value: 64.72491909385113 |
|
- type: manhattan_precision |
|
value: 84.03361344537815 |
|
- type: manhattan_recall |
|
value: 52.63157894736842 |
|
- type: max_accuracy |
|
value: 89.5 |
|
- type: max_ap |
|
value: 77.26879308078568 |
|
- type: max_f1 |
|
value: 65.71428571428571 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/cdscr-sts |
|
name: MTEB CDSC-R |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 93.18498922236566 |
|
- type: cos_sim_spearman |
|
value: 93.26224500108704 |
|
- type: euclidean_pearson |
|
value: 92.25462061070286 |
|
- type: euclidean_spearman |
|
value: 93.18623989769242 |
|
- type: manhattan_pearson |
|
value: 92.16291103586255 |
|
- type: manhattan_spearman |
|
value: 93.14403078934417 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-pl |
|
name: MTEB DBPedia-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 8.268 |
|
- type: map_at_10 |
|
value: 17.391000000000002 |
|
- type: map_at_100 |
|
value: 24.266 |
|
- type: map_at_1000 |
|
value: 25.844 |
|
- type: map_at_3 |
|
value: 12.636 |
|
- type: map_at_5 |
|
value: 14.701 |
|
- type: mrr_at_1 |
|
value: 62.74999999999999 |
|
- type: mrr_at_10 |
|
value: 70.25200000000001 |
|
- type: mrr_at_100 |
|
value: 70.601 |
|
- type: mrr_at_1000 |
|
value: 70.613 |
|
- type: mrr_at_3 |
|
value: 68.083 |
|
- type: mrr_at_5 |
|
value: 69.37100000000001 |
|
- type: ndcg_at_1 |
|
value: 51.87500000000001 |
|
- type: ndcg_at_10 |
|
value: 37.185 |
|
- type: ndcg_at_100 |
|
value: 41.949 |
|
- type: ndcg_at_1000 |
|
value: 49.523 |
|
- type: ndcg_at_3 |
|
value: 41.556 |
|
- type: ndcg_at_5 |
|
value: 39.278 |
|
- type: precision_at_1 |
|
value: 63.24999999999999 |
|
- type: precision_at_10 |
|
value: 29.225 |
|
- type: precision_at_100 |
|
value: 9.745 |
|
- type: precision_at_1000 |
|
value: 2.046 |
|
- type: precision_at_3 |
|
value: 43.833 |
|
- type: precision_at_5 |
|
value: 37.9 |
|
- type: recall_at_1 |
|
value: 8.268 |
|
- type: recall_at_10 |
|
value: 22.542 |
|
- type: recall_at_100 |
|
value: 48.154 |
|
- type: recall_at_1000 |
|
value: 72.62100000000001 |
|
- type: recall_at_3 |
|
value: 13.818 |
|
- type: recall_at_5 |
|
value: 17.137 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa-pl |
|
name: MTEB FiQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 16.489 |
|
- type: map_at_10 |
|
value: 26.916 |
|
- type: map_at_100 |
|
value: 28.582 |
|
- type: map_at_1000 |
|
value: 28.774 |
|
- type: map_at_3 |
|
value: 23.048 |
|
- type: map_at_5 |
|
value: 24.977 |
|
- type: mrr_at_1 |
|
value: 33.642 |
|
- type: mrr_at_10 |
|
value: 41.987 |
|
- type: mrr_at_100 |
|
value: 42.882 |
|
- type: mrr_at_1000 |
|
value: 42.93 |
|
- type: mrr_at_3 |
|
value: 39.48 |
|
- type: mrr_at_5 |
|
value: 40.923 |
|
- type: ndcg_at_1 |
|
value: 33.488 |
|
- type: ndcg_at_10 |
|
value: 34.528 |
|
- type: ndcg_at_100 |
|
value: 41.085 |
|
- type: ndcg_at_1000 |
|
value: 44.474000000000004 |
|
- type: ndcg_at_3 |
|
value: 30.469 |
|
- type: ndcg_at_5 |
|
value: 31.618000000000002 |
|
- type: precision_at_1 |
|
value: 33.488 |
|
- type: precision_at_10 |
|
value: 9.783999999999999 |
|
- type: precision_at_100 |
|
value: 1.6389999999999998 |
|
- type: precision_at_1000 |
|
value: 0.22699999999999998 |
|
- type: precision_at_3 |
|
value: 20.525 |
|
- type: precision_at_5 |
|
value: 15.093 |
|
- type: recall_at_1 |
|
value: 16.489 |
|
- type: recall_at_10 |
|
value: 42.370000000000005 |
|
- type: recall_at_100 |
|
value: 67.183 |
|
- type: recall_at_1000 |
|
value: 87.211 |
|
- type: recall_at_3 |
|
value: 27.689999999999998 |
|
- type: recall_at_5 |
|
value: 33.408 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa-pl |
|
name: MTEB HotpotQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 37.373 |
|
- type: map_at_10 |
|
value: 57.509 |
|
- type: map_at_100 |
|
value: 58.451 |
|
- type: map_at_1000 |
|
value: 58.524 |
|
- type: map_at_3 |
|
value: 54.064 |
|
- type: map_at_5 |
|
value: 56.257999999999996 |
|
- type: mrr_at_1 |
|
value: 74.895 |
|
- type: mrr_at_10 |
|
value: 81.233 |
|
- type: mrr_at_100 |
|
value: 81.461 |
|
- type: mrr_at_1000 |
|
value: 81.47 |
|
- type: mrr_at_3 |
|
value: 80.124 |
|
- type: mrr_at_5 |
|
value: 80.862 |
|
- type: ndcg_at_1 |
|
value: 74.747 |
|
- type: ndcg_at_10 |
|
value: 66.249 |
|
- type: ndcg_at_100 |
|
value: 69.513 |
|
- type: ndcg_at_1000 |
|
value: 70.896 |
|
- type: ndcg_at_3 |
|
value: 61.312 |
|
- type: ndcg_at_5 |
|
value: 64.132 |
|
- type: precision_at_1 |
|
value: 74.747 |
|
- type: precision_at_10 |
|
value: 13.873 |
|
- type: precision_at_100 |
|
value: 1.641 |
|
- type: precision_at_1000 |
|
value: 0.182 |
|
- type: precision_at_3 |
|
value: 38.987 |
|
- type: precision_at_5 |
|
value: 25.621 |
|
- type: recall_at_1 |
|
value: 37.373 |
|
- type: recall_at_10 |
|
value: 69.365 |
|
- type: recall_at_100 |
|
value: 82.039 |
|
- type: recall_at_1000 |
|
value: 91.148 |
|
- type: recall_at_3 |
|
value: 58.48100000000001 |
|
- type: recall_at_5 |
|
value: 64.051 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco-pl |
|
name: MTEB MSMARCO-PL |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 16.753999999999998 |
|
- type: map_at_10 |
|
value: 26.764 |
|
- type: map_at_100 |
|
value: 27.929 |
|
- type: map_at_1000 |
|
value: 27.994999999999997 |
|
- type: map_at_3 |
|
value: 23.527 |
|
- type: map_at_5 |
|
value: 25.343 |
|
- type: mrr_at_1 |
|
value: 17.192 |
|
- type: mrr_at_10 |
|
value: 27.141 |
|
- type: mrr_at_100 |
|
value: 28.269 |
|
- type: mrr_at_1000 |
|
value: 28.327999999999996 |
|
- type: mrr_at_3 |
|
value: 23.906 |
|
- type: mrr_at_5 |
|
value: 25.759999999999998 |
|
- type: ndcg_at_1 |
|
value: 17.177999999999997 |
|
- type: ndcg_at_10 |
|
value: 32.539 |
|
- type: ndcg_at_100 |
|
value: 38.383 |
|
- type: ndcg_at_1000 |
|
value: 40.132 |
|
- type: ndcg_at_3 |
|
value: 25.884 |
|
- type: ndcg_at_5 |
|
value: 29.15 |
|
- type: precision_at_1 |
|
value: 17.177999999999997 |
|
- type: precision_at_10 |
|
value: 5.268 |
|
- type: precision_at_100 |
|
value: 0.823 |
|
- type: precision_at_1000 |
|
value: 0.097 |
|
- type: precision_at_3 |
|
value: 11.122 |
|
- type: precision_at_5 |
|
value: 8.338 |
|
- type: recall_at_1 |
|
value: 16.753999999999998 |
|
- type: recall_at_10 |
|
value: 50.388 |
|
- type: recall_at_100 |
|
value: 77.86999999999999 |
|
- type: recall_at_1000 |
|
value: 91.55 |
|
- type: recall_at_3 |
|
value: 32.186 |
|
- type: recall_at_5 |
|
value: 40.048 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 70.9280430396772 |
|
- type: f1 |
|
value: 68.7099581466286 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 74.76126429051783 |
|
- type: f1 |
|
value: 74.72274307018111 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus-pl |
|
name: MTEB NFCorpus-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.348 |
|
- type: map_at_10 |
|
value: 12.277000000000001 |
|
- type: map_at_100 |
|
value: 15.804000000000002 |
|
- type: map_at_1000 |
|
value: 17.277 |
|
- type: map_at_3 |
|
value: 8.783000000000001 |
|
- type: map_at_5 |
|
value: 10.314 |
|
- type: mrr_at_1 |
|
value: 43.963 |
|
- type: mrr_at_10 |
|
value: 52.459999999999994 |
|
- type: mrr_at_100 |
|
value: 53.233 |
|
- type: mrr_at_1000 |
|
value: 53.26499999999999 |
|
- type: mrr_at_3 |
|
value: 50.464 |
|
- type: mrr_at_5 |
|
value: 51.548 |
|
- type: ndcg_at_1 |
|
value: 40.711999999999996 |
|
- type: ndcg_at_10 |
|
value: 33.709 |
|
- type: ndcg_at_100 |
|
value: 31.398 |
|
- type: ndcg_at_1000 |
|
value: 40.042 |
|
- type: ndcg_at_3 |
|
value: 37.85 |
|
- type: ndcg_at_5 |
|
value: 36.260999999999996 |
|
- type: precision_at_1 |
|
value: 43.344 |
|
- type: precision_at_10 |
|
value: 25.851000000000003 |
|
- type: precision_at_100 |
|
value: 8.279 |
|
- type: precision_at_1000 |
|
value: 2.085 |
|
- type: precision_at_3 |
|
value: 36.326 |
|
- type: precision_at_5 |
|
value: 32.074000000000005 |
|
- type: recall_at_1 |
|
value: 5.348 |
|
- type: recall_at_10 |
|
value: 16.441 |
|
- type: recall_at_100 |
|
value: 32.975 |
|
- type: recall_at_1000 |
|
value: 64.357 |
|
- type: recall_at_3 |
|
value: 9.841999999999999 |
|
- type: recall_at_5 |
|
value: 12.463000000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq-pl |
|
name: MTEB NQ-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.674 |
|
- type: map_at_10 |
|
value: 37.672 |
|
- type: map_at_100 |
|
value: 38.767 |
|
- type: map_at_1000 |
|
value: 38.82 |
|
- type: map_at_3 |
|
value: 33.823 |
|
- type: map_at_5 |
|
value: 36.063 |
|
- type: mrr_at_1 |
|
value: 27.839000000000002 |
|
- type: mrr_at_10 |
|
value: 40.129 |
|
- type: mrr_at_100 |
|
value: 41.008 |
|
- type: mrr_at_1000 |
|
value: 41.048 |
|
- type: mrr_at_3 |
|
value: 36.718 |
|
- type: mrr_at_5 |
|
value: 38.841 |
|
- type: ndcg_at_1 |
|
value: 27.839000000000002 |
|
- type: ndcg_at_10 |
|
value: 44.604 |
|
- type: ndcg_at_100 |
|
value: 49.51 |
|
- type: ndcg_at_1000 |
|
value: 50.841 |
|
- type: ndcg_at_3 |
|
value: 37.223 |
|
- type: ndcg_at_5 |
|
value: 41.073 |
|
- type: precision_at_1 |
|
value: 27.839000000000002 |
|
- type: precision_at_10 |
|
value: 7.5 |
|
- type: precision_at_100 |
|
value: 1.03 |
|
- type: precision_at_1000 |
|
value: 0.116 |
|
- type: precision_at_3 |
|
value: 17.005 |
|
- type: precision_at_5 |
|
value: 12.399000000000001 |
|
- type: recall_at_1 |
|
value: 24.674 |
|
- type: recall_at_10 |
|
value: 63.32299999999999 |
|
- type: recall_at_100 |
|
value: 85.088 |
|
- type: recall_at_1000 |
|
value: 95.143 |
|
- type: recall_at_3 |
|
value: 44.157999999999994 |
|
- type: recall_at_5 |
|
value: 53.054 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: laugustyniak/abusive-clauses-pl |
|
name: MTEB PAC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 64.5033304373009 |
|
- type: ap |
|
value: 75.81507275237081 |
|
- type: f1 |
|
value: 62.24617820785985 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/ppc-pairclassification |
|
name: MTEB PPC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 85.39999999999999 |
|
- type: cos_sim_ap |
|
value: 91.75881977787009 |
|
- type: cos_sim_f1 |
|
value: 87.79264214046823 |
|
- type: cos_sim_precision |
|
value: 88.68243243243244 |
|
- type: cos_sim_recall |
|
value: 86.9205298013245 |
|
- type: dot_accuracy |
|
value: 71.0 |
|
- type: dot_ap |
|
value: 82.97829049033108 |
|
- type: dot_f1 |
|
value: 78.77055039313797 |
|
- type: dot_precision |
|
value: 69.30817610062893 |
|
- type: dot_recall |
|
value: 91.22516556291392 |
|
- type: euclidean_accuracy |
|
value: 85.2 |
|
- type: euclidean_ap |
|
value: 91.85245521151309 |
|
- type: euclidean_f1 |
|
value: 87.64607679465777 |
|
- type: euclidean_precision |
|
value: 88.38383838383838 |
|
- type: euclidean_recall |
|
value: 86.9205298013245 |
|
- type: manhattan_accuracy |
|
value: 85.39999999999999 |
|
- type: manhattan_ap |
|
value: 91.85497100160649 |
|
- type: manhattan_f1 |
|
value: 87.77219430485762 |
|
- type: manhattan_precision |
|
value: 88.8135593220339 |
|
- type: manhattan_recall |
|
value: 86.75496688741721 |
|
- type: max_accuracy |
|
value: 85.39999999999999 |
|
- type: max_ap |
|
value: 91.85497100160649 |
|
- type: max_f1 |
|
value: 87.79264214046823 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/psc-pairclassification |
|
name: MTEB PSC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 97.58812615955473 |
|
- type: cos_sim_ap |
|
value: 99.14945370088302 |
|
- type: cos_sim_f1 |
|
value: 96.06060606060606 |
|
- type: cos_sim_precision |
|
value: 95.48192771084338 |
|
- type: cos_sim_recall |
|
value: 96.64634146341463 |
|
- type: dot_accuracy |
|
value: 95.17625231910947 |
|
- type: dot_ap |
|
value: 97.05592933601112 |
|
- type: dot_f1 |
|
value: 92.14501510574019 |
|
- type: dot_precision |
|
value: 91.31736526946108 |
|
- type: dot_recall |
|
value: 92.98780487804879 |
|
- type: euclidean_accuracy |
|
value: 97.6808905380334 |
|
- type: euclidean_ap |
|
value: 99.18538119402824 |
|
- type: euclidean_f1 |
|
value: 96.20637329286798 |
|
- type: euclidean_precision |
|
value: 95.77039274924472 |
|
- type: euclidean_recall |
|
value: 96.64634146341463 |
|
- type: manhattan_accuracy |
|
value: 97.58812615955473 |
|
- type: manhattan_ap |
|
value: 99.17870990853292 |
|
- type: manhattan_f1 |
|
value: 96.02446483180427 |
|
- type: manhattan_precision |
|
value: 96.31901840490798 |
|
- type: manhattan_recall |
|
value: 95.73170731707317 |
|
- type: max_accuracy |
|
value: 97.6808905380334 |
|
- type: max_ap |
|
value: 99.18538119402824 |
|
- type: max_f1 |
|
value: 96.20637329286798 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_in |
|
name: MTEB PolEmo2.0-IN |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 68.69806094182825 |
|
- type: f1 |
|
value: 68.0619984307764 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_out |
|
name: MTEB PolEmo2.0-OUT |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 35.80971659919028 |
|
- type: f1 |
|
value: 31.13081621324864 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora-pl |
|
name: MTEB Quora-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 66.149 |
|
- type: map_at_10 |
|
value: 80.133 |
|
- type: map_at_100 |
|
value: 80.845 |
|
- type: map_at_1000 |
|
value: 80.866 |
|
- type: map_at_3 |
|
value: 76.983 |
|
- type: map_at_5 |
|
value: 78.938 |
|
- type: mrr_at_1 |
|
value: 76.09 |
|
- type: mrr_at_10 |
|
value: 83.25099999999999 |
|
- type: mrr_at_100 |
|
value: 83.422 |
|
- type: mrr_at_1000 |
|
value: 83.42500000000001 |
|
- type: mrr_at_3 |
|
value: 82.02199999999999 |
|
- type: mrr_at_5 |
|
value: 82.831 |
|
- type: ndcg_at_1 |
|
value: 76.14999999999999 |
|
- type: ndcg_at_10 |
|
value: 84.438 |
|
- type: ndcg_at_100 |
|
value: 86.048 |
|
- type: ndcg_at_1000 |
|
value: 86.226 |
|
- type: ndcg_at_3 |
|
value: 80.97999999999999 |
|
- type: ndcg_at_5 |
|
value: 82.856 |
|
- type: precision_at_1 |
|
value: 76.14999999999999 |
|
- type: precision_at_10 |
|
value: 12.985 |
|
- type: precision_at_100 |
|
value: 1.513 |
|
- type: precision_at_1000 |
|
value: 0.156 |
|
- type: precision_at_3 |
|
value: 35.563 |
|
- type: precision_at_5 |
|
value: 23.586 |
|
- type: recall_at_1 |
|
value: 66.149 |
|
- type: recall_at_10 |
|
value: 93.195 |
|
- type: recall_at_100 |
|
value: 98.924 |
|
- type: recall_at_1000 |
|
value: 99.885 |
|
- type: recall_at_3 |
|
value: 83.439 |
|
- type: recall_at_5 |
|
value: 88.575 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs-pl |
|
name: MTEB SCIDOCS-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.688 |
|
- type: map_at_10 |
|
value: 10.23 |
|
- type: map_at_100 |
|
value: 12.077 |
|
- type: map_at_1000 |
|
value: 12.382 |
|
- type: map_at_3 |
|
value: 7.149 |
|
- type: map_at_5 |
|
value: 8.689 |
|
- type: mrr_at_1 |
|
value: 18.2 |
|
- type: mrr_at_10 |
|
value: 28.816999999999997 |
|
- type: mrr_at_100 |
|
value: 29.982 |
|
- type: mrr_at_1000 |
|
value: 30.058 |
|
- type: mrr_at_3 |
|
value: 25.983 |
|
- type: mrr_at_5 |
|
value: 27.418 |
|
- type: ndcg_at_1 |
|
value: 18.2 |
|
- type: ndcg_at_10 |
|
value: 17.352999999999998 |
|
- type: ndcg_at_100 |
|
value: 24.859 |
|
- type: ndcg_at_1000 |
|
value: 30.535 |
|
- type: ndcg_at_3 |
|
value: 16.17 |
|
- type: ndcg_at_5 |
|
value: 14.235000000000001 |
|
- type: precision_at_1 |
|
value: 18.2 |
|
- type: precision_at_10 |
|
value: 9.19 |
|
- type: precision_at_100 |
|
value: 2.01 |
|
- type: precision_at_1000 |
|
value: 0.338 |
|
- type: precision_at_3 |
|
value: 15.5 |
|
- type: precision_at_5 |
|
value: 12.78 |
|
- type: recall_at_1 |
|
value: 3.688 |
|
- type: recall_at_10 |
|
value: 18.632 |
|
- type: recall_at_100 |
|
value: 40.822 |
|
- type: recall_at_1000 |
|
value: 68.552 |
|
- type: recall_at_3 |
|
value: 9.423 |
|
- type: recall_at_5 |
|
value: 12.943 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/sicke-pl-pairclassification |
|
name: MTEB SICK-E-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 83.12270688952303 |
|
- type: cos_sim_ap |
|
value: 76.4528312253856 |
|
- type: cos_sim_f1 |
|
value: 68.69627507163324 |
|
- type: cos_sim_precision |
|
value: 69.0922190201729 |
|
- type: cos_sim_recall |
|
value: 68.30484330484332 |
|
- type: dot_accuracy |
|
value: 79.20913167549939 |
|
- type: dot_ap |
|
value: 65.03147071986633 |
|
- type: dot_f1 |
|
value: 62.812160694896846 |
|
- type: dot_precision |
|
value: 50.74561403508772 |
|
- type: dot_recall |
|
value: 82.4074074074074 |
|
- type: euclidean_accuracy |
|
value: 83.16347329800244 |
|
- type: euclidean_ap |
|
value: 76.49405838298205 |
|
- type: euclidean_f1 |
|
value: 68.66738120757414 |
|
- type: euclidean_precision |
|
value: 68.88888888888889 |
|
- type: euclidean_recall |
|
value: 68.44729344729345 |
|
- type: manhattan_accuracy |
|
value: 83.16347329800244 |
|
- type: manhattan_ap |
|
value: 76.5080551733795 |
|
- type: manhattan_f1 |
|
value: 68.73883529832084 |
|
- type: manhattan_precision |
|
value: 68.9605734767025 |
|
- type: manhattan_recall |
|
value: 68.51851851851852 |
|
- type: max_accuracy |
|
value: 83.16347329800244 |
|
- type: max_ap |
|
value: 76.5080551733795 |
|
- type: max_f1 |
|
value: 68.73883529832084 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/sickr-pl-sts |
|
name: MTEB SICK-R-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 82.60225159739653 |
|
- type: cos_sim_spearman |
|
value: 76.76667220288542 |
|
- type: euclidean_pearson |
|
value: 80.16302518898615 |
|
- type: euclidean_spearman |
|
value: 76.76131897866455 |
|
- type: manhattan_pearson |
|
value: 80.11881021613914 |
|
- type: manhattan_spearman |
|
value: 76.74246419368048 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (pl) |
|
config: pl |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 38.2744776092718 |
|
- type: cos_sim_spearman |
|
value: 40.35664941442517 |
|
- type: euclidean_pearson |
|
value: 29.148502128336585 |
|
- type: euclidean_spearman |
|
value: 40.45531563224982 |
|
- type: manhattan_pearson |
|
value: 29.124177399433098 |
|
- type: manhattan_spearman |
|
value: 40.2801387844354 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact-pl |
|
name: MTEB SciFact-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 52.994 |
|
- type: map_at_10 |
|
value: 63.612 |
|
- type: map_at_100 |
|
value: 64.294 |
|
- type: map_at_1000 |
|
value: 64.325 |
|
- type: map_at_3 |
|
value: 61.341 |
|
- type: map_at_5 |
|
value: 62.366 |
|
- type: mrr_at_1 |
|
value: 56.667 |
|
- type: mrr_at_10 |
|
value: 65.333 |
|
- type: mrr_at_100 |
|
value: 65.89399999999999 |
|
- type: mrr_at_1000 |
|
value: 65.91900000000001 |
|
- type: mrr_at_3 |
|
value: 63.666999999999994 |
|
- type: mrr_at_5 |
|
value: 64.36699999999999 |
|
- type: ndcg_at_1 |
|
value: 56.333 |
|
- type: ndcg_at_10 |
|
value: 68.292 |
|
- type: ndcg_at_100 |
|
value: 71.136 |
|
- type: ndcg_at_1000 |
|
value: 71.90100000000001 |
|
- type: ndcg_at_3 |
|
value: 64.387 |
|
- type: ndcg_at_5 |
|
value: 65.546 |
|
- type: precision_at_1 |
|
value: 56.333 |
|
- type: precision_at_10 |
|
value: 9.133 |
|
- type: precision_at_100 |
|
value: 1.0630000000000002 |
|
- type: precision_at_1000 |
|
value: 0.11299999999999999 |
|
- type: precision_at_3 |
|
value: 25.556 |
|
- type: precision_at_5 |
|
value: 16.267 |
|
- type: recall_at_1 |
|
value: 52.994 |
|
- type: recall_at_10 |
|
value: 81.178 |
|
- type: recall_at_100 |
|
value: 93.767 |
|
- type: recall_at_1000 |
|
value: 99.667 |
|
- type: recall_at_3 |
|
value: 69.906 |
|
- type: recall_at_5 |
|
value: 73.18299999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid-pl |
|
name: MTEB TRECCOVID-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.231 |
|
- type: map_at_10 |
|
value: 1.822 |
|
- type: map_at_100 |
|
value: 10.134 |
|
- type: map_at_1000 |
|
value: 24.859 |
|
- type: map_at_3 |
|
value: 0.615 |
|
- type: map_at_5 |
|
value: 0.9939999999999999 |
|
- type: mrr_at_1 |
|
value: 84.0 |
|
- type: mrr_at_10 |
|
value: 90.4 |
|
- type: mrr_at_100 |
|
value: 90.4 |
|
- type: mrr_at_1000 |
|
value: 90.4 |
|
- type: mrr_at_3 |
|
value: 89.0 |
|
- type: mrr_at_5 |
|
value: 90.4 |
|
- type: ndcg_at_1 |
|
value: 81.0 |
|
- type: ndcg_at_10 |
|
value: 73.333 |
|
- type: ndcg_at_100 |
|
value: 55.35099999999999 |
|
- type: ndcg_at_1000 |
|
value: 49.875 |
|
- type: ndcg_at_3 |
|
value: 76.866 |
|
- type: ndcg_at_5 |
|
value: 75.472 |
|
- type: precision_at_1 |
|
value: 86.0 |
|
- type: precision_at_10 |
|
value: 78.2 |
|
- type: precision_at_100 |
|
value: 57.18 |
|
- type: precision_at_1000 |
|
value: 22.332 |
|
- type: precision_at_3 |
|
value: 82.0 |
|
- type: precision_at_5 |
|
value: 81.2 |
|
- type: recall_at_1 |
|
value: 0.231 |
|
- type: recall_at_10 |
|
value: 2.056 |
|
- type: recall_at_100 |
|
value: 13.468 |
|
- type: recall_at_1000 |
|
value: 47.038999999999994 |
|
- type: recall_at_3 |
|
value: 0.6479999999999999 |
|
- type: recall_at_5 |
|
value: 1.088 |
|
language: pl |
|
license: apache-2.0 |
|
widget: |
|
- source_sentence: "query: Jak dożyć 100 lat?" |
|
sentences: |
|
- "passage: Trzeba zdrowo się odżywiać i uprawiać sport." |
|
- "passage: Trzeba pić alkohol, imprezować i jeździć szybkimi autami." |
|
- "passage: Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." |
|
|
|
--- |
|
|
|
<h1 align="center">MMLW-e5-base</h1> |
|
|
|
MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish. |
|
This is a distilled model that can be used to generate embeddings applicable to many tasks such as semantic similarity, clustering, information retrieval. The model can also serve as a base for further fine-tuning. |
|
It transforms texts to 768 dimensional vectors. |
|
The model was initialized with multilingual E5 checkpoint, and then trained with [multilingual knowledge distillation method](https://aclanthology.org/2020.emnlp-main.365/) on a diverse corpus of 60 million Polish-English text pairs. We utilised [English FlagEmbeddings (BGE)](https://huggingface.co/BAAI/bge-base-en) as teacher models for distillation. |
|
|
|
## Usage (Sentence-Transformers) |
|
|
|
⚠️ Our embedding models require the use of specific prefixes and suffixes when encoding texts. For this model, queries should be prefixed with **"query: "** and passages with **"passage: "** ⚠️ |
|
|
|
You can use the model like this with [sentence-transformers](https://www.SBERT.net): |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
from sentence_transformers.util import cos_sim |
|
|
|
query_prefix = "query: " |
|
answer_prefix = "passage: " |
|
queries = [query_prefix + "Jak dożyć 100 lat?"] |
|
answers = [ |
|
answer_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.", |
|
answer_prefix + "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.", |
|
answer_prefix + "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." |
|
] |
|
model = SentenceTransformer("sdadas/mmlw-e5-base") |
|
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False) |
|
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False) |
|
|
|
best_answer = cos_sim(queries_emb, answers_emb).argmax().item() |
|
print(answers[best_answer]) |
|
# Trzeba zdrowo się odżywiać i uprawiać sport. |
|
``` |
|
|
|
## Evaluation Results |
|
|
|
- The model achieves an **Average Score** of **59.71** on the Polish Massive Text Embedding Benchmark (MTEB). See [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for detailed results. |
|
- The model achieves **NDCG@10** of **53.56** on the Polish Information Retrieval Benchmark. See [PIRB Leaderboard](https://huggingface.co/spaces/sdadas/pirb) for detailed results. |
|
|
|
## Acknowledgements |
|
This model was trained with the A100 GPU cluster support delivered by the Gdansk University of Technology within the TASK center initiative. |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@article{dadas2024pirb, |
|
title={{PIRB}: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods}, |
|
author={Sławomir Dadas and Michał Perełkiewicz and Rafał Poświata}, |
|
year={2024}, |
|
eprint={2402.13350}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
